网络爬虫  该页面收集了与爬虫相关的框架和应用程序。

📂爬虫框架

  一款流行,高效,生态丰富的Python爬虫框架

Python41.85 k
scrapy/scrapy

  Elegant Scraper and Crawler Framework for Golang

翻译用于Golang的优雅的Scraper和Crawler框架

Go14.71 k
gocolly/colly

  python爬虫框架。简单易上手,自带在线编程和任务管理界面

Python13.91 k
binux/pyspider

  基于Scrapy和Redis的分布式爬虫框架

Python3.19 k
rmax/scrapy-redis
📂爬虫应用

  新浪微博爬虫(Scrapy、Redis)

Python2.87 k
LiuXingMing/SinaSpider

  一个种子嗅探器,它从 BitTorrent 网络获取人们下载音乐、电影、游戏、文档等等时所用的种子

Go2.81 k
fanpei91/p2pspider

  微信公众号爬虫

Python2.46 k
bowenpay/wechat-spider

  基于 webmagic 的 Java 爬虫应用

Java2.09 k
brianway/webporter

  豆瓣读书的爬虫

Python1.85 k
lanbing510/DouBanSpider

  🍥 Bilibili 用户爬虫

Python1.66 k
airingursb/bilibili-user
该分类下的开源项目

  一款流行,高效,生态丰富的Python爬虫框架

Python41.85 k
scrapy/scrapy

  Create agents that monitor and act on your behalf. Your agents are standing by!

翻译创建代理进行监视并代表您采取行动。您的代理商正在等待!

Ruby32.45 k
huginn/huginn

  👾 Fast and simple video download library and CLI tool written in Go

翻译👾快速,简单,干净的视频下载器

Go15.86 k
iawia002/annie

  Elegant Scraper and Crawler Framework for Golang

翻译用于Golang的优雅的Scraper和Crawler框架

Go14.71 k
gocolly/colly

  Python爬虫代理IP池(proxy pool)

Python12.07 k
jhao104/proxy_pool

  An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.

翻译使用Python编写的高级Twitter抓取和OSINT工具,不使用Twitter的API,可让您在逃避大多数API限制的同时抓取用户的关注者,关注者,推文等。

Python11.37 k
twintproject/twint

  News, full-text, and article metadata extraction in Python 3. Advanced docs:

翻译Python 3中的新闻,全文和文章元数据提取。高级文档:

Python10.09 k
codelucas/newspaper

  一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

Python9.87 k
shengqiangzhang/examples-of-web-crawlers

  A scalable web crawler framework for Java.

翻译Java的可伸缩Web搜寻器框架。

Java9.87 k
code4craft/webmagic