Open Source Projects with the Spider Topic

Spider

Topic

FlareSolverr—Proxy Server Bypassing CF Protection

This project can help developers bypass Cloudflare and DDoS-GUARD protection. It sets up a proxy ser

FlareSolverr

·Python·4 months ago

707

crawl4ai—LLM-Friendly Asynchronous Web Crawler Framework
2

This is an asynchronous web crawler framework developed with Python, capable of transforming website

unclecode

·Python·a year ago

4.6k

SeleniumBase—Comprehensive Browser Automation Framework
1

This project is a Python automation testing framework based on Selenium, integrating multiple functi

seleniumbase

·Python·a year ago

4.5k

helium—A Python Library for Simplifying Browser Automation

This project is a lightweight Python library based on Selenium, which makes writing browser automati

mherrmann

·Python·a year ago

2.5k

crawlee—Crawler Framework that Mimics Human Behavior

This is a web scraping and browser automation library characterized by its ability to write crawlers

apify

·TypeScript·a year ago

2.2k

Scrapegraph-ai—AI-based Python Web Scraper
2

This is a Python web scraping library powered by AI. Leveraging the capabilities of Large Language M

ScrapeGraphAI

·Python·a year ago

4.8k

katana—Out-of-the-box Spider Tool and Framework

This project is a web scraping framework written in Go, which can be used as a command-line tool or

projectdiscovery

·Go·a year ago

1.7k

undetected-chromedriver—Python Library to Bypass Anti-Scraping Detection
1

This is an optimized Selenium WebDriver patch specifically designed to prevent triggering anti-robot

ultrafunkamsterdam

·Python·a year ago

3.9k

DrissionPage—A Web Automation Tool Similar to Selenium
12

This is a Python-based web automation tool that supports Chromium-based browsers. It combines the fu

g1879

·Python·2 years ago

3.7w

URLFinder—A Rapid Tool for Extracting Webpage Information

This project is capable of quickly crawling information such as URLs and API interfaces from JavaScr

pingc0y

·Go·2 years ago

4.3k

EasySpider—A Visual Web Crawler Tool
6

This project allows users to perform automatic data collection/scraping without writing any code thr

NaiboWang

·JavaScript·2 years ago

2.2w

rod—Go 语言的网页自动化和爬虫库
1

该项目是 Go 语言封装的 DevTools 协议库，实现用 Go 语言操作浏览器，自动化之前需要手动完成的操作，比如：爬取客户端渲染的页面、端到端测试、自动填

go-rod

·Go·3 years ago

1.2w

colly—可能是最知名的 Go 爬虫框架

它拥有友好的 API 和丰富代码示例，短时间内即可上手。性能方面单核能达到 1K 请求/秒，还可以轻松管理请求方式、间隔和最大并发数，功能强大且优雅。

gocolly

·Go·3 years ago

1.9k

MechanicalSoup—自动与网站交互的轻量级 Python 库

我们写爬虫一般是请求+解析两步走，该项目将 Requests(请求) 和 BeautifulSoup(解析) 两大 Python 爬虫常用库，封装成一个浏览器对

MechanicalSoup

·Python·3 years ago

weiboSpider—Python 新浪微博爬虫

Python 写的微博爬虫，命令行直接启动。支持获取微博用户、内容字段丰富。虽然爬虫的实战项目很多，但能一直更新的很少，因为只要数据源变动爬虫就要跟进迭代。这个

dataabc

·Python·4 years ago

2.7k

- That's all for now, only these are available at the moment -

Recommend

Refresh