Open Source Projects with the Spider Topic

Spider

Topic

crawl4ai—LLM-Friendly Asynchronous Web Crawler Framework
1

This is an asynchronous web crawler framework developed with Python, capable of transforming website

unclecode

·Python·22 days ago

1.4k

SeleniumBase—Comprehensive Browser Automation Framework
1

This project is a Python automation testing framework based on Selenium, integrating multiple functi

seleniumbase

·Python·2 months ago

helium—A Python Library for Simplifying Browser Automation

This project is a lightweight Python library based on Selenium, which makes writing browser automati

mherrmann

·Python·3 months ago

1.5k

crawlee—Crawler Framework that Mimics Human Behavior

This is a web scraping and browser automation library characterized by its ability to write crawlers

apify

·TypeScript·3 months ago

1.3k

undetected-chromedriver—Python Library to Bypass Anti-Scraping Detection
1

This is an optimized Selenium WebDriver patch specifically designed to prevent triggering anti-robot

ultrafunkamsterdam

·Python·6 months ago

2.6k

Scrapegraph-ai—AI-based Python Web Scraper
1

This is a Python web scraping library powered by AI. Leveraging the capabilities of Large Language M

ScrapeGraphAI

·Python·4 months ago

DrissionPage—A Web Automation Tool Similar to Selenium
8

This is a Python-based web automation tool that supports Chromium-based browsers. It combines the fu

g1879

·Python·a year ago

3.4w

katana—Out-of-the-box Spider Tool and Framework

This project is a web scraping framework written in Go, which can be used as a command-line tool or

projectdiscovery

·Go·4 months ago

EasySpider—A Visual Web Crawler Tool
5

This project allows users to perform automatic data collection/scraping without writing any code thr

NaiboWang

·JavaScript·a year ago

1.9w

URLFinder—A Rapid Tool for Extracting Webpage Information

This project is capable of quickly crawling information such as URLs and API interfaces from JavaScr

pingc0y

·Go·a year ago

3.6k

rod—Go 语言的网页自动化和爬虫库
1

该项目是 Go 语言封装的 DevTools 协议库，实现用 Go 语言操作浏览器，自动化之前需要手动完成的操作，比如：爬取客户端渲染的页面、端到端测试、自动填

go-rod

·Go·2 years ago

1.1w

requests-html—人性化的解析 HTML 的 Python 库
1

写爬虫的小伙伴都感受过解析 HTML 的痛苦，常用工具 BeautifulSoup、lxml、Scrapy 的 selector 等。今天你有了新的选择 req

psf

·Python·4 years ago

1.1w

colly—可能是最知名的 Go 爬虫框架

它拥有友好的 API 和丰富代码示例，短时间内即可上手。性能方面单核能达到 1K 请求/秒，还可以轻松管理请求方式、间隔和最大并发数，功能强大且优雅。

gocolly

·Go·2 years ago

1.4k

weiboSpider—Python 新浪微博爬虫

Python 写的微博爬虫，命令行直接启动。支持获取微博用户、内容字段丰富。虽然爬虫的实战项目很多，但能一直更新的很少，因为只要数据源变动爬虫就要跟进迭代。这个

dataabc

·Python·3 years ago

1.9k

MechanicalSoup—自动与网站交互的轻量级 Python 库

我们写爬虫一般是请求+解析两步走，该项目将 Requests(请求) 和 BeautifulSoup(解析) 两大 Python 爬虫常用库，封装成一个浏览器对

MechanicalSoup

·Python·2 years ago

1.5k

- That's all for now, only these are available at the moment -

Recommend

Refresh