crawlab-team/crawlab

Distributed web crawler management platform

Article publishing with AI content optimization
crawlab-team/artipub
web scraping and browser automation with Crawlee
apify/crawlee
multi-engine web crawler for fast, large-scale data extraction
any4ai/AnyCrawl
list of web crawler and spider projects in many languages
BruceDone/awesome-crawler
site crawler and analyzer for SEO, security, performance, accessibility
janreges/siteone-crawler
Open-source tool for web crawling, scraping, and data extraction for LLMs and AI pipelines
unclecode/crawl4ai
selenium-powered web crawler and search engine
biolds/sosse
Python web scraping with built-in browser and proxy support
apify/crawlee-python
Python framework for web crawling and scraping with customizable spiders
scrapy/scrapy
CLI tool for crawling sites and mapping endpoints
projectdiscovery/katana