spider-rs/spider

High-performance Rust web crawler with AI automation

View on index · View in 3D Map

Related repos

• Automate web scraping spider creation with LLMs
• Web scraper for AI agents
• Build web scrapers with AI assistance
• list of web crawler and spider projects in many languages
• multi-engine web crawler for fast, large-scale data extraction
• Blocks AI crawlers from websites
• Open-source tool for web crawling, scraping, and data extraction for LLMs and AI pipelines
• Python framework for web crawling and scraping with customizable spiders
• Article publishing with AI content optimization
• AI-powered search, recommendations, and RAG API toolkit