ai-robots-txt/ai.robots.txt
Blocks AI crawlers from websites
View on index · View in 3D Map
Related repos
- Build web scrapers with AI assistance
- Scrapes websites while bypassing bot protections
- uBlock Origin filter list for AI content farms
- A blocklist for removing AI-generated content from image search results
- AI-powered Python tool for scraping websites and documents
- AI agents for internet research and task execution
- list of tools for searching the web using AI
- crawls websites to collect data for AI and custom GPTs
- fetch a whole website and save as text for ai projects
