Repository Showcase
A curated collection of interesting GitHub repositories
google/robotstxt
Google’s robots.txt parser and matcher
Related repos
- Tool for testing regular expressions with real-time results
- crawls websites to collect data for AI and custom GPTs
- Open-source tool for web crawling, scraping, and data extraction for LLMs and AI pipelines
- AI workflow platform for building chatbots and automating knowledge tasks
- AI-powered search, recommendations, and RAG API toolkit
- Agentic search tool for faster, smarter web results
- Open-source toolkit for agentic retrieval, research, and question answering over docs and web
- maxun: open source tool for training robots to extract web data without code
- Command-line fuzzy finder, searches files, text, git repos, environment, aliases, stdin
- open-source tool for automated search, reading, and reasoning
