JayBizzle/Crawler-Detect

Detects bots, crawlers, and spiders by analyzing user-agent strings and HTTP headers.

View on index · View in 3D Map

Related repos

• Analyze website defenses with Scrapfly Anti-bot Detector, a Chrome extension that identifies CAPTCHAs, anti-bot services like Cloudflare, and fingerprinting techniques such as Canvas and WebGL in real time.
• Web scraper with XPath extraction and domain spidering
• site crawler and analyzer for SEO, security, performance, accessibility
• Blocks AI crawlers from websites
• Scrapes websites and automates browsers
• Agentic search tool for faster, smarter web results
• High-performance Rust web crawler with AI automation
• Automate web scraping spider creation with LLMs
• Distributed web crawler management platform