Robots.txt
Validator & tester
Paste or fetch a robots.txt file and test any URL against any user-agent. See parse warnings, grouped rules, sitemaps, and the exact rule that allowed or blocked your path.
Live parsing • Path tester • No registration
What is robots.txt?
/robots.txt is the standard file at the root of a domain that tells crawlers which paths they're allowed to fetch and which they should leave alone. Search engines, AI training crawlers, and well-behaved scrapers all consult it before requesting pages.
A single misplaced Disallow: / can de-index an entire site overnight. This validator parses your file the same way Google does — grouping rules by User-agent, applying the longest-match-wins rule for Allow / Disallow, and supporting * and $ wildcards.
Test specific URLs against specific bots and see exactly which line decides the verdict.
Validate & test
Paste your robots.txt or fetch it from a live domain, then check any URL against any bot
Verification is only required when fetching from a URL — pasting content works without it.
Parse summary
Test a URL against a bot
Parsed groups
Sitemaps
Audit Robots.txt Across Your Whole Domain
Use Scrappey to fetch every site's robots.txt at scale, monitor changes, and validate that critical paths stay crawlable.
Why use Scrappey's robots.txt validator?
Match Google's behavior, catch mistakes before they hit production
Google-Spec Parsing
Groups by User-agent, supports * and $ wildcards, applies longest-match-wins for Allow vs Disallow.
Path Tester
Pick a URL and a bot, see the exact rule that allows or blocks it — including which line and which group decided.
Issue Detection
Catches missing colons, orphaned rules, bad paths, empty User-agents, and unknown directives before you ship.
Sitemap Discovery
Surfaces every Sitemap: directive with clickable links so you can sanity-check what crawlers will index.
Fetch Any Domain
Skip the CORS dance — fetch robots.txt from any public site and validate it in one click.
AI Bot Coverage
Preset list includes GPTBot, ChatGPT-User, CCBot, Claude-Web, anthropic-ai, and PerplexityBot for AI training audits.
Audit robots.txt in code
Use Scrappey to fetch robots.txt across many domains at once
import requests
# Fetch a robots.txt via Scrappey (handles blocked domains too)
api = "https://publisher.scrappey.com/api/v1?key=YOUR_API_KEY"
domains = ["example.com", "another-site.com"]
for d in domains:
res = requests.post(api, json={
"cmd": "request.get",
"url": f"https://{d}/robots.txt"
}).json()
print(d, "→", res["solution"]["response"][:200])Perfect for
SEO Audits
Confirm critical pages stay crawlable by Googlebot and Bingbot after deploys, redesigns, or staging swaps.
AI Bot Policy
Decide which AI training crawlers (GPTBot, CCBot, etc.) to block — and verify your rules actually do it.
Scraping Compliance
Check what your scraper is allowed to touch before you build it, so you stay within published crawl policy.
Pre-Launch QA
Catch the accidental "Disallow: /" before a launch — pipe staging robots.txt through this tool in CI.
Integrations
n8n
Automate workflows visually. Streamline data collection processes.
n8n Template
Pre-built template for modern websites. Simplifies Scrappey integration.
RapidAPI
Access via API marketplace. Easy integration with comprehensive docs.
Apify
Scalable actor-based automation. Reliable browser rendering.
MCP Server
AI-powered browser automation. Intelligent session management.
Scrappey CLI
Scrape from your terminal. One command, pipeable output, CI-ready.
Claude Code Skill
Portable skill for Claude Code + Codex. Browser-backed data access on demand.
LangChain
LangChain connector — clean web data for any chain or agent.
LlamaIndex
LlamaIndex reader — load modern web pages straight into RAG.
Zapier
Connect with 7,000+ apps. Automate workflows easily.
Make
Visual workflow automation. Connect with 1,000+ apps easily.
Start building with Scrappey
Try it free. No subscription, no credit card, instant setup — your trial is waiting.