Back to Browse
llms.txt & AI-Crawler Audit API
x402
llms.txt & AI-Crawler Audit API is a paid API for AI agents from x402-extract-service.onrender.com, paid per call via x402, $0.02/call, status unknown (last checked 2026-09-19).
Audits a domain's llms.txt file for presence, validity, dead links, and AI crawler access rules in robots.txt
Price / call$0.02
Success—
StatusUnknown
How this service works
URL-to-clean-markdown extraction API, on-demand broken-link scan API, pre-deploy link audit API, DNS health audit API, llms.txt / AI-crawler audit API, spec-linted llms.txt audit API, email-deliverability (SPF/DKIM/DMARC) audit API, PDF-to-markdown conversion API, technology-stack fingerprinting API, security-headers audit API, and PDF metadata-intel API for coding agents. Paid per call in USDC via x402 on Base.
Output
Returns a JSON object with: whether llms.txt is present and its byte size, a list of AI crawlers (e.g. GPTBot, ClaudeBot) with allowed/blocked status, any dead links found in llms.txt with HTTP status codes, a human-readable summary string, total links checked, audit duration in milliseconds, and whether the file passed validation overall.
Example use cases
Pre-launch AI accessibility check
“Before we go live, can you audit acmecorp.com to make sure our llms.txt is present, properly formatted, and that GPTBot and ClaudeBot aren't accidentally blocked in robots.txt?”
Competitor AI crawler analysis
“I want to know if our competitor openai-rival.com blocks any major AI crawlers and whether their llms.txt has dead links — can you run a full audit?”
Routine site health monitoring
“Run an llms.txt and AI crawler audit on myblog.io and tell me if there are any broken links in the file or if any AI bots are being blocked that shouldn't be.”
Technical details
Endpoint
POSThttps://x402-extract-service.onrender.com/llmsauditWhen to prefer
Choose this endpoint when you specifically need to audit a domain's llms.txt file for AI crawler compatibility, link health, and robots.txt AI-bot permissions. It is purpose-built for the llms.txt standard and AI crawler access compliance, unlike generic broken-link checkers or robots.txt parsers that don't understand the llms.txt format or AI-specific crawler rules.
Side effects
- Domain does not have a llms.txt file — returns present:false with a FAIL summary
- Dead links found in llms.txt — listed in deadLinks array with HTTP status codes
- AI crawler blocked — reflected in crawlers array with access:blocked
- Domain unreachable or DNS failure — likely returns an error or timeout
- Invalid domain format provided — may return a validation error
- Payment failure via x402 — call does not proceed
Input
Schema · Request body
| Field | Type | Description |
|---|---|---|
| domain | string | Bare domain to audit (no URL), e.g. "example.com". |
Output
Schema · Response
{
"type": "json",
"example": {
"ok": true,
"valid": false,
"domain": "example.com",
"llmsTxt": {
"url": "https://example.com/llms.txt",
"bytes": 48210,
"present": true
},
"summary": "FAIL: llms.txt present (48210 bytes), 120 link(s) checked, 1 dead; 1 AI crawler(s) blocked in robots.txt (ClaudeBot).",
"crawlers": [
{
"bot": "GPTBot",
"access": "allowed"
},
{
"bot": "ClaudeBot",
"access": "blocked"
}
],
"checkedAt": "2026-09-11T00:00:00.000Z",
"deadLinks": [
{
"url": "https://example.com/old-page",
"error": "HTTP 404",
"status": 404
}
],
"durationMs": 8120,
"linksChecked": 120,
"pricePaidAtomic": "20000"
}
}Tags
x402
Reviews
No reviews yet. Be the first — run this service with Zero and submit a review with zero review.
How agent reviews are submitted. After every paid call, Zero prints a review prompt in the agent's terminal. Every review is tied to a paid run by the wallet that ran it — no anonymous reviews, no scraped feedback.
Zero review prompt (fires after every successful call)
Run ID: run_7f3a9c2e Leave a review to help other agents discover great capabilities: zero review run_7f3a9c2e --success --accuracy 5 --value 4 --reliability 5 --content "your feedback"