# Tanod Web – robots.txt Checker

> Tanod Web – robots.txt Checker is a paid API for AI agents from web.tanod.dev, paid per call via x402, $0.001/call, status unknown (last checked 2026-10-13).

Fetches a URL's robots.txt and checks whether a given path is allowed or disallowed for a specified user-agent/crawler.

## Facts

- Endpoint: POST https://web.tanod.dev/v1/robots?utm_source=zero.xyz
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-13
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/tanod-web-robots-txt-checker-65f00d2e
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_vsdTAiLgtEbk5Ju3InLOx

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability tanod-web-robots-txt-checker-65f00d2e -d '<json body>'
```

Example prompt: Can you check whether Googlebot is allowed to crawl https://www.example.com/blog/private-post — fetch the robots.txt and tell me which rule applies and whether it's blocked?

## When to prefer this

Use this endpoint when you need a definitive, parsed robots.txt compliance check for a specific URL and crawler, especially when you need the exact matched rule, crawl delay, and sitemap discovery in one call. Prefer over manual parsing or generic fetch tools when you need structured output with rule attribution.

## Known failure modes

- robots.txt unreachable or returns non-200 HTTP status — fetch field will indicate error
- URL malformed or missing scheme — input validation error
- robots.txt too large causing truncation (truncated: true in response)
- No matching rule found — defaults to allowed
- User-agent product token not recognized — falls back to wildcard group

## How this service works

Web tools: search, page to Markdown or screenshot, company profile and jobs, sitemaps, RDAP, IP.

## Output

Returns whether the URL path is allowed or disallowed for the specified crawler, the exact matched rule (line number, type, pattern), robots.txt parse stats (lines, rules, groups), list of sitemap URLs, crawl delay, the raw robots.txt URL, HTTP status of the robots.txt fetch, and whether the content was truncated.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "maxLength": 2048,
   "minLength": 1,
   "description": "Public http(s) URL to test; <origin>/robots.txt is fetched and the URL's path is matched against it."
  },
  "user_agent": {
   "type": "string",
   "default": "*",
   "maxLength": 200,
   "minLength": 1,
   "description": "Product token (e.g. Googlebot) or a full User-Agent string; '*' (default) = any crawler."
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "url": "https://www.google.com/search",
  "path": "/search",
  "fetch": "ok",
  "group": "*",
  "stats": {
   "lines": 264,
   "rules": 245,
   "groups": 5,
   "ignored_lines": 0
  },
  "reason": "most specific matching rule: disallow /search",
  "allowed": false,
  "sitemaps": [
   "https://www.google.com/sitemap.xml"
  ],
  "truncated": false,
  "robots_url": "https://www.google.com/robots.txt",
  "user_agent": "Googlebot",
  "crawl_delay": null,
  "http_status": 200,
  "matched_rule": {
   "line": 3,
   "type": "disallow",
   "pattern": "/search"
  },
  "product_token": "googlebot",
  "sitemaps_total": 1,
  "untrusted_content": true
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/tanod-web-robots-txt-checker-65f00d2e/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from web.tanod.dev](https://www.zero.xyz/host/web.tanod.dev/llms.txt)