# Tanod sitepeek robots.txt Check

> Tanod sitepeek robots.txt Check is a paid API for AI agents from tanod.dev, paid per call via x402, $0.001/call, status unknown (last checked 2026-10-11).

Checks whether a given URL is allowed to be crawled by a specified user agent by fetching and parsing the site's robots.txt, returning the matching rule, crawl delay, and sitemaps.

## Facts

- Endpoint: POST https://tanod.dev/v1/robots?utm_source=zero.xyz
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-11
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/tanod-sitepeek-robots-txt-check-b15f7a55
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_Wc6HQl1Q-84pz-IGVqhKc

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability tanod-sitepeek-robots-txt-check-b15f7a55 -d '<json body>'
```

Example prompt: Can you check whether Googlebot is allowed to crawl https://example.com/private/page according to the site's robots.txt, and tell me what rule matches and if there's a crawl delay?

## When to prefer this

Use this endpoint when you need a precise, RFC 9309-compliant interpretation of robots.txt for a specific URL and user agent combination, including the matching rule, crawl delay, and sitemaps — especially before automated crawling or scraping to ensure compliance. Prefer it over manual parsing or generic HTTP tools when RFC 9309 compliance and structured output are required.

## Known failure modes

- robots.txt not found (404) — may default to allow-all
- URL is malformed or not HTTP/HTTPS — validation error
- robots.txt fetch times out or returns non-200 — service error
- user agent string exceeds 200 characters — input validation error
- Target server blocks the robots.txt fetch — may return error or empty result

## How this service works

One MCP server and HTTP API with 193 pay-per-call tools for AI agents (security checks, chain data, finance, aviation, documents), plus free uptime and MikroTik monitoring and 130+ free browser tools. No signup; free daily tier. Operated by an AI.

## Output

Returns whether the specified URL is allowed or disallowed for the given user agent, the specific robots.txt rule that matched, the crawl delay (if any), and a list of sitemaps declared in the robots.txt file.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "maxLength": 2048,
   "minLength": 1,
   "description": "Public http(s) URL to test; <origin>/robots.txt is fetched and the URL's path is matched against it."
  },
  "user_agent": {
   "type": "string",
   "default": "*",
   "maxLength": 200,
   "minLength": 1,
   "description": "Product token (e.g. Googlebot) or a full User-Agent string; '*' (default) = any crawler."
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "url": "https://www.google.com/search",
  "path": "/search",
  "fetch": "ok",
  "group": "*",
  "stats": {
   "lines": 264,
   "rules": 245,
   "groups": 5,
   "ignored_lines": 0
  },
  "reason": "most specific matching rule: disallow /search",
  "allowed": false,
  "sitemaps": [
   "https://www.google.com/sitemap.xml"
  ],
  "truncated": false,
  "robots_url": "https://www.google.com/robots.txt",
  "user_agent": "Googlebot",
  "crawl_delay": null,
  "http_status": 200,
  "matched_rule": {
   "line": 3,
   "type": "disallow",
   "pattern": "/search"
  },
  "product_token": "googlebot",
  "sitemaps_total": 1,
  "untrusted_content": true
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/tanod-sitepeek-robots-txt-check-b15f7a55/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from tanod.dev](https://www.zero.xyz/host/tanod.dev/llms.txt)