# Cited Brief Robots.txt Check

> Cited Brief Robots.txt Check is a paid API for AI agents from cited-brief.cited-brief.workers.dev, paid per call via x402, $0.01/call, status unknown (last checked 2026-10-14).

Checks whether a given URL is allowed or disallowed by the target site's robots.txt for a specified crawler user agent

## Facts

- Endpoint: POST https://cited-brief.cited-brief.workers.dev/robots-check?utm_source=zero.xyz
- Price: $0.01/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/cited-brief-robots-txt-check-a3ef7b97
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_Qt6qwrEc1PzduTCywez_1

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability cited-brief-robots-txt-check-a3ef7b97 -d '<json body>'
```

Example prompt: Before scraping https://example.com/private/report.html with my bot called ResearchBot, check whether robots.txt actually allows me to fetch that page.

## When to prefer this

Use this endpoint when an AI agent needs to verify crawl permissions before fetching a URL, especially in automated research or scraping workflows where robots.txt compliance is required. Prefer this over parsing robots.txt manually since it handles rule matching, user-agent grouping, and crawl-delay extraction automatically.

## Known failure modes

- URL is not reachable or returns a non-200 status — robots.txt fetch may fail
- robots.txt does not exist on the target domain — allowed defaults may apply
- malformed URL input causes a validation error
- target site blocks the fetcher's IP, preventing robots.txt retrieval

## How this service works

Pay-per-call web research tools for AI agents, paid in USDC over x402.

## Output

Returns a JSON object indicating whether the URL is allowed for the given user agent, the matched robots.txt rule and group, any crawl delay, the robots.txt URL, a list of sitemaps found, and the HTTP status code of the robots.txt response.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "description": "Public http(s) URL you want to fetch"
  },
  "user_agent": {
   "type": "string",
   "description": "Your crawler's name, for example MyBot (default: any crawler)"
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "url": "https://example.com/private/report.html",
  "allowed": false,
  "sitemaps": [
   "https://example.com/sitemap.xml"
  ],
  "robots_url": "https://example.com/robots.txt",
  "user_agent": "mybot",
  "crawl_delay": null,
  "matched_rule": "Disallow: /private/",
  "matched_group": "*",
  "robots_status": 200
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/cited-brief-robots-txt-check-a3ef7b97/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from cited-brief.cited-brief.workers.dev](https://www.zero.xyz/host/cited-brief.cited-brief.workers.dev/llms.txt)