# Robots.txt & Sitemap Inspector

> Robots.txt & Sitemap Inspector is a paid API for AI agents from api.delx.ai, paid per call via x402, $0.01/call, status unknown (last checked 2026-09-14).

Fetches and parses a domain's robots.txt rules and sitemap declarations to reveal crawlability, disallow paths, and indexing directives before scraping or indexing.

## Facts

- Endpoint: POST https://api.delx.ai/api/v1/x402/robots-inspect
- Price: $0.01/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/api-delx-ai-4a3ab788
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_zWIewSjKPH-U4b_ua58hk

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability api-delx-ai-4a3ab788 -d '<json body>'
```

Example prompt: Before I start scraping example.com, check its robots.txt to see which paths are disallowed and list any sitemap URLs it declares.

## When to prefer this

Use this endpoint when you need to respect or audit crawl policies before scraping, indexing, or building a sitemap graph for a domain. It is the right choice when an agent must check legal crawl boundaries, discover sitemap entry points, or verify whether a specific path is disallowed — especially as a pre-flight step before running a broader crawl.

## Known failure modes

- Domain has no robots.txt — returns empty or 404 signal
- Domain is unreachable or times out — network error response
- Malformed robots.txt — partial parse with best-effort output
- Invalid URL input — validation error returned
- Rate limit exceeded — HTTP 429 response

## How this service works

Read robots.txt rules and sitemap declarations before crawling or indexing a domain.

## Output

Returns the parsed contents of the domain's robots.txt including disallow and allow rules per user-agent, crawl-delay directives, and a list of sitemap URLs declared within the file — giving the agent a clear picture of what is and isn't permitted to crawl.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string"
  },
  "timeout": {
   "type": "integer",
   "maximum": 15,
   "minimum": 1
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "result": {
   "allow": [],
   "disallow": [],
   "sitemaps": [],
   "robots_url": "https://example.com/robots.txt"
  },
  "tool_name": "util_robots_inspect"
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/api-delx-ai-4a3ab788/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from api.delx.ai](https://www.zero.xyz/host/api.delx.ai/llms.txt)