# Delx Commerce — robots.txt Inspector

> Delx Commerce — robots.txt Inspector is a paid API for AI agents from commerce.delx.ai, paid per call via x402, $0.01/call, status unknown (last checked 2026-10-01).

Fetches and parses a website's robots.txt file, returning allowed paths, disallowed paths, and sitemap URLs

## Facts

- Endpoint: POST https://commerce.delx.ai/api/v1/x402/robots-inspect?utm_source=zero.xyz
- Price: $0.01/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-01
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/delx-commerce-robots-txt-inspector-ddc57728
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_Bh1vWu9AEzLDq6oPEmE3V

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability delx-commerce-robots-txt-inspector-ddc57728 -d '<json body>'
```

Example prompt: Can you inspect the robots.txt for https://example.com and tell me which paths are disallowed and what sitemaps are listed? Use a timeout of 10 seconds.

## When to prefer this

Use this endpoint when you need a fast, pay-per-call, no-signup way to fetch and parse a robots.txt file before crawling or scraping a site. Prefer it over writing custom HTTP fetch logic when you need structured output (allowed, disallowed, sitemaps) rather than raw text, and when operating in an agent context where micropayment-based access via x402/USDC is acceptable.

## Known failure modes

- Timeout if the site does not respond within the specified timeout window (max 15 seconds)
- robots.txt not found (404) resulting in empty or null result
- Network errors or DNS resolution failures for unreachable domains
- Malformed robots.txt that cannot be parsed cleanly
- Invalid URL input causing request rejection

## How this service works

Pay-per-result APIs for agents. No signup. Exact price. Verifiable delivery. USDC on Base + Solana via x402.

## Output

A JSON object containing arrays of allowed paths, disallowed paths, sitemap URLs, and the resolved robots.txt URL for the given domain

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string"
  },
  "timeout": {
   "type": "integer",
   "maximum": 15,
   "minimum": 1
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "result": {
   "allow": [],
   "disallow": [],
   "sitemaps": [],
   "robots_url": "https://example.com/robots.txt"
  },
  "tool_name": "util_robots_inspect"
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/delx-commerce-robots-txt-inspector-ddc57728/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from commerce.delx.ai](https://www.zero.xyz/host/commerce.delx.ai/llms.txt)