# Robots.txt Analyzer

> Robots.txt Analyzer is a paid API for AI agents from netintel-production-440c.up.railway.app, paid per call via x402, $0.01/call, status unknown (last checked 2026-09-13).

Fetches and parses a domain's robots.txt file, returning crawl rules by user-agent, sitemap URLs, crawl delay settings, and path permission checks.

## Facts

- Endpoint: GET https://netintel-production-440c.up.railway.app/robots-txt/analyze
- Price: $0.01/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-13
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/netintel-production-440c-up-railway-app-e1cf02df
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_R0X0KmLDOjSTdU9Zb0eZw

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability netintel-production-440c-up-railway-app-e1cf02df
```

Example prompt: Before I start scraping nytimes.com, can you check their robots.txt file and tell me whether the path /articles is allowed for a generic bot, and list any sitemaps they have?

## When to prefer this

Use this endpoint when an agent needs to respect crawl policies before scraping a website, locate sitemap URLs for a domain, check whether a specific path is accessible to a particular bot, or audit a domain's robots.txt configuration for compliance grading. Prefer this over manual HTTP fetching when structured parsing, multi-field extraction, and path permission evaluation are needed in a single call.

## Known failure modes

- Domain has no robots.txt file — found=false returned with 404 status
- Domain is unreachable or DNS resolution fails — connection error
- Invalid domain format provided — validation error on input
- Path check returns ambiguous result when no matching user-agent rule exists — defaults to allowed
- Robots.txt file is malformed or non-standard — partial parse with best-effort results

## How this service works

Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.

## Output

Returns whether robots.txt was found, parsed crawl rules grouped by user-agent, a list of sitemap URLs, crawl delay settings, a path permission check result for the specified path and user-agent, the raw content preview, the robots.txt URL, HTTP status code, a compliance grade, and a numeric score.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "domain"
     ],
     "properties": {
      "path": {
       "type": "string",
       "description": "URL path to check permission for (e.g. /api/data)"
      },
      "domain": {
       "type": "string",
       "description": "Domain name to fetch robots.txt from (e.g. example.com)"
      },
      "user_agent": {
       "type": "string",
       "description": "Bot name to check rules for (default: *)"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object",
     "properties": {
      "found": {
       "type": "boolean"
      },
      "grade": {
       "type": "string"
      },
      "rules": {
       "type": "array",
       "items": {
        "type": "object"
       }
      },
      "score": {
       "type": "number"
      },
      "domain": {
       "type": "string"
      },
      "findings": {
       "type": "array",
       "items": {
        "type": "object"
       }
      },
      "sitemaps": {
       "type": "array",
       "items": {
        "type": "string"
       }
      },
      "path_check": {
       "type": "object"
      },
      "robots_url": {
       "type": "string"
      },
      "status_code": {
       "type": "number"
      },
      "raw_content_preview": {
       "type": "string"
      }
     }
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "found": true,
  "grade": "B",
  "rules": [
   {
    "allow": [
     "/public/"
    ],
    "disallow": [
     "/private/",
     "/admin/"
    ],
    "user_agent": "*",
    "crawl_delay": null
   }
  ],
  "score": 85,
  "domain": "example.com",
  "findings": [],
  "sitemaps": [
   "https://example.com/sitemap.xml"
  ],
  "path_check": {
   "path": "/private/page",
   "allowed": false,
   "rule_type": "disallow",
   "user_agent": "*",
   "matched_rule": "Disallow: /private/"
  },
  "robots_url": "https://example.com/robots.txt",
  "status_code": 200,
  "raw_content_preview": "User-agent: *\nDisallow: /private/\n..."
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/netintel-production-440c-up-railway-app-e1cf02df/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from netintel-production-440c.up.railway.app](https://www.zero.xyz/host/netintel-production-440c.up.railway.app/llms.txt)