# Oromi robots.txt Agent-Access Checker

> Oromi robots.txt Agent-Access Checker is a paid API for AI agents from agents.oromi.co.uk, paid per call via x402, $0.002/call, status unknown (last checked 2026-09-14).

Checks whether a given URL is accessible to a specified web crawler agent by parsing and evaluating the site's robots.txt rules

## Facts

- Endpoint: GET https://agents.oromi.co.uk/api/web/robots-check
- Price: $0.002/call
- Payment: x402
- Status: unknown
- Last checked: 2026-09-14
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/oromi-robots-txt-agent-access-checker-b82d8f2f
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_IKmixdYCaRrPk_GjPjESn

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability oromi-robots-txt-agent-access-checker-b82d8f2f
```

Example prompt: Before I scrape https://www.bbc.co.uk/news, can you check whether the default wildcard crawler agent is actually allowed to access that URL according to the site's robots.txt?

## When to prefer this

Use this endpoint when an AI agent or automation pipeline needs to verify robots.txt compliance before crawling or scraping a URL, especially in agentic workflows where legal and ethical crawling behavior must be enforced programmatically. Prefer this over manual robots.txt parsing when you need a reliable, machine-payable micro-service that handles edge cases like missing robots.txt files and wildcard rules. Particularly useful before data collection tasks targeting UK websites alongside Oromi's other UK-focused data services.

## Known failure modes

- URL is unreachable or returns a non-200 status — may result in error or default-allow behavior
- robots.txt file is missing — typically treated as allow-all
- Malformed URL input causes validation error
- User-agent token not recognized — falls back to wildcard rules
- Network timeout fetching the target site's robots.txt

## How this service works

Machine-payable APIs for AI agents, paid per call in USDC via the x402 protocol: UK business data (Companies House), UK property market data (HM Land Registry), website agent-readiness audits, and crypto market context.

## Output

Returns a JSON object with an 'allowed' boolean indicating whether the specified crawler agent is permitted to access the given URL under the site's robots.txt rules, and a 'matched_rule' field showing which specific robots.txt directive triggered the decision (null if no specific rule matched).

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "description": "Full URL to check"
      },
      "agent": {
       "type": "string",
       "description": "Crawler user-agent token (default *)"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "allowed": true,
  "matched_rule": null
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/oromi-robots-txt-agent-access-checker-b82d8f2f/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from agents.oromi.co.uk](https://www.zero.xyz/host/agents.oromi.co.uk/llms.txt)