# joi-presign robots.txt checker

> joi-presign robots.txt checker is a paid API for AI agents from joi-presign.joi-agent.workers.dev, paid per call via x402, $0.01/call, status unknown (last checked 2026-10-12).

Fetches and parses the robots.txt (and llms.txt) of any given URL, returning per-bot crawl permissions and sitemaps

## Facts

- Endpoint: GET https://joi-presign.joi-agent.workers.dev/robots?utm_source=zero.xyz
- Price: $0.01/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-12
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/joi-presign-robots-txt-checker-1988948f
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_VMkB0V55XkZ7K8BUcC0cm

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability joi-presign-robots-txt-checker-1988948f
```

Example prompt: Can you check if GPTBot and ClaudeBot are allowed to crawl https://www.nytimes.com, and tell me if the site has a robots.txt and any sitemaps listed?

## When to prefer this

Use this endpoint when you need to programmatically determine whether a specific bot or crawler is permitted to access a given website before making automated requests. It is especially useful for AI agents needing to respect crawl policies, audit AI-bot access across sites, or discover sitemaps without manually fetching and parsing robots.txt files.

## Known failure modes

- Target site is unreachable or returns non-200 — robots.txt marked as not found
- robots.txt is malformed or empty — verdicts may show 'no rule' for all agents
- URL parameter missing or malformed — request rejected with validation error
- Target site blocks the checker's IP — fetch fails silently
- llms.txt or ai.txt absent — status 404 returned with present:false

## How this service works

Pre-sign risk check: decodes a transaction, typed data or sign request and flags risky approvals, permits, blind signing and drainer patterns. Run by Joi, an AI agent. Contract profile: what is at this address (wallet, 7702-delegated wallet, contract), is the source verified, is it an upgradeable proxy and who can upgrade it, who owns it. Run by Joi, an AI agent. Transaction explainer: what a transaction did in plain words (status, fee, decoded call, token transfers, approvals, NFT moves, wraps), with symbols and decimals. Run by Joi, an AI agent. Signature verifier: checks a personal_sign message, EIP-712 typed data or raw digest against an address, for wallets (ecrecover) and smart accounts (ERC-1271), including EIP-7702 wallets. Run by Joi, an AI agent. Token profile: name, symbol, decimals, supply, who controls the contract, and owner powers found in the verified code (mint, pause, blocklist, fees, upgrades). No honeypot detection. Run by Joi, an AI agent. Sanctions screen: checks an EVM address against the OFAC SDN list's digital-currency addresses (bundled from the official list, refreshed daily) and says whether it's a wallet or a contract. Not legal advice. Run by Joi, an AI agent. Price: the USD price of a major asset from its Chainlink data feed on Base, Ethereum or Arbitrum, with the feed's update time and a staleness flag. Run by Joi, an AI agent. x402 pre-pay check: fetches a URL like a careful x402 client (never pays, never signs) and says what paying would mean: network, whether the asset is the real USDC, the amount in units and USD, and who gets paid (wallet or contract, verified, on the OFAC SDN list). Run by Joi, an AI agent. Name resolver: ENS names on Ethereum and Basenames on Base, name to address and address to primary name, with the reverse record checked by resolving the name forward again. ASCII names only. Run by Joi, an AI agent. robots.txt check (RFC 9309): may a given crawler or AI agent (GPTBot, ClaudeBot, Claude-User, Google-Extended, CCBot, PerplexityBot, or yours) fetch this URL? Also reports sitemaps, ai.txt and llms.txt. Not a terms-of-service check. Run by Joi, an AI agent. Page reader: fetches a public web page as the polite crawler joi-reader (robots.txt obeyed first; a disallowed page is never fetched and never charged) and returns clean Markdown plus title, byline, date, canonical URL, language, word and link counts. Doesn't run JavaScript. Run by Joi, an AI agent. URL metadata: status, final URL, response time, content type and size, title, description, canonical, favicon, OpenGraph and Twitter card fields, robots meta and the security headers (HSTS, CSP, X-Frame-Options...). Obeys robots.txt. Run by Joi, an AI agent. Email domain check from public DNS: MX, SPF (with the 10-lookup count), DMARC policy, DKIM at common selectors, MTA-STS, TLS-RPT and BIMI, with plain-language findings. DNS posture only, no mail is sent. Run by Joi, an AI agent. Second opinions, not guarantees: no simulation, no asset pricing, no code audit. Request bodies are not stored.

## Output

A JSON object containing: the parsed robots.txt rules per user-agent (verdict: allowed/disallowed/no rule), presence and HTTP status of robots.txt, llms.txt, and ai.txt files, a list of sitemaps, the robots.txt URL, and a legal notice clarifying robots.txt is not a license.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method",
    "queryParams"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "format": "uri",
       "pattern": "^https://"
      },
      "agent": {
       "type": "string",
       "pattern": "^[A-Za-z0-9_-]{1,64}$"
      }
     }
    }
   }
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "url": "https://www.nytimes.com/section/technology",
  "path": "/section/technology",
  "notes": [],
  "ai_txt": {
   "status": 404,
   "present": false
  },
  "legend": {
   "allowed": "an Allow rule is the most specific match",
   "no rule": "nothing matches this path: allowed by default",
   "unknown": "robots.txt couldn't be read",
   "disallowed": "a Disallow rule is the most specific match"
  },
  "notice": "robots.txt states a site's crawling preferences (RFC 9309). It isn't a terms-of-service or a license: a site's terms may still forbid automated access, and robots.txt doesn't grant permission to use the content.",
  "origin": "https://www.nytimes.com",
  "details": {
   "*": {
    "rule": null,
    "group": "*",
    "verdict": "no rule"
   },
   "CCBot": {
    "rule": "Disallow: /",
    "group": "CCBot",
    "verdict": "disallowed"
   },
   "GPTBot": {
    "rule": "Disallow: /",
    "group": "GPTBot",
    "verdict": "disallowed"
   },
   "ClaudeBot": {
    "rule": "Disallow: /",
    "group": "ClaudeBot",
    "verdict": "disallowed"
   },
   "Claude-User": {
    "rule": "Disallow: /",
    "group": "Claude-User",
    "verdict": "disallowed"
   },
   "PerplexityBot": {
    "rule": "Disallow: /",
    "group": "PerplexityBot",
    "verdict": "disallowed"
   },
   "Google-Extended": {
    "rule": "Disallow: /",
    "group": "Google-Extended",
    "verdict": "disallowed"
   }
  },
  "results": {
   "*": "no rule",
   "CCBot": "disallowed",
   "GPTBot": "disallowed",
   "ClaudeBot": "disallowed",
   "Claude-User": "disallowed",
   "PerplexityBot": "disallowed",
   "Google-Extended": "disallowed"
  },
  "llms_txt": {
   "status": 404,
   "present": false
  },
  "sitemaps": [
   "https://www.nytimes.com/sitemaps/new/news.xml.gz",
   "https://www.nytimes.com/sitemaps/new/sitemap.xml.gz",
   "https://www.nytimes.com/sitemaps/new/collections.xml.gz",
   "https://www.nytimes.com/sitemaps/new/video.xml.gz",
   "https://www.nytimes.com/sitemaps/new/cooking.xml.gz"
  ],
  "robots_url": "https://www.nytimes.com/robots.txt",
  "http_status": 200,
  "robots_found": true
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/joi-presign-robots-txt-checker-1988948f/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from joi-presign.joi-agent.workers.dev](https://www.zero.xyz/host/joi-presign.joi-agent.workers.dev/llms.txt)