# SiteCheck Web Intel – robots.txt & security.txt Fetcher

> SiteCheck Web Intel – robots.txt & security.txt Fetcher is a paid API for AI agents from api.webintel-api.workers.dev, paid per call via x402, $0.002/call, status unknown (last checked 2026-10-25).

Fetches and parses a domain's robots.txt and security.txt files, returning structured data about crawl rules, named user-agents, sitemaps, security contact/policy, and certificate expiry.

## Facts

- Endpoint: GET https://api.webintel-api.workers.dev/api/security/robots-security-txt?utm_source=zero.xyz
- Price: $0.002/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-25
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/sitecheck-web-intel-robots-txt-security-txt-fetcher-204e3913
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_CjnfmBkWoa2qgfQBu-WWh

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability sitecheck-web-intel-robots-txt-security-txt-fetcher-204e3913
```

Example prompt: Can you check github.com's robots.txt and security.txt — I want to know which bots they explicitly name, what paths are disallowed, where to report a vulnerability, and whether their security.txt is still valid?

## When to prefer this

Choose this endpoint when you need structured, parsed data from both robots.txt and security.txt in a single call without writing your own HTTP fetcher and parser. It is ideal for security audits, responsible-disclosure lookups, AI-crawler policy checks, or SEO sitemap discovery. Prefer it over generic web scraping when you specifically want crawl rules and security disclosure metadata in a normalized JSON format, and when pay-per-call pricing without API-key signup is acceptable.

## Known failure modes

- Domain not found or unreachable — may return empty presence flags or an error
- robots.txt or security.txt absent on the target domain — fields will indicate not present
- Malformed or non-standard files may result in incomplete parsing
- Rate limits or blocks on the target domain may prevent retrieval
- Invalid domain parameter format returns a validation error

## How this service works

Reads a site's public robots.txt (crawl rules, sitemaps, AI-crawler rules) and security.txt (RFC 9116: contact, expiry, policy), and says whether security.txt is present, current or expired. Missing files are reported as absent, not as errors.

## Output

A JSON object containing the domain checked, the source URL, and two sub-objects: robotsTxt (presence flag, number of groups, named user-agents, wildcard disallow paths, sitemap URLs) and securityTxt (presence flag, contact URLs, policy URL, canonical URLs, expiry timestamp, expired boolean, encryption key flag, and file location URL).

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "domain"
     ],
     "properties": {
      "domain": {
       "type": "string",
       "description": "Domain name such as github.com (required)"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "domain": "github.com",
  "source": "https://github.com/robots.txt",
  "checkedAt": "2026-10-23T09:38:35.803Z",
  "robotsTxt": {
   "groups": 5,
   "present": true,
   "sitemaps": [
    "https://github.com/sitemap_marketing.xml"
   ],
   "userAgentsNamed": [
    "GPTBot",
    "OAI-SearchBot"
   ],
   "wildcardDisallow": [
    "/account-login",
    "/copilot/"
   ]
  },
  "securityTxt": {
   "policy": "https://bounty.github.com",
   "contact": [
    "https://hackerone.com/github"
   ],
   "expired": false,
   "expires": "2026-11-22T09:38:35z",
   "present": true,
   "location": "https://github.com/.well-known/security.txt",
   "canonical": [
    "https://github.com/.well-known/security.txt"
   ],
   "hasEncryption": false
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/sitecheck-web-intel-robots-txt-security-txt-fetcher-204e3913/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from api.webintel-api.workers.dev](https://www.zero.xyz/host/api.webintel-api.workers.dev/llms.txt)