# Northline WebKit – robots.txt Check

> Northline WebKit – robots.txt Check is a paid API for AI agents from northline-webkit.northline-c86.workers.dev, paid per call via x402, $0.001/call, status unknown (last checked 2026-10-31).

Fetches and parses a URL's robots.txt to determine crawl permissions for any user-agent, including known AI crawlers like GPTBot and ClaudeBot.

## Facts

- Endpoint: GET https://northline-webkit.northline-c86.workers.dev/v1/robots/check?utm_source=zero.xyz
- Price: $0.001/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-31
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/northline-webkit-robots-txt-check-3b1e1de9
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap__z4l1V8J-2cQQzJf7zwBB

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability northline-webkit-robots-txt-check-3b1e1de9
```

Example prompt: Check if https://www.nytimes.com is allowed to be crawled — specifically whether GPTBot is permitted according to the site's robots.txt.

## When to prefer this

Use this endpoint when an agent needs to programmatically verify crawl permissions before fetching or indexing web content, especially when AI-specific crawler compliance (GPTBot, ClaudeBot) matters. Ideal for building ethical web scrapers, content pipelines, or SEO tools that must respect robots.txt. Prefer over manual HTTP fetching because it handles parsing, rule matching, and AI-crawler-specific logic in a single call with no signup required.

## Known failure modes

- URL is unreachable or returns non-2xx — may return error or assume allow-all
- Domain has no robots.txt — typically treated as fully allowed
- Malformed robots.txt — partial parsing may occur
- Invalid or missing URL parameter — returns 400-level error
- Private/internal URLs not publicly accessible — connection timeout or error

## How this service works

Pay-per-call web and domain utilities for AI agents: URL to Markdown, link preview metadata, links, uptime/redirect check, DNS, email-domain check, WHOIS/RDAP, robots.txt and a one-call domain report. x402 (USDC on Base), no API key, no signup.

## Output

Returns a JSON object indicating whether the given URL path is allowed for the specified user-agent, a list of sitemap URLs declared in robots.txt, per-crawler allow/deny flags for known AI crawlers (GPTBot, ClaudeBot, etc.), and the specific matched rule from the robots.txt file.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "$schema": "https://json-schema.org/draft/2020-12/schema",
 "required": [
  "input"
 ],
 "properties": {
  "input": {
   "type": "object",
   "required": [
    "type",
    "method"
   ],
   "properties": {
    "type": {
     "type": "string",
     "const": "http"
    },
    "method": {
     "enum": [
      "GET"
     ],
     "type": "string"
    },
    "queryParams": {
     "type": "object",
     "required": [
      "url"
     ],
     "properties": {
      "url": {
       "type": "string",
       "minLength": 3,
       "description": "Public http(s) URL (scheme optional)"
      },
      "userAgent": {
       "type": "string",
       "default": "*"
      }
     }
    }
   },
   "additionalProperties": false
  },
  "output": {
   "type": "object",
   "required": [
    "type"
   ],
   "properties": {
    "type": {
     "type": "string"
    },
    "example": {
     "type": "object"
    }
   }
  }
 }
}
```

## Response schema (JSON Schema)

```json
{
 "type": "json",
 "example": {
  "allowed": false,
  "sitemaps": [
   "https://www.nytimes.com/sitemaps/new/news.xml.gz"
  ],
  "aiCrawlers": {
   "GPTBot": false,
   "ClaudeBot": false
  },
  "matchedRule": "Disallow: /"
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/northline-webkit-robots-txt-check-3b1e1de9/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from northline-webkit.northline-c86.workers.dev](https://www.zero.xyz/host/northline-webkit.northline-c86.workers.dev/llms.txt)