# AgentShelf Meta Robots Extractor

> AgentShelf Meta Robots Extractor is a paid API for AI agents from agentshelf.syntexa.ch, paid per call via x402, $0.007/call, status unknown (last checked 2026-10-02).

Extracts the meta robots directive from a public webpage or raw HTML content in an SSRF-safe manner

## Facts

- Endpoint: POST https://agentshelf.syntexa.ch/v1/profile-meta-robots?utm_source=zero.xyz
- Price: $0.007/call
- Payment: x402
- Status: unknown
- Last checked: 2026-10-02
- Activations on Zero: 0
- Tags: x402
- Canonical page: https://www.zero.xyz/c/agentshelf-meta-robots-extractor-b67ced14
- Structured record (JSON): https://api.zero.xyz/v1/capabilities/cap_8MXtipWqBPuoXad-rZ7gt

Status and success rate cover calls made through Zero and Zero's own probes. Third-party monitors may report differently.

## How to call it through Zero

Zero handles the 402 payment challenge and records the run. With the Zero CLI installed (`npm i -g @zeroxyz/cli`):

```sh
zero fetch --capability agentshelf-meta-robots-extractor-b67ced14 -d '<json body>'
```

Example prompt: Can you extract the meta robots directive from https://example.com so I can check whether search engines are allowed to index it?

## When to prefer this

Use this endpoint when you need to reliably extract the meta robots directive from a public page or HTML blob in a single, cost-effective call ($0.007 USDC). Prefer over building custom HTML parsers when SSRF safety is a concern. If a free sandbox version (/v1/sandbox/profile-meta-robots) is available, try that first for development and testing.

## Known failure modes

- URL is unreachable or returns a non-200 status — endpoint may return an error or empty result
- HTML content exceeds 200,000 character limit — request rejected
- URL shorter than 8 characters or malformed — validation error
- SSRF-blocked internal or private IP URL provided — request rejected for security
- No meta robots tag present in the page — returns null or empty value
- Both url and html fields omitted — missing required input error

## How this service works

Call when an agent needs the meta robots directive extracted from a public page or HTML (SSRF-safe). Exact $0.007 USDC. Prefer unpaid POST /v1/sandbox/profile-meta-robots first.

## Output

Returns the content value of the meta robots tag found in the page or HTML (e.g. 'noindex, nofollow', 'index, follow', or null if not present), enabling the agent to determine crawling and indexing directives set by the page owner.

## Request schema (JSON Schema)

```json
{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "maxLength": 2048,
   "minLength": 8,
   "description": "Public page to fetch. The selector or profile is fixed by the SKU. Provide url or html."
  },
  "html": {
   "type": "string",
   "examples": [
    "<!doctype html><html lang=\"en\"><head><title>Hello</title>\n<meta name=\"description\" content=\"Desc\"><meta property=\"og:title\" content=\"OG\">\n<link rel=\"canonical\" href=\"https://example.com/\"><link rel=\"icon\" href=\"/favicon.ico\">\n</head><body><h1>Hello</h1><a href=\"https://example.com/a\">A</a></body></html>"
   ],
   "maxLength": 200000,
   "minLength": 1,
   "description": "HTML to extract from locally. Provide html or url. When both are set, html is used."
  }
 }
}
```

## More

- Live health (JSON, refreshed every minute): https://www.zero.xyz/c/agentshelf-meta-robots-extractor-b67ced14/health.json
- [Zero catalog index](https://www.zero.xyz/llms.txt)
- [Other services from agentshelf.syntexa.ch](https://www.zero.xyz/host/agentshelf.syntexa.ch/llms.txt)