Citiview Website Crawler

Choose this endpoint when you need to extract textual content from multiple pages of a website in one call, especially for documentation sites, blogs, or content archives. It is ideal when you want clean Markdown output rather than raw HTML, and when following internal links automatically saves you from orchestrating individual page fetches. Prefer it over single-page scrapers when the target content spans many URLs under a shared domain or path prefix.

• Start URL is unreachable or returns non-200 — crawl fails or returns empty results
• robots.txt disallows crawling — pages are skipped or crawl is blocked
• Path prefix does not match any linked pages — only the start page is returned
• Max pages exceeded silently — results capped at 100
• Store link expires after 7 days — downstream consumers must fetch before expiry
• JavaScript-heavy SPAs may not render fully — content may be incomplete