100 Services from crawlspur.halowerk.com

Checks whether a given URL is allowed to be crawled according to the site's robots.txt file.

Checks whether a paginated URL is allowed to be crawled according to the site's robots.txt rules.

Checks whether a given URL rule is permitted by the site's robots.txt file

Checks the X-Robots-Tag header value found on a robots.txt file at a given URL.

Checks the X-Robots-Tag HTTP header value on an image resource to assess its indexability directive.

Checks the X-Robots-Tag HTTP header value on a canonical URL object for SEO indexability directives.

Checks and returns the X-Robots-Tag HTTP header value found on a given parameter URL, as part of SEO crawlability auditing.

Checks the X-Robots-Tag HTTP header value on a given page URL and returns the crawlability finding for that page object.

Checks the X-Robots-Tag header value for a given URL rule object to assess its crawl/indexing directives.

Checks the meta-robots tag value on a given page URL and returns the detected finding.

Checks whether a parameter-based URL has a self-referencing canonical tag pointing to itself.

Checks whether a given sitemap URL is permitted (allowed) by the site's robots.txt file.

Checks the X-Robots-Tag HTTP header value on a news document URL and returns the crawl/indexing directive found.

Checks whether a URL or domain is technically indexable based on its robots.txt file.

Checks the meta-robots directive value for a given parameterized URL to assess its crawlability and indexability signals.

Checks the meta-robots tag value for a given URL rule object, returning the crawl/indexing directive found at that URL.

Checks whether an image object on a webpage is technically indexable by search engine crawlers.

Checks whether a given webpage URL is allowed to be crawled according to the site's robots.txt file

Checks the HTTP status code of a sitemap URL, reporting whether it is accessible and returning the correct response.

Checks the X-Robots-Tag header value on a sitemap object for SEO crawlability analysis

Checks whether a webpage's canonical tag correctly points to itself (self-referencing canonical URL) on a given target URL.

Checks the HTTP status of the sitemap URL declared in a website's robots.txt file

Checks the HTTP status of the sitemap for a given page URL on a website.

Checks whether a pagination page URL has a self-referencing canonical tag pointing to itself

Checks whether the canonical URL of a given page is permitted (not blocked) by the site's robots.txt rules.

Checks the technical indexability status of a given web page URL, returning a crawl/indexability audit result.

Checks whether a sitemap object has a self-referencing canonical address (i.e. the canonical URL points to itself).

Checks whether a given URL (with parameters) is allowed to be crawled according to the site's robots.txt file

Checks whether a given sitemap URL is itself listed (included) in another sitemap or sitemap index.

Checks the HTTP status of the sitemap referenced by a canonical URL object on a given page.

Checks the meta-robots directive value on a video object at a given URL for SEO crawlability assessment.

Checks the X-Robots-Tag HTTP header value for video objects on a given URL to assess crawlability directives.

Checks whether a video resource URL is permitted to be crawled according to the site's robots.txt file

Checks and validates the delivered content type of a canonical URL target for SEO technical auditing.

Checks whether a canonical URL is included in the target website's XML sitemap.

Checks the declared crawl-delay directive as it applies to an image object on a given URL or domain.

Checks whether a video object (page) is present/included in the website's XML sitemap.

Checks whether a given parameterized URL is included in the target website's sitemap.

Checks the X-Robots-Tag header value on pagination objects at a given URL.

Checks whether a given page URL is listed (contained) in the website's sitemap.

Checks the meta-robots directive value on a sitemap object for a given URL.

Checks the technical indexability status of the canonical element on a given URL or web object.

Checks the technical indexability status of a news document URL, returning a structured crawl audit finding.

Checks whether a parameterized URL is technically indexable by search engines (e.g. not blocked by robots.txt, noindex directives, or other crawl barriers).

Checks the meta-robots tag value found on an image object at a given URL for SEO crawl and indexing analysis.

Checks the technical indexability of a URL based on URL-rule criteria (e.g. URL patterns, robots rules, canonical signals).

Checks and returns the delivered HTTP content-type of an image object at a given URL.

Checks the meta-robots directive value on a canonical URL object for SEO crawlability analysis

Checks the meta-robots directive value found in a website's robots.txt file object

Checks the declared crawl-delay directive for a sitemap object and returns the audit finding.

Checks whether a webpage (Seite) has a self-referencing canonical tag pointing to itself, as an SEO crawl audit finding.

Checks and validates the delivered content type of a sitemap object at a given URL.

Checks a sitemap URL for technical indexability findings, assessing whether the sitemap and its referenced pages are crawlable and indexable.

Checks the meta-robots tag value on a news document URL and returns the SEO crawl directive finding.

Checks the meta-robots tag value on paginated pages of a given URL, returning an SEO crawlability finding.

Checks the technical indexability of pagination objects on a given URL using SEO crawl analysis.

Checks the declared crawl delay setting found in the robots.txt or relevant directives for a given parameter-style URL.

Checks the number of URL address parameters found on a given page object via SEO crawl analysis.

Checks the delivered content type of a news document URL and returns a structured audit finding about the HTTP content-type header.

Checks a paginated URL and reports the number of address (URL) parameters present on that page.

Checks whether an image object on a webpage has a self-referencing canonical URL (i.e. the canonical tag points back to the page itself).

Checks whether a robots.txt file contains a self-referencing canonical URL declaration, returning an SEO audit finding for the target URL.

Checks whether a URL rule object has a self-referencing canonical address (canonical tag pointing to itself).

Checks whether a video page's canonical tag correctly points to itself (self-referencing canonical validation).

Checks the declared crawl-delay directive as it applies to a news document URL, verifying whether a crawl delay is specified for that resource.

Checks a robots.txt file to detect and extract the declared Crawl-Delay directive for a given URL or domain.

Checks and reports the declared crawl-delay directive for a given web page or URL target.

Checks the number of URL query parameters found in sitemap entries for a given website

Checks a URL rule object for the finding 'number of URL parameters' (Zahl der Adressparameter) — i.e. counts query string parameters on a given URL.

Checks whether a given image URL is allowed to be crawled according to the site's robots.txt rules.

Checks whether a news document URL is allowed to be crawled according to the site's robots.txt rules.

Checks whether a given image URL is listed (included) in a website's XML sitemap.

Checks whether a news document URL is included in the website's sitemap.

Checks whether a paginated URL is present (included) in the website's sitemap.

Checks whether a website's robots.txt file is listed/included in its XML sitemap.

Checks whether a given URL rule (url-regel) appears in the sitemap of a target website.

Checks the HTTP status of the sitemap for a given news-document URL, verifying whether the sitemap is reachable and returning the correct status code.

Checks and returns the HTTP status of the sitemap entry for a pagination object at a given URL.

Checks the HTTP status of a sitemap referenced by or associated with a given URL parameter, as part of SEO technical auditing.

Checks and reports the delivered content type of a web page URL, verifying what MIME/content type the server actually returns for that page.

Checks the declared crawl-delay directive for a canonical URL by inspecting the robots.txt or related crawl configuration at the target address.

Checks whether a video resource is technically indexable by search engine crawlers.

Checks whether a news document page has a self-referencing canonical URL (i.e., the canonical tag points to the page itself).

Checks the HTTP status code returned for a URL rule object in the context of sitemap crawling.

Checks and validates the delivered content type of a robots.txt file at a given URL.

Checks the declared crawl delay setting for a URL rule object by fetching and analyzing the relevant robots.txt directive.

Checks the delivered content type (MIME type) of a given parameter URL by fetching it and inspecting the HTTP response headers.

Checks the delivered content type of a URL rule object via HTTP inspection

Checks and validates the delivered content type of a video resource by fetching it and inspecting the HTTP response headers.

Checks the number of URL query parameters on the canonical tag of a given web page or object.

Checks a given URL and counts the number of URL query parameters present (Zahl der Adressparameter) for the parameter-url object type.

Checks the number of URL/query parameters found on a robots.txt file URL for a given web target.

Checks the HTTP status of the sitemap associated with an image ('bild') object on a given URL.

Checks the number of URL query parameters on a video page for SEO and crawlability analysis.

Checks whether a pagination URL or page declares a crawl-delay directive in its robots.txt or HTTP headers.

Checks the HTTP status of a video object's sitemap entry for SEO crawl analysis.

Checks and reports the delivered content type (MIME type) of a paginated page object at a given URL.

Checks the number of URL query parameters on a news document page as an SEO crawlability signal.

Checks the number of URL address parameters found on an image object via web crawling analysis.

Checks the declared crawl delay directive for video objects on a given URL or domain.