CrawlwiseBot: information for website owners

Crawlwise Data is a small API operated by Crawlwise. Customers use it to read public sitemaps and feeds and to identify the technologies a web page uses. This page explains what our requests look like and how to control them.

How to recognize us

Every request we make to your site sends this User-Agent:

Mozilla/5.0 (compatible; CrawlwiseBot/1.0; +https://crawlwisedata.com/bot.html)

The robots.txt product token is CrawlwiseBot. Requests come from Netlify's serverless infrastructure (shared cloud IP addresses), so please identify us by User-Agent rather than by IP. Earlier versions identified themselves as WebDataAPI/0.1; we still honor robots.txt groups for WebDataAPI.

What we fetch

Only what a customer asks for, one request at a time:

• /robots.txt, always checked first;
• your sitemap files (from robots.txt Sitemap: lines or common paths such as /sitemap.xml), up to 50 files per request;
• your RSS/Atom/JSON feed (and the home page, to discover it);
• a single page (plus public DNS records) to detect technologies.

We don't follow links to crawl your site, submit forms, log in, run JavaScript, solve CAPTCHAs, or try to get around bot protection. If your server answers with an error (for example 403 or 429), we return that error to the customer and don't retry. Each fetch has a 15-second timeout and a size limit.

How we handle robots.txt

• We follow RFC 9309. If your robots.txt has a group for CrawlwiseBot, we obey that group. Otherwise we obey the User-agent: * group. Allow/Disallow use longest-match, with * and $ wildcards.
• If robots.txt returns a 5xx server error, we treat the whole site as disallowed. If it returns 404 (or another 4xx), we treat the site as allowed, as the standard says.
• We check robots.txt for the host of the URL the customer asked for, and separately for every other host that a sitemap index points to. We follow at most 5 HTTP redirects and never fetch private or internal network addresses.

How to block us

To block only us:

User-agent: CrawlwiseBot
Disallow: /

To block only part of your site, for example your feeds:

User-agent: CrawlwiseBot
Disallow: /feed

Changes take effect on the next request; we don't cache robots.txt between customer requests. You can also ask us to add your domain (and all its subdomains) to our opt-out list, and we will then refuse all customer requests for it, whatever your robots.txt says. Use the opt-out form or email Crawlwise Data support — crawlwise.support@gmail.com with the domain. We act on opt-out requests within 2 business days, usually sooner.

Report a problem

If you think our bot misbehaved (too many requests, ignored robots.txt, or was used for abuse), please tell us the domain, the approximate time (with time zone) and, if possible, log lines showing our User-Agent. Use the report form or email Crawlwise Data support — crawlwise.support@gmail.com. We investigate every report and can suspend the customer account involved.