# Crawlwise Data — agent brief

Crawlwise Data is a pay-as-you-go HTTP API for website change signals and structured web metadata.

Positioning: "Change events for the machine-readable web." We check feeds, sitemaps and tech fingerprints on a schedule and POST signed JSON to your app (or AI agent) when something changes. On-demand lookups still work. Failed/blocked/robots-disallowed calls are free. Noop monitor re-checks cost 2 credits (5 for techstack).

Runtime: deterministic Node.js 20 on Netlify. No headless browser. No LLM at request time.

Base URL: https://crawlwisedata.com
OpenAPI: https://crawlwisedata.com/openapi.json
Human docs: https://crawlwisedata.com/docs.html
Pricing HTML / JSON: /pricing.html · /v1/pricing
Bot info (CrawlwiseBot): /bot.html
Operator: Crawlwise · Crawlwise Data support — crawlwise.support@gmail.com

## Authentication

- Header: `Authorization: Bearer cw_live_...` (test mode: `cw_test_...`; older `wd_live_` keys still work) or `X-API-Key: cw_live_...`
- Keys are shown once (at checkout or in the trial response); only a SHA-256 hash is stored
- Free trial key: see "Free trial key" below (1,000 one-time credits, no card)
- Without a key, data endpoints run in demo mode (20/IP/day, smaller max, no monitors; Retry-After until 00:00 UTC)
- Keyed data endpoints do not send CORS headers (server-side use only)

## Free trial key

POST /v1/trial with Content-Type: application/json and body { "email": "you@company.com", "acceptTerms": true }
(acceptTerms: true accepts the Terms of Service at /terms.html. Without it the key answers 403 terms_required until you POST /v1/terms/accept {"acceptTerms":true} with the key.)
→ 201 { "ok": true, "trial": true, "accountId": "acct_...", "apiKey": "cw_test_...", "credits": 1000, "limits": {...} }

- One trial per normalized email (case, +tags, Gmail dots ignored) every 30 days; per network 5 attempts/hour and 10/day (Retry-After never more than 24 h); global daily cap of 25 issued trials (refused requests never use a slot); a 429 says until when trials are paused (same time as Retry-After) and offers the $10 pack (100,000 credits) instead
- Disposable / reserved email domains and domains without MX records are refused (400, error.reason disposable_email | domain_has_no_mail)
- Already used: 409 trial_not_available (error.reason email | ip | network)
- Trial limits: 20 req/min, 2 concurrent, 5 monitors, 1 active key. If a call costs more than your balance, it's refused (402) and you're charged nothing
- GET /v1/account shows accountType "trial" and a trial object (label FREE TRIAL KEY, creditsGranted, creditsLeft)
- Buying a pack/plan from the dashboard on the same account keeps the key and ends trial status
- We never send email. GET /v1/trial returns the current rules
- Web form: https://crawlwisedata.com/#trial
- See a change during the trial (about 50 credits): monitor the built-in demo target, send a free test event, run the check now (see "See a change" below)

## Data endpoints (GET)

### /v1/techstack?url=
Params: dns (default true), minConfidence 0-100, implied (default true)
Cost: 30 credits on success
Returns technologies with version, confidence, categories, detectedBy signals; optional DNS providers
SaaS domain-verification DNS TXT records (openai-domain-verification=, anthropic-domain-verification, cursor-domain-verification, MS=, stripe-verification=, …) are NOT technologies: they are returned separately in data.verifications (confidence 0, signal "dns-txt-domain-verification", alsoDetectedAsTechnology) and never appear in technologies / technologyNames / categories. SPF includes (v=spf1 include:sendgrid.net) still count as mail-sending technologies

### /v1/feed?url=
Params: max, include, exclude, publishedAfter, content, monitor
Cost: 10 credits (or 2 on monitor re-check with no new items)
Autodiscovers feed from HTML link rel=alternate. monitor=NAME → baseline then new items only

### /v1/sitemap?url=
Params: max (default 1000, max 10000), include/exclude regex, lastmodAfter, monitor
Cost: 10 + 1 per URL returned; on change, bills per changed URL: delivering a sitemap change needs at least 11 credits (10 base + 1 per URL); a check that finds no change costs 2. Carried-over URLs from a partial delivery cost 1 each (no second base fee). Purchases: sign in and buy from /dashboard.html?buy=<item> or POST /v1/checkout (server-created Checkout Session bound to your account).
Monitor baseline (first call for a monitor name): flat 10 credits for any site size; only 10 is reserved, so a trial can baseline a 10,000-URL sitemap. Returns counts + a 20-URL sample (data.monitor.baselinePricing "flat", urlsInSnapshot). 50 flat baselines/account/day (10 on a trial), then per-URL pricing with the full list
monitor=NAME → new / updated / removed. Truncated results do not update snapshot and never report "removed"

## Monitors & webhooks

POST /v1/monitors JSON body:
{ "endpoint": "feed|sitemap|techstack", "name": "my-list", "url": "https://...", "webhookUrl": "https://...", "schedule": true, "intervalSec": 900 }

Crawlwise runs schedule=true monitors and POSTs:

{
  "id": "evt_...",
  "type": "monitor.change",
  "created_at": "...",
  "detected_at": "...",
  "account_id": "acct_...",
  "monitor": { "endpoint": "feed", "name": "my-list", "url": "..." },
  "change": { "kind": "feed.new_items" | "sitemap.diff" | "techstack.diff", ... },
  "credits": { "charged": N }
}

Headers on delivery:
- X-Crawlwise-Signature: t=<unix>,v1=<hex hmac-sha256 of `${t}.${rawBody}`>
- X-Crawlwise-Delivery, X-Crawlwise-Timestamp, X-Crawlwise-Event
- User-Agent: Crawlwise-Webhooks/1.0

Test event (free, 10/hour/account): POST /v1/monitors/test { "endpoint": "feed", "name": "my-list" }
→ POSTs { "type": "monitor.test", "test": true, ... synthetic change ..., "credits": { "charged": 0 } } to the monitor webhookUrl, signed like real events, plus header X-Crawlwise-Test: 1. One attempt, 8 s timeout, no redirects, non-public addresses refused. Response: delivered, status, eventId, and sent.body + sent.headers (exact signed body and X-Crawlwise-Signature)

Check now: POST /v1/monitors/run { "endpoint": "feed", "name": "my-list" }
→ runs one check with scheduled-check billing; if it changed and webhookUrl is set, a real signed monitor.change is POSTed (one attempt). 429 + Retry-After inside the minimum interval (the same on every plan); 30 runs/hour/account. Works with or without schedule:true

Get secret: GET /v1/webhooks/secret
Rotate: POST /v1/webhooks/secret/rotate
Recent attempts: GET /v1/webhooks/deliveries?limit=25
DELETE /v1/monitors?endpoint=feed&name=my-list

Scheduled checks, usually within an hour, plus check-now on demand (the scheduler is a 5-minute cron that can run late); use POST /v1/monitors/run to check now. intervalSec is a lower bound, not a guarantee.

## See a change (trial)

Built-in demo target on this host (GET /v1/demo-target): https://crawlwisedata.com/demo-target/feed.xml gains one item every 5 minutes; https://crawlwisedata.com/demo-target/sitemap.xml adds one URL, removes one and updates one lastmod every 5 minutes.

1. POST /v1/monitors { "endpoint": "feed", "name": "demo", "url": "https://crawlwisedata.com/demo-target/feed.xml", "webhookUrl": "https://your.app/hook", "schedule": true }
2. POST /v1/monitors/test { "endpoint": "feed", "name": "demo" } → signed monitor.test at your webhook (free)
3. POST /v1/monitors/run { "endpoint": "feed", "name": "demo" } → baseline (10 credits)
4. After 15 minutes (the minimum interval, the same on every plan), POST /v1/monitors/run again → "changed": true and a real signed monitor.change POST (10 credits)

## Account & public

- POST /v1/trial — free trial key (no auth), GET /v1/trial — trial rules
- GET /v1/demo-target — URLs of the built-in demo feed/sitemap that change every 5 minutes
- GET /v1/account — balance, plan limits, keys, top-up links, accountType (trial | standard)
- GET /v1/pricing — creditUsd, prices, packs, subscriptions, demo/limits, refund note
- GET /v1/health — { ok, storage, storageMs, time } (503 if storage fails)
- GET /v1/status — component status for /status.html

## Credits (1 credit = $0.0001)

- techstack 30 · feed 10 · sitemap 10+1/URL
- monitor noop: 2 (feed/sitemap) or 5 (techstack)
- sitemap monitor baseline: flat 10 (any size) · test event: 0
- delivering a sitemap change needs at least 11 credits (10 base + 1 per URL); a check that finds no change costs 2
- Packs (one-time, item ids for POST /v1/checkout): pack_10 = $10 for 100,000 credits; pack_25 = $25 for 262,500 credits
- Plan (monthly): starter = $19/month for 250,000 credits/month (one plan)
- pack_10: 100,000 credits for $10, one-time. Credits never expire while your account is active. A sitemap change costs at least 11 credits; a check with no change costs 2.
- pack_25: 262,500 credits for $25, one-time (5% bonus). Credits never expire while your account is active. A sitemap change costs at least 11 credits; a check with no change costs 2.
- starter: 250,000 credits every month for $19, about 43 monitors. Scheduled checks, usually within an hour, plus check-now on demand. Starter credits carry over for one billing period, then expire.

## MCP (local)

Package name: crawlwise-mcp (stdio). Env: CRAWLWISE_API_KEY, optional CRAWLWISE_BASE_URL.
No key yet? Get a free trial key with POST /v1/trial (see above) and set it as CRAWLWISE_API_KEY.
Tools: techstack, feed, sitemap, list_monitors, create_monitor, test_monitor_webhook, run_monitor, get_account.
Install from local path or npm when published. Config snippets in the package README for Claude Desktop and Cursor.

## Crawler ethics

- User-Agent CrawlwiseBot; robots.txt always respected (RFC 9309)
- No login, forms, JS rendering, CAPTCHA solving or bot-protection bypass
- Site owners: /bot.html for opt-out
- Private/reserved IPs blocked after DNS and redirects

## Errors

JSON `{ ok:false, error:{ code, message, billable:false } }`. Common codes: invalid_params, invalid_key, insufficient_credits, trial_not_available (409), robots_disallowed, no_data_found, target_error, rate_limited, timeout, internal_error. Respect Retry-After on 429.


## Free browser tools

- https://crawlwisedata.com/tech-stack.html
- https://crawlwisedata.com/sitemap-checker.html

## Compare / use cases

- https://crawlwisedata.com/crawlwise-vs-visualping.html — API change alerts vs visual UI monitors
- https://crawlwisedata.com/crawlwise-vs-builtwith.html — monitoring wedge vs historical tech DB (we do not claim to beat BuiltWith’s database)
- https://crawlwisedata.com/website-change-alerts-for-ai-agents.html — signed webhooks + OpenAPI/MCP for agents