API documentation
Every endpoint and parameter, with cURL, JavaScript and Python examples. Base URL: https://crawlwisedata.com
Quickstart: zero to first call (and first webhook)
Five minutes to a successful keyed call and a scheduled monitor that POSTs a signed change event.
- Try it without a key (free demo, 10 calls per day per IP):Shell
curl "https://crawlwisedata.com/v1/techstack?url=wordpress.org"Or use the free browser tools: tech stack lookup · sitemap & feed checker. Compare: vs Visualping · vs BuiltWith · for AI agents.
- Get a key: request a free trial key (1,000 one-time credits, no card) with the form below or
POST /v1/trial, or buy a credit pack or monthly plan on the pricing page. Either way the key is shown once. Copy it.Shellcurl -X POST "https://crawlwisedata.com/v1/trial" -H "Content-Type: application/json" -d '{"email":"you@company.com","acceptTerms":true}'Get a free trial key
1,000 free credits, one time (about 33 tech-stack lookups or 100 feed reads). No card. One key per email every 30 days; per network up to 5 requests an hour and 10 a day. The trial flow never emails you.
- One successful keyed call (copyable):
export CRAWLWISE_KEY=cw_test_... curl -G "https://crawlwisedata.com/v1/techstack" -H "Authorization: Bearer $CRAWLWISE_KEY" -d url=wordpress.orgconst res = await fetch('https://crawlwisedata.com/v1/techstack?url=wordpress.org', { headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}` }, }); const json = await res.json(); if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`); console.log(json.data.technologyNames, json.charged);import os, requests r = requests.get( 'https://crawlwisedata.com/v1/techstack', params={'url': 'wordpress.org'}, headers={'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}, timeout=60, ) data = r.json() if not data.get('ok'): raise RuntimeError(data.get('error')) print(data['data']['technologyNames'], data['charged']) - Create a monitor + webhook (Crawlwise runs the schedule; you receive signed POSTs when something changes):
curl -X POST "https://crawlwisedata.com/v1/monitors" -H "Authorization: Bearer $CRAWLWISE_KEY" -H "Content-Type: application/json" -d '{ "endpoint": "sitemap", "name": "competitor", "url": "https://example.com", "webhookUrl": "https://your.app/hooks/crawlwise", "schedule": true, "intervalSec": 900 }' # Fetch the HMAC secret used in X-Crawlwise-Signature curl -H "Authorization: Bearer $CRAWLWISE_KEY" "https://crawlwisedata.com/v1/webhooks/secret"const headers = { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}`, 'Content-Type': 'application/json', }; const mon = await fetch('https://crawlwisedata.com/v1/monitors', { method: 'POST', headers, body: JSON.stringify({ endpoint: 'sitemap', name: 'competitor', url: 'https://example.com', webhookUrl: 'https://your.app/hooks/crawlwise', schedule: true, intervalSec: 900, }), }).then((r) => r.json()); const secret = await fetch('https://crawlwisedata.com/v1/webhooks/secret', { headers: { Authorization: headers.Authorization }, }).then((r) => r.json()); console.log(mon, secret.secret);import os, requests h = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"} mon = requests.post('https://crawlwisedata.com/v1/monitors', headers={**h, 'Content-Type': 'application/json'}, json={ 'endpoint': 'sitemap', 'name': 'competitor', 'url': 'https://example.com', 'webhookUrl': 'https://your.app/hooks/crawlwise', 'schedule': True, 'intervalSec': 900, }, timeout=60).json() secret = requests.get('https://crawlwisedata.com/v1/webhooks/secret', headers=h, timeout=60).json() print(mon, secret['secret'])Verify deliveries with the Node/Python snippets under Monitors & change alerts (
X-Crawlwise-Signature). - See it work right away:
POST /v1/monitors/testsends a free signed sample event to your webhook, and the built-in demo target produces a real change every 5 minutes. Walkthrough: See a change during your trial. - Check your balance any time with
GET /v1/accountor in the dashboard. - Agents: import OpenAPI, or run the local
crawlwise-mcpMCP server (see AI agents).
See a change during your trial
You don't need a site that happens to change, or to wait for a schedule. With a trial key (1,000 credits) the whole loop costs about 50 credits:
- Create a monitor on the built-in demo target (
/demo-target/feed.xmlgains an item every 5 minutes;/demo-target/sitemap.xmladds, removes and updates a URL every 5 minutes):Shellcurl -X POST "https://crawlwisedata.com/v1/monitors" -H "Authorization: Bearer $CRAWLWISE_KEY" -H "Content-Type: application/json" -d '{"endpoint":"feed","name":"demo","url":"https://crawlwisedata.com/demo-target/feed.xml", "webhookUrl":"https://your.app/hooks/crawlwise","schedule":true}' - Send a test event (free, 10/hour): a signed
monitor.testPOST to your webhookUrl, signed exactly like real events. The response shows the exact body and signature sent, so you can check your verifier.Shellcurl -X POST "https://crawlwisedata.com/v1/monitors/test" -H "Authorization: Bearer $CRAWLWISE_KEY" -H "Content-Type: application/json" -d '{"endpoint":"feed","name":"demo"}' - Take the baseline now (10 credits for a feed; a sitemap baseline is a flat 10 credits for any site size):Shell
curl -X POST "https://crawlwisedata.com/v1/monitors/run" -H "Authorization: Bearer $CRAWLWISE_KEY" -H "Content-Type: application/json" -d '{"endpoint":"feed","name":"demo"}' - After the minimum interval (the same on every plan; a 429 includes Retry-After), run it again. The demo feed has new items, so the response shows
"changed": trueand Crawlwise POSTs a real signedmonitor.changeevent to your webhook (10 credits). Scheduled monitors do the same on their own.
Your own sites work the same way. For a big sitemap, set max high enough for the whole site (up to 10,000) on the baseline: it is still a flat 10 credits, and later checks bill only for changed URLs.
Authentication
Send your key as Authorization: Bearer cw_live_... (or X-API-Key: cw_live_...). Keys start with cw_live_ (live) or cw_test_ (test mode); older wd_live_ keys keep working. Keys are 40 characters, and we store only a SHA-256 hash, so a key can never be shown again. If you lose a key, recover a new one; if it leaks, rotate it (POST /v1/keys/rotate). Without a key, data endpoints run in demo mode with smaller limits. Dashboard sessions (cws_..., 30 minutes) work for account endpoints only. Never put your key in public front-end code: keyed endpoints don't send CORS headers, on purpose.
Responses
All responses are JSON. Success: { "ok": true, "endpoint", "mode": "paid"|"demo", "charged": {credits, usd}, "balance": {credits, usd} | null, "data": {...} }. Response headers: X-Credits-Charged, X-Credits-Remaining, X-RateLimit-Limit, X-RateLimit-Remaining; Retry-After on 429.
Errors and status codes
Every error has the same shape and is never billed:
{ "ok": false, "error": { "code": "insufficient_credits", "message": "Insufficient credits", "billable": false, ... }, "charged": { "credits": 0, "usd": 0 } }| code | HTTP | Meaning / what to do |
|---|---|---|
invalid_params | 400 | A parameter is missing, unknown or out of range. Fix the request. |
invalid_url | 400 | The url parameter is not a valid URL. |
unsupported_url | 400 | Not allowed: non-http(s) scheme, non-default port, credentials in the URL, internal hostname, or a private/reserved IP (also after DNS resolution and redirects). |
invalid_key | 401 | Missing, malformed, revoked or unknown API key/session. |
auth_required | 401 | This endpoint needs a key (or the demo is disabled). |
insufficient_credits | 402 | Balance too low for this call. One-off GET /v1/sitemap needs 10 + max credits up front (lower max). Monitor checks (scheduled, check-now and ?monitor=) reserve only the cheapest possible check (2 credits; 5 for techstack; 10 for a first snapshot) and are refused only below it; the real cost is charged when the check finishes, from the balance available then, and a sitemap change larger than that balance delivers the URLs it covers and carries the rest over. Includes needed, available and topUp links; nothing is charged. |
account_suspended | 403 | Negative balance (top up), open payment dispute, or suspended for abuse (contact support). |
robots_disallowed | 403 | The site's robots.txt disallows this URL for us (or robots.txt returned a 5xx error). We never fetch it. |
not_found | 404 | Unknown route or object. |
method_not_allowed | 405 | Wrong HTTP method. |
trial_not_available | 409 | POST /v1/trial: email already used in the last 30 days (details.reason). Network limits answer 429 rate_limited with Retry-After (at most 24 h). Use the free demo or buy a pack. |
signup_not_available | 409 | POST /v1/signup: a free account was already created for this email in the last 30 days (details.reason). Per-network signup limits (hour/day, Cloudflare WARP = one network) return 429 with Retry-After of at most 24 hours. Sign in with your key or contact support. |
payload_too_large | 413 | Request body too large. |
no_data_found | 422 | The target was reachable but had no feed/sitemap we could parse. |
target_error | 422 | The target failed: DNS error, connection refused, TLS error, HTTP ≥ 400 (often bot blocking), response too large, or too many redirects. |
rate_limited | 429 | Per-minute, concurrency, monitor-frequency, demo or form limit reached. Respect Retry-After. |
internal_error | 500 | Our fault. You were not charged. Retry with backoff. |
feature_unavailable | 503 | A dependency (e.g. payments) is not configured or temporarily unreachable. |
timeout | 504 | The target did not answer within the time limit. |
Limits
| Free trial | Pay as you go (packs) | Starter | Pro | Business | |
|---|---|---|---|---|---|
| Requests per minute per account | 20 | 60 | 120 | 300 | 600 |
| Concurrent requests | 2 | 3 | 5 | 10 | 20 |
| Monitors | 5 | 100 | 100 | 300 | 1,000 |
| Minimum monitor interval | 15 min | 15 min | 15 min | 15 min | 15 min |
All plans: max 25 s per call, 15 s per outbound request, 5 redirects, page ≤ 5 MB, feed ≤ 10 MB and 200 items, sitemap file ≤ 15 MB (after decompression), ≤ 50 sitemap files, ≤ 60 MB total and ≤ 10,000 URLs per call, 3 active keys. Only http(s) on ports 80/443 to public IP addresses. robots.txt is always respected.
Demo (no key): 10 calls/day per IP, 60/day per /24 (IPv4) or /64 (IPv6) network, 500/day in total; feed ≤ 15 items, sitemap ≤ 75 URLs, no monitors. Demo attempts count even when they fail. On 429, Retry-After is seconds until 00:00 UTC.
Costs
1 credit = $0.0001. Tech stack 30 credits, feed 10 credits, sitemap 10 + 1/URL. Monitor re-checks with no change: 2 credits (feed/sitemap) or 5 (techstack). When a change is found, the full endpoint price applies (sitemap bills per changed URL): delivering a sitemap change needs at least 11 credits (10 base + 1 per URL); a check that finds no change costs 2. If your balance covers only part of a change, the URLs it pays for are delivered and the rest carry over; delivering carried-over URLs later costs 1 credit each (no second base fee). A refusal is free and the monitor shows change detected, top up to fetch remaining N URLs. Sitemap monitor baseline (first snapshot): flat 10 credits for any site size (50/day per account, 10 on a trial; then per URL). Test events (POST /v1/monitors/test): free. Failed, blocked, timed-out, robots-disallowed and invalid calls cost nothing. Credits are reserved when a call starts and only the actual cost is deducted when it succeeds. Packs: $10 = 100,000, $25 = 262,500. Monthly: Starter $19 = 250,000/month, Pro $49 = 750,000/month, Business $149 = 2,500,000/month. Details: pricing.
pack_10: 100,000 credits for $10, one-time. Credits never expire while your account is active. A sitemap change costs at least 11 credits; a check with no change costs 2.pack_25: 262,500 credits for $25, one-time (5% bonus). Credits never expire while your account is active. A sitemap change costs at least 11 credits; a check with no change costs 2.starter: 250,000 credits every month for $19. Up to 100 monitors, 120 requests/min, 5 concurrent requests. Scheduled checks, usually within an hour, plus check-now on demand. Plan credits carry over for one billing period, then expire.pro: 750,000 credits every month for $49. Up to 300 monitors, 300 requests/min, 10 concurrent requests. Scheduled checks, usually within an hour, plus check-now on demand. Plan credits carry over for one billing period, then expire.business: 2,500,000 credits every month for $149. Up to 1,000 monitors, 600 requests/min, 20 concurrent requests. Scheduled checks, usually within an hour, plus check-now on demand. Plan credits carry over for one billing period, then expire.
Monitors & change alerts
We run them for you. POST /v1/monitors with schedule: true and an https webhookUrl. Crawlwise checks the target on its schedule (scheduled checks, usually within an hour, plus check-now on demand) and POSTs a signed JSON event when something changes (new feed items, sitemap add/update/remove, tech stack add/remove). Noop checks cost 2 credits (5 for tech). Get your signing secret from GET /v1/webhooks/secret.
Verify deliveries (Node):
import { createHmac, timingSafeEqual } from 'node:crypto';
function verify(secret, rawBody, header) {
const m = /^t=(\d+),v1=([0-9a-f]{64})$/i.exec(header || '');
if (!m) return false;
const expected = createHmac('sha256', secret).update(`${m[1]}.${rawBody}`).digest();
const got = Buffer.from(m[2], 'hex');
return got.length === expected.length && timingSafeEqual(got, expected);
}Python:
import hmac, hashlib, re
def verify(secret, raw_body, header):
m = re.match(r'^t=(\d+),v1=([0-9a-f]{64})$', header or '', re.I)
if not m: return False
expected = hmac.new(secret.encode(), f'{m[1]}.{raw_body}'.encode(), hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, m[2].lower())Retries. A delivery gets one attempt with a ~5 s timeout; redirects are not followed. If it fails (non-2xx, timeout, network error), we retry on later scheduler ticks with backoff (about 1 min, 5 min, 30 min, 2 h, 6 h; 6 attempts in total). A retry resends the same event: the same body, the same X-Crawlwise-Event id and the same X-Crawlwise-Delivery id, re-signed with a fresh timestamp (t= in X-Crawlwise-Signature, also in X-Crawlwise-Timestamp) on every attempt, so a timestamp tolerance check (we suggest rejecting t= older than 5 minutes) still passes on late retries. Retries never re-run the check and never charge again. Deduplicate on X-Crawlwise-Delivery (or the event id).
You can still poll yourself with ?monitor=NAME on feed/sitemap (same noop pricing), or call POST /v1/monitors/run to check a monitor now and get the webhook if it changed. POST /v1/monitors/test sends a free signed sample event (type monitor.test, header X-Crawlwise-Test: 1). Delete with DELETE /v1/monitors.
AI agents & OpenAPI
Machine-readable surface for agents and codegen:
/openapi.json(also YAML) — OpenAPI 3 for every public endpoint plus themonitor.changeandmonitor.testwebhook event schemas./llms.txtand/llms-full.txt— short indexes for LLM crawlers.- Free trial key for agents:
POST /v1/trialwith{"email":"…","acceptTerms":true}returns a key with 1,000 one-time credits (one per email every 30 days; per-network hourly and daily limits for shared egress). Set it asCRAWLWISE_API_KEY. - MCP: local stdio package
crawlwise-mcp(tools: techstack, feed, sitemap, list_monitors, create_monitor, test_monitor_webhook, run_monitor, get_account). Install it withnpx crawlwise-mcpand add it to Claude Desktop / Cursor. Auth: envCRAWLWISE_API_KEY. No AI runs inside Crawlwise — thin HTTP wrapper. - Billing for agents: Your AI agent calls Crawlwise with your Crawlwise API key, and every call uses your Crawlwise credits. No AI runs on our side. Free test events and failed calls cost nothing; without a key the data tools run in the limited free demo.
Data endpoints
GET /v1/techstack
Detects the technologies used by one web page (CMS, e-commerce platform, frameworks, analytics, CDN, server, and email/DNS providers from DNS records). Uses 7,600+ open-source fingerprints and reports which signal matched. SaaS domain-verification TXT records (openai-domain-verification=…, MS=…, stripe-verification=…) only prove someone set up an account for the domain, so they are NOT reported as technologies: they appear in data.verifications with confidence 0 and never in technologies, technologyNames or categories. SPF includes (v=spf1 include:sendgrid.net) still count as mail senders.
| Parameter | Rules | Meaning |
|---|---|---|
url | required | Page to analyze. A bare domain means https://domain/. |
dns | optional, default 1 | 0 skips the MX/TXT/NS/SOA/CNAME lookups. |
minConfidence | optional, 0-100, default 0 | Drop detections below this confidence. |
implied | optional, default 1 | 0 hides technologies that are only implied by others (e.g. PHP implied by WordPress). |
curl \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
"https://crawlwisedata.com/v1/techstack?url=wordpress.org&minConfidence=50"const res = await fetch('https://crawlwisedata.com/v1/techstack?url=wordpress.org&minConfidence=50', {
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}` },
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.get('https://crawlwisedata.com/v1/techstack', params={'url':'wordpress.org','minConfidence':'50'}, headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){
"ok": true, "endpoint": "techstack", "mode": "paid",
"charged": { "credits": 30, "usd": 0.003 },
"balance": { "credits": 99970, "usd": 9.997 },
"data": {
"url": "https://wordpress.org/", "domain": "wordpress.org", "statusCode": 200, "title": "Blog Tool, Publishing Platform, and CMS",
"technologyCount": 6, "technologyNames": ["WordPress", "Nginx", "PHP", "MySQL"],
"technologies": [{ "name": "WordPress", "version": null, "confidence": 100, "categories": ["CMS", "Blogs"], "implied": false, "detectedBy": ["meta", "html"] }],
"categories": { "CMS": ["WordPress"] },
"verifications": [{ "name": "OpenAI", "confidence": 0, "signal": "dns-txt-domain-verification", "record": "TXT", "categories": ["AI"], "alsoDetectedAsTechnology": false }],
"verificationNote": "verifications = SaaS domain-verification DNS TXT records. They show an account was once set up for this domain, not that the site uses the product, so they are excluded from technologies (confidence 0)."
}
}GET /v1/feed
Reads an RSS 2.0, Atom, RDF or JSON Feed and returns one normalized schema. Pass a website URL and the feed is discovered from its <link rel="alternate"> tag. With monitor=NAME each call returns only items you have not seen before under that name.
| Parameter | Rules | Meaning |
|---|---|---|
url | required | Feed URL or website URL. |
max | optional, 1-200 (demo: 1-15), default 50 (demo 20) | Maximum items returned. |
include | optional, repeatable, max 10 | Keep items whose title/summary contains any of these keywords (case-insensitive). |
exclude | optional, repeatable, max 10 | Drop items containing any of these keywords. |
publishedAfter | optional | A date (2026-09-01) or relative period (7 days, 12 hours). |
content | optional, default 0 | 1 includes full item HTML (contentHtml). |
monitor | optional (keys only), 1-40 of A-Z a-z 0-9 _ - | New-items-only mode. The first call is the baseline and returns everything currently in the feed. |
curl \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
"https://crawlwisedata.com/v1/feed?url=https://blog.example.com&monitor=my-list&include=release"const res = await fetch('https://crawlwisedata.com/v1/feed?url=https://blog.example.com&monitor=my-list&include=release', {
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}` },
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.get('https://crawlwisedata.com/v1/feed', params={'url':'https://blog.example.com','monitor':'my-list','include':'release'}, headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){
"ok": true, "endpoint": "feed", "mode": "paid",
"charged": { "credits": 10, "usd": 0.001 },
"balance": { "credits": 99960, "usd": 9.996 },
"data": {
"feedUrl": "https://blog.example.com/feed.xml", "discoveredFrom": "https://blog.example.com/", "title": "Example Blog", "itemsInFeed": 20,
"items": [{ "itemId": "https://blog.example.com/p/1", "title": "Release 2.0", "link": "https://blog.example.com/p/1", "publishedAt": "2026-09-27T12:00:00.000Z", "author": "Ann", "categories": ["news"], "summary": "...", "enclosures": [] }],
"monitor": { "name": "my-list", "baseline": false, "newItems": 1 }
}
}GET /v1/sitemap
Extracts every URL a site lists in its XML, gzipped or plain-text sitemaps (found through robots.txt Sitemap lines or common locations), with lastmod, changefreq, priority, image/video counts and hreflang alternates. With monitor=NAME you get only new, updated and removed URLs since the last call.
| Parameter | Rules | Meaning |
|---|---|---|
url | required | Site URL or a sitemap URL. |
max | optional, 1-10,000 (demo: 1-75), default 1,000 (demo 100) | Maximum URLs returned. |
include | optional, repeatable, max 10, regex ≤200 chars | Keep URLs matching any regex (case-insensitive). Backreferences and nested quantifiers are rejected. |
exclude | optional, repeatable | Drop URLs matching any regex. |
lastmodAfter | optional | Keep URLs whose lastmod is on/after this date. |
monitor | optional (keys only) | Change-detection mode. The first call (baseline) costs a flat 10 credits for any site size and returns counts plus a 20-URL sample (monitor.urlsInSnapshot); 50 flat baselines per account per day (10 on a trial), then per-URL pricing. Set max high enough for the whole site on the baseline: it costs the same. If a result is truncated the snapshot is not updated and "removed" is not reported. |
curl \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
"https://crawlwisedata.com/v1/sitemap?url=example.com&max=5000&monitor=competitor"const res = await fetch('https://crawlwisedata.com/v1/sitemap?url=example.com&max=5000&monitor=competitor', {
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}` },
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.get('https://crawlwisedata.com/v1/sitemap', params={'url':'example.com','max':'5000','monitor':'competitor'}, headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){
"ok": true, "endpoint": "sitemap", "mode": "paid",
"charged": { "credits": 13, "usd": 0.0013 },
"data": {
"site": "example.com", "discoveredVia": "robots.txt", "sitemapsFetched": 2, "truncated": false, "errors": [],
"urls": [{ "url": "https://example.com/new-page", "lastmod": "2026-09-21", "changefreq": null, "priority": null, "imageCount": 0, "videoCount": 0, "alternates": [], "sitemapUrl": "https://example.com/sitemap.xml", "changeType": "new" }],
"monitor": { "name": "competitor", "baseline": false, "counts": { "new": 1, "updated": 1, "removed": 1, "unchanged": 950 }, "snapshotSaved": true }
}
}Account
GET /v1/account
Balance (pack vs subscription credits, trial status: accountType "trial" | "standard" and a trial object with creditsGranted / creditsLeft for free trial keys, reserved, expiring grants), totals (purchased, consumed, clawed back, expired), monthly usage, the last 25 calls, plan and limits, subscription status, keys (prefix only), billing events, personal top-up/subscribe links and support info.
curl \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
"https://crawlwisedata.com/v1/account"const res = await fetch('https://crawlwisedata.com/v1/account', {
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}` },
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.get('https://crawlwisedata.com/v1/account', headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){
"ok": true, "accountId": "acct_...", "status": "active", "suspension": null,
"balance": { "credits": 349970, "usd": 34.997, "packCredits": 100000, "subscriptionCredits": 249970, "reserved": 0, "available": 349970,
"subscriptionGrants": [{ "credits": 249970, "grantedCredits": 250000, "expiresAt": "2026-11-28T00:00:00.000Z", "plan": "starter" }] },
"totals": { "creditsPurchased": 100000, "subscriptionCreditsGranted": 250000, "creditsConsumed": 30, "requests": 2 },
"plan": { "id": "starter", "name": "Starter", "limits": { "requestsPerMinute": 120, "concurrentRequests": 5, "maxMonitors": 500, "monitorMinIntervalSec": 900 } },
"subscription": { "id": "sub_...", "status": "active", "plan": "starter", "currentPeriodEnd": "2026-10-28T00:00:00.000Z", "cancelAtPeriodEnd": false },
"keys": [{ "id": "key_ab12cd34ef56", "prefix": "cw_live_AbCd…", "active": true }],
"topUp": [{ "label": "$10 - 100,000 credits", "item": "pack_10", "url": "https://crawlwisedata.com/dashboard.html?buy=pack_10", "checkout": { "method": "POST", "path": "/v1/checkout", "body": { "item": "pack_10" } } }]
}POST /v1/session
Exchanges an API key for a dashboard session token (cws_...) valid for 30 minutes. Sessions work for account endpoints only (not data endpoints) and are revoked when any key is revoked or rotated.
curl -X POST \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
"https://crawlwisedata.com/v1/session"const res = await fetch('https://crawlwisedata.com/v1/session', {
method: 'POST',
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}` },
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.post('https://crawlwisedata.com/v1/session', headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "token": "cws_...", "expiresAt": "2026-09-28T20:30:00.000Z", "accountId": "acct_..." }POST /v1/account/delete
Permanently deletes the account: cancels any subscription immediately (no further charges), revokes all keys, deletes monitors, usage history and the balance (remaining credits are forfeited). Payment records stay in Stripe for accounting.
| Parameter | Rules | Meaning |
|---|---|---|
confirm | body, required | Must be exactly "DELETE". |
curl -X POST \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
-H "Content-Type: application/json" \
-d '{"confirm":"DELETE"}' \
"https://crawlwisedata.com/v1/account/delete"const res = await fetch('https://crawlwisedata.com/v1/account/delete', {
method: 'POST',
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}`, 'Content-Type': 'application/json' },
body: JSON.stringify({"confirm":"DELETE"}),
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.post('https://crawlwisedata.com/v1/account/delete', json={'confirm':'DELETE'}, headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "deleted": "acct_...", "canceledSubscriptions": [], "note": "Account data deleted and all keys revoked. Payment records remain in Stripe for accounting." }Free trial
POST /v1/trial
Issues a free trial API key with 1,000 one-time credits. The key is returned once in the response (we store only a hash). Emails are normalized (case, +tags and Gmail dots ignored); disposable or reserved domains and domains without mail servers are refused. Network limits: 5 requests per hour and 10 per day per network (an IPv4 address or an IPv6 /48; relay services such as Cloudflare WARP count as one network), 20 keys per day per /24, and 25 issued trial keys per UTC day site-wide (only issued keys count; refused requests never use a slot). A 429 says until when free trials are paused (the same time as Retry-After, never more than 24 hours) and offers the $10 pack (100,000 credits) with a checkout link (details.buy). Trial keys work on every endpoint with trial limits (20 req/min, 2 concurrent, 5 monitors, 1 active key) and can never go below zero: a call that needs more credits than are left returns 402. Buying a pack or plan from the dashboard keeps the key and ends the trial limits. Send Content-Type: application/json. Refusals: 409 trial_not_available (reason email), 400 invalid_params (reason disposable_email | domain_has_no_mail), 429 rate_limited (reason ip_attempts | ip_attempts_day | network | global). Trial keys must accept the Terms: send acceptTerms: true, or accept later with POST /v1/terms/accept using the key (until then 403 terms_required). GET /v1/trial returns the current rules.
| Parameter | Rules | Meaning |
|---|---|---|
email | body, required | Your email address. The trial flow never emails you. |
acceptTerms | body, recommended | true = you accept the Terms of Service and Privacy Policy (/terms.html). Without it the key is issued with termsRequired: true and calls answer 403 terms_required until you POST /v1/terms/accept {"acceptTerms":true} with the key. |
curl -X POST \
-H "Content-Type: application/json" \
-d '{"email":"you@company.com","acceptTerms":true}' \
"https://crawlwisedata.com/v1/trial"const res = await fetch('https://crawlwisedata.com/v1/trial', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({"email":"you@company.com","acceptTerms":true}),
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
r = requests.post('https://crawlwisedata.com/v1/trial', json={'email':'you@company.com','acceptTerms':True}, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){
"ok": true, "trial": true, "accountId": "acct_...", "keyId": "key_...", "apiKey": "cw_test_...", "credits": 1000,
"note": "Trial key: shown once, copy it now. We store only a hash.",
"limits": { "requestsPerMinute": 20, "concurrentRequests": 2, "maxMonitors": 5, "activeKeys": 1 }
}Keys
POST /v1/keys
Creates an additional key (max 3 active). The key is returned once.
curl -X POST \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
-H "Content-Type: application/json" \
-d '{}' \
"https://crawlwisedata.com/v1/keys"const res = await fetch('https://crawlwisedata.com/v1/keys', {
method: 'POST',
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}`, 'Content-Type': 'application/json' },
body: JSON.stringify({}),
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.post('https://crawlwisedata.com/v1/keys', json={}, headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "keyId": "key_ab12cd34ef56", "apiKey": "cw_test_...", "note": "Shown once. Store it securely." }POST /v1/keys/rotate
Issues a new key and revokes ALL other keys and sessions immediately. Use this if a key may be compromised.
curl -X POST \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
-H "Content-Type: application/json" \
-d '{}' \
"https://crawlwisedata.com/v1/keys/rotate"const res = await fetch('https://crawlwisedata.com/v1/keys/rotate', {
method: 'POST',
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}`, 'Content-Type': 'application/json' },
body: JSON.stringify({}),
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.post('https://crawlwisedata.com/v1/keys/rotate', json={}, headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "keyId": "key_...", "apiKey": "cw_test_...", "note": "Shown once. All previous keys and dashboard sessions were revoked." }POST /v1/keys/revoke
Disables one key ({"keyId":"key_..."}) or all keys ({"all":true}). Revocation is immediate. After revoking all keys, use key recovery to get a new one.
| Parameter | Rules | Meaning |
|---|---|---|
keyId | body, or | Key id from GET /v1/account. |
all | body | true to disable every key. |
curl -X POST \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
-H "Content-Type: application/json" \
-d '{"keyId":"key_ab12cd34ef56"}' \
"https://crawlwisedata.com/v1/keys/revoke"const res = await fetch('https://crawlwisedata.com/v1/keys/revoke', {
method: 'POST',
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}`, 'Content-Type': 'application/json' },
body: JSON.stringify({"keyId":"key_ab12cd34ef56"}),
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.post('https://crawlwisedata.com/v1/keys/revoke', json={'keyId':'key_ab12cd34ef56'}, headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "revoked": ["key_ab12cd34ef56"] }Monitors
GET /v1/monitors
Lists your monitors (name, endpoint, target, schedule, webhook, last/next run).
curl \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
"https://crawlwisedata.com/v1/monitors"const res = await fetch('https://crawlwisedata.com/v1/monitors', {
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}` },
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.get('https://crawlwisedata.com/v1/monitors', headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "count": 1, "monitors": [{ "endpoint": "feed", "name": "my-list", "target": "https://blog.example.com/feed.xml", "schedule": true, "enabled": true, "webhookUrl": "https://example.com/hooks", "intervalSec": 900, "lastRunAt": "2026-10-05T14:00:00.000Z", "nextRunAt": "2026-10-05T14:15:00.000Z" }], "scheduling": { "noopCredits": { "feed": 2, "sitemap": 2, "techstack": 5 } } }POST /v1/monitors
Create or update a monitor (PATCH /v1/monitors updates an existing one; 404 if it does not exist). With schedule:true and an https webhookUrl, Crawlwise runs the check for you and POSTs a signed change event when something changes. endpoint may be feed, sitemap or techstack. intervalSec has the same minimum on every plan (900 seconds, 15 minutes); scheduled checks run usually within an hour, plus check-now on demand. Deprecated: sending `target` instead of `url` still works for one release; the response then carries a `warnings` entry and the headers `Deprecation: true` and `X-Crawlwise-Warning: 299 - "target is deprecated; use url"` (no standard `Warning` header). An intervalSec below 900 is not refused: it is raised to 900 and the response says so (a `warnings` entry with code `interval_clamped` and `X-Crawlwise-Warning: 299 - "intervalSec 300 clamped to 900"`). webhookUrl gets the same public-address check as url when the monitor is created or updated: private, loopback, link-local and cloud metadata addresses (also written as IPv4-mapped IPv6, octal or hex, or a hostname that resolves to one) are refused with 400 invalid_params (`field: webhookUrl`).
| Parameter | Rules | Meaning |
|---|---|---|
endpoint | body, required | feed | sitemap | techstack |
name | body, required | 1-40 of A-Z a-z 0-9 _ - |
url | body, required | Target site, feed or sitemap URL |
webhookUrl | body | https URL that receives signed POSTs |
schedule | body | true = Crawlwise runs it (requires webhookUrl) |
enabled | body | false pauses scheduled runs |
intervalSec | body | Desired seconds between checks (minimum 900, the same on every plan) |
max | body | Optional max items/URLs for feed/sitemap |
curl -X POST \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
-H "Content-Type: application/json" \
-d '{"endpoint":"feed","name":"my-list","url":"https://blog.example.com","webhookUrl":"https://example.com/hooks/crawlwise","schedule":true,"intervalSec":900}' \
"https://crawlwisedata.com/v1/monitors"const res = await fetch('https://crawlwisedata.com/v1/monitors', {
method: 'POST',
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}`, 'Content-Type': 'application/json' },
body: JSON.stringify({"endpoint":"feed","name":"my-list","url":"https://blog.example.com","webhookUrl":"https://example.com/hooks/crawlwise","schedule":true,"intervalSec":900}),
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.post('https://crawlwisedata.com/v1/monitors', json={'endpoint':'feed','name':'my-list','url':'https://blog.example.com','webhookUrl':'https://example.com/hooks/crawlwise','schedule':True,'intervalSec':900}, headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "created": true, "monitor": { "endpoint": "feed", "name": "my-list", "schedule": true, "webhookUrl": "https://example.com/hooks/crawlwise", "nextRunAt": "..." } }DELETE /v1/monitors
Deletes a monitor and its stored snapshot, freeing a monitor slot. Webhook deliveries still waiting for a retry are cancelled (never sent); the response always includes cancelledDeliveries (0 when there were none). Retries also stop when the account is suspended.
| Parameter | Rules | Meaning |
|---|---|---|
endpoint | required | feed, sitemap or techstack |
name | required | Monitor name. |
curl -X DELETE \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
"https://crawlwisedata.com/v1/monitors?endpoint=feed&name=my-list"const res = await fetch('https://crawlwisedata.com/v1/monitors?endpoint=feed&name=my-list', {
method: 'DELETE',
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}` },
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.delete('https://crawlwisedata.com/v1/monitors', params={'endpoint':'feed','name':'my-list'}, headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "deleted": { "endpoint": "feed", "name": "my-list" }, "cancelledDeliveries": 0 }POST /v1/monitors/test
Sends a signed sample event (type "monitor.test", test: true, synthetic change shaped like the monitor endpoint) to the monitor's webhookUrl right now, signed with your account secret exactly like real change events, plus header X-Crawlwise-Test: 1. One attempt, 8 s timeout, redirects are not followed, non-public addresses are refused. The response echoes the exact body and X-Crawlwise-Signature that were sent, so you can test your verifier.
| Parameter | Rules | Meaning |
|---|---|---|
endpoint | body, required | feed | sitemap | techstack |
name | body, required | Monitor name (must have a webhookUrl). |
curl -X POST \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
-H "Content-Type: application/json" \
-d '{"endpoint":"feed","name":"demo"}' \
"https://crawlwisedata.com/v1/monitors/test"const res = await fetch('https://crawlwisedata.com/v1/monitors/test', {
method: 'POST',
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}`, 'Content-Type': 'application/json' },
body: JSON.stringify({"endpoint":"feed","name":"demo"}),
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.post('https://crawlwisedata.com/v1/monitors/test', json={'endpoint':'feed','name':'demo'}, headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){
"ok": true, "test": true, "delivered": true, "status": 200, "eventId": "evt_test_...", "deliveryId": "deliv_...",
"charged": { "credits": 0, "usd": 0 },
"sent": { "headers": { "x-crawlwise-signature": "t=1791000000,v1=<hex>", "x-crawlwise-test": "1" }, "body": "{\"id\":\"evt_test_...\",\"type\":\"monitor.test\",\"test\":true, ...}" }
}POST /v1/monitors/run
Checks a monitor immediately. If something changed since the last check and the monitor has a webhookUrl, a real signed "monitor.change" event is POSTed (one attempt). Works with or without schedule:true. Bound by the minimum monitor interval, the same on every plan (429 with Retry-After), and 30 runs per hour per account. Failed checks are free.
| Parameter | Rules | Meaning |
|---|---|---|
endpoint | body, required | feed | sitemap | techstack |
name | body, required | Monitor name. |
curl -X POST \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
-H "Content-Type: application/json" \
-d '{"endpoint":"feed","name":"demo"}' \
"https://crawlwisedata.com/v1/monitors/run"const res = await fetch('https://crawlwisedata.com/v1/monitors/run', {
method: 'POST',
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}`, 'Content-Type': 'application/json' },
body: JSON.stringify({"endpoint":"feed","name":"demo"}),
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.post('https://crawlwisedata.com/v1/monitors/run', json={'endpoint':'feed','name':'demo'}, headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){
"ok": true,
"run": { "endpoint": "feed", "name": "demo", "ok": true, "baseline": false, "changed": true, "credits": 10, "status": "changed",
"change": { "kind": "feed.new_items", "newItems": 2, "items": [ ... ] },
"delivery": { "ok": true, "id": "deliv_...", "eventId": "evt_...", "status": 200, "error": null } },
"charged": { "credits": 10, "usd": 0.001 }, "balance": { "credits": 978, "usd": 0.0978 },
"note": "Change found; webhook delivered."
}GET /v1/demo-target
A built-in target on this host that changes every 5 minutes: /demo-target/feed.xml gains one item, and /demo-target/sitemap.xml gains one URL, drops one and updates one lastmod. Point a monitor at it to see a real diff and webhook during a trial.
curl \
"https://crawlwisedata.com/v1/demo-target"const res = await fetch('https://crawlwisedata.com/v1/demo-target');
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
r = requests.get('https://crawlwisedata.com/v1/demo-target', timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "feed": "https://crawlwisedata.com/demo-target/feed.xml", "sitemap": "https://crawlwisedata.com/demo-target/sitemap.xml", "changesEverySec": 300 }Webhooks
GET /v1/webhooks/secret
Returns the HMAC-SHA256 secret used to sign change-alert deliveries (header X-Crawlwise-Signature: t=<unix>,v1=<hex>).
curl \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
"https://crawlwisedata.com/v1/webhooks/secret"const res = await fetch('https://crawlwisedata.com/v1/webhooks/secret', {
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}` },
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.get('https://crawlwisedata.com/v1/webhooks/secret', headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "secret": "<64 hex chars>", "header": "X-Crawlwise-Signature", "format": "t=<unix>,v1=<hex hmac-sha256 of `${t}.${rawBody}`>" }POST /v1/webhooks/secret/rotate
Replaces the webhook signing secret. Update your verifier immediately.
curl -X POST \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
-H "Content-Type: application/json" \
-d '{}' \
"https://crawlwisedata.com/v1/webhooks/secret/rotate"const res = await fetch('https://crawlwisedata.com/v1/webhooks/secret/rotate', {
method: 'POST',
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}`, 'Content-Type': 'application/json' },
body: JSON.stringify({}),
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.post('https://crawlwisedata.com/v1/webhooks/secret/rotate', json={}, headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "secret": "<new 64 hex chars>" }GET /v1/webhooks/deliveries
Recent webhook deliveries (status, latency, errors). Each delivery keeps an attempts history (last 10 attempts: time, HTTP status, error text). Bodies and signatures are never stored.
| Parameter | Rules | Meaning |
|---|---|---|
limit | optional, 1-100 | Max rows (default 25). |
curl \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
"https://crawlwisedata.com/v1/webhooks/deliveries?limit=25"const res = await fetch('https://crawlwisedata.com/v1/webhooks/deliveries?limit=25', {
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}` },
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.get('https://crawlwisedata.com/v1/webhooks/deliveries', params={'limit':'25'}, headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "count": 1, "deliveries": [{ "id": "deliv_...", "eventId": "evt_...", "ok": true, "status": 200, "attempt": 2, "at": "...", "attempts": [{ "at": "...", "status": 500, "error": "HTTP 500" }, { "at": "...", "status": 200, "error": null }] }] }Billing
POST /v1/checkout
Creates a Stripe Checkout Session on our server for THIS account (bound by the account id we set in the session metadata), so the purchase is added to your existing balance. Only sessions created this way are credited automatically. Items: pack_10 ($10, 100,000 credits), pack_25 ($25, 262,500 credits), starter ($19/month, 250,000 credits/month). Subscriptions are refused if you already have one (change plan in the portal instead).
| Parameter | Rules | Meaning |
|---|---|---|
item | body, required | pack_10 | pack_25 | starter |
curl -X POST \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
-H "Content-Type: application/json" \
-d '{"item":"pack_25"}' \
"https://crawlwisedata.com/v1/checkout"const res = await fetch('https://crawlwisedata.com/v1/checkout', {
method: 'POST',
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}`, 'Content-Type': 'application/json' },
body: JSON.stringify({"item":"pack_25"}),
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.post('https://crawlwisedata.com/v1/checkout', json={'item':'pack_25'}, headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "url": "https://checkout.stripe.com/c/pay/cs_test_...", "sessionId": "cs_test_..." }POST /v1/checkout/start
Sign-up fallback, only when no sign-up method is available (email codes off and no OAuth provider): creates an account with 0 credits and returns its API key once, plus the Checkout URL for the item. The key works only for checkout and account reads until the purchase completes, then it carries the paid credits. Limited to 30 attempts per hour per network (an IPv4 address or an IPv6 /48; relay services such as Cloudflare WARP count as one network) and 300 per hour site-wide, separately from POST /v1/trial; over the limit you get 429 rate_limited ("Too many checkout attempts from your network, try again in N minutes.") with Retry-After. The key is never put in a URL. Otherwise 409 signup_not_available (sign up first).
| Parameter | Rules | Meaning |
|---|---|---|
item | body, required | pack_10 | pack_25 | starter |
acceptTerms | body, required | true |
curl -X POST \
-H "Content-Type: application/json" \
-d '{"item":"pack_10","acceptTerms":true}' \
"https://crawlwisedata.com/v1/checkout/start"const res = await fetch('https://crawlwisedata.com/v1/checkout/start', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({"item":"pack_10","acceptTerms":true}),
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
r = requests.post('https://crawlwisedata.com/v1/checkout/start', json={'item':'pack_10','acceptTerms':True}, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "accountId": "acct_...", "apiKey": "cw_...", "credits": 0, "pendingPurchase": true, "keyNote": "This is your API key. It starts working as soon as your purchase completes. Save it now, it's shown once.", "checkoutUrl": "https://checkout.stripe.com/c/pay/cs_test_...", "sessionId": "cs_test_..." }POST /v1/recover/account-link
Key recovery for an account created at checkout: emails a single-use recovery link (1 hour) for THAT account only, to the checkout email saved on that account when its purchase completed. Accounts are never looked up or merged by email. Same response whether or not a link was sent.
| Parameter | Rules | Meaning |
|---|---|---|
accountId | body, required | acct_... |
curl -X POST \
-H "Content-Type: application/json" \
-d '{"accountId":"acct_..."}' \
"https://crawlwisedata.com/v1/recover/account-link"const res = await fetch('https://crawlwisedata.com/v1/recover/account-link', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({"accountId":"acct_..."}),
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
r = requests.post('https://crawlwisedata.com/v1/recover/account-link', json={'accountId':'acct_...'}, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "sent": "If this account has a saved checkout email, a single-use recovery link (valid 1 hour) was sent to it." }POST /v1/portal
Returns a Stripe Customer Portal link: change plan, cancel at period end, update card, download invoices.
curl -X POST \
-H "Authorization: Bearer $CRAWLWISE_KEY" \
-H "Content-Type: application/json" \
-d '{}' \
"https://crawlwisedata.com/v1/portal"const res = await fetch('https://crawlwisedata.com/v1/portal', {
method: 'POST',
headers: { Authorization: `Bearer ${process.env.CRAWLWISE_KEY}`, 'Content-Type': 'application/json' },
body: JSON.stringify({}),
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
headers = {'Authorization': f"Bearer {os.environ['CRAWLWISE_KEY']}"}
r = requests.post('https://crawlwisedata.com/v1/portal', json={}, headers=headers, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "url": "https://billing.stripe.com/p/session/..." }Key recovery
POST /v1/recover/start
Starts key recovery for the email used at checkout. Returns a code and a Stripe portal login link and sets an HttpOnly cookie. Use the /recover.html page; it does this for you.
| Parameter | Rules | Meaning |
|---|---|---|
email | body, required | Checkout email. |
curl -X POST \
-H "Content-Type: application/json" \
-d '{"email":"you@example.com"}' \
"https://crawlwisedata.com/v1/recover/start"const res = await fetch('https://crawlwisedata.com/v1/recover/start', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({"email":"you@example.com"}),
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
r = requests.post('https://crawlwisedata.com/v1/recover/start', json={'email':'you@example.com'}, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "code": "CW-7K2P-9QXM", "expiresAt": "...", "portalLoginUrl": "https://billing.stripe.com/p/login/...?prefilled_email=..." }POST /v1/recover/status
Poll after updating your name in the portal. Returns state pending | verified (with a NEW key, once) | used | expired | none.
curl -X POST \
-H "Content-Type: application/json" \
-d '{}' \
"https://crawlwisedata.com/v1/recover/status"const res = await fetch('https://crawlwisedata.com/v1/recover/status', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({}),
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
r = requests.post('https://crawlwisedata.com/v1/recover/status', json={}, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "state": "verified", "accountId": "acct_...", "apiKey": "cw_test_...", "note": "All previous keys were revoked. This key is shown only once." }Public
GET /v1/pricing
Machine-readable prices, packs, subscriptions, limits and refund policy.
curl \
"https://crawlwisedata.com/v1/pricing"const res = await fetch('https://crawlwisedata.com/v1/pricing');
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
r = requests.get('https://crawlwisedata.com/v1/pricing', timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "creditUsd": 0.0001, "prices": { ... }, "packs": [ ... ], "subscriptions": [ ... ], "limits": { ... } }GET /v1/health
Liveness + storage write/read probe. 200 when healthy, 503 when storage fails.
curl \
"https://crawlwisedata.com/v1/health"const res = await fetch('https://crawlwisedata.com/v1/health');
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
r = requests.get('https://crawlwisedata.com/v1/health', timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "storage": "ok", "storageMs": 42, "time": "2026-09-28T19:00:00.000Z" }GET /v1/status
Component status used by /status.html.
curl \
"https://crawlwisedata.com/v1/status"const res = await fetch('https://crawlwisedata.com/v1/status');
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
r = requests.get('https://crawlwisedata.com/v1/status', timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "status": "operational", "checks": { "api": "ok", "storage": "ok", "payments": "ok", "checkout": "open", "demo": "enabled" } }POST /v1/contact
Sends a message to support (stored securely; we reply by email if you give one).
| Parameter | Rules | Meaning |
|---|---|---|
message | body, required | 10-5000 characters. |
email | body, optional | Where we should reply. |
category | body, optional | support | billing | abuse | opt-out | privacy | security | deletion | other |
targetSite | body, optional | For opt-out/abuse reports: the affected site. |
curl -X POST \
-H "Content-Type: application/json" \
-d '{"email":"you@example.com","category":"support","message":"My monitor returns no items, can you help?"}' \
"https://crawlwisedata.com/v1/contact"const res = await fetch('https://crawlwisedata.com/v1/contact', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({"email":"you@example.com","category":"support","message":"My monitor returns no items, can you help?"}),
});
const json = await res.json();
if (!json.ok) throw new Error(`${json.error?.code}: ${json.error?.message}`);
console.log(json);import os, requests
r = requests.post('https://crawlwisedata.com/v1/contact', json={'email':'you@example.com','category':'support','message':'My monitor returns no items, can you help?'}, timeout=60)
data = r.json()
if not data.get('ok', r.ok):
raise RuntimeError(data.get('error'))
print(data){ "ok": true, "id": "msg-2026-09-28T19:00:00.000Z-Ab12Cd", "note": "Thanks. We usually reply within 2 business days." }