Async jobs & webhooks

A long crawl, a batch of a hundred pages or an AI extraction can outlast your HTTP client's timeout. Send them as jobs: the API answers 202 Accepted at once, runs the request in the background, and keeps the result for 7 days.

Send a jobCopy Anchor Link

Add ?async=true to /scrape, /batch or /crawl, the body stays the same:

curl "$SCRAPE_ENGINE_URL/scrape?async=true&webhook_url=https://example.com/hooks/jobs" \
  -H "Authorization: Bearer $SCRAPE_ENGINE_KEY" \
  -H "Content-Type: application/json" \
  -d '{"target_url": "https://example.com", "fields": ["h1"]}'

Or POST /v1/jobs with { "path": "/crawl", "body": { … }, "webhook_url": "…" }.

{
  "job": {
    "id": "64ba08ab-a430-4a45-b028-4b4e3cbd2513",
    "path": "/scrape",
    "status": "queued",
    "webhook_url": "https://example.com/hooks/jobs",
    "created_at": "2026-10-26T08:33:17.490Z"
  }
}

A job is billed like the same request sent directly, when it runs. It waits its turn in the queue (your plan's concurrency applies) instead of being refused when you send many at once; 1,000 can wait or run at a time.

Get the result (polling)Copy Anchor Link

GET /v1/jobs/{id}: status goes queued → running → done (or failed, cancelled). Once finished, result holds the response, exactly what the request would have answered, meta included:

{
  "id": "64ba08ab-a430-4a45-b028-4b4e3cbd2513",
  "status": "done",
  "http_status": 200,
  "credits": 1,
  "started_at": "2026-10-26T08:33:17.912Z",
  "finished_at": "2026-10-26T08:33:18.405Z",
  "result": { "target_url": "https://example.com", "data": ["Example Domain"], "meta": { … } }
}

• GET /v1/jobs lists your jobs, newest first (?status=running, ?limit=, ?before=<created_at>), without their results.
• DELETE /v1/jobs/{id} cancels a job that's waiting or running: it stops on the engine at once, and only what it used until then is billed.
• Every job is also in the dashboard, under Automation → Async jobs.

Get the result (webhook)Copy Anchor Link

With webhook_url, the API POSTs the job and its result there when it's finished:

{
  "event": "job.finished",
  "job": { "id": "64ba08ab-…", "status": "done", "http_status": 200, "credits": 1, … },
  "result": { "target_url": "https://example.com", "data": ["Example Domain"], "meta": { … } }
}

• It's signed: X-Signature: sha256=<hex> is the HMAC-SHA256 of the raw body with your account's webhook signing secret (in the dashboard, under Async jobs, where you can rotate it). See how to check it.
• Answer with a 2xx within 10 seconds. Otherwise it's sent again after 1 minute, 5 minutes and 30 minutes, then given up; you can send it again from the dashboard.
• The webhook must be a public http(s) address; redirects aren't followed.