Batch
POST /batch scrapes many pages in one call, in parallel. List the pages in targets; fields and options apply to all of them:
This request (several targets, or HTML) is edited in JSON.
{
"succeeded": 3,
"failed": 1,
"results": [
{
"target_url": "…/demo/shop?page=1",
"status": 200,
"data": { "title": "Demo shop — page 1" },
…
},
{
"target_url": "…/demo/shop?page=2",
"status": 200,
"data": { "title": "Demo shop — page 2" },
…
},
{
"target_url": "…/demo/product",
"status": 200,
"data": { "title": "Desk lamp" },
…
},
{
"target_url": "…/demo/missing",
"error": { "code": "HTTP_ERROR", "status": 404, "message": "…" }
}
]
}results are in the order of targets, each a scrape response or an error. The batch itself answers 200 as long as the request is valid: check each result.
Different settings per page
A target can be an object with its own fields and options. They're merged over the top-level ones, like steps: an option replaces the shared one, headers and cookies are combined:
{
"fields": { "title": "h1" },
"options": { "proxy": { "country": "DE" } },
"targets": [
"https://example.com/a",
{ "target_url": "https://example.com/b", "options": { "browser": true } },
{
"target_url": "https://example.com/c",
"fields": { "price": { "selector": ".price", "transform": ["price"] } }
}
]
}Options
| Property | Type | Default | Description |
|---|---|---|---|
targets | array | — | 1 to 100 URLs or { target_url, fields, options } objects |
fields | array or object | — | Fields for every target without its own |
options | object | — | Options for every target |
concurrency | integer | 5 | Pages scraped at once, 1 to 10 |