Pagination
Follow a site's next links, merge the lists, and serve pages of any list, on any request.
options.pagination works on any /scrape. It can read several pages of a site, serve one page of a long list, or both.
{
"target_url": "https://example.com/products",
"fields": { "products": { "selector": ".product", "items": { "name": "h2" } } },
"options": { "pagination": { "next": "ai", "max_pages": 10, "page": 2, "per_page": 50 } }
}| Property | Type | Default | Description |
|---|---|---|---|
next | true, "ai" or a CSS selector | none | How to find the next link. Without it, only the first page is read |
max_pages | integer | 5 | Site pages read at most, 1 to 50, within your plan |
list | string | the longest | The list field page and per_page apply to |
page | integer | 1 | Which page of the list to return |
per_page | integer | the whole list | Items per page, up to 1000 |
Finding the next link
- A CSS selector: the link to follow.
true: the usual patterns (rel="next", "next" labels and classes)."ai": the AI writes a selector on the first page. It's remembered for that site, like AI fields, so later requests don't call the model, and rewritten if it breaks. Billed as AI only when the model runs.
List fields of every page are merged; other fields come from the first page. Each site page read is billed as a page.
The answer
"pagination": { "field": "products", "page": 2, "per_page": 50, "pages": 4, "total": 180, "has_next": true, "fetched": 10, "next_url": "https://example.com/products?p=11" }fetched is the number of site pages read. next_url is set when max_pages stopped before the end: start a request there to continue. If a later page fails, what was read is returned with a warning.
For a whole site, see Crawling.