A REST API as a process source: pagination, filters and incremental reads

What do you need?

A GET endpoint that answers with JSON holding a list of records, each with at least a case id, a step (or status) and a time. For an API that needs sign-in, create a Header Auth, Query Auth, Basic Auth or Bearer Auth credential in Duaer; a public API needs none.

How do you set it up?

1. As an owner or admin, open Data sources, select Add source and choose REST API under API.
2. Enter the endpoint (without a query) and the records path, such as data.items; leave it empty when the answer is the list.
3. Add filters when needed: each goes as a query parameter with every request, such as status = open.
4. Choose the pagination: None, Page number, Offset, Cursor (the path of the next cursor in the answer), Next page address in the answer, or Link header, and enter the parameter names the API uses.
5. When the API can filter by a field such as the last change, enter the incremental parameter (such as updated_since). After Read fields, choose the incremental field (such as updated_at): each read fills the parameter with the largest value read so far and reads only what changed since.
6. Select Read fields; once you see the columns and a few records, select Next, choose the case id, step and time as for a spreadsheet, and confirm. Read the fields again after any change.

How often does it read, and how much?

Each run reads at most 20 pages of at most 10 MB each; a long list finishes over several runs, and the next round starts from the largest incremental value read. Without an incremental parameter each round starts at the first page again; a step is never written twice.

Is it safe?

• Direct calls use https only and never reach private, local or link-local addresses, checked again at every redirect.
• A system inside your network is read only through an on-premises data gateway that lists its address; http is allowed there.
• Credentials go only to the API’s own host: a redirect or a next page on another host stops the read and says why.
• Reading fields in the wizard uses only a credential you may use yourself.

What are the limits?

• GET and JSON only.
• Nested objects become a.b.c columns, three levels deep; lists are kept as JSON text; 200 columns at most.