Currently, Zendesk offers offset-based pagination for almost all of its current REST API endpoints. While this is fine for most small scale use cases (e.g., requests that don’t go above 10 pages worth of data), it puts an exponential strain on infrastructure when used at large scales with high page offsets. These high offset requests lead to high response times and poor performance because for each new page requested, the work to assemble all the previous pages still needs to happen first.
To provide a better developer experience and faster performance for our customers, we will soon be migrating primarily to cursor-based pagination and implementing limits on very high page offset requests.
- What is cursor-based pagination?
- How and why are we making these changes?
- Limits for offset-based pagination
- How can I begin implementing these changes?
- Which endpoints are supported?
What is cursor-based pagination?
Cursor-based pagination works by using a pointer to a specific item in a dataset to mark your progress. Essentially this acts as a sort of bookmark that lets you request the next batch of items in the dataset by providing the cursor on subsequent requests. Instead of getting the next page to iterate through items, you provide the cursor given in the previous response. This has the benefit of allowing faster results for high volumes.
How and why are we making these changes?
In the near future Zendesk will be making changes that enhance large scale API usage. This includes making cursor-based pagination widely available, starting with our Users, Tickets, and Tags endpoints before eventually adding the majority of our endpoints throughout 2021.
Offset- based pagination will continue to be offered with a significantly limited scope and depth. This includes limiting very high offset requests (requests above 1,000 pages), which will have a variety of outcomes – most positive – for heavy consumers of our APIs.
Benefits of cursor-based pagination include:
- Requests will no longer suffer the extremely high timeouts encountered when making requests with very high page numbers.This will result in faster performance for almost all integrations that require going through entire sets of resources page by page.
- Requests using cursor-based pagination will no longer present the risk to infrastructure that high page offset-based pagination presents, from service outages to incidents due to strain on infrastructure.
- For integrations that use offset-based pagination to export an entire set of resources, switching to cursor-based pagination will be relatively easy, requiring a minimal amount of rework to prepare integrations for the switch. Instead of grabbing a next_page parameter, the integration can use the cursor parameter for a new page.
Please see this page for a more detailed comparison.
Limits for offset-based pagination
Zendesk would like all traffic that can use cursor-based pagination to move away from offset-based pagination.
To encourage this shift, beginning September 15, 2021, Zendesk will implement rate limits to requests that go over 1,000 pages (100,000 resources). For all requests above 1,000 pages, accounts will be limited to 10 requests per minute.
This will slow down (but not stop) requests that disproportionately tax our infrastructure. If a request over 1,000 pages is made faster than 10 times per minute, a 429 response will be given and the request will need to be retried later. The error message will also indicate the amount of time remaining before users can submit their next request and continue the search.
Requests made using cursor-based pagination will not have a similar limit and will perform faster than high page offset requests.
How can I begin implementing these changes?
We recommend that you plan to switch to cursor-based pagination as soon as possible. See Paginating lists using cursor-based pagination. After you have a chance to look at the changes and discuss them internally, we ask that you review a few items:
- Ensure that integrations that will continue to use offset-based pagination do so within our limits.
- Review any portions of your integrations that use the API specifically to gather all available data or just report on changes. These are usually ideal candidates for data warehousing.
- If using the endpoint to scrape all available data, please begin work on switching to cursor-based pagination.
- If using the API to report on daily changes, please begin to use our Incremental APIs, which would allow for a high volume of all relevant data to this use case. Using cursor-based pagination for your initial export and then switching to the Incremental API to capture changes will significantly reduce the calls needed to gather relevant data.
- If currently making requests to multiple pages in parallel, please consider the above incremental option, as parallelized pagination is not supported and very taxing to our infrastructure.
Which endpoints are supported?
For a list of endpoints that support cursor-paged pagination, see Endpoints with CBP capability in the announcement about offset-based pagination limits.
If after reviewing this you have any further questions or concerns, we would be happy to discuss them with you and your engineering and integrations teams.
Thank you for taking the time to go over these points. We look forward to working with you in the near future to build better, faster, and more scalable tools to better serve your customers.