Recent searches
No recent searches
Zendesk Support API rate limit on the last batch of records
Posted Nov 19, 2024
I'm using a custom Airbyte connector to extract tickets from Zendesk Support. I'm using the Search Export endpoint to try and retrieve all tickets created in a day. This will be used to do a massive extraction of previous years. Doesn't matter the amount of tickets, I'm always receiving a rate limit exception when reading the last batch of tickets. In the example below, there should be around 17k tickets on Nov 18th. (Btw, we're not using the official zendesk support source connector because when we tried to use it for such a big retroactive extraction it was bugging and taking forever before even a million tickets).
2024-11-19 18:40:15 platform > Records read: 5000 (190 MB)
2024-11-19 18:44:26 platform > Records read: 10000 (381 MB)
2024-11-19 18:48:34 platform > Records read: 15000 (571 MB)
2024-11-19 18:50:35 source > Backing off _send(...) for 0.0s (airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: Request URL: https://<subdomain>.zendesk.com/api/v2/search/export.json?filter%5Btype%5D=ticket&page%5Bafter%5D=<next_token>&per_page=100&query=created_at%3E%3D2024-11-18T00%3A00%3A00Z+created_at%3C%3D2024-11-18T23%3A59%3A59Z, Response Code: 429, Response Text: {"error":"APIRateLimitExceeded","description":"Rate limit for Export API (api/v2/search/export) exceeded. Please wait 1 minute and try again."})
2024-11-19 18:50:35 source > Retrying. Sleeping for 20 seconds
2024-11-19 18:50:56 source > Backing off _send(...) for 0.0s (airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: Request URL: https://<subdomain>.zendesk.com/api/v2/search/export.json?filter%5Btype%5D=ticket&page%5Bafter%5D=<next_token>&per_page=100&query=created_at%3E%3D2024-11-18T00%3A00%3A00Z+created_at%3C%3D2024-11-18T23%3A59%3A59Z, Response Code: 429, Response Text: {"error":"APIRateLimitExceeded","description":"Rate limit for Export API (api/v2/search/export) exceeded. Please wait 1 minute and try again."})
2024-11-19 18:50:56 source > Retrying. Sleeping for 80 seconds
2024-11-19 18:52:42 source > Backing off _send(...) for 0.0s (airbyte_cdk.sources.streams.http.exceptions.UserDefinedBackoffException: Request URL: https://<subdomain>.zendesk.com/api/v2/search/export.json?filter%5Btype%5D=ticket&page%5Bafter%5D=<next_token>&per_page=100&query=created_at%3E%3D2024-11-18T00%3A00%3A00Z+created_at%3C%3D2024-11-18T23%3A59%3A59Z, Response Code: 429, Response Text: {"error":"APIRateLimitExceeded","description":"Rate limit for Export API (api/v2/search/export) exceeded. Please wait 1 minute and try again."})
2024-11-19 18:52:42 source > Retrying. Sleeping for 20 seconds
Does someone knows what could be causing this rate limit always on the final “batch” of records? And also why even after sleeping for more than a minute it keeps happening indefinitely?
0
1 comment
Greg Katechis
Hi Joao! I took a look at our logs and I wasn't seeing anything that was particularly useful in explaining what is going on here. One thing that I did notice is that you're using offset pagination to move through large chunks of data, which is something that we do not recommend. I would encourage you to read this documentation to understand some of the limitations and then take a look at this tutorial that will show you how to use cursor-based pagination instead. Our API documentation for the search export endpoint no longer even lists offset pagination as a valid option, so it's really difficult to say how we expect this to work at this time.
If you would like a more in-depth analysis of what is happening with the rate limits, I would recommend reaching out to our Advocacy team directly, since they can ask you more detailed questions about your account that we can't cover here in our forums.
0