Cursor Based Incremental API 504 Issue

Hey Greg,

Apologies for the delay in my response, but I wanted to provide an update that we were able to utilize your recommendation to get all the necessary data loaded (specifically, setting 'per_page' = 250)! Thank you for your time an assistance!

- Raja

Hi Raja! I checked your account and did see a significant number of 504s when hitting the incremental exports APIs, so I can at start by stating that you are definitely seeing a higher incidence than would be expected. Generally speaking, when we see an elevated number of 504s as a result of slow response from our servers, the two main issues are 1) a service degradation/service incident or 2) very large db calls as a result of the query. Obviously there are a lot of other things that could cause this, but when we see a pattern of errors over the course of a month, the trend is indicative of the second issue with large db calls. 

To that end, there are some things that you may be able to do to help limit the volume of data being retrieved and thus hopefully reduce/eliminate the 504s. The first thing that I would recommend doing is including a `per_page` parameter and setting it to 200 to start and see how that performs. As the default `per_page` is set to 1,000, this could be taxing if your tickets contain a lot of ticket fields and tags.

The other suggestions are dependent on whether or not you are side-loading additional resources, such as `metric_sets` and that suggestion would be to either test this by eliminating the side-load and seeing if you return consistent results. If you do see good results, add the side-load back in and combine it with the fine-tuning of the `per_page` parameter and you should begin to find a sweet spot of consistent results. 

If one/both of the methods above work for you and you then see a decline in the future, starting by cutting the `per_page` parameter in half should be a good test as to whether the issue is with our service or if it's with the size of the resulting db query...if cutting it in half doesn't work (and you didn't add hundreds of thousands of new ticket fields!), it's probably a service issue. If it does work, fine-tuning the `per_page` parameter again should get you where you need to go.

All of that said, this is almost more of a dance than it is science in your situation. For almost all of our customers, using incremental exports with cursor-based pagination will work exactly as expected. Since your account has so much data, we're really going to be in a situation where that fine-tuning back-and-forth may be necessary. I'll loop in your Premier team so that they're aware of what's going on and if this continues to be an issue, they can work with the relevant engineering resources to investigate more long-term solutions.

Let me know how the above goes for you!