Announcing enhanced Explore performance with a 37-month data retention limit

Hi all, while this won't be a solution for everyone, we've come up with an Explore recipe that might be helpful for some folk affected by this change. See Using constants to add historical data to reports.

Hello 1263169416370 , 

We appreciate your interest in exploring potential solutions. Unfortunately, there is no solution without a trade-off. Splitting data into two datasets will create extra complexity but will not provide substantial performance gain because it will still be stored in the same database.

Hi 1263082140049, thanks for your reply.

I haven’t heard from you directly, but even if executions for 37 months+ are only 2% of all reporting, that again isn’t hugely surprising given the tendency towards recency in most daily reporting requirements. However, it’s still in my view a highly misleading stat to use, as that 2% - even though not frequently used - have outsized importance when they are for historical reporting. I just don’t think it’s a good metric to try to justify this change. 

I can’t help coming back to my earlier thought on this: ‘really this seems to be because Zendesk haven’t got the processing power - which in this day and age just shouldn't be an issue. It seems to me that it’s a ‘Zendesk problem’ that Zendesk are making a ‘Customer problem’…’

Looking at the Announcement pages, the article announcing this change has more than twice as many downvotes as any other article in announcements in the past 6 months (which is as far back as I checked). 

Customers clearly aren't happy about this at all. Surely there has to be a solution, or this decision has to be reconsidered? Zendesk can’t simply cut off everything more than 3 years old - it's not what anyone other than Zendesk has agreed, and fundamentally changes the utility of Explore / Insights. 

Is it possible to do the below? Or is the performance not based on the dataset? 

Is there any opportunity to instead of culling the data completely, give customers the option of an additional dataset? e.g. the default dataset would only have the last 3ish years of data, and by default reports would all use that set. We'd get the best performance for the majority of reports. BUT, if you needed to have one report showing historical data from e.g. 7 years ago, you could create a report that uses the full-historical-data dataset. Slightly slower maybe, but it can then be added to the required dashboard(s) and people can keep everything in Explore.

Carmen,  I checked your ticket with our Support team, and they have provided more details now. I hope this addresses your concerns. 

Jon,  you have a valid point. For sure, there are some useful reports that look at the time ranges longer than 37 months. However, this is the trade-off we need to make to provide good performance and reliability.  Believe me, we considered and investigated many other solutions. Unfortunately, it is impossible to opt-out from this change per account basis because the data pruning mechanism is applied on the data source level that is account agnostic. 

I want to clarify this data point “only 2% of reports being filtered by the time ranges longer than 37 months”.  It refers to the report executions, so it means that only 2% of all report executions looked at time ranges longer than 37 months. For example, if you have a report filtered by 30 days but while using it in the dashboard, you filtered it by 4 years, this report execution is included in this 2%. I will pull the list of the reports that had a longer time range executed in your account and will send it to you via a proactive ticket. 

Eli, thank you for your feedback. Your account is indeed not large enough to experience the performance issues with Explore.  As mentioned above, it is impossible to opt-out from this change per account basis because of how the source data is stored.  

I would also like the ability to opt-out of this 37-month data retention limit, whether for an entire account or for individual reports. This change makes Explore largely useless for reporting on annual trends. Furthermore, I've never experienced performance issues with Explore, possibly because our ticket volumes are lower relative to larger companies. In that regard, this change is a strict (and major) downgrade for us.

Hi Eugene, thanks for the response, although with respect I think the logic here is somewhat misplaced, particularly around this part: 

‘Our analysis shows that most users focus on recent data, with only 2% of reports being filtered by the time ranges longer than 37 months. This supports our decision to introduce data retention for maintaining a high-quality reporting experience.’

I don't think that measurement shows the full story. We barely have any saved reports that go back longer than 37 months as a saved default; most are 12 or 24 months (so our account is probably even below 2%). However, that doesn't mean that we don't frequently use those template reports to change the duration and pull important historical data by adjusting the date filters, without saving. We need that long-term data, and we need to be able to manipulate it in Explore to give key historical insights and trends. 

Even using the 2% figure at face value, 2% is not nothing. It also makes sense to me that most reports are shorter-term than that to focus on current metrics. But it doesn't mean that the longer term reports aren't extremely useful, even though there's fewer of them. I don't think it's a good way to justify the change, I'm sure others would agree, and I hope that feedback can be shared. 

Please can consideration be given to an opt-out for this for customers who need a longer historical period, even if the default for the majority is to cut-off at 37 months? We don't mind a trade-off on slower speed, but it's business critical for us to be able to pull ad-hoc historical insights for longer periods. 

Thank you for the clarification, we have had different responses from your support team, may want to check our support ticket https://support.zendesk.com/hc/en-us/requests/13249456

Thank you for sharing your feedback. We understand that this adjustment may raise concerns, and we want to assure you that your input is invaluable to us.

We'd like to offer an important clarification: This data retention policy will solely impact data within the Explore analytics tool. Other Zendesk features like Agent workspace, Search, Help Center and APIs will not be impacted.
 
The decision to implement data retention in Explore was not made lightly. It was a careful decision aimed at enhancing the overall performance and reliability of the platform.  We observed up to 85% faster report execution times on the accounts with 37-month data retention. While everyone will experience improved performance and stability, accounts with higher volumes of data and longer history will likely see the most significant gains. This enhancement is crucial for providing a more efficient reporting experience for all users and enables Zendesk to continue enriching Explore with new data.
 
Our analysis shows that most users focus on recent data, with only 2% of the executed reports being filtered by the time ranges longer than 37 months. This supports our decision to introduce data retention for maintaining a high-quality reporting experience.
 
We appreciate your understanding and continued support as we strive to improve our product. Please continue to provide feedback and comments in this thread, we will respond in a timely manner. If you have technical concerns, please reach out to your Account Manager or a member of our Support team who would be glad to assist you. 
 
Thank you for being a valued member of our community and Zendesk customer.
 

Can you please clarify that this is limited to statistics within Explore, for example, will our agents using dashboard still be able to access tickets older than the 37 months while working in the agent workspace dashboards?  ie. if they want to refer to a historical email from a customer, will this no longer be available?