SUMMARY
On April 08, 2024 from 13:37 UTC to 18:20, Tymeshift customers across all Pods experienced delays in processing some activities and timely tracking of Agent Activity being prevented.
Timeline
April 08, 2024 04:11 PM UTC | April 08, 2024 09:11 AM PT
We are receiving reports that some Tymeshift customers are experiencing issues tracking Agent Activity, and are stuck on tracking specific tasks. We will provide further updates shortly.
April 08, 2024 04:26 PM UTC | April 08, 2024 09:26 AM PT
We have confirmed an issue affecting some Tymeshift customers causing delays in processing some activities and preventing timely tracking of Agent Activity. We will provide another update in one hour or as soon as we have new information to share.
April 08, 2024 05:24 PM UTC | April 08, 2024 10:24 AM PT
We are still investigating the issue affecting some Tymeshift customers causing delays in processing activities, preventing timely Agent Activity tracking. We will share additional updates in one hour or when we have new information to share.
April 08, 2024 06:26 PM UTC | April 08, 2024 11:26 AM PT
We have addressed the immediate impact for affected Tymeshift customers and Agent Activity tracking should be accurately updated at this time; however, in order to implement this fix we have had to temporarily disable Teams and Locations data within Agent Activity until the root cause is fully addressed. Our investigation will continue tomorrow, April 9, 2024 and we will provide further updates at that time.
April 09, 2024 04:56 PM UTC | April 09, 2024 09:56 AM PT
The teams and location data within Agent Activity have been re-enabled for Tymeshift customers. Agent Activity tracking should now include teams and location data. We will provide a final update once the recalculation of activities - attaching the appropriate teams and locations to them, is completely finished.
April 17, 2024 03:01 PM UTC | April 17, 2024 08:01 AM PT
We have successfully completed the recalculation script to resolve the teams and locations mismatch issue. All affected accounts have had their teams and locations updated for the period between April 8th, 2024, 13:37, and April 9th, 2024, 18:00. Additionally, we have manually optimized the ClickHouse agents table and recalculated adherence for all accounts. With these measures in place, we are marking this incident as resolved and apologize for any inconvenience caused.
POST-MORTEM
Root Cause Analysis
An underperforming query in an internal service caused delayed responses resulting in the delayed processing of some activities.
Resolution
To fix the issue, the query was disconnected. This impacted reports containing groupings and filtering by Teams and Locations.
Remediation Items
- Run recalculations in reports containing Teams and Locations filtering and groupings once the primary issue is resolved [Complete]
- Explore possibility of load testing more before new releases [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.
1 comment
Charlotte Kobler
Post-mortem published April 26, 2024
0