SUMMARY
Between 12:00 and 21:50 UTC on September 21, 2023, and then again between 22:45 UTC on September 21, 2023, and 00:50 UTC on September 22, 2023, Chat & Messaging customers using Agent Workspace, and some Support customers experienced 5xx level errors, green screen errors, and issues accessing and connecting to these products.
Timeline
Sep 21, 2023
19:06 UTC | 12:06 PT
Our team is investigating reports of some customers receiving errors when accessing Chat. We will provide an update when we have more information.
19:25 UTC | 12:25 PT
We have confirmed an issue causing errors when attempting to access Chat & Messaging within Agent Workspace across multiple pods, and our team is actively investigating. Further updates will be posted shortly.
19:55 UTC | 12:55 PT
Our teams continue to investigate the issue causing errors when attempting to access Chat & Messaging within Agent Workspace across multiple pods. We will share another update within the next hour or as soon as we have new information to share.
20:54 UTC | 13:54 PT
We observed a temporary drop in the error rates preventing access to Chat & Messaging in Agent Workspace; however, these errors have returned and our team is still working towards uncovering a root cause. We will provide further updates as we learn more.
21:23 UTC | 14:23 PT
Further investigation has uncovered an issue with our CDN provider, and we are seeing impact beyond Chat & Messaging within Agent Workspace. Please be on the look out for 5xx level errors and "green screen" errors across multiple products. We will continue to post updates as our investigation progresses.
22:09 UTC | 15:09 PT
We are happy to report that the issue causing 5xx level errors, green screen errors, and access issues in Chat & Messaging within Agent Workspace has been resolved following a fix from our CDN provider. Thank you for your patience during our investigation.
Sep 22, 2023
00:20 UTC | 17:20 PT
Following the fix that had been deployed earlier by our CDN partner, we are once again receiving reports of access issues in Chat & Messaging, as well as detecting increased error rates on our dashboards. We are in the process of working with our CDN provider right now to get more details, and will provide our next update within the hour or if we have earlier updates to share.
00:39 UTC | 17:39 PT
We have confirmation that Zendesk customers across multiple pods may experience 5xx level errors, green screen errors, and access issues in Chat & Messaging within Agent Workspace. Our CDN partner is currently working to resolve the issue and we are continuing to monitor at our end. We will provide another update once we know more.
00:58 UTC | 17:58 PT
We are glad to pass on that a fix has been put in place by our CDN provider and we are seeing improvements at our end, as well as stable metrics on our dashboards. We will continue to monitor and provide another update when we have more to share.
03:04 UTC | 20:04 PT
We have continued to work with our CDN partner on their mitigation efforts on the performance issues. We will subsequently be following up with them for an RCA when that becomes available post issue resolution. There has been observed stability from our dashboards for a while now, and we will keep monitoring actively at our end. Next update when we have more information to share.
12:11 UTC | 05:11 PT
We continue actively monitoring this with our CDN provider for any further issues and will provide updates as necessary. At this point in time, things continue stable. Thanks for your ongoing patience.
Sep 23, 2023
14:27 UTC | 07:27 PT
The incident related to customers experiencing errors when accessing Chat & Messaging within Agent Workspace has been fully resolved. We appreciate your patience throughout.
POST-MORTEM
Root Cause Analysis
This incident was caused by a networking issue with our content delivery network provider.
Resolution
Our content delivery network provider released an update which resolved the issue.
Remediation Items
Zendesk has assigned no remediation items following this incident.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.
1 Comments
Postmortem published October 9, 2023.
Article is closed for comments.