Service Incident - July 20, 2022 - Chat in Agent Workspace | EU Pods - Server Disconnection errors

SUMMARY

On July 20, 2022 from 7:18 UTC to 9:54 UTC, Chat Agent Workspace customers across EU Pods (multiple geological locations including NL, GB, FR & ES), experienced transient disconnection and saw growl notification of chat server disconnections. 

Timeline

08:30 UTC | 01:30 PT

We’re currently investigating issues with Chat getting disconnected for Agent Workspace customers. Workaround is to clear cache and cookies and refresh browser. More information to follow.

08:43 UTC | 01:43 PT

Our team confirmed a spike in errors between 7-8 am UTC for Chat in Agent Workspace on EU Pod customers. If the workaround mentioned previously doesn’t work, please let us know. We appreciate your patience.

09:11 UTC | 02:11 PT

As our team continues to investigate the root cause of this issue we are now seeing another spike in errors. We’re working on understanding and finding a fix for this issue with Chat disconnections in Agent Workspace in EU Pods. More information as soon as available.

09:56 UTC | 02:56 PT

We appreciate your patience while we work through these error spikes for EU Poded customers related to Chat disconnections in Agent Workspace. The last spike has subsided and we continue monitoring as well as investigating the root cause. More information in 1h or when available.

11:04 UTC | 04:04 PT

We are happy to report that the issue with error spikes for EU Poded customer related to Chat disconnection has now been resolved. Thank you for your patience.

POST-MORTEM

Root Cause Analysis

This incident was caused by what seems to be a customer local network anomaly. Several services appeared unavailable and disconnected while monitoring still showed them active. No issue could be found with Zendesk or its vendors. The issue self-resolved within minutes of the error spikes. 

Resolution

No action was taken by Zendesk to resolve the issue. 

Remediation Items

1. Create an alert to notify the responsible team when similar disconnections occur again.

 

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.