Service Incident - October 31, 2022 - SunCo & Support | All Pods - SunCo, Web Widget and Mobile SDK Messaging Connection errors

Summary

Between October 30, 2022 at 23:47 UTC and October 31, 2022 at 01:50 UTC, customers using the Sunshine Conversations API, Web Widget and Mobile SDK Messaging on all Pods experienced connection errors when end users attempted to chat with agents. 

Timeline

01:08 UTC | 18:08 PT

We are investigating an issue preventing the end-user Web Widget loading in multiple Pods, which may impact Messaging conversations. You may also experience an issue contacting our support team via Messaging. Please monitor status.zendesk.com for updates.

01:34 UTC | 18:34 PT

We are working on fixing a potential cause of the issue preventing the Web Widget loading for customers on multiple Pods. You might still also experience issues contacting our support team via Messaging. More to come in 30 minutes.

01:58 UTC | 18:58 PT

We are observing extensive recovery of the Web Widget connections across all Pods. Our team continues to monitor the issue and we will send a final update soon.

02:33 UTC | 19:33 PT

We are happy to report that the issue impacting the Web Widget is resolved. Thanks for your patience while we worked through this issue today.

Root Cause Analysis

This incident was caused by a configuration change that was improperly rolled out and that impacted all pods at different times within the incident period. Web Widget was initially impacted at the start of the rollout while the Sunshine Conversations API was impacted in the second phase of the rollout.

Resolution

To fix this issue, we aborted the release and reverted the change to the previous stable version. Recovery was observed at 01:50 UTC following the final rollback.

Remediation Items

1. Improve our configuration change process to prevent similar issues in the future.
2. Improve internal documentation on the change process to ensure it is strictly followed.
3. Review standard soak time for internal proxy changes.
4. Review monitoring and alerts for impacted services.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.