Service Incident - July 28, 2022 - Chat & Guide | Agent Workspace, SunCo & Guide Delays and Errors

Summary

On July 28, 2022 from 11:43 UTC to 12:05 UTC, Agent Workspace and Sunshine Conversations-only customers may have seen delays, missed chats, and issues appending transcripts to tickets. Guide customers may also have experienced errors trying to connect to the Help Center.

 

Timeline

13:05 UTC | 06:05 PT

From 11:43 UTC to 12:05 UTC, Agent Workspace across multiple Pods and SunCo only customers might have experienced a delay in SunCo events (affecting Whatsapp and AnswerBot) & an increase in missed Chats. This also affected Guide [1/2]

13:05 UTC | 06:05 PT

customers who might have seen a “500 Internal Server Error” message while trying to access the Help Center. This issue has now been resolved. We apologise for any inconvenience. [2/2]

14:05 UTC | 07:05 PT

Additional customer impact confirmed for today’s SI: Agent Workspace customers might also see Chat transcript updates failed during the incident duration. We will work on the backfill for those as soon as possible.

POST-MORTEM

Root Cause Analysis

This incident was caused by a lack of resource availability when migrating traffic away from legacy systems.

Resolution

The issue was self-resolved; backlogged requests were processed within a few minutes.

Remediation Items

1. Improve resource allocation for similar situations in the future.
2. Add additional testing scenarios and soaking time for similar changes.

 

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.