Service Incident - August 29, 2022 - Talk | Pod 27 - Ticket creation issue

SUMMARY

On August 29, 2022 from 16:00 to 16:58 UTC, Talk customers on Pod 27 experienced issues with ticket creation after placing calls in Talk. 

Timeline

16:53 UTC | 09:53 PT

We are investigating reports of Talk tickets not being created for Talk calls on Pod 27. We will provide another update in 15 minutes or when we have more information.

17:06 UTC | 10:06 PT

We are seeing improvements in the processing of Talk tickets on Pod 27. Next update in 30 minutes or when we have new information.

17:21 UTC | 10:21 PT

The issue impacting Talk ticket creation on Pod 27 is now fully resolved. Sorry for the inconvenience this may have caused you and your team.

POST-MORTEM

Root Cause Analysis

This incident was caused by a spike in Talk requests following an improperly scoped change, causing a backlog in the background job that processes ticket creation. 

 

Resolution

To fix this issue, another update was rolled out to properly scope the original change.

Remediation Items

1. Update monitoring and alerting to lower the threshold for response.
2. Created additional logging to look for similar issues in the future.

 

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.