SUMMARY
Between December 20, 2023 from 12:12 UTC to December 29, 2023 23:30 UTC, some customers engaging with the Zendesk Z Bot widget may have experienced their messages not creating follow up tickets in Zendesk.
Timeline
17:25 UTC | 09:25 PT
We are aware of an issue that is preventing some tickets from being created following conversations with the Zendesk Z Bot Widget used to contact Zendesk Support. We are failing over to the web form while our team investigates. Additional updates will be provided when we have new information to share.
23:19 UTC | 15:19 PT
We've re-enabled the Zendesk Z Bot Widget used to contact Zendesk Support at support.zendesk.com and will continue to monitor performance. We will provide a final update when the issue is fully resolved.
03:20 UTC | 19:20 PT
We are happy to report that the issue affecting access to our Zendesk Z Bot Widget for support (support.zendesk.com) is now fully resolved. Thanks for your patience while we worked on fixing this issue.
POST-MORTEM
Root Cause Analysis
This incident was caused by an escaped defect, where the events processing service generated errors when trying to parse webhook events with a null value within their metadata.
Resolution
To fix this issue, the code logic was updated so that null values in metadata objects would be accepted. We continued to monitor the situation and confirmed in the logs that the errors on Pod 26 were no longer occurring after this change.
Remediation Items
- Update Sunshine Conversation public documentation to accept null values in metadata objects [Scheduled]
- Adding additional monitoring alerts for such errors [Done]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.
1 comment
Eugene Khoo
Post-mortem published January, 19, 2024.
0