Service Incident - July 12, 2022 - Support | All Pods - API Issues

Summary

On July 12, 2022 from 02:30 UTC to 07:44 UTC, customers using the Support API to create or update users may have been presented with a 500 ‘Internal server error’ message. 

Timeline

09:52 UTC | 02:52 PT

We are aware of an issue that prevented the creation or updates of users via the Support API today between 2.30 AM UTC and 7.44 AM UTC. We have deployed a fix for this and are happy to report that this is now resolved. We apologise for any inconvenience caused.

POST-MORTEM

Root Cause Analysis

This incident was caused by an update which assumed the presence of a “User-Agent” header on all requests, as this header is used in testing. 

Resolution

To fix this issue, the update was rolled back.

Remediation Items

1. Improve testing scenarios to account for a wider range of use cases.
2. Add further layers of monitoring and alerting in testing environments.
3. Update response playbook to improve resolution times.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.