SUMMARY
On July 12, 2023 from 01:24 UTC to 17:41 UTC, customers on all pods may have experienced failed API calls when making calls to the ‘organizations’ endpoint to perform bulk creation of organization records.
Timeline
16:19 UTC | 09:19 PT
Since approximately 01:24 UTC on July 12, 2023, bulk requests to create, update or delete many organizations have been silently failing with a 200 response, with no organizations being created. Our team has a fix prepared and we will follow up as soon as it has been released. Thank you for your patience in the meantime.
17:54 UTC | 10:54 PT
We are happy to report that the fix for silently failing bulk organization jobs has now been deployed across all pods. Any bulk requests for organizations between 01:24 and 17:41 UTC will need to be re-requested to successfully create, update or delete the organizations. We apologize for this inconvenience, and thank you for your patience.
POST-MORTEM
Root Cause Analysis
This incident was caused by an escaped defect that resulted in the process of fixing another known issue.
Resolution
To fix this issue, a fix was released that reverted the escaped defect, and Organization bulk actions started working as expected again.
Remediation Items
- Enable Organization regression tests [Scheduled]
- Update runbooks to assist in more expedient resolution of similar issues in future [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.
1 Comments
Post-mortem published July 20, 2023.
Article is closed for comments.