On July 12, 2023 from 01:24 UTC to 17:41 UTC, customers on all pods may have experienced failed API calls when making calls to the ‘organizations’ endpoint to perform bulk creation of organization records.
16:19 UTC | 09:19 PT
Since approximately 01:24 UTC on July 12, 2023, bulk requests to create, update or delete many organizations have been silently failing with a 200 response, with no organizations being created. Our team has a fix prepared and we will follow up as soon as it has been released. Thank you for your patience in the meantime.
17:54 UTC | 10:54 PT
We are happy to report that the fix for silently failing bulk organization jobs has now been deployed across all pods. Any bulk requests for organizations between 01:24 and 17:41 UTC will need to be re-requested to successfully create, update or delete the organizations. We apologize for this inconvenience, and thank you for your patience.
Root Cause Analysis
This incident was caused by an escaped defect that resulted in the process of fixing another known issue.
To fix this issue, a fix was released that reverted the escaped defect, and Organization bulk actions started working as expected again.
- Enable Organization regression tests [Scheduled]
- Update runbooks to assist in more expedient resolution of similar issues in future [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.