SUMMARY
On August 30, 2021 from 23:13 UTC to August 31, 2021 13:29 UTC, a small group of our customers using an undocumented, alternate way to set ticket requester on ticket creation, may have experienced receiving 400 errors when trying to create tickets via our Zendesk Support API.
Timeline
14:32 UTC | 07:32 PT
Between Aug 30, 23:13 UTC and Aug 31, 13:29 UTC some customers across multiple Pods may have encountered failed ticket creation errors (400 Bad Request) via the Support API. No further errors should occur as a fix has been deployed.
POST-MORTEM
Root Cause Analysis
This incident was caused by a change to our API endpoints as part of our constant efforts to increase our existing security standards even higher and future proof our systems.
While not documented, before this change the requester parameter could be set in an alternative way when creating tickets. This change would have brought the behaviour in line with our existing documentation.
Resolution
To fix this issue, our team reverted this change for now. Once the rollback happened, the errors stopped and functionality returned to normal for our impacted customers.
Remediation Items
- Update out internal logging and monitoring for the specific parameters that caused this behaviour.
- Review why this rollout only impacted certain customers and how to avoid recurrence for them so we are able to deploy this change.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.