Summary
On April 19, 2021 from 8:15 AM UTC to 8:38 AM UTC, Zendesk Sell customers using a none standalone version of Sell, experienced receiving 502 errors when opening the product.
Timeline
09:01 UTC | 02:01 PT
We are happy to report that the issue of 502 errors received by some customers when using Zendesk Sell has been fixed.
08:43 UTC | 01:43 PT
We are investigating reports of some customers receiving 502 errors when opening Zendesk Sell. More updates to follow
POST-MORTEM
Root Cause Analysis
This incident was caused by a configuration issue with our exportation of logs. As the previous configuration of Sell log tooling did not send all it’s data to our monitoring system via User Datagram Protocol (UDP) and it was set up to block buffering traffic.
The heavy traffic on the blocked UDP buffering hindered DNS resolution, this meant all requested DNS from the impacted servers timed out, and users received a 502 error.
Resolution
To fix this issue, our engineering team restarted the impacted servers and services returned to normal.
Remediation Items
- Correct logs tooling configuration to allow for UDP buffering.
- Add extra monitoring for the behaviour.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.