Summary
On December 5, 2025, we received reports from customers regarding difficulties accessing their Zendesk instances due to 5XX errors. The root cause was identified as a major outage with our Content Delivery Network (CDN) provider.
Timeline
December 5, 2025 09:35 UTC | December 5, 2025 01:35 AM PT
We are currently investigating reports of multiple Zendesk services returning 5XX errors. Our engineering team has been notified that our Content Delivery Network (CDN) provider is also actively investigating the issue. We apologize for the inconvenience and will provide updates as they become available.
December 5, 2025 10:02 UTC | December 5, 2025 02:02 AM PT
Following the recent 5XX errors linked to our CDN provider, we have observed recoveries in the impacted Zendesk Support services after their fix was applied. Our engineering team is actively monitoring the issue to ensure stability. We apologize for any disruption this may have caused and will continue to provide timely updates.
December 5, 2025 11:19 UTC | December 5, 2025 03:19 AM PT
We are pleased to inform you that, following the fix implemented by our CDN provider, Zendesk Support services have been fully restored and the issue resolved. We sincerely appreciate your patience as our team diligently addressed this matter.
Root Cause Analysis
This incident was caused by a major outage with our Content Delivery Network (CDN) provider, disabling an internal tool prior to global rollout of the security change which precipitated the 5xx errors.
Resolution
The issue was resolved after our CDN provider implemented and deployed a fix on their platform, which addressed the 5XX errors affecting Zendesk Support services. No intervention was required from the Zendesk team.
Remediation Items
- Review and improve the global failover process to prioritize it over regional failover at the country or state level.
- Investigate alternative signals for internet resilience when current tracing methods do not indicate failure.
- Enhance alert system resilience during global service outages with additional paging options and failover mechanisms.
- Update the global failover runbook to include clear guidance on when manual activation of global failover is necessary.
- Review the fallback system used in the Classic Analytics platform to ensure reliability.
- Remove the customer “empggroup” from the legacy Scheduler’s blacklist to restore normal processing.
- Improve the Scheduler runbook with clear steps for analyzing data, identifying blacklisted customers, and removing them from the blacklist.
- Add caching to the Feature Flipper on the status page to reduce the number of system queries and improve performance.
- Optimize slow database queries responsible for loading incident history on the status page.
- Set appropriate timeouts for database queries to prevent delays and improve overall responsiveness.
FOR MORE INFORMATION
For current system status information about Zendesk and specific impacts to your account, visit our system status page. If you have additional questions about this incident, contact Zendesk customer support.