SUMMARY
On February 14, 2025 from 11:30 UTC to February 15, 6:08 UTC, Sell customers using the Office 365 integration via OAuth experienced issues sending and syncing emails in Sell.
TIMELINE
February 15, 2025 06:07 AM UTC | February 14, 2025 10:07 PM PT
We're happy to report that all outbound email issues impacting Sell have been resolved. Thank you for your patience.
February 15, 2025 02:08 AM UTC | February 14, 2025 06:08 PM PT
We have rolled a fix for the outbound email issues experienced. We will monitor the situation. Thank you for your patience
February 15, 2025 01:54 AM UTC | February 14, 2025 05:54 PM PT
Our team have identified an issue with the outbound email service impacting Sell customers. They are currently working on a fix. More to come soon.
POST-MORTEM
Root Cause Analysis
This incident was caused by the expiration of OAuth2 secrets generated in the Microsoft Azure portal on February 14, 2025. The secrets were still in use by the services, leading to failures in outbound email functionality and email syncing for Sell customers.
Resolution
To resolve the issue, the team rotated the expired secret credentials and redeployed the affected services, which restored the email functionality for customers.
Remediation Items
- Implement Expiration Alerts: Create a monitor that raises alerts for expiring OAuth2 credentials one month in advance to ensure timely rotations.
- Provide Azure Access: Grant access to the Azure portal for the Sell team to facilitate quicker responses to similar issues in the future.
- Create Latency Monitor: Develop a latency monitor for the Mailman sendout process to identify delays in email delivery.
- Implement Backoff Mechanism: Introduce a backoff mechanism to handle authentication problems more gracefully, reducing the impact of similar incidents.
By addressing these items, the team aims to prevent future occurrences of similar issues and enhance overall service reliability.
FOR MORE INFORMATION
For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.
1 comment
Bob Novak
Postmortem published February 27, 2025
0