SUMMARY
From December 11, 2023 from 15:30 UTC to December 12, 2023 00:40 UTC, Support customers were unable to receive two-factor authentication codes via SMS and were unable to sign in.
Timeline
17:46 UTC | 09:46 PT
We are investigating reports of 2-Factor Authentication not sending SMS to complete the authentication process. This is also impacting the creation of new 2-FA SMS methods to authenticate. More information will be posted shortly.
17:55 UTC | 09:55 PT
We have confirmed an issue with 2-Factor Authentication SMS functionality, and our team is actively investigating. In the meantime, we have found that authenticator apps are not affected and can function as a replacement, if it is an option. Further updates will be posted soon.
18:23 UTC | 10:23 PT
While our team is working on the issue with native 2-Factor Authentication SMS functionality, in order to temporarily bypass this behavior an admin would need to disable native 2-FA for all agents, or move to using an authenticator app. We will continue to post updates on our investigation as we learn more.
19:27 UTC | 11:27 PT
Our team continues to investigate the issue affecting native 2-Factor Authentication SMS reception. To clarify our earlier message, admins can toggle the requirement for 2-FA for the account; however, agents would still need to disable this setting on their profile to log in without the SMS using this authentication method. Adjusting 2-FA to using an authenticator app is still available as a workaround for this issue.
20:04 UTC | 12:04 PT
Our engineers have identified the root cause of the 2-Factor Authentication issue and are currently working on a fix. We will provide another update when we have more information.
20:51 UTC | 12:51 PT
Our engineers continue to actively work on a fix for the 2-Factor Authentication issue. We will provide another update in one hour or when we have new information to share.
22:01 UTC | 14:01 PT
The resolution of the 2-Factor Authentication issue remains a top priority for our engineering team as they continue to work towards deploying a fix. We will provide another update when we have new information to share.
01:59 UTC | 17:59 PT
We are pleased to inform you that a solution for the Two-Factor Authentication issue has been successfully tested and implemented across all Pods. We have also received confirmations verifying that 2FA codes are now being received as expected. We appreciate your patience and understanding throughout this process.
POST-MORTEM
Root Cause Analysis
This incident was caused by a certificate rotation exercise in our service partner’s API, with the result that Zendesk was unable to send SMS messages via the API for the impacted duration.
Resolution
To fix this issue, an updated version of the certificate was uploaded. Once the deployment was completed, calls to the API were successful once again.
Remediation Items
- Adding an alert for 2FA SMS worker failures [Scheduled]
- Adding additional logging for better visibility on which accounts are affected by errors [Scheduled]
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.
1 comment
Eugene Khoo
Post-mortem published December 27, 2023.
0