SUMMARY
From August 7, 2022 from 17:55 UTC to September 8, 2022 17:07 UTC, WhatsApp customers, across multiple Pods, may have experienced their business profile page returning HTTP 500 responses and error messages when trying to access their settings via the Support Admin Center & Sunshine Conversations-only interfaces.
Timeline
13:28 UTC | 06:28 PT
We’re currently investigating and working on a solution for issues affecting customers using Support Admin Center & Sunshine Conversations-only that are getting error messages when trying to edit Whatsapp settings. More information in 1h or when we have more details.
14:01 UTC | 07:01 PT
As a possible workaround for the issue regarding inability to edit Whatsapp settings, if you have a Sunshine Conversations License, please make changes through API calls via https://docs.smooch.io/rest/v1/#whatsapp-message-templates. More updates as we know more.
14:52 UTC | 07:52 PT
We appreciate your patience while we continue working through the issues affecting customers being unable to edit their Whatsapp settings. We are in contact with our provider in order to get this resolved as soon as possible. Another update in 1h or if we have more information.
15:50 UTC | 08:50 PT
Our engineers continue to work with our provider to resolve the issue causing Support Admin Center & Sunshine Conversations users to be unable to update their WhatsApp settings. Next update in 2 hours or when we have new information.
17:52 UTC | 10:52 PT
Our engineers have escalated the issue causing Support Admin Center & Sunshine Conversations users to be unable to update their WhatsApp settings to our provider. Next update in 2 hours or when we have new information.
20:40 UTC | 13:40 PT
The issue causing Support Admin Center & Sunshine Conversations users to be unable to update their Whatsapp settings is now fully resolved. Sorry for the inconvenience this may have caused you and your customers.
POST-MORTEM
Root Cause Analysis
This incident was caused by a software patch installed by our service provider, resulting in HTTP 500 response errors.
Resolution
To fix this issue, our service provider partner rolled back the defective patch.
Remediation Items
- Adding steps to the 4xx and 5xx errors runbook to verify 3rd party outages.
FOR MORE INFORMATION
For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.