Service Incident - November 1, 2022 - Support (Agent Workspace, Social Messaging add-on) | Multiple Pods - Unable to send messages via the WhatsApp channel

SUMMARY

Between November 01, 2022 from 01:40 UTC to November 08, 2022 19:00 UTC, Zendesk Support customers on multiple pods were unable to send messages through the WhatsApp messaging channel. This included customers using the Support Agent Workspace interface, or the Support standard interface with the Social Messaging add-on.

Timeline

02:24 UTC | 19:24 PT

We are currently investigating reports of issues sending messages through the WhatsApp messaging channel. More info to come.

02:35 UTC | 19:35 PT

We continue to investigate the issues in Agent Workspace impacting WhatsApp message sending and retrieval. Our team is investigating at the highest priority. Next update in 30 mins.

03:00 UTC | 20:00 PT

Our team continues to investigate the issue with WhatsApp in Agent Workspace. We will provide the next update in 60 minutes or when we have a substantive update to share.

03:58 UTC | 20:58 PT

We are working with our service provider on the WhatsApp (Agent Workspace) issue and have narrowed down the path of investigation. We appreciate your patience as we work towards fixing this issue. Next update in 60 mins or when an update is available.

04:57 UTC | 21:57 PT

Our service provider continues to work through the WhatsApp issue in Agent Workspace. While a recent attempt to resolve the issue was unsuccessful, teams are fully engaged and continue to troubleshoot at the highest priority. Next update in 60 mins.

06:02 UTC | 23:02 PT

Our service provider has identified the cause of the WhatsApp issue and is working on a fix. We will provide the next update as soon as we have more information to share. Thanks again for your patience and understanding today.

POST-MORTEM

Root Cause Analysis

This incident was caused by an expired token used to authenticate with our WhatsApp service partner.

Resolution

To fix this issue, we worked with our service partner to generate this token.  After the token was successfully generated, we continued to monitor until recovery was observed.

Remediation Items

1. Working with our service partner on quicker response times on critical issues [Pending]

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.