Service Incident - February 10th, 2021 - Composer Issue

SUMMARY

On February 10th, 2021 from 8:47 UTC to 16:40 UTC, some Sunshine Conversations customers using Agent Workspace were unable to reply to messages, receiving the error message “You can’t reply on this channel because it’s not available."

Timeline

16:43 UTC | 08:43 PT
We are happy to say the issue that is displaying an error for some customers when attempting to reply to messages using Agent Workspace has been resolved. Please message us at [e-mail] if you continue seeing issues.

15:51 UTC | 07:51 PT
We are rolling back a deploy to resolve the issue that is displaying an error for some customers when attempting to reply to messages using Agent Workspace. We will provide an update once this has been completed.

15:19 UTC | 07:19 PT
We are continuing to investigate the cause of the issue that is displaying an error for some customers when attempting to reply to messages using Agent Workspace. We will provide updates.

14:59 UTC | 06:59 PT
We are currently aware of an issue for some customers experiencing an error when attempting to reply to messages using Agent Workspace. We are investigating this and will provide updates.

Root Cause Analysis

This incident was caused by a deployed change to composer permissions which unintentionally impacted Sunshine Conversations customers using Agent Workspace. 

Resolution

Engineers rolled back the deployed change to restore functionality to impacted accounts in the immediate and appropriately flagged all accounts to prevent the issue from recurring. 

Remediation Items

1. Build additional validation audits for account settings
2. Create additional checks and automated flagging for Sunshine Conversations customers
3. Document additional processes for internal teams.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.