Justin

SUMMARY

On March 5, 2024 from 19:50 UTC to 22:06 UTC, some Talk customers may have experienced connectivity issues including the inability to make, receive or transfer calls. 

 

Timeline

21:36 UTC | 13:36 PT

We are investigating reports of Talk connectivity issues including the inability to make, receive or transfer calls. Next update in 15 minutes or when we have more information.

22:06 UTC | 14:06 PT

Our engineers continue to investigate reports of Talk connectivity issues including the inability to make, receive or transfer calls. Next update in 30 minutes or when we have more information.

22:32 UTC | 14:32 PT

Our investigation continues into the Talk related issues resulting in the inability to make, receive, or transfer calls. Next update in 1 hour or when we have new information to share.

23:31 UTC | 15:31 PT

Our engineers have rolled back a change and we are seeing some recovery on our end. We will continue to monitor performance over the next 24 hours and provide an update at that time.

March 6 - 10:59 UTC | 02:59 PT

Talk functionality is fully restored and confirmed functioning normally by both customer feedback and our internal monitoring. We appreciate your patience during this time. This incident is now resolved.

 

POST-MORTEM

Root Cause Analysis

This incident appears to have been caused by a local ISP issue for some customers in the US.

Resolution

To fix this issue, we initially rolled back a recent Talk change; however, our investigation points to an ISP outage which more accurately aligns with the window of impact.

Remediation Items

1. Improve monitoring and alerting around outbound call failures.
2. Work with our provider to adjust alert thresholds so we can be notified faster in similar situations.
3. Improve troubleshooting runbooks to more accurately assess scope.

 

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.

SUMMARY

On March 5, 2024 from 15:30 UTC to 18:25 UTC, Sunshine Conversations customers across all Pods using Social Messaging channels may have experienced a significant disruption in service due to an outage at Meta. This outage primarily affected WhatsApp, with substantial impacts also on Facebook Messenger and Instagram, leading to a partial interruption in outbound traffic through these channels.

Timeline

16:30 UTC | 08:30 PT
We have confirmed an issue impacting all social messaging channels powered by Meta, including Facebook, Messenger, WhatsApp, and Instagram. We will provide another update when our partner provider has resolved the behavior.

17:51 UTC | 09:51 PT
We are beginning to see recovery from the issue impacting use of all social messaging channels powered by Meta. We will continue to work with our partner provider and monitor the situation until full resolution.

19:15 UTC | 11:15 PT
We are happy to report that the issue causing delays in incoming and outgoing messages on all social messaging channels powered by Meta has been resolved, and these channels are processing messages as expected at this time. Thank you for your patience as we worked with our partner provider.

 

POST-MORTEM

Root Cause Analysis

This incident was caused by an issue with Meta's services, specifically impacting the WhatsApp Cloud API. The exact root cause was not disclosed by Meta, but they acknowledged the disruption and worked on restoring services. During this time, Sunshine Conversations' monitoring systems detected a high occurrence of errors when attempting to send messages through the affected channels.

Resolution

The resolution of the outage was entirely dependent on Meta's recovery efforts. Sunshine Conversations monitored the situation closely and provided updates to customers as they became available. Once Meta announced that services were restored, Sunshine Conversations verified that outbound traffic resumed to normal levels and that messages were being successfully delivered through the social messaging channels.

Remediation Items

1. Improve monitoring for outbound traffic health.
2. Review and update the incident response plan to include alternative communication channels with Meta in case of their primary support portal being unavailable.

 

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.

SUMMARY

POST-MORTEM

Root Cause Analysis

This incident was caused by an escaped defect, where the events processing service generated errors when trying to parse webhook events with a null value within their metadata.

Resolution

To fix this issue, the code logic was updated so that null values in metadata objects would be accepted.  We continued to monitor the situation and confirmed in the logs that the errors on Pod 26 were no longer occurring after this change.

Remediation Items

1. Update Sunshine Conversation public documentation to accept null values in metadata objects [Scheduled]
2. Adding additional monitoring alerts for such errors [Done]

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.

SUMMARY

On November 28, 2023 from 13:14 UTC to 17:14 UTC, Zendesk Talk customers on multiple pods experienced the Talk/phone icon going missing, with some customers seeing Explore reports showing high numbers of missed calls. 

Timeline

POST-MORTEM

Root Cause Analysis

This incident was caused by a software fix that contained an escaped defect that was not visible during the testing process.  This fix involved a variable that was not always available during the loading of the voice activity, causing the loading to crash and the call icon to not appear in scenarios where agents had third party apps installed.

Resolution

To fix this issue, we performed a rollback to the previous software version, and had confirmation of recovery.

Remediation Items

1. Adding a test scenario to check call icon loads when third party apps are present. [Scheduled]
2. Adding monitors and alerts when voice activity fails. [Scheduled]

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.

SUMMARY

On November 16, 2023 from 18:02 UTC to 20:00 UTC some Support customers in Pods 13, 17, 19, 23, 28, and 29 encountered delays or a total stoppage in receiving inbound emails. The delay durations varied from 15 to 60 minutes between the sending of emails and ticket creation in Zendesk via Google-managed services.

Timeline

18:53 UTC | 10:53 PT
We are investigating reports of inbound emails not processing for customers on Pods 28 and 29. We will provide additional information shortly.

18:57 UTC | 10:57 PT
We have confirmed an issue that is causing delays in inbound email processing for customers on Pods 13, 19, 23, 28, and 29. Our team is investigating and we will provide further updates as soon as they're available.

19:33 UTC | 11:33 PT
Our team continues to investigate the issue causing inbound email processing delays on Pods 13, 17, 19, 23, 28, and 29. We are working diligently to mitigate impact, and will be sure to share new information as soon as we can.

19:54 UTC | 11:54 PT
We are beginning to see improvement in the issue causing delays in inbound email processing on Pods 13, 17, 19, 23, 28, and 29. Our team will continue to monitor to ensure full recovery. 

21:14 UTC | 13:14 PT
We have resolved the issue causing inbound email delays for customers on Pods 13, 17, 19, 23, 28, and 29, and inbound emails are processing as expected at this time. Thank you for your patience during our investigation.

 

POST-MORTEM

Root Cause Analysis

This incident was caused by the Mail Fetcher service experiencing connectivity problems with Gmail, disrupting inbound mail processing in Support, where Gmail's 302 Moved responses were interpreted as failures by the liveness probe, indicating to the container orchestrator that the Pods were unhealthy. This led the orchestrator to replace the Pods and halt mail processing in the associated containers, causing inbound mail delays or stoppages.

Resolution

To fix this issue, inbound mail traffic was restored after Gmail stopped blocking those health checks, thus allowing Support inbound email to finish creating its Pods and start processing mail again. Not long after, inbound mail queues caught up and traffic started flowing as normal.

Remediation Items

1. Improve existing implementation tools for email health checks. 
2. Create additional alerts. 
3. Add correction code lines on specific applications.

 

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us via ZBot Messaging within the Widget.

Hey shelley,

I suspect that this could be achieved using a custom attribute to filter out tickets with singular engagements. 

Hello Oanh Chau, 

You should be able to make a report like this in the Support: Tickets Dataset. The screenshot below is a bare bones idea of what metrics and attributes you would need to use: 

The Test Tag Inclusion metric I have here uses the following formula to display only solved tickets with the tags you specify: 

IF (([Ticket status - Unsorted] = "Solved" OR [Ticket status - Unsorted] = "Closed")) 
AND (INCLUDES_ANY([Ticket tags], "your_tag_here"))
THEN [Ticket ID]
ENDIF

You would need to make two custom metrics for your Large and Super Large tags, so they will show as separate columns in your report. 

Hey Bobby Koch, 

The backlog snapshots occur at midnight daily. To view the most current totals for outstanding tickets, we recommend using the Tickets dataset. 

Hey Joel Sandi, 

To answer your first question, you should be able to report on ticket information with an Explore Report using the Guide: Knowledge Capture dataset. There is a set of attributes categorized as "ticket" that allow you to look closer at ticket data from the Knowledge Capture App. 

As for your second question, the articles linked per ticket metric, as well as all the metrics under the "Knowledge Capture" Dashboard tab, are meant to gauge performance metrics of the Knowledge Capture application, primarily. From a business value perspective, the articles linked per ticket metric could show pain points for your staff in your Knowledge Base. I would assume that ideally, you would want this number to be as close to 1 as possible (1 article for each ticket). 

Hey Jacqui, 

Are you using bookmarks to capture the desired time range in this dashboard?