Incident Management part 2: How Zendesk manages service incidents

This is part 2 of the Overview of incident management at Zendesk. This guide contains the following parts:

• Part 1: How Zendesk service issues become service incidents
• Part 2: How Zendesk manages service incidents (this article)
• Part 3: Monitoring a public Zendesk service incident
• Part 4: Post-resolution incident analysis and reporting

In this article, part 2, you'll get an understanding of how the Zendesk teams respond to service incidents within our products based on severity levels. Zendesk takes a comprehensive approach in understanding an incident--from its root cause to the total impact to affected customers--and communicates the appropriate level of detail.

This article contains the following sections:

• Incident severity
• Incident response team structure
• Communication timelines for incidents
• Low severity incident process

Incident severity

One of the key decisions made when a service incident is created is assigning the incident’s severity. The severity of an incident determines how and which Zendesk teams address the issue and how it is communicated to customers who are affected.

Zendesk uses a system that groups service incidents into 5 severity levels based on the characteristics of the incident:

Zendesk Severity Rating System

Different escalation paths and teams are engaged to investigate, communicate and remediate the incident based on severity level. This ensures the right level of rigor is given to each incident. The diagram below describes the key activities that happen during and after an incident is cleared based on its severity level:

Process by Severity Level

While high severity incidents go through rigorous analysis and remediation activities, every incident - regardless of severity level - goes through a real-time response and investigation process. That produces:

• Updates to the Zendesk status page when the incident is public
• Root cause analysis and incident remediations
• Zendesk (internal) incident report

Zendesk service availability incident example

Here is an example of how Incident Severity is set by Zendesk and how Zendesk teams respond internally:

Incident Discovery and Response Example

In the example, you see the following workflow:

1. The Zendesk Network Operations Center (ZNOC) identified an issue when system alerts showed service nodes in Pod 17 could not be reached by requests. The Zendesk Network Engineering team verified the access issues were affecting customer services directly and quickly realized the Support, Guide and Talk services for multiple customers were not operating as expected. A new Zendesk service incident was created.
2. This incident was known to affect two customers when it was initially created, but because of the nature of the outage, more customers were experiencing the issue and began to raise their issues with Zendesk Customer Support. The incident was assigned a severity rating of 1 by the Engineering team - a high priority incident that requires immediate attention. 
3. The Incident Response on-call team was paged immediately. Within minutes of incident creation, an Incident Manager gathered information and assembled additional engineering resources to troubleshoot and fix the service incident.

Incident response team structure

Zendesk has a dedicated global Incident Response team to ensure that every incident is shepherded through the service incident management process and escalated to the appropriate levels of Zendesk leadership, as warranted. 

Incident Management Roles and Responsibilities

This team structure enables Zendesk to conduct a thorough analysis of the incident with technical resources and communicate in real time to customers through Zendesk Customer Support. 

Communication timelines for incidents

Zendesk is invested in making sure incidents are properly communicated and resolved in appropriate timeframes for the customer. We have established internal timelines for the distribution of incident details. The timeline is based on the severity level of the incident and the service incident management stages.

Stage	Response Timelines
Public Announcement	Within 15 minutes of the incident called
Incident Updates	Every 30 minutes until service is restored or as new information becomes available
Event Analysis  (for Severity 0 and 1 incidents)	Within 48 hours of incident resolution
Root Cause Analysis	Within 72 hours of incident resolution
Public Incident Retrospective	Within 96+ hours of incident resolution

Incidents that have a Severity rating of 0 - 2 are considered high severity incidents. When a high severity incident occurs, the global on-call Incident Response Team is available 24/7 to respond to these incidents. The team consists of the following roles:

Incident Response Team Roles

Global Incident Response Team Locations

As the on-call team is paged, the incident diagnosis starts within minutes of the incident being declared. A Slack channel and Zoom call are created to enable response team communication in real-time. As the Incident Response team triages and scopes the incident, on-call engineering teams are paged based on what products and services are affected.

A public post on the Zendesk status page is made within 15 minutes of incident creation to keep customers informed about the known incident. Updates are posted every 30 minutes thereafter until resolution as new information becomes available. Depending on the issue and how much new information is identified, this cadence may be reduced or lengthened as needed.  Customers can monitor active service incidents on the Zendesk Status page - that process is described in part 3 of this guide. 

In addition to our global on-call incident response teams, Zendesk has established processes for leadership notification and escalation. If a high severity incident fits certain criteria, we enable the next level of response, which is Crisis Management.

Zendesk service availability incident example continued

In continuation of using the service availability incident as an example, this is how the incident response progressed through the Incident Management process at Zendesk:

Service Availability Incident Response Timeline

As you can see in the example, once the incident was created in the Zendesk Incident Portal, a series of automated actions were taken:

• The Incident Response on-call team was paged to respond to the incident

• An incident Slack channel was automatically created and the Incident Response on-call team was added to the dedicated Slack channel

• A Zoom call was automatically started and posted to the Slack channel for all responders to join

• An Event Summary Document was automatically created to document the incident and share progress internally to Zendesk

On the Zoom call, the Incident Manager validated the initial severity and confirmed the scope and impact of the issue. 

It was quickly determined multiple container nodes in Pod 17 were not accessible and could not be used by dependent services including Support, Guide and Talk products. One node type in particular had no available replicas in other pods. This would eventually cause these products to become unresponsive for multiple customers.

The ZNOC paged the appropriate Network engineering team to the Zoom call to begin investigating how to solve the immediate problem of restoring service and API access to customers.  Edge engineering SMEs were also paged and joined the call. Within 5 minutes, a fix was identified and deployed so the affected nodes were again accessible to API calls and services. 

Zendesk Customer Support created a problem ticket to track the customer reports. This ticket was added to the incident Slack channel to quickly allow for new reports to be added as they came in.

While the investigation was continuing, the Incident Escalation Manager created and published the first public update to the Zendesk Status Page 12 minutes after incident creation. 

First Service Availability Incident Post on Zendesk System Status Page

While the teams investigated the incident, customer reports that came in were linked to the main problem ticket associated with the incident. This allowed the Incident Response team to send updates to all impacted customers when they made public notifications.

The Network engineering team determined a change to how certificates were generated and used was responsible for the incident and took the following actions to restore service to affected customers:

• Deactivated unreachable nodes

• Created new service nodes with properly referenced certificates

• Verified that new service nodes were accessible for services and through API calls

• Monitored inbound traffic to see that inbound requests were now being handled appropriately

As the incident progressed, two more public updates were made: One 14 minutes after incident creation and another 63 minutes after incident creation. The public communication history along with published incident retrospective information can be found on the Service Notifications page for the incident.

As shown in the example, high severity incidents go through a rigorous process where root causes are determined and remediation items are created for Product Engineering teams to fix the underlying problem that caused the incident. This analysis happens during our incident retrospective and is discussed in more detail in the Post Resolution Incident Analysis section.

Low severity incident process

Lower severity service incidents (level 3-4) are less critical because they affect a smaller number of customers and do not prevent those customers from using business critical functions of Zendesk products. These incidents are addressed according to the guidelines above, but are not posted to public channels.

Severity 3 incidents are handled in much the same way as severity 0-2 incidents. Expected response times are extended because of the reduced business impact. Even though the on-call team is not paged, these incidents are handled through specific  Zendesk incident Slack channels associated with the supporting product engineering team(s), and the teams tend to respond as quickly as higher severity incidents. Most severity 3 incidents do not use public communication channels. Instead, Zendesk Customer Support teams reach out to customers using proactive notifications if specific action is required from a subset of customers.

Severity 4 incidents do not directly affect customer use of Zendesk services, but have the potential to do so if not addressed. These incidents are created as proactive responses to potential issues. Product engineering teams engage the same way as they do with the severity 3 process.

Learn more

This completes Part 2, How Zendesk manages service incidents, of the Overview of incident management at Zendesk.

If you'd like to learn more, you can move on to the next part of this guide: Part 3: Monitoring a public Zendesk service incident.