Relevant to Zendesk customers with accounts activated prior to November 1, 2016
What happened?
We recently were alerted by a third party regarding a security matter that may have affected the Zendesk Support and Chat products and customer accounts of those products activated prior to November of 2016. Once we became aware of this information, the Zendesk Security teams and our external forensics experts launched a comprehensive investigation into the incident. While our investigation is still ongoing, on September 24, 2019, we determined that information belonging to a small percentage of customers was accessed prior to November of 2016.
We identified approximately 15,000 Zendesk Support and Chat accounts, including expired trial accounts and accounts that are no longer active, whose account information was accessed without authorization prior to November of 2016. Information accessed included some personally identifiable information (PII) and other service data. We have found no evidence that ticket data was accessed in connection with this incident.
The information exposed from these databases included the following data, potentially up to November 1, 2016:
- Email addresses, user names, and phone numbers of agents and end-users of certain Zendesk products.
- Agent and end user passwords that were hashed and salted - a security technique used to make them difficult to decipher - potentially up to November 2016. We have found no evidence that these passwords were used to access any Zendesk services in connection with this incident.
UPDATE: We have also determined that certain authentication information was accessed for a set of approximately 7,000 customer accounts, including expired trial accounts and accounts that are no longer active. Upon further analysis, we also found an error and identified a group of customers who had a small number of TLS certificates accessed, almost all of which are currently expired.
Here is the information impacted:
- Transport Layer Security (TLS) encryption keys provided to Zendesk by customers
- Configuration settings of apps installed from the Zendesk app marketplace or private apps. This may include integration keys used by those apps to authenticate against third party services.
We deeply regret that this incident occurred. The safety and security of our customers and their data is of paramount importance to us. Our goal is to communicate this information as quickly as possible with transparency and guidance on how to address. We will be updating and sharing more information in this blog post as it becomes available.
What has been done to remedy the situation?
At this stage, our security team and a third party forensics team are still completing their in-depth investigation and analysis. Based on the information we have to date, we have also taken the following actions:
- Engaging a team of outside forensic experts to validate this security matter and to determine the exact data and information that was exposed.
- Activating our internal data security response team and protocol. This team continues to investigate with full resources dedicated to determining how this exposure occurred.
- Informing law enforcement and the appropriate global regulatory agencies.
- Informing all impacted customers directly and sharing the steps we are taking to safeguard their accounts and data, and additional actions they can take themselves.
- Implementing as a precautionary measure password rotations for certain end users and agents created prior to November 1, 2016.
- In the next 24 hours, Zendesk will start implementing password rotations for all active agents in Support and Chat, and all end users in Support created prior to November 1, 2016. We recommend users change their password ahead of time for convenience. This password rotation will impact all other products which share authentication with Support, including Guide, Talk and Explore.
- Upon their next login, each of these users will be required to create a new password. Note that if you utilize basic authentication into the Zendesk or Chat APIs through your password, you will need to reestablish you connection to those APIs upon changing your password. This password rotation will not impact the use of API or OAuth tokens.
- You will not be impacted by this if we have been able to identify that you have updated your password since November 1, 2016 or if you use Single Sign-on.
What should I do?
If you have received an email from us saying that you had an account prior to November 1st, 2016, we recommend the following steps:
- If you installed a Zendesk Marketplace or private app prior to November 1, 2016 that saved authentication credentials such as API keys or passwords during installation, we recommend that you rotate all credentials for the respective app. You can pull the list of app installations and their corresponding installation dates using /v2/apps/installations.json API endpoint.
- In addition, if you uploaded a TLS certificate to Zendesk prior to November 1st of 2016 which is still valid, we recommend you upload a new certificate, and revoke the old one.
- While we have no indication at this time that other authentication credentials were accessed, customers may want to consider rotating authentication credentials used in Zendesk products prior to November 1, 2016. API Tokens in Chat do not need to be rotated.
Is my Zendesk data secure?
While no security measure can be considered to be 100% effective, Zendesk has significantly invested in its security program since 2016. Since then, we have made significant investments in our Security Program, including rolling out additional protection of sensitive personal data by implementing and aligning log and data retention with the General Data Protection Regulation (GDPR); deploying Customer Controlled User Assumption (CCUA), which restricts the access Zendesk employees have to customer service data and more than doubling the size of our security team.
At this stage, our security team and a third party forensics team are still completing their in-depth investigation and analysis of this incident. Once this investigation has been completed, we will also aim to share more information.
How does the password rotation work?
In the next 24 hours, Zendesk will start rotating credentials for those agents and end users that had not rotated their credentials since November 1st, 2016, and are not using Single sign-on.
This password rotation will impact all other products which share authentication with Support, including Guide, Talk and Explore. Upon their next login, each of these users will be required to create a new password. We recommend users to change their password ahead of this time for convenience.
Note that if you utilize basic authentication into the Zendesk or Chat APIs through your password, you will need to reestablish your connection to those APIs upon changing your password. This password rotation will not impact the use of API or OAuth tokens. You will not be impacted by this if we have been able to identify that you have updated your password since November 1, 2016 or if you use Single Sign-on.
My Zendesk account was created after November 1, 2016. Does that mean I am not affected?
Correct. We have no evidence to indicate that accounts created after November 1, 2016 were affected.
Was my Zendesk Service Data compromised?
We have not established that PII or other Service Data of customers was accessed other than those 10,000 accounts we have specifically identified. If we have determined that your account was impacted, we have specifically notified you and are sharing the steps we are taking to safeguard your account and data, and additional actions you can take.
What Zendesk products are affected?
We have no evidence that products other than Support and Chat were impacted. Please note that the password rotation we are implementing will also impact all other products which share authentication with Support, including Guide, Talk and Explore. BIME, Connect, Sell, and Smooch were not impacted and will not be impacted by our password rotation.
Do I need to report this incident to my Data Protection Supervisory Authority?
If we have determined that your Service Data, including any PII, was compromised, we have specifically communicated with you as to that determination. Otherwise, we have found no evidence that PII or other Service Data was compromised.
Zendesk’s customers are the Data Controllers of Service Data, and Zendesk is a Data Processor of that Service Data when it is performing the Zendesk Service. This means that it is up to each customer to determine if it is required under Article 33 of the General Data Protection Regulation 2016/679 (GDPR) to notify its Supervisory Authority, depending on whether the relevant risk thresholds have been met under the GDPR. We will make available all information we have for you to help you make this determination.
Can you tell me what specifically was compromised/a list of data that was compromised?
If we have determined that your Service Data, including any PII, was compromised, we have specifically communicated with you as to that determination and will work with you to provide more specifics. If we have not specifically communicated to you that your Service Data was compromised, we have no evidence that it was.
Where will you be posting updates and post-mortem?
We will continue to post important updates to this help center article as needed, including a public post-mortem once it is completed. Once this investigation has completed, we will aim to share more information through our web site and blog.
Where can I find more information about your security programs?
Customers who would like to obtain more information or answers to specific questions about our security program can review the Security portion of our web site at https://www.zendesk.com/product/zendesk-security/.