Zendesk on Zendesk is a day-long discussion about a specific topic and how Zendesk Support uses Zendesk. Each session is hosted by a member of our Support team.
In this month's Zendesk on Zendesk, Jon details how our support team tackles the challenges of communicating across timezones, offices, and teams while supporting a large and diverse customer base. Learn about:
- History and background on different communication tools and applications at Zendesk
- Details about key Flowdock flows and their use cases
- Integrating Flowdock and Zendesk to display and update key ticket information
This session is hosted by Jon Brummel, a Tier 3 Support Engineer in our San Francisco office.
See all of the Zendesk on Zendesk series discussions.
Welcome back to Zendesk on Zendesk! As companies evolve and expand, team collaboration and communication is a very unique and interesting struggle.
As we grew from our humble roots, with three founders gathered over a literal door which was made into a work table, to a global company with footprints in the Americas, APAC, and EMEA with hundreds of employees, we faced a conundrum: how could we communicate with one another—easily, elegantly, and efficiently with as little friction as possible?
When Zendesk was an infant, things were much simpler — you could literally look up and be face to face with a colleague! As things expanded, this would rapidly become unfeasible.
After a short stint in Boston, and finally settling into San Francisco, Zendesk hired our first customer advocates. Yes, that’s right: customer advocates . Here at Zendesk, we don’t just have a support team to answer the technical, financial, and educational needs of our clients—we’re also here to advocate on the behalf of the customer. We are your voice in critical interactions with our product management team, operations, and engineering team in absentia.
Our team rapidly expanded from a handful of agents in San Francisco to London, Melbourne, and, today, Latin America, Madison, Dublin, Copenhagen, and Manila!
With a global team, we had several key challenges in regard to inter-team communication. Email is great, but it’s not an ideal channel for a large customer base, or support team. (That’s why you use Zendesk, right?) Email is also a particularly bad mechanism for inter-team communication and knowledge sharing.
For static information, including documentation, internal processes, and other materials, we use a variety of tools, including email, Confluence , Google Docs , Box , and we drink our own champagne with our own beloved Help Center (of which, we have both a public and an Internal section—if you’re interested, we can share more in a future post!)
The challenge however, is finding a suitable mechanism for transferring information on multiple fronts:
One-to-one: communication between team members
One-to-many: both for real time interactions, as well as asynchronous information for teams that will be coming online soon—specifically, incident communication and tracking of significant issues (infrastructure or code-based) that all team members need to have access to
There are myriad reasons that you would need a conference with another team member. It can be as simple as a quick information transfer or as complex as an in-depth collaborative effort and project. Often, there are occasions when you must connect with a colleague in a time-sensitive way and you might not be able to simply walk over to their desk.
In addition to the cross-continental challenges we face, many of our facilities are comprised of not only several buildings, but multiple floors. In addition to open floor plans and public working spaces, our team is highly mobile with team members occasionally working remotely, or other times they simply move to another portion of the facility and you have no idea where to find them.
The tools we’ll be discussing today have been instrumental in not only facilitating real-time communication among various colleagues in remote regions of the globe, but literally fifty feet away (this might have happened to me…), but you might not know they’re on the same floor, since they’re working in an open space, in the common area. I often like to sneak down to the basement, where it’s nice and open, and often pretty quiet. If I have a major deadline or project I can really knuckle down and focus, but still be available to my team as needed—thanks to the tools we’ll be discussing here today.
Have you ever been part of an email thread where they CC everyone in the company, your college roommate, your dog, and your great-aunt Phyllis? It’s never fun, and adds up to a lot of extra information that can be a bit overwhelming.
With several of the tools we’ll discuss here today, we can share key information with the global team in a non-intrusive fashion, and put it in a medium that can be consumed as desired and when needed.
As our team and our needs grew, we found ourselves moving through a variety of tools before finally settling in on one tool to rule them all for the Support and Engineering team. Okay…well, mostly one tool… (It’s complicated—you’ll see.)
A bagful of tools
While doing research for this article, I spoke with one of our earliest support representatives from San Francisco and was presented with the following anecdote:
Team communication in the early days?! Easy! We just walked over to their desk or yelled at them across the aisle ;)
I’m not going to lie—to this day, SneakerNet continues to be one of the most powerful tools in my arsenal. Simply walking over and chatting with a colleague is priceless . There are nuances of human communication and tone that are impossible to express in a digital medium.
That said, out of respect for my colleagues’ schedules, I do use traditional digital channels whenever appropriate, and try to save the Emergency visits to Operations only for critical issues, or for my nearby colleagues when they are available.
Skype became the primary method in which our Support team would both accept inbound and make outbound calls to customers for quite some time. To this day it continues to be a preferred method of one-to-one chat for the Support Team, and while we have migrated to Zendesk Voice as our voice solution, Skype continues to be a popular method of internal communication, and is a great vehicle for a fallback voice solution, if needed.
The video-conferencing features in Skype are extremely popular in Zendesk, particularly for our support team. We use Skype daily to conference in colleagues who are working remotely. In fact, our friends in Dublin and London hold a daily joint stand-up meeting where they conference via Skype, iPads, and Bluetooth speakers.
Our first formal tool for team communication was Campfire . Essentially, it was used in conjunction with Skype, but its primary focus was the team chat component which facilitated teamwork and knowledge exchange during the troubleshooting process.
IRC (Internet Relay Chat)
An interesting challenge that we faced for some time as Zendesk grew was that each team had a familiarity and comfort level with their own tool of choice. Our developers, as is likely not surprising, preferred to use a tool called Internet Relay Chat (IRC) , a standard chat client on the internet since the 1980s.
Due to the close relationship between the engineering team and our customer advocates, this posed a bit of a challenge. Each team was using a different tool and not able to readily communicate with one another, aside from comments in tickets, or if the team were to use an additional tool.
This brings our tool tally up to three for the Support team: Skype, Campfire, IRC, and naturally, a fourth tool preferred by our sales team.
To keep in touch with their clients and potential leads, email is a powerful tool for the sales team. As a result, an integration with Google Messenger, and later, Google Hangouts was the natural tool of choice, as it elegantly interfaces with the gMail interface.
To this day, it continues to be not only the preferred method of messaging for our sales and IT team, but also a quite popular method of video conferencing for all teams outside of the support organization. Any time a cross team video conference is held, a Google Hangout is often the default channel.
One tool to rule them all
As you can see, we had quite a variety of channels occurring simultaneously, and a central vehicle was needed.
Flowdock was a beautifully simple solution for our team at Zendesk, one that both our engineering and support teams could get onboard with. It featured a number of integrations that we use daily to show code deploys, Airbrake alerts from our infrastructure, Twitter @Mentions and more!
In addition to their prebuilt integrations, they also have a robust API, which can be leveraged in a number of ways, including integrating with your Zendesk .
In Flowdock, a flow is a separate room that can be created inside of your Organization. We have created quite a few flows here at Zendesk, but something that we have found quite useful, is to create flows that correspond with:
- Our Zendesk Groups
- Geographic region
- Our About field
Custom fields—your new best friend.
What is an About field? Phew, I thought you’d never ask!
Every product, business, or organization has a variety of areas of focus, even within one product or service. In our Zendesk instance we have created a truly massive custom ticket field that has a variety of areas of focus that surround not only the specific features in Zendesk, but also about subject areas that surface throughout the lifecycle of a customer relationship.
This area really deserves a post of it’s own, but I’ll give you a bit of insight into our friend, the About field. First, it contains no less than 270 options . That… is one big field. Second, it is under regular review, typically it is reviewed, and updated on an annual basis by our Support Operations team.
There are a couple of great lessons to be gleaned here:
- A field of this nature is fantastic for identifying tickets of a similar nature for reporting, staffing, and and ticket volume analysis. If you’re not using something similar, it would be a great idea to consider implementing one! Custom fields have a significant advantage over tags when you’re building your Insights reports
- Each drop down field option is associated with a tag — as you know, tags can have a giant impact in your Zendesk workflow, as they’re not valuable only for searching for tickets , but you can build custom actions based of of them with triggers and automations .
We won’t cover all 270 field values, but here’s a few ideas of things you might find useful — but, remember, this is Top Secret — you only can view this, if you promise to use it, should it enhance your business:
A great example of how we utilize custom flows may be found in the relationship between our About field, and our targeted flows: one that is quite popular is our Reporting flow, of which, the product champions, product manager, and agents working those tickets can converse about a highly specialized area of the product, without cluttering out our main forum across teams, the Customer Advocate flow.
Each primary area of the product has at least one (and sometimes several) About fields for it, and a corresponding Flow for that area of the product.
Zendesk Voice, for example, has several:
are a boon to your organization, and can also be used
— if you’d like to implement a similar set of fields, simply use the Syntax:
Top Level::Next Level::Item Here
You an go as deep as you’d like by adding more levels separated by a
— but a good rule of thumb is to use as many levels as you need, but as few as you can get away with.
While Zendesk utilizes a number of flows, a few stand out:
This flow is a bit like drinking from a fire hose . All global advocates congregate here, in addition to product managers, sales team members, services, customer success, and more. It’s a fantastic forum to ping the hive mind for not only specific technical help, but also to both notify agents of an issue, as well as see if an issue may be trending.
Whenever a Problem ticket is sent up to our Engineering team, the advocate creates a post in the Customer Advocates flow to notify both online agents and agents in other global offices who will be coming online later.
An advocate post is codified by internal processes, and is in the following format:
All of the information in our Problem post, comes directly from a ticket in our own Zendesk instance.
- #Problem : Directly tied to the ticket type of the post
- #388315 : Ticket ID for the problem
- Brief description of the problem : Taken directly from the ticket title
- ** https://support.zendesk.com/agent/#/tickets/388315 :** Direct link to the ticket, so the team can easily visit it with a single click
- #nico : Specific agent tag of the Agent who posted the ticket — with this information, it’s a simple matter to reach out to the posting agent for clarification, or to identify if you may be working on a related issue
Our Operations team uses this flow for communication about key infrastructure issues, traffic loads, and deploys. It’s where the developers and network operations agents exchange information about the health and status of the Zendesk infrastructure. When our advocate team needs to reach out to Operations and Engineering, this is our first, and best mechanism to respectfully approach them.
This flow is a rotating role staffed by a developer who not only processes inbound Problem tickets from the customer advocate team for its severity. It’s also available in the flow for consultation by other developers and the support team. By having a dedicated developer on-call in the flow, it allows the other developers to focus on their coding duties at hand without multiple interruptions.
If there is a critical service issue that is multiple customers, a security issue, or an issue that is discovered that affects critical business functions, our team will sound a Red Alert , that begins a sudden flurry of activity. The Pager Duty service is employed to rouse key members of our team, whereupon the Zencident flow goes immediately from a ghost town to a mission control center as a support manager, a support incident lead, developer operations agents, network operations engineers, and sustaining engineers converge to address a critical issue that has been identified by a team member.
Having a dedicated flow and destination allows an uninterrupted flow of communication, as well as a place that other agents may observe the status and progress of an incident as it is diagnosed, addressed, and communicated with our valued customers.
The present, and the future
Our team is constantly committed to refining our internal processes and becoming more efficient. One of our advocates cleverly took advantage of the external target capability of Zendesk and created a trigger/target combination that would automatically update one of our flows when a new Problem ticket was posted. This could be highly helpful to notify agents, since diagnosing an issue early allows all other incoming tickets to be attached to the Problem ticket via the incident ticket feature in Zendesk. By ensuring these tickets are sent to the proper destination as soon as the issue is diagnosed, we save both advocate time and customer frustration, making sure they’re updated as soon as we have a fix.
Last fall, I partnered with my colleague Ben Evans to take this great idea and level up via the Zendesk Apps Framework to create FlowBot , a digital assistant that automatically posts Problem tickets and updates their statuses:
This posts the Problem ticket in the proper format and, when it’s solved out by our team, changes the
, letting the team know that this issue is no longer in play. In addition to having the information posted immediately, it saves the advocate team an incredible amount of time as they don’t have to post in Flowdock each time they create a new Problem ticket. Even better, we don’t have the risk of any legacy
tickets floating around for months after the issue was solved out if an advocate forgets to update the original post.
By a clever use of Zendesk, Flowdock, and the Zendesk Apps Framework, we were able to enhance our global team communication and save our advocate team time.
It is truly remarkable to see agents descend upon a problem ticket in real-time, thanks to a integration between Zendesk ticketing, the Flowbot app, and Flowdock, as suddenly I witnessed several agents converge, in order to assess the issue, and help minimize customer impact.
Now that you know our story, what’s yours?
- How do you keep in touch with your teams?
- What are the specific challenges that you’ve encountered on your journey?
- How did you address them?
- How do you share knowledge amongst teams?
- What are the specific pitfalls and success strategies that you’ve evolved in your teams?
- What tools are working well for you?
- What processes like FlowBot have you tried, or can you think up that would enhance your team’s experience, and productivity?