Using Liquid Markup to A/B Test Your Triggers
Here’s the scenario: You want to make a change to how one of your workflows operates. Maybe you want to try out some new messaging for when a customer writes in about a particular issue. Or maybe you want to try routing tickets to a different group of associates. The easy part is doing it, but how do you validate whether or not the test was successful?
Normally, you would need to run one option for a few weeks or months, then run the other for the same amount of time. Not only is that an awful long time to spend on a single test, it doesn’t isolate the variable, leaving you with poor data to draw conclusions from. Instead of running consecutive tests, you can use a little bit of Liquid Markup and JSON to do an actual, concurrent A/B test in Zendesk.
Our Case
In our case, we require users to choose an issue type when submitting a ticket to our Zendesk. We wanted to test splitting tickets for one issue type between our own pre-written email responses and our partner's machine learning system. We knew our re-open rate and set an ambitious goal to beat that rate.
The Process
Step 1: Create an HTTP Target
Before you start building anything, you need to create an HTTP Target that points back to your own Zendesk account.
- Navigate to Admin > Extensions.
- Create a new HTTP Target extension.
- Enter https://subdomain.zendesk.com/api/v2/tickets/update_many.json?ids={{ticket.id}} as your URL
- Select PUT as the target method.
- Select JSON as the content type.
- Add your username and password to the Basic Authentication section, then click Save.
This has a drawback, though: if the user whose username and password you're using to authenticate this request leaves the company or otherwise has their Zendesk access deactivated, your target will stop working. If at all possible, dedicate one of your Zendesk seats for these kinds of callback requests to prevent outages. (Or come up with a transition plan for when your admins change.)
Also, note that we’re using the “tickets/update_many” endpoint here, rather than the regular tickets endpoint. The reason for that is because we'll be adding tags to the ticket, which requires the update_many endpoint.
Step 2: Create or A/B Test trigger
Now that you have a JSON notifier in place, we need to build a trigger that places tickets into your A/B test.
- Create a new trigger and add “Ticket > Is > Created” as a condition to the ALL section.
- Optionally, if you don’t want all of your tickets to get tossed into the A/B test, add any other extra criteria. In our case, we’re only A/B testing on tickets submitted with a particular tag, so we added a tag condition.
- Set your action to Notify Target and choose the HTTP Target you just set up.
- Add this JSON response:
{% assign randomizer = ticket.id | modulo:2 %}
{% case randomizer %}
{% when 0 %}
{“ticket”: {“additional_tags”:[“control”]}}
{% when 1 %}
{“ticket”: {“additional_tags”:[“experiment”]}}
{% endcase %}
Wait, What?
Let’s break down what that JSON payload actually does. Liquid is a markup language built by Shopify and written in Ruby that allows you to incorporate a handful of logical expressions into your Zendesk triggers. In this case, we’re using Liquid to accomplish a few things:
First, we’re using Randomizer to assign a random number value to the newly created ticket. We’ve specified modulo:2 to tell Randomizer that there are two possible values for it to create.
When Randomizer settles on 0, we’re calling that ticket the Control and adding a control tag to the tag list using the additional_tags endpoint. The Control is the current behavior that you’re testing against. This should act as normal, working through your existing workflow or process.
When Randomizer settles on 1, we’re calling that ticket the Experiment and adding an experiment tag to the tag list using the additional_tags endpoint. The Experiment will be using your new behavior.
This effectively splits your ticket volume, 50/50, into two buckets that can be routed in two different ways. Want to weight the test more? Increase the modulo number and add more when statements. They don’t need to be different — you could add modulo:4 and have three different “when” statements that give you the control and only one that gives you the experiment (splitting your volume 75/25), like this:
{% assign randomizer = ticket.id | modulo:4 %}
{% case randomizer %}
{% when 0 %}
{“ticket”: {“additional_tags”:[“control”]}}
{% when 1 %}
{“ticket”: {“additional_tags”:[“control”]}}
{% when 2 %}
{“ticket”: {“additional_tags”:[“control”]}}
{% when 3 %}
{“ticket”: {“additional_tags”:[“experiment”]}}
{% endcase %}
Step 3: Add Your New Behavior
Now that you’ve split your volume, you can create a trigger that only acts on that experiment tag. You may want to update any existing triggers that would otherwise act on these tickets to not fire on the experiment tag, just to be safe.
In our case, because we wanted to test our partner's response system, we created a new trigger with their JSON target and added the experiment tag as a required condition. We also added the control tag to the existing trigger, to prevent it from firing on the experiment.
That's a relatively simple use-case. Anything that can act on a ticket based on the presence of a tag—triggers, automations, SLAs, skills—can be tested with this method.
Reporting on Your Test
The beauty of this solution is that you can easily report on the control and experiment for as long as the test is running. In Insights or Explore, whatever you happen to be using, build a report based on your success metrics. Is it reopens? Average handle time? Decide on your KPI and build a report that shows data for both the control and experiment.
For our test, we wanted to track total tickets received and total reopens for both our control and our experiment. From that, we can derive the percentage reopened. If the experiment’s reopen percentage is less than the control’s reopen percentage, we’ve got ourselves a winner. Otherwise, we go back to the drawing board.
-
This is great - thanks for sharing Lance!
-
I agree with Chris! This is awesome :)
Thanks Lance!
-
I agree with everyone above - this is great
-
I'm getting a
{"error":"Unprocessable Entity","message":"Server could not parse JSON"}Any insight?
also it's an HTTP target not HTML target.. i assume thats correct
-
Excellent Write up Lance!
Not only about the clever tip but also of your writing style and engaging information delivery.
-
I can't wait to try this. Fantastic tip!
-
Hello,
I am getting the following issue. can someone help ?
-
@... Good question! You can ignore that error message, the JSON target is trying to validate your code as JavaScript rather than Liquid. As long as the JavaScript between the {% when %} lines is valid, you're good.
-
@...
I'm having some trouble getting this to add additional tags.
When creating the HTTP target, I replace subdomain with my own subdomain, correct?
https://subdomain.zendesk.com/api/v2/tickets/update_many.json?ids={{ticket.id}}
It may be that I am not enabling basic authentication - is that required? -
@... Yes, you'll replace the subdomain with your own ZD subdomain. And you do need to enable auth and enter your username and password in order to pass an API call to Zendesk that updates a ticket.
-
Thank you @...
I enabled auth - how will I know that the API call is passed? Will successful sends update on the extensions page?
I'm still not getting additional tags on created tickets. Maybe it is because I'm using google connector to authenticate? -
@... - Thanks for this great info, But now i seem to be having the same issue as @... above.
See his post here
Thanks
Ash
-
@... @... Sounds like you might have some special characters in your JSON body. That's particularly common when using something like Notepad that changes your quotation marks to curly quotes (rather than straight quotes).
I'd recommend reviewing the characters in your JSON and adjusting as needed.
-
@... is correct, the Unprocessable Entity error was caused by a malformed JSON. It seems the copy and pasting of the JSON example from this article replaced the quotation marks with
“
instead of
"
-
It should be HTTP target, not HTML
Great write up been using this for years
-
Thanks @...! Lance has updated his article.
-
Hi All,
I'm trying to establish an integration between two Zendesk instances, however I get the following error.
Request:
POST /api/v2/tickets.json HTTP/1.1
Response:
{"error":"Unprocessable Entity","message":"Server could not parse JSON"}Below is what I sent in API request:
{% case ticket.group.id %}
{% when standard_group_id %}
{% assign group = 'secure_group_id' %}
{% else %}
{% assign group = '' %}
{% endcase %}
{
"ticket":
{
"subject": "{{ticket.title}}",
"group_id": {{group}},
"comment": {
"body": "{{ticket.comments_formatted}}"
},
"requester": "{{ticket.requester.email}}",
"tags": [
"escalated_from_standard",
"ticket_stub_required"
],
"custom_fields": [{"id": escalated_by_id, "value": "{{current_user.email}}"}]
}
}Can anyone please help me?
Regards,
Kuldeep
-
Hey there @...!
I'd be happy to take a look with you with that payload. A few different things standing out from what I can see:
- I'm assuming that the secure_group_id string is something you've added just as a placeholder for your public comment here, so as long as your payload itself is including valid group IDs in your conditions like {% when 12345 %}, then feel free to ignore this bullet point!
- For your else statement: If you set the value of your group variable to be blank or only single quotes, that may lead to the unprocessable entity errors you've been running into, as your payload will need a group ID entered in order for it to be accepted. Depending on your workflow needs, you could try setting a fallback group ID for your group variable whenever its else statement returns true, like {% assign group = ticket.group.id %}.
- For your custom fields object, we'll want to set the values of custom fields by listing their field IDs rather than their field names (since field names can change at any time while a field ID will be permanent).
Since you mentioned setting up an integration between 2 Zendesk instances though: How would you feel about skipping this HTTP target-based workflow altogether, and instead setting up a ticket sharing agreement using our guide here?
Ticket sharing would allow you to get pretty close to the payload setup you're currently using, and you could also sync custom field data from one Zendesk instance to another. You could also build out trigger conditions and actions to automatically share certain tickets with the other Zendesk instance using a trigger's "Share ticket with" action, worth a look!
-
Hi @...,
Thank you for your answer.
I apologize for having responded late. I'm wondering if you could create a support ticket so that we can have a discussion about the alternative.
Looking forward to hearing back!
-
Hi all,
Thanks for the tip Lance! Amazing.
However, when testing the extension I keep getting HTTP/1.1 401 Unathourized
I am using my e-mail/token as username and my newly generated token as password.
Am I correct in assuming that I should leave the JSON body blank when creating the HTTP target and that the request is PUT with the URL written exactly like this: https://subdomain.zendesk.com/api/v2/tickets/update_many.json?ids={{ticket.id}}
Or should I replace {{ticket.id}} with a ticket id?
All help is much appreciated
-
Maybe this is you, maybe not - But I came across this article elsewhere that seems to have lifted your content word for word - https://roca.work/blog/ab-testing-in-zendesk
Vous devez vous connecter pour laisser un commentaire.
21 Commentaires