Combating spam submitted via web service

Return to top
Have more questions? Submit a request

99 Comments

  • J.H.

    One thing you should all do is report the domains to the domain Registrar -- for bitbiz.xyz that is Namecheap -- abuse@namecheap.com. The more reports, the faster they will respond and take down the site.

    Any time you deal with spam, the people hosting it should be notified. While I'm sure Zendesk is working on this too, it definitely helps when there are multiple reports. It will go a long long way to stop the behavior. Bitbiz is also touting instagram services, so report to instagram too.

    Also, why are you trying to circumvent this recommendation? Having a configuration that you knowingly are aware can be used as a way for malicious actors to send spam is dangerous and being irresponsible as a netizen.

    Doing the above will minimally effect you or your customers, so you should reconsider your "workarounds". Its your company domain names that are being used to send spam -- you should eliminate the possibility of it happening.

    3
  • pstrauss

    We just started getting a bunch of these as well. Can someone explain how they could be marked as submitted via API if our API access is locked down by token and Oauth? I just want to make sure this isn't a larger security issue with the Zendesk API.

    1
  • Ryan W
    Zendesk team member

    Hey pstrauss,

    The endpoint being used by the spammers is the one your webform submit a request form and web widget uses. Due to the nature of contact forms, these do not require authentication (and therefore not locked down by oauth at this time). There is no indication of any larger security issue (outside of the configuration of triggers which fire on ticket creation) in this instance, the spammers are unfortunately just exploiting expected behavior.

    The volume of the spam you're experiencing can be explained by rotating I.P.s, emails, etc, which circumvents traditional rate limiting.

    Feel free to write into support if you have any concerns, and we will do our best to alleviate them.

    Thanks.

    0
  • Jim Stalder

    pstrauss -  Zendesk does allow the ability to create tickets via anonymous methods.    The API token and Oauth is for read, but not write access.   Specifically, https://developer.zendesk.com/rest_api/docs/support/requests#create-request     Us admins can turn this ability off via end-user settings (https://support.zendesk.com/hc/en-us/articles/203663806)     However, in turning it off, you also disable the ability to receive email from unregistered sources (e.g. new customers).   In my perfect world, Zendesk would differentiate between these two methods for anonymous requests and allow individual settings for each method.   Spammers could be doing the same thing via email, but that hasn't been an issue for me yet....

    *disclaimer.  Above are my opinions and understanding of how things work.   I could be incorrect.

    2
  • pstrauss

    Understood, Ryan. We have implemented the recommended fixes to the autoresponse email triggers. I was unclear on one thing - do we also have to remove the two fields in question from trigger messages sent by our agents when they update the status of a ticket as well?

    0
  • Ryan W
    Zendesk team member

    Jim Stalder -- That sounds right on the money to me, and I'll pass on your feedback to the appropriate places -- Thanks for providing those links!

     

    pstrauss -- You should only need to remove from triggers which sends the notification to end-users that has the ticket is created condition within it (i.e. Notify Requester of Received Request). We've done this in our own account, so feel free to write into Support [at] zendesk.com for an example.

      --

    Ticket update triggers (I.e. Notify Requester of comment update) are completely fine to keep and should remain unaffected. It's important that you keep those there to ensure your end users are receiving your response.

    Hopefully that helps!

    0
  • Scott

    First off: ZENDESK NEEDS TO FIX THIS. PERIOD.

    It's irresponsible at best to have such an easy exploit open in every customer account by default. It's utterly absurd that customers can't kill off the "Web Service" channel with a simple setting. 

    Get your act together.

     

    Now, we need to get things working right. So, here's how I've mitigated this attack using the techniques others have discussed in prior posts. It takes two triggers to do it. (One new trigger and an edit to your automated response trigger.)

    This will work for you if you aren't using any of their Web Service stuff for tickets. 

    First Trigger:

    The first trigger I set up snags anything created via their "Web Services (API). This is a new trigger created in Zendesk as follows:

     

    This does 2 things:

    1. Marks the ticket "Solved" (I might change this to "Closed" after I'm 100% comfortable with it working as designed.)

    2. Applies a "tag" for these spammy tickets  (I used a tag I called "Instagram spam" since most of these spams seem to mention Instagram for whatever stupid reason. You can use whatever you like.)

     

    IMPORTANT NOTE: You'll want this to process first, so be sure to move this trigger to the FIRST position in your triggers so that it's the first thing Zendesk processes on all new tickets.

     

    Move a trigger to the first position by clicking the three-dot icon on the right-hand side and selecting "Move to first position".

     

    Second Trigger

    Next, you're going to need to stop Zendesk from emailing a notice to the spammer's target. That's their whole motive, so we want to make sure these spam tickets don't get that benefit!

    Do so by filtering on the new tag we just applied to the bad ticket:

     

    This has the impact of stopping Zendesk from sending the "Notify requester" email for tickets that come via the spammy channel (Web Services).

    You should repeat adding this tag-based filtering for other triggers that send out emails, like ones notifying your agents.

     

    This filter combo seems to be working OK for me. 

     

    Unfortunately, Zendesk's customer support didn't provide this detailed of a solution, which is ironic, but whatever. 

    Hopefully, these notes help someone else out there!

     

    Still, I must re-iterate: 

    ZENDESK, YOU REALLY NEED TO FIX THIS ASAP. 

    This whole fiasco completely embarrassed me as I'd just vouched for how GOOD Zendesk was with my boss and then this happens. Ugh. Not cool. Not cool at all.

     

    Thanks,

    Scott

    10
  • Sheryl T

    Scott - marking a spam ticket as solved is going to mess up your statistics.  I would just use triggers to send all the spam messages to a spam view as I did and then mark them as spam periodically then delete completely.

    1
  • Scott

    Sheryl T: Yep. I plan to do just that after observing it for a bit, once I feel 100% confident that the filter is truly working as I expect it to. 

    0
  • Sheryl T

    Removing Notify Requester of comment update fields does nothing.  I even deactivated the whole trigger. The messages are sent with non-existent email addresses, so how could that ever help?  ZenDesk is making stuff up to placate people.  They need to fix the problem.  I cannot take anonymous messages off as we have new clients sign up for our program all day and night, and they may need support.

    1
  • Scott

    FYI - I don't see any option to mark a ticket as "spam" or suspend a ticket using a Zendesk trigger.

    The best option I see is to make it "Solved" or "Closed". Am I missing something in there?

     

    0
  • Sheryl T

    Scott - yes, my filters worked fine throughout the night.  I woke up to 64 tickets in my new SPAM view, and I was not notified of any of them by text or email as I made those exceptions in my email and text triggers also.  It is just a pain to mark only 30 messages at a time as spam and then 30 messages at a time as permanently deleted, but fortunately they are coming in rather slowly as opposed to a number of months ago when there were thousands at a time.

    1
  • Scott

    I see what you're doing now. You aren't automating the part where you mark them as spam, you're just filtering the view, selecting and then manually marking them "spam". Makes sense. Good idea.

     

    Zendesk still needs to fix this crap!

    1
  • Ronny Hofsøy

    Unfortunately, Scott’s very clever mitigation suggestion is not possible if you`re using Essential due to the fact that Triggers on that plan is not that advanced. 

    The ZD plarform has effectively been rendered useless for both our company, but even worse, our many client that use ZD on our recommodations. Spam is 100:1 as of now, and is causing agents to a level of frustration I have never experienced before.

    ZD, please implement a fix asap.

    I wonder if this can and/or will be fixed, or if we have to shut down ZD and terminating the contract entirely.

    1
  • Martin Cox

    Ronny --

    See my earlier post about not using a Triggers method for halting the flow of spam to your techs. If you support users over a limited number of domains you can whitelist those domains and set a requirement for users to register with your support portal. This has the net effect of putting all the spam tickets in the Suspend queue where you can review/delete while allowing your end-users to submit as they always have been.

    0
  • Sheryl T

    We support individual clients all over the world and cannot whitelist domains.  The emails are showing random IPs also, so blacklisting those does not help.  I just put back our chat widget and notify requester emails as stopping those is useless too.  Filtering the spam into a view and filtering them from going to email or text is what is helping us.

    0
  • Ryan W
    Zendesk team member

    Hey Scott,

    Thanks for the details there, and apologies if you are not getting the full information in your ticket. I will check in on how our advocacy team is handling them and go from there.

    This is an effective way to mitigate them from ending up in your queue.

    As for having an option to close the requests Endpoint -- We're looking into it. This has some pretty big implications, but please rest assured we're looking at all options right now.

    ----

    Sheryl T -- On the note about the "changing your Notify Requester of received request trigger doing nothing" --- The entire reason why this information is given is because the spammers are not targeting you, they're targeting the requesters they're using on each ticket. By having your placeholders within your triggers on ticket creation, you effectively create an open mail relay, which allows these spammers to use you as a conduit for spam.  By removing them, you remove their incentive to target you. While this doesn't stop your spam immediately, it absolutely will help you be less of a target for spam.

    ---

    Ronny Hofsøy -- We're definitely looking for a solution, and apologies for the frustration. I'm going to reach out to you to help you explore some options. Watch out for a ticket shortly.

    -1
  • Jonathan L

    Ryan, we aren’t the target a poorly designed Zendesk api that’s left open is the problem.

    The fix needs applying to the entire ZD platform not just the odd account. This is a ZD issue not ours! Sort it out before you loose customers! If this is not resolved within the next 24 hours we will be moving to Help Scout as they will migrate our data and the system is more secure.

    2
  • Martin Cox

    Looks like the Zendesk spam filter is now classifying these bogus tickets as a "Malicious pattern" and tossing them into the Suspend queue.

    3
  • Jonathan L

    That doesn’t solve the issue, it’s still sending them to the end user or the attackers intended destination as your company name ...

    2
  • Martin Cox

    Agreed in terms of the primary issue not being solved. Disagree in terms of them using our portal to send on behalf of our name. They're stuck in the Suspend queue.

    I have to imagine that an  API change can't be taken lightly as to the overall impact of already deployed applications. One of those things that's probably easier said than done.

    2
  • Jonathan L

    Martin sorry I misunderstood what you said, ok is this a confirmed fix across all PODS? I have shut down the support desk over the last 48hrs as it got so bad ... a target threatened me with reporting me to the ICO even though I didn’t send the message.

    I agree API changes should be planned but they have known about the issue for 12 months, no action has been taken apart from a half baked work around that doesn’t fix the problem.

    I might come across as a little stroppy but my business cannot function properly without a support desk solution that works properly. I’m not new to the platform I have been using it for 8 years solid with over 19000 tickets across 2 accounts. Spam is frustrating and distracting for engineers on a daily basis.

    Let’s face it.. life is hard enough without clearing spam all night when we should be down time.

    2
  • Martin Cox

    I get it, I do. My first Zendesk ticket dates back to March 29th, 2008 when they were still essentially a startup in Denmark and the grandfathered plan I'm in will have me probably forever. lol

    I prefer to see forums as a place to collaborate workarounds and push support for the bigger, long-term fixes.

    0
  • Jason Wallis

    Mine are now going into the "suspended tickets" status and folder.... isn't that problem solved?  By Zendesk - thanks Zendesk!

    Unless I'm mistaken I don't see how they're still sending the response emails - Jonathan L are you seeing that yours are still immediately responding before they get captured as suspended?  I had to unsuspend by recovering a ticket to check and it seemed as though it only sent the auto response once I recovered it. 

    I'm assuming this fix by Zendesk

    1- doesn't send the email that the spammers want in the first place

    2- doesn't mess up my statistics for solved cases etc

    3- didn't require me writing any triggers

    4- puts them all into a nice little folder where I can see them and laugh at their feeble efforts to hijack my email

    unless I'm totally wrong about the above . . . 

     

    THANKS ZENDESK!!!

    2
  • Yunus Unia

    So far what I have done is required the option for users to signup when submitting using the webform, as it is a rarely used option at my company.

    All submissions from there have gone to the Suspended tickets view, from which I can just simply delete the e-mails, but still triggers a message to sign up sent to the e-mail addresses provided in the webform.

    It doesn't screw up any statistics or inflate our ticket ID values, but email addresses are still going to get an email requesting them to signup to submit a ticket.

    Zendesk needs to fix this ASAP. For a company that charges a premium for its service, it certainly isn't delivering solutions that feel premium.

    edit: It seems that genuine users would still have to register to submit their requests, which I'm debating on whether or not that's a good idea.

    Now I'm even more frustrated.

    0
  • Sheryl T

    Mine seems to have stopped completely as of 2-3 hours ago.  Jason Wallis - yes your observations are true except that the spam needs to be stopped before it comes to you!  I would delete everything in your Suspended view also.  Hopefully the problem is now completely solved.

    0
  • Scott

    Sheryl T - Ours stopped yesterday as well. Starting at ~17:14 Eastern Time (US), those spam tickets are going right to the "Suspended Tickets" list with a "cause of suspension" being "Malicious pattern detected".

     

    So, the primary exploit still exists: There is open access to Zendesk's anonymous API somewhere. However, the impact is mitigated by these tickets getting filtered out by the system.

    In my opinion, Zendesk should still fix the problem with its API, but (in our case) the impact is mitigated now.

     

    1
  • Sheryl T

    I had 2 more in the middle of the night that did NOT go to Suspended but went to my custom SPAM view. I had nothing in Suspended overnight. I shut off my Notify Receiver of Received Request trigger on Saturday night.

    1
  • Patrick

    The attack seems to be over for us as well. It mostly stopped just before 1PM yesterday, with a single spam email coming in at 2:15AM this morning. Thankfully, my filter caught all of them after it was setup. I will probably adjust it to be more inline with what Scott suggested and leave it running.

    For now, that isn't an issue. In a few months when my company's new site launches we will be using the API to create tickets directly from our site.Hopefully Zendesk will have this exploit plugged by then.

    EDIT; Nevermind, 2 new spam tickets just got created.

    1
  • Ryan W
    Zendesk team member

    Hey All, 

    We hear you loud and clear and doing our best to mitigate the spam. I don't have anything new to share at this time, but rest assured that we're formulating a plan and will share with you once we have solidified a plan of action.

    In the meantime, to decentivize the spammers, the instructions in this article will play a very big role in avoiding more spam (though not immediate, as the spammers may not update their "target" list fast enough).

    Additionally, while we have been reaching out to domain registrars, and link hosts from our end, I would encourage you to report the spam links as well from your end (though ultimately optional), as more reports will give more context.



    1

Please sign in to leave a comment.

Powered by Zendesk