Welcome back to The Rabbit Hole Podcast. Today on the show, we dive into part five in the “Stop Doing Stuff” series. Our aim is to help programs to stop DDoS’ing themselves with the hope that if you stop doing the wrong thing, you’ll become a better developer and engineer in your organization. Inside today’s episode we will be talking about what DDoS is, why people are DDoS'ing themselves, and how to prevent yourself from being DDoSed in the first place. We have to do our best to prevent these issues so that we can still provide the quality content that the organization wishes to provide. So stay tuned as we unpack each of these questions, and find a way to solve the problem together. For all this and more, keep listening!
Key Points From This Episode:
Transcript for Episode 110. Stop DDoS'ing Yourself
[0:00:01.9] MN: Hello and welcome to The Rabbit Hole, the definitive developer’s podcast in fantabulous Chelsey, Manhattan. I’m your host, Michael Nunez. Our co-host today.
[0:00:09.8] DA: Dave Anderson.
[0:00:10.8] MN: Our producer.
[0:00:12.0] WJ: William Jeffries.
[0:00:12.7]MN: Today, we’re going to say again, “Stop DDoS'ing yourself.”
[0:00:18.3] WJ: Stop doing it.
[0:00:19.2] DA: It’s part five in the stop doing stuff series.
[0:00:21.6] MN: Yeah, we’ve got a series going on, which is like you start doing things, you stop doing things and you’ll be a good developer engineer in your organization. You need to stop DDoS'ing yourself and today we’ll be talking about what is DDoS, why are people DDoS'ing themselves, and how to prevent you from being DDoSed in the first place.
[0:00:42.7] WJ: DDoS is like a better version of the DOS operating system, right?
[0:00:47.5] MN: No it’s not.
[0:00:48.8] DA: I’m not sure about that.
[0:00:50.5] MN: DOS was great, it’s the reason why I got into computers.
[0:00:54.5] DA: MS-DOS, CD.
[0:00:56.4] MN: CD../.. you got to do all sorts of things to navigate. Let’s jump right to it. What is DDoS? I have a definition right here, just to be clear. Does anyone want to share their definition of DDoS?
[0:01:09.4] DA: What it means to me?
[0:01:10.6] MN: What does DDoS mean to you, Dave?
[0:01:13.3] DA: Like the Thanksgiving holiday? What does Thanksgiving mean to me?
[0:01:17.1] WJ: Let’s all go around in a circle and share. I think it stands for distributed denial of service, yes?
[0:01:23.7] MN: Yes, that is correct.
[0:01:24.8] WJ: So I think a regular denial of service attack is where you flood a server with so much traffic that it falls over and it can’t serve real customers or people who are actually supposed to be interacting with that server.
[0:01:36.7] DA: Right, that’s easy to do. Like you just don’t listen to that guy who is being really annoying.
[0:01:43.9] WJ: Ban him. That’s where the extra D comes in. You throw in that extra D when people are banning you.
[0:01:52.8] MN: You get distributed denial of service where you have more than one person. Is that the criteria where it’s more than one?
[0:02:00.6] WJ: I think it has to be a lot. I think if like two people did it, people would be like, “Ah, this is regular, you just get the one D.”
[0:02:07.3] MN: You just regular DoS. You ain’t DDoS.
[0:02:08.4] WJ: This is not a double D.
[0:02:09.7] MN: You're just DOS. Yeah, pretty much what you mentioned. I think you definitely got the definition down to a T, William. Great job. Awesome. This guy DDoS’s, I don’t know if you know.
[0:02:22.0] WJ: I don’t, I swear, it’s very illegal.
[0:02:25.4] DA: I see you the other day DDoS'ing but that was the different circumstance. Because like I think you were doing it in a controlled fashion for –
[0:02:33.2] WJ: I was DDoS'ing myself, yeah. It was on purpose.
[0:02:37.0] DA: Yeah. I mean, there’s sometimes that you want to put a lot of strain on your system and understand what the limits are through a load test, which there are services that will help you generate a lot of load on your server, flood.io or BlazeMeter where they have like applications that libraries that you can use locally like a locust to spawn up multiple processes and spam your servers.
[0:03:07.1] MN: You mentioned flood.io and BlazeMeter are different services that would help you DDoS but as long as you’re not doing it to another individual, I think it’s okay. Is that correct? I don’t want to give people commands to go and do some illegal shit after they listen to The Rabbit Hole.
[0:03:24.5] WJ: Yeah, you very much should only do this to your own websites. If you do it to somebody else’s website, you’re very mean.
[0:03:31.2] DA: I think you have to have like a token setup so that they can verify that it is something that you own.
[0:03:36.8] WJ: Yeah, for like load.io or flood.io. Any of the paid services, they do check, they make sure that — you have to confirm in some way that you are the owner of the website.
[0:03:48.2] DA: Put a C name or something like that on your DNS? Yeah, but that’s like a great way to figure out what the extent is, as long as you’re not doing it at four AM when [inaudible] duties are active.
[0:03:58.7] MN: Man. Yeah, don’t do it at four AM.
[0:04:02.1] DA: Then you effectively are DDoS'ing.
[0:04:04.9] WJ: You’re DDoS'ing your fellow engineers, that’s very mean.
[0:04:08.9] MN: That’s horrible.
[0:04:09.6] DA: No fingers pointed.
[0:04:13.2] WJ: Yeah, if you do it with Bees with Machine Guns, you can spin up easy two instances on AWS and point them at anything and then normal check, that’s literally the name of the library, Bees with Machine Guns.
[0:04:27.8] MN: Bees with Machine Guns? That sounds terrifying! Oh my god. I’m already scared of bees; this is like imagine the damage.
[0:04:37.4] WJ: Don’t arm them.
[0:04:39.2] MN: With machine guns.
[0:04:41.8] WJ: They would be very tiny machine guns.
[0:04:43.7] MN: No, I mean, A, machine guns are machine guns.
[0:04:48.4] WJ: I don’t know if they have the opposable thumbs for that.
[0:04:52.5] MN: We don’t know that, I don’t want to find out.
[0:04:55.7] DA: It’s like sharks with lasers attached.
[0:04:58.9] MN: Sharks with lasers. We could get down to some of the examples, famous examples of DDoS attacks throughout history.
[0:05:07.3] DA: It’s a story in history. It really is. I mean, a lot of us use GitHub and I think that every time that this service is even momentarily impacted, I am acutely aware of it. I’m sure you guys are aware of it too.
[0:05:23.3] MN: Oh yeah, you try to get pushed and it’s just like longer than usual and you’re like, “Wait, what’s going on?” Go to status.github.com and then everything is red. That was terrifying, that happened early in the year I think in 2018.
[0:05:38.6] DA: Yeah, like around February, end of February this year, it was the largest DDoS attack in history.
[0:05:46.4] MN: If you're listening, why would you do it to GitHub? What did GitHub do to you?
[0:05:49.3] WJ: It was SourceForge.
[0:05:51.8] MN: SourceForge.
[0:05:54.3] WJ: This is a revenge.
[0:05:57.2] MN: Oh man, that was pretty catastrophic in ways that slowed nearly everyone’s productivity down. If you were programming on that day.
[0:06:03.9] WJ: You were not programming on it, you had to stop.
[0:06:05.6] MN: Yeah, you had to stop programming on that day when you were trying to program on that day.
[0:06:10.3] WJ: It was a very good excuse to screw around. Like, “I literally cannot push any code right now.”
[0:06:18.6] DA: Yeah, GitHub did a pretty good job of like doing an incidence response and like being public about the analysis of the issue and what the steps they were taking now and in the future to respond to it. I think in the end, it was like pretty manual intervention that they had to do but they outlined some steps that they’d like to do in the future to improve.
I think the pattern for that particular attack was a memcached amplification attack.
[0:06:46.1] MN: Woah, what?
[0:06:49.4] DA: That’s like –
[0:06:50.1] WJ: That sounds very scary.
[0:06:51.3] MN: Yeah.
[0:06:51.5] DA: Yeah, that was a vocabulary word for me, I had to look that up.
[0:06:57.5] WJ: Is this North Koreas new secret weapon?
[0:07:00.9] DA: Basically there was a vulnerability that was posted for memcached. I think in the week prior by SourceForge to this attack. Basically you could send spoofed requests to a memcache server, any memcache server, with an IP that you wanted to attack and because UDP isn’t validated and it’s super quick, it would just be like, memcache is just a really responsive service and can cache very large objects. It would just be like, “Oh, sure, I got you, shoot her right across.”
[0:07:34.6] MN: No problem.
[0:07:36.0] DA: So it’s like really one of the notes they had in this document we can link to it is just that the size of the request versus the size of the response made it really profitable to just ban these servers that weren’t even GitHub’s to cause them trouble.
[0:07:53.8] WJ: So they were using potentially your memcached D server to spam GitHub and you didn’t even know?
[0:08:01.3] DA: Yeah, exactly. This was a vulnerability that had to be addressed in a lot of thought writers. Folks like DigitalOcean and AWS had to take measures to disable UDP and not expose memcached to like the public.
[0:08:22.2] MN: SourceForge, revenge, it must be it.
[0:08:25.0] WJ: I blame them. Let’s start that rumor.
[0:08:30.0] MN: Started here. You’ve heard it here first, folks. The rumors are real.
[0:08:35.4] DA: You guys remember the Dyn outage in 2016?
[0:08:42.1] MN: No, what is that?
[0:08:43.0] DA: That was like maybe like a month after I started at Stride, I was at my first client and again, GitHub was down.
[0:08:51.5] MN: Well I would remember that but it happened rarely. Oh yeah, that happened. But what happened with Dyn exactly?
[0:08:57.7] DA: Dyn, that was a botnet of Internet of things to vices.
[0:09:04.3] WJ: That’s so cool.
[0:09:07.9] MN: Hold on. So you said, Internet of things and botnet. Let’s talk about those real quick.
[0:09:12.0] DA: So many exciting words in these.
[0:09:13.3] MN: Struck those. So what’s a botnet exactly?
[0:09:16.9] WJ: A botnet is a network of devices that you have captured in some way. If you're a hacker or a script kitty and you are able to penetrate X number of people’s computers, you’re hacking grandma, you’re hacking whoever and you install some kind of software on there that allows you to take over that machine and use it to make requests, then you can add them to a list of all of the machines that you’ve hacked and then you can send commands simultaneously to all of them to go and do something in unison. That’s called a botnet. So you can direct thousands of machines that you’ve hacked all over the world to go and send requests to just one server in a DDoS attack.
[0:10:02.2] MN: Right, and an IOT device is like a device that’s connected to the Internet, right? Internet of things.
[0:10:07.2] DA: Yeah, like my lightbulbs.
[0:10:10.6] MN: And the fridge. Whoever’s got a fridge out there connected to the Internet.
[0:10:16.4] DA: You know. I don’t have an Internet connected tooth brush but I have seen them, they do exist.
[0:10:23.1] WJ: Just stop owning Internet connected devices.
[0:10:26.2] MN: Right.
[0:10:27.4] WJ: They’re just going to be used to DDoS you.
[0:10:29.9] DA: But I can turn off my lightbulbs right now.
[0:10:31.9] MN: Yeah, with a simple command.
[0:10:34.6] DA: I am part of the problem now.
[0:10:39.6] MN: I’m glad we got a lot of those words out the way because I was kind of confused; what was the amplification of botnets and IOT’s and the likes?
[0:10:49.9] DA: So I heard something about another big attack which was a pulse wave attack.
[0:10:53.6] WJ: Yeah, the pulse wave attack, that’s a really cool one. Normally, if you do a botnet attack, it takes a while to spin up all of the bots that have acquired, all those zombie machines that you control and that is slow ramp up time gives networks a chance to scale, especially with features like auto scaling on AWS or other providers and so with pulse wave attacks, what they’ll do is they’ll pick several targets and then they will alternate. So when it’s your turn to get DDoS'ed to the bot net is fully ramped up.
So they’re not getting that laser beam as you’re charging, you are getting the laser beam when it is full forced.
[0:11:36.5] MN: 100%.
[0:11:37.9] WJ: And then they’ll rotate off of you and so you’re servers will calm down and be like, “Okay you were good. All right, let’s take stock what happened?” and then bam, it hits you again.
[0:11:51.0] DA: Like death ray.
[0:11:51.5] MN: When you least expect it, geez.
[0:11:53.2] WJ: Yeah, you know, I was looking at traffic using Charles Proxy recently and there are a ton of requests that are going out for my machine. I think this is actually pretty common and probably has to do with the fact that I have approximately a thousand tabs opened at once but when you look at Charles Proxy even when you are doing nothing, there’s a suspicious amount of traffic happening.
[0:12:16.4] MN: Yeah.
[0:12:17.2] DA: Yeah, I closed all my Chrome tabs just now and my battery shot up.
[0:12:24.9] WJ: We are the botnet.
[0:12:26.3] MN: Exactly. I often use the Chrome, where you right click on the tab and you close all tabs to the right, that’s always been helpful. So helpful. How does an organization or a company accidentally DDoS themselves so we could tell them to stop?
[0:12:42.7] WJ: Well I think if you push code that runs in the browser and that code potentially makes an Ajax request, which is kind of a thing that we all do all the time.
[0:12:53.1] MN: All the time.
[0:12:53.7] WJ: You are at risk of accidentally DDoS'ing yourself because now you have code running on browsers all over the world for all of your users. So if that’s pointed it to service that you maintain and you’re not expecting serious load from it or if you don’t have a good exponential backup stretch or some kind of a backup strategy if that service starts to get under heavy load and backup, you can absolutely bring your own services down.
[0:13:17.2] MN: So be careful what you actually deploy because it could very well be the end of your demise.
[0:13:22.2] WJ: Yeah there was a really great episode of the Ruby Rogues where they interviewed the people from trackJS and they talked about a really great example of this because they have code that tracks error messages in the browser for people with front end apps and so if you start catching a lot of errors and not batching them properly, you can do really, really serious traffic.
[0:13:45.8] MN: Oh wow.
[0:13:47.0] DA: Yeah and that’s important service that a lot of people are relying on to give and get information about what’s going on.
[0:13:54.5] WJ: Yeah it was a great episode.
[0:13:55.5] DA: Cool, yeah. I guess another way people can DDoS themselves in a nice way sometimes is if you have a small website and it’s not properly set up and some large publication finds that about you, like Reddit. The Reddit hug of death is real.
[0:14:15.1] MN: The Reddit hug of death. Such a nice way to put the fact that you and all of Reddit brought down a website.
[0:14:23.0] DA: Yeah, with all like legitimate traffic that is distributed, service is being denied but it is actually like, it is traffic that you could have served happily if you were just prepared for the moment.
[0:14:35.2] MN: Right, I think before it was called the Reddit hug of death, it was called slash-dotting or slash-dot.
[0:14:41.6] WJ: Yeah the slash-dot effect.
[0:14:43.5] MN: Yeah, I learned about that when I was Googling the Reddit hug of death. Why do people do this? Why is this a thing? Why would you want to do this?
[0:14:52.8] DA: That sounds like an evil genius kind of thing.
[0:14:56.6] MN: Yeah it’s like –
[0:14:57.0] DA: I think the Bees with Machine Guns sound pretty appropriate.
[0:15:00.6] MN: Yeah that just sounds terrifying. I mean, I read online part of the reason, one of the reasons why you would want to do this is because I mean we’ve all probably been on at some point in time on that side of a DDoS where the client that we are working on is getting DDoS'ed or what have you and the entire engineering team could be so occupied on fixing it that who knows what kind of malware is being injected into JavaScript or something that can be overlooked because you are trying to mitigate this DDoS attack.
[0:15:35.5] WJ: Yeah or you could be trying to make some money, get that paper. Get that Bitcoin, rather.
[0:15:40.2] MN: Oh yeah, get that Bitcoin rather. We can’t talk about that right now. The prices are a little low but I know that it was often times people would want to DDoS the crypto currency exchanges to get a cheaper price by just flooding the market with requests that were not valid or not real so like the bids would then go lower and lower. I think it happened in summer of 2016 where Ethereum went from $150 to $7.
[0:16:10.2] WJ: And then right back up?
[0:16:11.7] MN: And then right back up, yep.
[0:16:13.2] WJ: So you could have made a lot of money if you were at the right place at the right time there.
[0:16:16.6] MN: Yeah, if you happen to make those bids of like –
[0:16:19.0] WJ: If you happened to know that that was about to happen.
[0:16:21.4] MN: Yeah, I mean, so make your bids, make it really low and hope one day that a crash comes so that you hopefully those bids are acknowledged and you make money. I don’t know? I mean, I have no idea how that worked but it has happened before in the past where DDoS'ing a crypto exchange caused fluctuations in the market, which is nuts.
[0:16:40.2] DA: I was reading that some people in less regulated places might grow a botnet and not use it. They would just have it and maintain it and like –
[0:16:52.4] WJ: Feed it, water it.
[0:16:55.2] DA: And then put it up for sale to the highest bidder like whoever wants to go for a ride on the botnet can point it at whoever they want to.
[0:17:05.0] WJ: Yeah, I think that is probably really profitable. You could probably make a pretty penny doing that.
[0:17:10.7] MN: Destroy websites? That’s so mean.
[0:17:13.0] DA: Yeah, it’s probably easier to make money legitimately. But you know that’s life. Yeah so what are some ways you can mitigate DDoS? It seems like it is such a bigger problem than any of us?
[0:17:29.1] WJ: You can use a CDN.
[0:17:30.9] DA: That is a way to fight fire with fire I guess. Like if the attack is distributed having a distributed solution, it would surely help.
[0:17:39.3] WJ: You could drop packets from anybody who seems suspicious?
[0:17:43.2] DA: Yeah that’s true. I guess like, if you are using a distributed solution like Akamai or a cloud flare, they are in a better position to identify a pattern.
[0:17:53.7] WJ: Is it Akamai or Akamai? I don’t know.
[0:17:55.6] MN: I think it might be Akamai, Akamai, you know it?
[0:18:03.2] DA: Yeah, well anyway so those providers are in a better position to do that kind of analysis in a really broad way and figure out what the bad patterns are and who the bad actors might be than just your poor old engineering team. But there are on Prem solutions you can get.
[0:18:20.7] WJ: Like what?
[0:18:21.7] MN: Hardware. If you have hardware, but who has hardware now?
[0:18:26.1] MN: You can use hardware to prevent DDoS?
[0:18:28.8] DA: Yeah, you can filter requests as they’re coming in.
[0:18:31.6] MN: Oh interesting.
[0:18:32.4] DA: Through like a hardware device.
[0:18:34.1] WJ: But how do you know which ones are real customers?
[0:18:36.6] MN: That’s the crazy part.
[0:18:38.4] DA: Yeah that is the secret sauce I guess.
[0:18:40.5] MN: I think you have to identify the IP where it comes from, have they visited other websites or if it is just like a brand new botnet that’s doing all sorts of things. I have no idea to be honest.
[0:18:50.9] WJ: Does it look like a real browser, do they have a user agent?
[0:18:53.9] MN: Yeah that might be — looking at some of that metadata might be helpful in identifying where this person, him or her, is currently doing this thing. Just a fun fact, it is Akamai and if you Google Akami and you’re in fantabulous Chelsea, Manhattan it leads you to a Japanese Restaurant.
[0:19:13.0] DA: Oh okay that’s what I was thinking. I’m just really hungry for sushi right now. Yeah so like just tickle my brain, I did remember like on Prem assist solution that I came across. It’s called a new star.
[0:19:25.8] MN: Ooh what’s that?
[0:19:26.6] DA: And they have cloud based and on prem assist. So they have a hardware thing and a software solution as well. So yeah, there are things out there but easier to go with the service than try to solve the problem yourself when it’s distributed.
[0:19:44.8] MN: We understand that there are bad actors out there in the world who want to bring down these websites. If you are listening, stop. We have to do our best to prevent these issues so that we can still provide the quality content that the organization wishes to provide.
[END OF EPISODE]
[0:20:02.8] MN: Follow us now on Twitter @radiofreerabbit so we can keep the conversation going. Like what you hear? Give us a five-star review and help developers just like you find their way into The Rabbit Hole and never miss an episode, subscribe now however you listen to your favorite podcast. On behalf of our producer extraordinaire, William Jeffries, and my amazing co-host, Dave Anderson and me, your host, Michael Nunez, thanks for listening to The Rabbit Hole.
[END]
Links and Resources:
