Unless you’re William and write perfect code every time, as a developer, you’ll likely have encountered a bug or two in your time. One way to get rid of these pests, is through a bug bash, which we talk about in today’s show. While we have come a long way from the threat of physical bugs destroying our computers, errors are part of the development process. We kick the episode off by exploring what a bug bash is. In its basic form, it is presenting a group of people with an application or website before a big release to get them to interact with the product and find any edge cases developers may have missed. Although it’s a useful way to fix bugs, it is not a substitute for proper testing and QA. Then, we get into some of the logistics of a bug bash, like who to invite, where to host it, and ways to keep it fun. It is important to view the bug bash as a development milestone, and teams should give themselves a pat on the back for hard work. It can be easy to spiral into crisis mode, thinking there are too many bugs to fix, which is why it’s important to have a proper system at the bash. We round the show off by talking about the documentation process and the different roles people should play in the event. So, get out there, and go smash those bugs!
Key Points From This Episode:
Transcript for Episode 159. Bug Bash
[INTRODUCTION]
[0:00:01.9] MN: Hello and welcome to The Rabbit Hole, the definitive developer’s podcast. Live from the boogie down Bronx. I’m your host, Michael Nunez. Our co-host today.
[0:00:09.3] DA: Dave Anderson in Jersey City.
[0:00:11.0] MN: And our producer.
[0:00:12.2] WJ: William Jeffries in Seoul, South Korea.
[0:00:15.0] MN: Today, we’ll be talking about bug bash. That’s probably the best party you’ll ever have as a software engineer at work. And we’re going to talk about what it is, when to do it and what the process looks like.
[0:00:26.4] DA: Got to smash those bugs. The only good bug is a dead bug.
[0:00:29.1] MN: That is correct. Before we begin, we got William on the line, again from Seoul. How’s it going? How are you?
[0:00:34.9] WJ: It’s going all right. Just like recovering from some food poisoning. I had some blowfish.
[0:00:42.3] DA: That’s really tempting fate.
[0:00:45.2] WJ: I did not know that I was going to be eating blowfish. The restaurant, apparently only serves blowfish.
[0:00:52.4] MN: You got surprise blowfish.
[0:00:55.2] WJ: I got surprised blowfish, yes
[0:00:58.2] MN: Man, you’re fighting off your own personal bug that you’re dealing with. Hope that you recover as soon as possible.
[0:01:05.1] DA: I hope that you bash that bug, you know? Stomach bug.
[0:01:08.5] MN: What was it called in Korean? Boku?
[0:01:10.1] WJ: Bogeo.
[0:01:13.0] DA: Don’t go chasing bogeo. Stick to the bulgogi that you’re used to.
[0:01:18.2] MN: You got any tips, feedback on how to come back and feel better after some – after feeling begeo, feel free to tweet us.
[0:01:27.3] WJ: Yeah, that would be great.
[0:01:30.3] MN: People would even love to know.
[0:01:32.5] WJ: Yeah, much appreciated.
[0:01:34.9] MN: But we’re talking about bug bashing today. First and foremost, we can start by defining what is a bug bash?
[0:01:41.6] DA: What is a bug? I don’t even know what a bug is. I’ve never had one of those.
[0:01:46.5] MN: I think when we were –
[0:01:46.8] WJ: I write perfect code the first time. Every time.
[0:01:51.9] MN: Well, the term goes back many moons ago, when we were writing this outline, I was really curious as to why it was called the bug in the first place. And apparently, when computers were the size of a room, William at the time who writes perfect code may have written perfect code, it was still possible that a bug could go into the machine and mess up the electromechanically parts of the computer.
If you look up software bug, you’re probably see a picture of a moth that was stuck in a computer which caused it to do weird things. That is probably why it’s called the bug till this day.
[0:02:29.4] DA: Kind of happy we don’t have to deal with that anymore.
[0:02:31.9] MN: Right. If you write perfect code, then you may not have to worry about bugs anymore but you were writing perfect code, what is it like in 1970, you’re probably going to have to fight bugs with a bug swatter or something like that. I don’t know what that’s called.
[0:02:46.7] WJ: What was the name of the woman who invented the term debugging? Guys remember, it’s just like an admiral –
[0:02:51.7] MN: Grace Hopper?
[0:02:52.7] WJ: Grace Hopper.
[0:02:52.9] MN: Yes, Grace Hopper.
[0:02:54.4] DA: Yeah. They are, properly attributed to Grace Hopper in the 1940s from Wikipedia.
[0:03:00.7] WJ: She’s doing like COBOL back in the day.
[0:03:03.2] MN: Writing them COBOL.
[0:03:05.0] WJ: Actually, that was probably pre-COBOL. She did COBOL later. Didn’t she invent COBOL or something?
[0:03:09.8] MN: It’s possible. Very much possible, we can definitely do the Google. Led to the COBOL language, that is correct. We’re fast typers over here at the Rabbit Hole. Especially doing a live since we’re home in front of our computers doing this. We’re all hoping you’re safe.
[0:03:27.4] DA: Way more distractions. So, Googleable.
[0:03:31.2] MN: What is a bug bash? Now we know that we have bugs that we need to bash, not physically anymore with the moth, kinks in our code that we need to fix.
[0:03:40.6] DA: Yeah, I guess the idea of a bug bash is that before you have a big release, you want to get the software in front of as many eyes as possible to try and see if there are any edge cases that you might have missed, any problems that might still exist. You would just try to have people play with the code, try to break it and record the findings.
[0:04:02.7] WJ: Play with the code or play with the app?
[0:04:03.5] DA: Play with the app.
[0:04:05.5] MN: With the application up, you can have a group of individuals going to the website, to the application and playing with it and making sure that everything’s fine, especially when you have a big release. And I think the way in which you run the bug bash can determine how often people will come to help you in finding these bugs so that you’re less likely to introduce these bugs to your user when it comes time to release it to the world.
I think we mentioned before when would be a good time to run the bug bash. If you have a really big feature and you want to get it to as many eyes as possible before releasing it, it tends to be a good time when you want to run a bug bash. And everyone gets a chance to play with the application and make sure that if there are any particular edge cases that weren’t caught by the engineering team, then it gets documented and then that can also be worked on before you release it to your users.
[0:04:58.0] WJ: Yeah, just to note that this is not a substitute for proper testing and QA and all that good stuff like you still need a good software development process with unit tests and integration tests. And also, if you are releasing a huge feature that has you so scared that you take everybody in the company and have them try and break it, maybe you should not roll this feature act to all of your customers all at once.
[0:05:19.3] MN: Right.
[0:05:20.7] WJ: Future flag that. Do some beta testing.
[0:05:23.5] DA: Yeah, there’s definitely something to be said for having like trying as much as possible to do an incremental rollout. Like we were talking in episode 154 about releasing as a big bang release and some alternatives to that. So, bug bash can be helpful if you have to do it but definitely consider some alternative patterns for releasing a software.
[0:05:48.0] WJ: Yeah, you could do both. You could future flag it and roll it out to 1% of your users. And also, do a bug bash.
[0:05:53.9] MN: You don’t have to do it, it’s not exclusive to having a bug bash, you can do it in many different ways as well.
[0:05:58.9] DA: Yeah, totally. Any time that we wouldn’t want to have a bug bash?
[0:06:01.9] MN: I think with what William said before, pretty much, the bug bash is there to find bugs that may have gone through the system. But you need to make sure that once those bugs are caught that they’re part of the automated testing suite, that it’s test driven to make sure that it doesn’t happen again. You don’t want to heavily rely on manual QA process for your application, when it could be done automatically.
This could be very useful given if you had the interval of once a quarter or periodically when you release a new feature, having a bug bash every two weeks could be very costly at the end when you could have used that time to be building up the new feature.
[0:06:44.2] WJ: Also. Probably the day before a major launch or press release.
[0:06:50.1] MN: Yeah.
[0:06:50.7] DA: Just sounding the alarm. “My god. Get in here.” You don’t want to bug bash then.
[0:06:56.3] MN: Try to do it as early as possible before the release where you can catch those bugs safely and then have it part of your QA process.
Yeah, let’s talk about, what the actual process? Suppose I now about bug bashing and I want to try one of those for a feature that I’m going to – that we’re going to release. How do you start this bug bashing process?
[0:07:15.0] DA: I guess the first up is just picking some kind of a date and time that works for a good amount of people. Planning it, maybe like a week in advance. Letting everybody know, like picking out who is going to be part of your audience or part of the participants for the bug bash. And setting aside that time for the core group of people to be there to support that.
[0:07:39.2] WJ: Yeah, I think if you have some full-time QA people or people who specialize in QA, involving them in a leadership role and organizing can be really helpful because there’s actually a lot of nuance and subtly that goes into testing things properly. Knowing how to write and test the scenario.
Also, familiarity with the bug tracking system. And also, familiarity with the expected behavior off the application for – if you’re making a call to everybody in the whole company, like Bill from accounting may not actually know how to use the app and what it is supposed to do it. Something may look like bugs, but to a customer it with domain knowledge actually seem quite normal.
[0:08:19.4] DA: Right, yeah. Definitely need to make sure that if you are like including people who aren’t domain experts like they’re new to it, your people who support them with the presence of the subject matter expert who can like guide them through that. Or like some good documentation of an overview of what kind of features might be available. What the general expected behavior might be.
[0:08:42.3] WJ: If you have full-time QA person, you know, they can use that opportunity to teach people about the stuff and that can be helpful for engineers who maybe haven’t had as much experience with different kinds of testing like input testing and all those things that QA engineers do or think about when they’re trying to break an application.
[0:09:03.2] DA: Totally, yeah. It’s definitely a good time – Since you’re bringing people together, any other time that you bring people together like to do contact sharing. People will learn about this new feature about how it’s expected to behave. And also, learn like you’re saying with the general process of what a good test looks like. How to QA and think about breaking something.
[0:09:27.0] MN: Just want to mention that it’s also the idea that this could be for both tech and non-tech people. Just know that when you send the invites that you have, the right people that are there but that doesn’t mean that they both have to be in the tech space or in the team to be a part of this. The invites are out.
And the next thing you would want is like a place to do it, I guess before BC, before coronavirus, we would have it in a meeting room, where we would be able to have everyone show up to the same space comfortably would be the idea to have a bug bash.
[0:10:01.7] WJ: If you’ve got you know, a number of people that can fit into a room together. If it’s a super big company, maybe everybody is at their desks and you have a video call going.
[0:10:11.9] DA: Yeah, even now, video call would definitely be good. Like just having like one definitive place, where if you are bashing bugs then this is the place where you can communicate a Slack a channel or a video call.
[0:10:27.5] MN: Yeah, I think I’ve only been invited to a bug bash two or three times. I think the way that we have ran it was everyone had their machines like in front of them as they were using the application and when a bug was found and it was – you were able to repeat the bug on the laptop, is when you would do it in the room so that everyone kind of sees the process and then have a conversation as to why that bug happened in the first place.
I thought that that was like really cool because then, everyone can asynchronously use the app and how they see fit and then when the bug happens, that’s when they were able to then show it to everyone to see what the deal is. And then you have the technical conversation as to why it’s happening. And then you have the QA person in there that could write the ticket or the project manager to determine like whether it’s high priority to fix right now or be able to do that later like that kind of stuff happened after the big TV reveal.
[0:11:21.0] DA: Yeah. That’s something to celebrate too, right? Since this is a big group activity, it’s a good opportunity to like, kind of focus on fun and team culture like team-bonding. If you have like a big TV where you’re sharing this bug, then that’s a good way to celebrate like, “Hey, we’re doing what we set out to do. We’re finding issues that we can address and make our software better.”
[0:11:43.0] MN: Yeah, definitely. So, if you have everyone in the room, finding bugs, you got to make it fun. I’ve been in bug bashes that had pizza, everyone loves pizza, that’s a thing that people eat. It’s good and nutritious.
[0:11:55.6] DA: I think it’s the bottom of the food pyramid. It’s actually a food pizza I guess, right?
[0:12:00.1] MN: Exactly.
[0:12:00.4] DA: They came in and they revised it. It is actually a pizza.
[0:12:02.9] MN: Yeah, no the food pyramid is in a shape of a slice so it has it. But there is usually food and beverages. Prizes also like if someone is able to find the most bugs, you get an Amazon gift card. Thanks for finding them. We’ll get to work and fix those things before we release it to the users. If you think about it like the 50 dollars and a gift card upfront to someone is still less expensive than your users, right? You could have hundreds of thousands of users seeing this very same bug too. So, you mitigate it.
[0:12:34.7] DA: Right it is a really small cost. How much is a developer hour of work and all of that like all dollar hours of time that people could have wasted with faulty software? And that is something you can do even now if you are not able to share food or a tasty beverage, you can still have a gift card for a food delivery or online shopping what have you.
[0:12:58.1] MN: I mean when you are at home, you control your food and beverages yourself. You got no one stopping you. So, enjoy the bug bash as much as you can when you are at home. The prize can still be done electronically. And as we mentioned before, suppose you do find a bug that is when collecting that information about the error that everyone was able to witness and replicate, if it valid. And we as a team have to figure out what are the ways to fix this bug, what are the things we would have to do so that we can fix it before we release it to our users.
[0:13:29.6] DA: Right, having some kind of standard form like a Google Form or a spreadsheet that they can enter all the relevant information about what environment am I seeing this bug in, what are the steps I took to reproduce it, what did I expect to see and what did I actually see just those basic things that you need in order to reproduce and understand what the problem actually is.
So, imagine if you have a big TV or a screen share that is a good time to walk through and capture all of those things. So that when you are triaging the blog and prioritizing it then you can have developers work on it later.
[0:14:08.1] WJ: So, at bug bashes you guys have been to, how much of the emphasis was on finding the bugs and how much was the emphasis on fixing them?
[0:14:16.3] MN: I think there was a lot more emphasis on finding them. The way that I ran, we try to find as many bugs as possible for an hour, got them on a list, like Dave mentioned it was like a Google Sheets. And then the project manager was in the room and the product manager was in the room for us to determine the user story that needed to be written and what priority needed to take within those bugs. And then starting the sprint. The next sprint, we pulled those tickets in, in that order.
[0:14:46.0] DA: Yeah, fixing in my experience we just kind of – it’s been a long time since I have done one. But the last time I did a bug bash we were working on a store front for ordering pizza. And so, part of our bug bash was ordering multiple pizzas. And so, we started with pizza and then we ended up doing more pizza but you know we were just going through the process over and over again and trying to figure out what the edge cases were logging the bugs.
[0:15:16.3] MN: Did you end up delivering a ton of pizzas to your place and then that was the food?
[0:15:21.0] DA: Yeah that was like a regular thing where somebody was like, “Oh I had to test a feature so there is a pizza.” It is 10 AM.
[0:15:28.2] MN: Nice.
[0:15:29.0] DA: Somebody please eat my bug pizza.
[0:15:30.9] MN: Oh man, I mean that doesn’t sound appetizing. But yes.
[0:15:36.5] DA: Not bug pizza. What about you, William?
[0:15:39.2] WJ: I think similarly the emphasis primarily on finding bugs. And then I think well-run bug bashes just like some kind of an expectation already set before and about how much time you are going to spend them on fixing bugs. Like what bugs actually have to be fixed and which ones can live on at the backlog for a while. If you don’t have a plan then sometimes people can start to panic and they’re like, “Oh my god there are all these bugs. We can’t launch.”
[0:16:08.2] MN: Oh no.
[0:16:09.2] WJ: Well, I mean we knew that there were a lot of bugs. There have always been bugs. There will always be bugs.
[0:16:15.8] DA: In the beginning there was bugs.
[0:16:18.3] MN: There was bugs and they were crushed and then they appeared again.
[0:16:22.0] WJ: Right, you got to have some sort of system. You should have categories for different levels, the severity of the bug and some sense of what is the release block or what is critical and what is major and what is minor.
[0:16:37.2] DA: Right. Yeah understanding what the impact and maybe the potential complexity of the fix is going to be. I like that point that you said about having someone on hand who is going to make that call. That is something that we didn’t go into too much depth. But you know it is good to have someone who is owning this like some kind of different roles around how you’re going to facilitate the bug bash. So, there is a pretty good article, a blog post from Usersnap that we can link in the shownotes.
There is actually a bunch of good blog post out there we can link a couple of them. But they suggest some different roles that I don’t think I have had such formal roles on the past. But like I think it is a good general structure where one person is the bug master. They are facilitating and keeping people on track. There is the testers. There is the people invited and someone might be responsible for managing the backlog of bugs and asking for clarification if there is missing information. People setting up environments and making sure that the people who are doing the testing are not blocked by anything.
Those all seem like pretty reasonable roles.
[0:17:47.1] WJ: Yeah. I have said it before, I will say it again. I know this is a little bit of a tangent but I think I am a big fan of bug budgets. It is normal for there to be bugs in any system, and the amount of and severity of those bugs is something that you should be managing. A million minor bugs will still create a bad user experience. So, you know it is not like just because a bug is minor that you don’t have to address it.
But at the same time, it is like even a major bug that there is a work around for, it may need to live in production for some period of time if there are other more pressing things that the company needs to prioritize. Having some sort of logic to that so that it isn’t like some product manager making a call and then getting in trouble later, if somebody is disagreeing with them, I think is helpful.
[0:18:40.1] DA: Like having some kind of buy-in from management and the stakeholders about like what it means to be releasing quality software.
I think another thing that we touched on before is like having some documentation. So, it is accessible for people to get started quickly with minimal effort from the people running the bug bash.
Do you have some thoughts on the things that might be good to include in that kind of getting started document?
[0:19:09.2] MN: I guess like what to bring. What you are expected to do. Maybe like case by case or feature by feature, you would have the feature overview of the particular bug bash that will probably be in the invitation. It will be a little different than any other invitation, but just the general rules. I think you mentioned before like if you have roles that you want to assign to people just like a description of what each roles tend to do.
So that the person who is reading the documentation gets an understanding of what they are expected if they are x role. When everyone gets an understanding of what they are expected to do, when we start the bug bash like everyone is ready to go essentially.
[0:19:46.4] DA: Yeah that totally makes sense. What about testing scenarios? Do you guys think that is important at all in a bug bash?
[0:19:53.8] MN: I don’t think that I have seen a bug bash that had testing scenarios. It was just more like, “Hey, here is the feature. Find out what is wrong with it,” kind of thing. It was very Wild Wild West, if you will.
I am curious to see how bug bashes are ran with the testing scenarios and see what are the differences and how the findings found when using and testing scenarios.
[0:20:13.3] WJ: I think testing scenarios are really helpful when you’re doing to QA work. I think that trying to ramp – And it depends on who is participating in your big bash. Trying to ramp up a bunch of non-technical people who haven’t really done QA work before on how to provide a testing scenario I think is going to take most of the time in the bug bash. So maybe that level of formality is not really justified.
However, if you are an engineer then I think writing a testing scenario is really helpful. Putting your thoughts down around what is the expected behavior and how are we going to verify it beforehand makes going through the testing process easier. And I think also provides you with reassurance that you are not crazy when you run into an intermittent bug or a bug with some slightly unexpected but still rationalizable behavior. I think also it generally makes for a better bug reports if you’re doing it and then filing a bug report including the test scenario.
[0:21:15.0] DA: Right, you get steps to reproduce for free out of that.
[0:21:19.5] WJ: Yeah. And expected and actual behavior is really critical for a bug report.
[0:21:24.0] DA: Yeah, I think part of the thing that’s exciting about a bug bash is having people go a little bit off script and trying to use the feature and in a way you didn’t entirely anticipate if you have testing scenarios then that could be a good starting place for people to see how to interact with the software. But I think even if you had them it is still good to go a little bit off-script and try and probe at the edges of the application.
But if you don’t have testing scenarios at all and you have people who aren’t subject matter experts like it is probably good just to have a short overview of what the features are, what all the knobs and buttons and text inputs mean and what you are trying to get out of it.
[0:22:08.4] WJ: Yeah. I think having people record their screens when they find bugs is really helpful. If everybody has QuickTime or some kind of a video, a screen recorder on there already installed like if you have them that QuickTime that comes preinstalled.
[0:22:21.2] DA: Oh yeah, Click Time? Oh, QuickTime, not ClickTime.
[0:22:25.5] WJ: Yeah that could be really helpful for documenting the bug and verifying if it is really a bug.
[0:22:31.8] MN: Yeah. So, kind of knowing exactly when you find a bug during a bug bash and having and documenting it correctly, what test scenarios hopefully with recorded video, you’ll soon see those very same stories in your sprint, following sprint I imagine. And engineers will get right to work in fixing those bugs. Bug bashes are a lot of fun. I think like when you’re with the team and kind of showing off the work that has been done for X amount of time.
I imagine everyone feels really good when you’re almost at the finish line and finishing all of the critical bugs that may have appeared had the team not run the bug bash.
[0:23:10.1] DA: Right. Yeah definitely a good way to celebrate something that is coming up and build some anticipation in the company about what kind of cool features are rolling out to users. Get a good feeling about what you are rolling out that yeah you can feel confident that that is a good quality product.
[0:23:28.3] MN: So, get out there. Bash some bugs. Make it happen. Crush them.
[0:23:33.8] DA: Smoosh it.
[0:23:34.6] MN: Smoosh it.
[0:23:35.4] WJ: Exterminate.
[END OF INTERVIEW]
[0:23:38.2] MN: Follow us now on Twitter @radiofreerabbit so we can keep the conversation going. Like what you hear? Give us a five-star review and help developers like you find their way into The Rabbit Hole and never miss an episode, subscribe now however you listen to your favorite podcast. On behalf of our producer extraordinaire, William Jeffries and my amazing co-host, Dave Anderson and me, your host, Michael Nunez, thanks for listening to The Rabbit Hole.
[END]
Links and Resources:
