Today we will be talking about toasting to failures. We will get into what a toast to failure is and share some stories that pertain to failing and we will, of course, toast to them. We don’t have any drinks or apple juice in the recording session with us, but we will pretend that we have glasses and we will put them in the air as we toast to our failures. A toast to failure is when one fails at his or her particular task or at their job but it’s celebrated, which we feel is very important. It’s a little counterintuitive and you don’t want to just go out there and fail all the time. So stay tuned as we go over how failures should not only be celebrated, but should also be seen as a learning opportunity so that you and your colleagues can learn from the mistake. All this and more inside today’s episode.
Key Points From This Episode:
Transcript for Episode 284. Toast to Failure (Replay)
[0:00:01.9] MN: Hello and welcome to The Rabbit Hole, the definitive developer’s podcast in Fantabulous Chelsey, Manhattan. I’m your host, Michael Nunez. I have my co-host today.
[0:00:10.0] DA: Dave Anderson.
[0:00:10.8] MN: Our producer.
[0:00:11.7] WJ: William Jeffries.
[0:00:13.3] MN: Today we’ll be talking about toasts to failures. We will get into what is a toast to failure and share some stories that pertain to failing and we will toast to them. We don’t have any drinks or apple juice in the recording session with us, but we will make pretend that we have glasses and we’ll put them in the air when we toast to failures.
[0:00:35.2] WJ: We do have wine though.
[0:00:36.3] DA: We definitely have champagne, this is champagne, what are you talking about?
[0:00:38.8] MN: Magically champagne. We would toast to champagne now.
[0:00:44.6] WJ: Can we get some like drinking sounds?
[0:00:46.2] DA: Post production?
[0:00:47.2] MN: Exactly.
[0:00:48.1] DA: Corks pop.
[0:00:50.3] MN: This is a failure right here guys, right here. We knew we would record this, but we forgot and got champagne. We officially got champagne.
[0:01:00.5] DA: Yeah, so what is a toast to failure? I feel like I’ve heard this around the office before at Stride?
[0:01:07.4] MN: A toast to failure is when one fails at his or her particular task or at their job but it’s celebrated. I think that’s like important.
[0:01:18.1] DA: Yeah.
[0:01:18.6] MN: To celebrate a failure.
[0:01:19.9] DA: It’s a little counterintuitive.
[0:01:21.0] MN: Right. You don’t want to just go out there and fail all the time. That’s not –
[0:01:25.4] WJ: You’ve got to learn from it. You toast to a failure because it’s a learning opportunity. You have to learn from it otherwise, you know, you have to make a toast to failure about failing to learn form that failure.
[0:01:34.0] DA: Right, you can’t just wake up every day and fail, you have to actually improve and fail in different ways.
[0:01:39.5] MN: Right. But the important part about sharing that failure is, being able to learn from that failure and then have your peers also learn from your failure so that they don’t fail in that particular situation as well.
[0:01:55.0] WJ: It’s like a broadcasted learning.
[0:01:57.3] DA: Right.
[0:02:00.0] MN: Like, “This happened to me so if you’re ever in this situation, don’t do what I did.”
[0:02:05.2] DA: Yeah, that’s pretty fun. I mean, like I really enjoy that part of our company meeting when everyone gets together and we share failures. It’s often pretty funny sometimes, you know? It’s a situation you’re not in, so you can enjoy it.
[0:02:21.4] MN: Yeah, exactly. It’s like, “Oh, that didn’t happen to me but thanks for getting in that situation and teaching me about that failure that just happened.”
[0:02:30.5] DA: Yeah, I think it does make things a lot more relatable too. It kind of helps reduce kind of imposter syndrome because like, people are afraid like, especially in a really knowledge heavy job such as programming that you need to know all the things, have all the answers and be correct all the time. But often, people don’t know and you know, people that you respect do make mistakes. Having them share those mistakes really sets the tone.
[0:03:02.8] WJ: I think it promotes humility, because everybody knows about all of your failures. So you can’t really get quite as big of an ego, which is a good thing for a company culture.
[0:03:12.2] MN: The fact that someone is – it’s not like you go around in a circle and ask every single individual to, “Hey, you give me a failure right now that you have to share with everyone." It’s another step where the person volunteers to share that failure. That’s like another step of humility. Like, “Oh, I too make mistakes. Here’s what happened.”
[0:03:32.7] DA: If you wanted to have this kind of a meeting in your company or in your group or with your significant other, how would you go about doing this?
[0:03:45.5] WJ: I think the person organizing it is ideally the leader and the leader goes first to set the tone and make sure that everybody feels comfortable. You know, that’s just – we’ve talked about this on previous episodes, that’s an effective way of getting good participation.
[0:04:00.7] MN: Right, the bigger the failure from the leader, the more people would feel like the things that has happened to them isn’t as bad so they can also share those things.
[0:04:09.9] WJ: Yeah, it makes it safe.
[0:04:11.0] MN: Yeah. You know, one example that I hear happen all the time is, you know, “That one time that I broke production.” Like some junior developer, here’s that and you know, they freak out if they broke production one time but if they hear from a senior developer that that’s happened on many occasion, even including where they work right now, they will feel more at ease when and if that time ever happens.
[0:04:37.1] WJ: Yeah, I think it’s good to have alcohol involved to sort of you know, loosen people up. You’ve got to have some nonalcoholic options.
[0:04:44.0] MN: Oh okay, apple juice? Get some apple juice.
[0:04:47.3] WJ: Liquid courage.
[0:04:47.9] MN: Yeah, I don’t think I’ve had a toast to failure like with alcohol before it but definitely the conversation happens and then you continue it at the bar is like usually what happens and then you realize you fail a lot. The more you drink the more you hear, “And that one time was crazy. That day sucked and this is what happened.”
[0:05:08.7] DA: Yeah, I definitely feel like the point that you made like having a safe space and making sure that leaders are sharing and everyone’s on the same page and not judging, that’s super important. It could be kind of awkward if someone who everyone respects or fears is off on the side and just kind of like scratching notes. And you’re like, “Oh, okay. Well Timmy, we’ll talk about this later.”
[0:05:33.3] MN: You did what now? Let’s speak about that?
[0:05:36.4] WJ: It’s going on your permanent file.
[0:05:39.1] MN: I mean, that too, right? You have to make sure that you work at a company that isn’t like that.
[0:05:43.8] WJ: Or make it a company that isn’t like that. Toast to failures are a good way of bringing that culture about. Then the next time somebody goes to call you out on making a mistake and tries to punish you for it, rather than a giving you the space to learn from it and grow and be better you can just point out all of the worst failures that other people on the company have had.
[0:06:05.4] DA: Yeah, that’s true. Also like, I guess, if you’re promoting a culture that is the opposite of culture that toast failures then people are going to feel like they can never fail, they can’t ask for help if they don’t know. So then they can admit that they don’t know something or they’re not confident, they just need to go out and do it and then maybe fail, but like stupid under the rug and not tell anyone.
[0:06:30.1] WJ: Yeah, secret failures are not good failures.
[0:06:31.7] DA: Yeah.
[0:06:33.3] DA: You have dishonesty, not humility, not learning, this is the anti-Stride.
[0:06:40.8] MN: The anti-Stride.
[0:06:41.6] WJ: You guys want to just like do one right now?
[0:06:43.2] DA: Yeah, sure.
[0:06:44.3] MN: Yeah, I think I always got one in the pocket in case the situation comes up, so I’ll go ahead and just share it.
[0:06:50.9] DA: Yeah, we’ll just pop the champagne.
[0:06:52.1] MN: There you go. Let’s just make that post production please. Right?
[0:06:57.2] DA: Clinking my glass right now.
[0:06:59.5] MN: I was at a client working on a feature that was looked to be added in the future. This particular place constantly rolled their code base on to master’s so it was, you know, you do your test, make a PR, right? Unit test, you merge it straight to master and some production and then everything is free to go.
I was working on adding this new feature and kind of overlook the story and thought it was just like, “Oh that was misestimated as a three. But that really was like a one-point story, it wasn’t really that difficult so I’ll just do it this way.” You know, PR happened, you got it merged in. Everything was good in the world in the afternoon on a Thursday and at 3:30 on that day someone I’ve never seen before comes down to the floor and asks for my tech lead to ask for a meeting with that person.
Because apparently I had introduced like a show stopper bug and that pretty much stopped the entire company. I just didn’t know that I actually introduced this bug, even with all the coverage that I had. It was really important because they had an event that would happen over the weekend and we needed to get this fixed before Friday and I’m like, “Wait, we have those little thing, how does it work?”
But it inadvertently caused like a chain of events that would cause this bug to exist and unfortunately, some else got chewed out for the work that I did but eventually it got to me and like to fix this work. So we ended up cherry picking that particular work out for the weekend and then we continued it soon after. It was still in between sprints, so that was like okay for me to continue working on other things.
But, to be able to bring someone down form business to say, “Hey, we need this, this will definitely bring down revenue,” made me realize, “Oh, I can think of something as small as this but it can definitely change the path of a particular client at the event with the revenue for this one little change that I did.”
[0:09:02.1] DA: Yeah, bring the suits down upon you.
[0:09:03.7] MN: Yeah, no. Don’t ever – if you see suits that are normally not there, you know, be weary.
[0:09:09.3] WJ: What did you learn?
[0:09:10.1] MN: Well, I learned to be sure like if you do receive a story and you feel like it’s too good to be true, to make sure you can speak to the people, your project manager or the product owner to kind of fully understand the story that you just picked up.
Rather than just saying, “Oh, it’s just this? Let me do that,” and then whatever work that I did uncover because of that book, I was able to talk to the product owner to then write stories to fix up those things after. It was like a tight communication with the people who are writing the stories and ensuring that you understand it yourself is something that I learned real fast.
[0:09:50.2] DA: Nice.
[0:09:50.0] MN: Anyone else? Come on, you can fallow that. I can’t fail…
[0:09:53.4] DA: We’re good, we’re at the the end of the podcast now. Yeah, for me in another life, I was a Java developer and I used to work on like enterprise applications and those were often legacy applications where there’s like, not really most test coverage at all.
[0:10:13.5] MN: Those are fun.
[0:10:14.8] DA: A lot of manual testing and all that fun stuff that goes along with that. We had to do a platform upgrade and the way that these projects often get prioritized, it’s a big amount of work, it’s a legacy application, a lot of manual testing. It’s expensive and nobody wants to pay for it until it’s December. Because everyone’s budget is rolling up and so we started this project. We did a great job I think, we tested everything very thoroughly and staging, fixed all of the bugs that we identified and then we go to prod release the week before Christmas and we do the cut over, seems like it’s fine. You know, there are numerous interfaces and all kinds of crazy stuff so I’ll check in all the boxes. Come in on Monday, no users were able to use the application.
[0:11:04.7] MN: Oh my god.
[0:11:06.8] DA: No database transaction. So we upgraded the application server and the transactions were not having a session. So if you fired a query and then you fired another query then they were not related and like the session thing was just not working with the database.
I was like, “What’s going on, this never happened in staging.” We’re trying to figure it out. Clock’s ticking. I was scheduled to go on vacation and I’m like, “Okay, I guess I’m canceling my vacation, try to figure it out.” 11th hour we figured out that it was a configuration setting that was different between production and staging.
[0:11:49.2] WJ: Oh my god, classic.
[0:11:50.8] MN: Classic.
[0:11:51.7] DA: It was just a bullion, you know? It was true false.
[0:11:54.8] MN: My gosh.
[0:11:57.1] DA: You know, we were going through all this complicated procedures and like process to like roll back, but then we found out that it was this thing and so we’re like, “Okay. We’re going to stick with it, fix it and yeah.” Then, you know, the big takeaway from that is that short services can be good but you should trust but verify. Like when the platform team’s like, “Don’t worry about it, we got you. Let’s figure it out.” I really wanted all the details and also those bullions, they’ll get you.
[0:12:29.1] MN: Configurations alone is just like, you hear a story about failing and someone, now, I’m going to look at all my configurations and make sure they work, if something’s broken, always go back to the configurations.
[0:12:39.3] DA: Yeah. XML, YAML, all that fun stuff.
[0:12:43.4] MN: Will, you have a failure we can toast to?
[0:12:46.3] WJ: Yeah, absolutely. So I started a nonprofit a while back that helps people in low income housing who don’t have access to adequate heat and all the software is open source and I was the only developer on the project for, you know, significant stretches of time. So in order to kind of approximate pair programming and, you know, have some contact with other developers, I started live streaming my own code.
So I would work, at the time, there was this site called livecoding.tv. I think they’ve either shut down or renamed, or themselves have been bought. But at the time it was livecoding.tv and so I setup a stream and while I was working, I would just livestream whatever it is that I was doing. It was cool. People would show up in my channel and they would comment and they would ask questions and sometimes they would have helpful suggestions and help me debug stuff.
It was really a lot of fun and then for some reason I needed to remote into production, which you know, surprisingly wasn’t the fail that is already kid of – I feel like, what are you doing? Tunneling into production and like making changes from the console. But I had the rails console open in production. I was messing around and I had a query for a user data and then people’s personal information came up on the screen like names of people in low income housing who are like, in housing court fighting their landlords. Some of whom who’s landlords don’t know that they were using our product to gather evidence.
[0:14:18.3] MN: Oh my gosh.
[0:14:20.9] WJ: I just live streamed the video on the internet!
[0:14:25.2] MN: Oh snap.
[0:14:26.3] WJ: Immediately shut down the stream and went to delete the video files. I was like, “Oh my god, this is awful.” Crisis averted though, I stopped it very quickly. There was like maybe two people in the channel. I’ve deleted the records.
[0:14:44.4] MN: Okay.
[0:14:43.8] WJ: So then, I went back to doing it and the exact same thing happened again. Exact same thing happened again.
[0:14:49.3] MN: Oh my god.
[0:14:50.1] WJ: It’s like, “OH my god, I’m not learning. Stop.”
[0:14:54.5] DA: Just started streaming again or?
[0:14:55.7] WJ: Well I started streaming and then I like did another query that I wasn’t expecting to return in any user data and I got back user data.
[0:15:02.1] MN: Oh my gosh.
[0:15:02.0] WJ: It was like names and addresses. It was like the worst thing that you could leak. So, you know, I shut it down. “Shut it down!”
[0:15:10.8] MN: Shut it down now.
[0:15:12.8] WJ: Then I went and I edited just to make absolutely sure it could never happen again, I edited the way that the rails console displayed user objects to eliminate any personally identifiably information. I tested that, had unit test coverage so that I could live stream my code. Then went back in and we started the feed.
[0:15:34.9] DA: Oh that PII.
[0:15:36.5] WJ: PII man. You know, I learned perils of live streaming if you’re going to do it, if you’re working on open source and you can live stream, I highly recommend it. If there’s any kind of personally identifiable information. First of all, don’t use prod. Don’t use prod data when you’re web developing, don’t tunnel into prod. But if by some crazy reason you end up in a situation where you have to, be very careful that you don’t accidentally live stream it to the world.
[0:16:02.1] MN: Right, be offline when you do it, if you can. When you go into production, that is possible.
[0:16:07.7] DA: Go into your learning from your closet. Live stream from there to yourself.
[0:16:15.2] MN: Man, yeah, those were awesome examples of failures that we all earn form each other. Mine is to pay close attention to the stories that were written and ensure that everything is documented well enough. For Dave, configurations are a pain in the neck but you have to make sure that everything you look over when you do a conversion transfer is A okay.
Make sure you do that and first off, SSH in production is a bad idea but live streaming as well while doing that is just as bad because of people’s personal information. I mean, those are just like one of many things.
[0:16:54.0] DA: Yeah, it’s a double whammy.
[0:16:55.4] MN: This is just one of the many things that we as developer and consultants have on a daily basis and when you share that information with other people and you would get interesting stories on what other people have failed and how you can learn from those failures.
I think toasts to failures are amazing and we should all toast to that and hopefully we can add up post production sound to the toast that we just had to failing. Even like the process of doing that, what the people in the room is pretty interesting because then now, we know some of the struggles, we may even felt some of those struggles that as Dave or as William was explaining, we may have been in that situation and like, “Oh my god, yes, I understand.” Or, we will look out for that situation if it comes up at work.
[0:17:40.6] DA: Yeah, you know, raw humans, we’re all learning as we go.
[0:17:47.0] MN: Great, so we got a lot of stories of toasting to failure and this is the end of the episode. Hopefully this episode isn’t one of my toast to failures. I’m hoping that this was an awesome episode for everyone to listen.
[0:17:59.8] DA: Oh, it was definitely awesome.
[0:18:00.6] WJ: We’ll find out later when it drops.
[0:18:03.1] MN: Yeah, we’ll find out how many people one starred or something. I mean, I’m sure it’s great. Yeah. Finishing up, I’d like to thank my cohost, Dave, thank you for sharing your toast to failure and toast into it.
[0:18:17.7] DA: Yeah, always a pleasure filming with you.
[0:18:21.0] MN: William, thank you for sharing your failure as well, I hope that our listeners get something out of it, I sure did, don’t live stream and go to production.
[0:18:30.1] WJ: Yeah, well, thanks for starting us off, you set the right tone.
[0:18:33.4] MN: I tried, I really did.
[0:18:34.9] DA: Fearless leader.
[0:18:35.9] MN: Yeah, if you have any failures you would like to share in 140 characters or less, hit us up at Twitter.com/radiofreerabbit. This is The Rabbit Hole, we’ll see you next time.
Links and Resources:
