Continuous integration is the peanut butter to the jelly of continuous delivery, and it’s also the topic of today’s episode. We are talking all things CI, including some of the benefits and downsides and a whole lot more. We kick off with a definition of CI, where we discuss how it helps reduce bugs. No-one would say no to having fewer bugs, and it is by far one of the biggest advantages of CI. As great as CI is, it is not just something that can be done spontaneously. It needs proper systems to make sure that it does what it needs to do and does not slow things down. We then touch on some of the big drawbacks, which include time and cost. Although it is not a perfect system, you should not wait to set it up. Be sure to tune in today!
Key Points From This Episode:
{{addCallout()}}
Transcript for Episode 266. Continuous Integration
[INTRODUCTION]
[0:00:01.9] MN: Hello and welcome to The Rabbit Hole, the definitive developer’s podcast. Living large in New York. I’m your host, Michael Nunez. Our co-host today.
[0:00:09.2] DA: Dave Anderson.
[0:00:11.0] MN: And our producer.
[0:00:12.4] WJ: William Jeffries.
[0:00:13.8] MN: Today, we’ll be talking about continuous integration.
[0:00:18.3] DA: That’s right, it is the peanut butter to the jelly of continuous delivery.
[0:00:24.3] MN: Yeah. We can’t talk about CI without talking about CD and this is probably going to be a two-part series that we would do because you got to mash it up altogether. We’ll be talking about what is continuous integration, what are the benefits, risk and many other things.
Gentlemen, does anyone want to define what is continuous integration before we move forward?
[0:00:49.7] WJ: Yeah, I could do a definition of continuous integration. Continuous integration is merging your code back into master frequently. The idea is, by regularly integrating your code into the shared repository, you get short-lived branches that don’t diverge too far from everybody else’s code and therefore you get fewer merge conflict and fewer bugs.
[0:01:17.8] DA: I love not having bugs. That’s one of my favorite things.
[0:01:21.5] MN: That definitely is one of our favorite things. The ideas that you and your team have all these systems in place that allows a seamless continuous integration like ecosystem for your code base, right?
You can’t wake up – you wake up in the morning and is like, “All right, we’re just going to merge all the time,” and are you continuous integrating that way, is that what that would be or –
[0:01:47.8] DA: Yeah, I guess that’s like kind of like the evolution of continuous integration where at it’s like the most simple thing, if you’re continuously bringing everything together in the same place, then you are continuously integrating it but that doesn’t mean that it isn’t like a big pile of junk. There’s a lot of methodologies that people use to validate the changes before integrating them automatically that are pretty inseparable with the whole concept.
[0:02:22.6] WJ: Yeah, I think when people say CI, they’re usually talking about a server and that server is in charge of taking the code from your branch and running a bunch of tests in lint and whatever other stuff you want to run before you allow somebody to merge into master.
Then often times, that CI server is also responsible for running the tests after it merges into master and maybe doing a bunch of other stuff, sometimes kind of unrelated stuff, sometimes your CI server, especially if it’s Jenkins turns into like just where you put all kinds of random jobs.
[0:03:02.7] DA: Yeah, just like a big Swiss army knife of groovy scripts. Yeah, I guess a lot of things with the CI server does for you automatically, you could do yourself locally. You could run automatic unit tests and integration test locally. You could run your Lint locally. You could do type setting like staff analysis.
[0:03:28.4] WJ: Unless you have like a three-hour test suite and then you need parallelizing on CI.
[0:03:33.4] MN: Yeah, those computers are so much faster than my 2015 MacBook Pro bro, just like throw it up to the servers, they got it, don’t worry about it.
[0:03:43.4] DA: Yeah, that’s true. I guess that is like kind of a challenge sometimes with really big test suites. Like if you have a monolith codebase that’s grown super huge and maybe incorporates like all of the different services that you're running. Even when you’re changing like one little bit, you got to run all of the integration tests and all of the unit tests and whatever and then no one wants to run it locally because I have Jenkins or I have Circle or Travis or whatever service you’re using to run it.
It’s faster to push it up than go for a walk, come back, see what happened.
[0:04:28.2] MN: I mean, can we think about a number in which it changes from “I could run it on my local machine” to “Let me push it up and have Jenkins do it for me?” Is there a time, do you guys have in mind, is it three minutes like if the test runs longer than three minutes then it’s like, “I’m running it and I’m going for a walk” or like, what’s that number for you?
[0:04:53.5] WJ: Man, like 10 seconds, I’m very impatient. Whatever, let CI do this.
[0:05:04.7] MN: I guess –
[0:05:04.6] WJ: I mean, the other thing is, it will spike your CPU usage, it will spike your memory, it will slow down all of your other applications. If you have a notification setup so that the build CI tells you when your test run is done.
I’ll just run the one test that I’m working on at a time. Maybe the one test file but writing the whole suite? No.
[0:05:33.6] MN: This guy’s working on a big code base, I could tell. He is just not having it, he’s just not having it.
[0:05:39.1] DA: Yeah. I mean in the ideal world, I think for me, if everything’s running another 10 minutes then I’m fine with it, and I’m not going to run everything all the time but you know, hopefully my editor is running the lint checks and doing formatting and type checking as I’m going so I don’t even need to run that but maybe I’ll run it once.
I have ended up in situations where like, even that is incredibly slow because there’s so much code.
[0:06:10.2] WJ: I think that it needs to be done in the time it makes to make a sandwich.
[0:06:16.6] DA: Sandwich time.
[0:06:16.6] MN: Sandwich time.
[0:06:18.7] WJ: Not like a fancy sandwich, I’m talking PB and J, none of this like club. None of that madness.
[0:06:24.3] DA: Just really going to the metaphor and what I’m hearing is that William can make a really freaking fast sandwich, right? He’s like a ninja just chopping the bread in the air and the peanut butter is getting in there. Maybe he’s using Goober, like the combination peanut butter and jelly.
[0:06:40.5] MN: My gosh, no.
[0:06:41.6] DA: Like a mad man. Not the Goober PB and J.
[0:06:48.2] WJ: I think I just have PB and J on the brain because I went to a burger place here in Korea and they had a peanut butter and jelly burger.
[0:07:00.3] MN: My gosh.
[0:07:00.5] DA: My gosh. Wait, and jelly and peanut butter?
[0:07:03.4] MN: And burger.
[0:07:03.9] WJ: Yeah, it was like, they advertise it as like a peanut butter and blueberry burger and I was like, “What are you doing? This is so upsetting.”
[0:07:13.5] DA: And it was good?
[0:07:15.2] WJ: Well, I had to buy it because I mean, you can’t not try that, right?
[0:07:19.2] MN: Yeah, you can’t walk by that sign and be like “no” You have to be like, “I have experienced that,” and then walk on. You experienced it and would you get it again I guess is the question.
[0:07:29.6] WJ: I would not get it again because it’s a sin but I will admit that even though it was sacrilegious, I did enjoy it.
[0:07:37.6] MN: Yeah.
[0:07:37.7] DA: I once got a burger that was on two slices of Ellio’s pizza. Although, I feel like this metaphor is breaking down now.
[0:07:46.5] MN: My gosh.
[0:07:49.1] WJ: Is that a sandwich? Is that a sandwich?
[0:07:51.6] MN: It turns a ridiculous segue.
[0:07:52.4] DA: A burger between two slices of pizza.
[0:07:57.3] MN: All right, one more burger and then we’ll cover, the ridiculous burger I’ve had is a burger where the buns are two grilled cheese sandwiches.
[0:08:05.0] WJ: My god, that’s brilliant.
[0:08:07.3] MN: It was called the excalibuger. Yes, it was great.
[0:08:11.4] DA: Wait, okay, well okay, those are other podcast episodes. The burger on the Ellios and the burger on the grilled cheese but we’re talking about burgers with peanut butter and jelly where the peanut butter is a continuous integration, the jelly is the continuous delivery and I don’t know what the burger is but –
[0:08:31.7] MN: Deliciousness.
[0:08:32.5] WJ: Your code.
[0:08:33.1] MN: Yeah, your code. Your burger is your code, exactly.
[0:08:37.1] DA: It’s just protein.
[0:08:38.9] MN: I think we definitely covered like the benefits of having small pieces of work to get it merged in, it’s probably a – having that system in place so you can then deliver your code as fast as possible can’t happen unless you have the continuous integration portion nailed down.
Is anyone here familiar with setting up continuous integration in their projects before?
[0:09:08.8] DA: Yeah, I think a couple of times.
[0:09:09.2] WJ: Yeah, I’ve had to do that.
[0:09:10.6] DA: Yeah.
[0:09:12.0] MN: Because I’ve only been lucky enough to join a team that already had it and like a part of me has been thinking of just like tearing it up. “I’m just going to delete this,” and then try to do it over but I don’t know if that’s something that I really want to do.
[0:09:26.5] DA: Just delete circle.yaml or whatever it is and just see if you can arrive at the same place. I mean it is not too hard. If you are using like a SaaS solution, then there’s often kind of off-the-shelf configurations that will mostly get you where you need to go.
[0:09:47.1] WJ: Yeah, I think if you’ve used a CI or CD server before and you know what you want it to do, figuring out how to configure it to do that thing is surprisingly intuitive. At least, you know, if you are using one of the mainstream servers.
[0:10:04.3] DA: Yeah, and I always – like I have my checklist where it’s like, “Okay, I got to run the tests.” Like that’s – you got to and then I’m going to do the linting, code formatting, maybe type checking. I think that is the four things that I really want right off the bat and then I’ll just write the script locally and then if it works locally, then just the next thing to get it to work in the CI land.
[0:10:42.1] WJ: Yeah, I think setting it up to do a deployment is more complicated. Getting it to just run your test and lint and everything, usually yeah, you could definitely delete your circle yml.
[0:10:56.4] MN: Such as like delete my travis.yml right now, push to production and then go to sleep. Are you telling me I should do it?
[0:11:05.1] DA: Well actually, you can and if you’ve never made changes to the configuration like often with any of these services that are running on hooks like if they’re using the configuration that is in the repo, then you could make a new branch and run it off of that branch and see what happens when you make the changes.
[0:11:29.9] MN: Right, so just like create a new branch and then you know, fudging up the travis.yml will run the build in that branch only with those configurations?
[0:11:43.9] DA: You delete it and that’s green and then I don’t know, you just make it fail for whatever reason and see what happens.
[0:11:50.8] WJ: Yeah, pro tip, if you ever want to deploy broken code to production, you can always just delete the line in your travis.yml that runs the test and push.
[0:12:01.0] DA: Right, I mean this is –
[0:12:02.3] WJ: It will use the new travis definitions.
[0:12:04.3] DA: This is a new hack. You know, you don’t just have to delete the test now or just comment it out like you can leave the tests there but just delete the travis or at least have it in configuration.
[0:12:15.9] MN: Yes, oh man, teaching people how to break their systems. That’s what we’re all about.
[0:12:21.3] DA: This is chaotic evil on the developer alignment chart.
[0:12:26.0] MN: Yes, oh man and so, I do want to touch on – I think we haven’t mentioned before, I’d ask the question with the time. You know, how much time would it take when you are impatient in fixing it and whatever number, I know we have mentioned William. You are very impatient, 10 seconds tops and you got to get – then you’re like just pushing it up to the cloud I think –
[0:12:51.6] WJ: Maybe five.
[0:12:52.3] MN: Maybe five, even faster but I think the – that may seem like a drawback to having continuous integration by having your test being – you know, having an extensive amount of tasks to make sure that your system is running correctly but we all agree that the tests are important regardless of how long their going to take and then we’ll need to kind of assess how do we make those tests. How do we ensure that we can get those tests happen faster?
It is probably one of the drawbacks that one may have to think about when dealing with continuous integration but I feel there is so much benefits to having CI, a CI system set up for your engineer so that they can deliver and ship new features as fast as possible.
[0:13:37.6] WJ: Yeah, I mean you just need to make sure that people aren’t accidentally breaking the build and if it doesn’t run automatically, people are going to forget to do it. Somebody is going to merge code that breaks a really critical workflow that you had test coverage over because they forgot to run the test. The only way to guarantee against that is to automate it.
[0:13:59.6] MN: Right and while they, you know, another drawback that I am thinking of could be costs. I don’t – is there a cost money to do a Travis? I think a Jenkins is an open source, right?
[0:14:12.0] WJ: Jenkins is open source. I mean if you are going to deploy it to a server you are still paying for the server. Travis has free tier but I think if you are code base is private, which if it’s for work it probably is. You are going to have to pay some nominal amount of money. It’s small, it’s worth it, do it.
[0:14:30.9] MN: Right, the drawbacks does not outweigh the benefit of the many things you can do when you have continuous integration set-up.
[0:14:40.2] DA: Yeah, I mean sometimes the pricing comparison couldn’t get a little bit confusing because like it may not be clear like you may pay extra for more features like if you are trying to do those parallel builds that will get you over the finish line faster because you have 24 hours of tests but if you spin up like you know, 48 pods and it will be done in half an hour, you’re going to pay a lot more per build most likely.
[0:15:12.3] WJ: Yep, at what team size do you think it’s worth it to parallelize?
[0:15:17.4] MN: At what team size?
[0:15:20.6] WJ: I mean I feel like if it is just one developer even if the test suite takes like, you know, 10 hours to run, how much benefit do you really get out of parallelizing? You don’t need to merge this often because there are fewer, it is just one or two developers. How often do they actually need to merge code? How often are there really conflicts?
[0:15:44.5] DA: Yeah, I mean you might end up running into problems with concurrent builds because I know some services only allow you to have two builds running or four builds running depending on your plan. You have to put more money into it in order to have more builds running on different branches. If people are just pushing up on their branch and waiting for CI to tell them what the state of the world is, then they might not get that feedback or you may have to cancel someone else’s build on their branch in order to see what happened with yours and then –
Yeah, you can kind of see that kind of dynamic start to play out a bit more as you scale the team and as your build also gets slower. If you have a really fast build then it is probably not going to be a problem but you know, there’s some calculus there. I’m pretty sure Isaac Newton would have a formula for this.
[0:16:41.7] MN: Yeah, I don’t think I have a number either. I just think that the earlier you do it like the longer you put it off, the more slower the build will get for individuals to do, be able to run in parallel. The earlier the better, so maybe four like I have in mind but that number is in my head but I don’t know. I am not Isaac Newton, I don’t – that number just came to me. Four, I’m going to say four.
[0:17:06.8] WJ: The other thing is the longer you wait to make your tests safe to run in parallel, the harder it’s going to be to adapt parallelization.
[0:17:14.7] MN: Yeah.
[0:17:15.3] DA: Right, you could shoot yourself in the foot with dependencies that you don’t even know exists.
[0:17:21.6] MN: Right.
[0:17:22.4] DA: Yeah, it is always a good thing to work on optimizing tests. I would do it if it was just me because I would be really frustrated. I wouldn’t want to wait on that build. I’ll be like, “Whose fault is this?” and I’d be like, “It’s mine.”
[0:17:37.0] MN: It’s me.
[0:17:38.6] DA: It’s me.
[0:17:40.5] MN: William over here doesn’t want to wait longer than five seconds for a build. You got to do it fast.
[0:17:47.7] WJ: It’s messing with my cycle time, man.
[0:17:49.0] MN: There you go. You got to make a fast sandwich in those five seconds or else. With continuous integration, we have to continue the continuous movement I would say. The next step would be the continuous delivery portion, so now you got your code and it’s all deliver into and you run your test. You run your lint, you run your – and everything runs well. You are using your Jenkins or your Travis or any of the services that are out there but now you need to deliver that bad boy. Deliver that fresh sweet feature that you’re going to give to your customers.
[0:18:28.2] DA: That burger.
[0:18:29.3] MN: Oh yeah, that burger. When the peanut butter hits the jelly and your code is slapped in the middle, that burger, you got to need some good continuous deployment. It catches it on our next episode when we talk about continuous deployment.
[END OF INTERVIEW]
[0:18:44.4] MN: Follow us now on Twitter @radiofreerabbit so we can keep the conversation going. Like what you hear? Give us a five star review and help developers like you find their way into The Rabbit Hole and never miss an episode, subscribe now however you listen to your favorite podcast. On behalf of our producer extraordinaire, William Jeffries and my amazing co-host, Dave Anderson and me, your host, Michael Nunez, thanks for listening to The Rabbit Hole.
[END]
Links and Resources:
