In today’s episode, we get together with William Jeffries to ask all our most pressing questions relating to GitOps. Tuning in you’ll hear why GitOps uses a pull-based system instead of a push-based system and how GitOps can facilitate better security practices through an improved threat model. William explains when and how to apply GitOps to your project and why he is so excited about the work being done by the Cloud Native Computing Foundation. For all this and much more, tune in today as we get to the bottom of what GitOps means for the future of development!
Key Points From This Episode:
{{addCallout()}}
Transcript for Episode 214. GitOps
[INTRODUCTION]
[0:00:01.9] MN: Hello and welcome to The Rabbit Hole, the definitive developer’s podcast. Living large in New York. I’m your host, Michael Nunez. Our co-host today.
[0:00:09.3] DA: Dave Anderson.
[0:00:10.1] MN: And our producer.
[0:00:11.3] WJ: William Jeffries.
[0:00:13.8] MN: Today, we’ll be talking about GitOps, what is GitOps? William, it’s been a while, thanks for coming on again to the show. How you been?
[0:00:23.3] WJ: I’ve been good, it’s good to be back. I’m in Budapest now and so time zones are – it’s just much easier to schedule than from Seoul and so.
[0:00:34.0] DA: Yeah. There’s so many times we spoke to a sleepy William. We had to really do a lot of warm-ups and jumping jacks.
[0:00:43.4] WJ: Yeah, it’s like the middle of the afternoon right now. I’m ready, let’s do this.
[0:00:48.8] DA: Yeah, the moment we started talking with you, you’re like, “Yeah, I’m doing this thing with GitOps” and I’m like, “How is that any different than every structure as code?” Which you're like, “Why is everyone kind of asking the same question?”
[0:01:02.9] WJ: It’s literally – I mean, I understand, because that’s the question I asked when somebody told me about GitOps. It’s like, I think it’s just a badly named thing.
[0:01:10.5] MN: Why did you do new things?
[0:01:13.6] WJ: Why did you name it GitOps? This is just a bad name.
[0:01:16.4] DA: Yeah.
[0:01:18.9] MN: What is GitOps, right? I could go from the GitOps.tech, I’ll read the first sentence. GitOps is a way of implementing continuous deployment for cloud native applications. I think William gave us that definition too in which Dave’s rebottle was. How is that any different than infrastructure as code? Right? William, you’ve been working in this mindset and GitOps for some time now, yeah?
[0:01:44.2] WJ: Yeah, I’m fully bought in at this point, I think it’s an improvement. I mean, the way that we think of deployment pipelines is you merge your code to master and then that triggers a CI job that then goes and does a one-time deployment or one-time apply of your changes to your infrastructure and then you forget about it, that’s it. You don’t think about it again until the next time you go to deploy a change.
One problem with that is that things can drift. If you go two weeks without deploying, when you go to do the next deploy, there’s been two weeks of drift. That’s just an inherent for in a push-based system. With GitOps, it flips the script in it’s pull-based instead of push-based. The infrastructure is constantly pulling your get repo and anytime that the infrastructure defined in your get repo is different from the actual state of your infrastructure and production, it will reconcile, it will try and get your production infrastructure to match what’s in GitHub so you have like a kind of a constant self-healing that happens.
[0:03:02.2] DA: When you're saying like, your infrastructure is pulling your get repo, you’re not talking about my CI server, my Jenkins server has like a chrome job on it that runs every 10 minutes to pull it down and run the scripts. You’re talking about actual – my API server is itself concerned with – is it setup properly.
[0:03:28.3] WJ: Right, there are different CI tools that you can use that are designed specifically for Kubernetes using this GitOps framework. The most popular ones at the moment are flux and Argo CD and I guess you could argue that it’s really not that different from having Jenkins just pulling every 10 minutes. I think if you really boil it down, that’s kind of what’s going on but it is fundamentally different in that the CI serve like the CI service is a part of your cluster, it’s inside your cluster.
Your threat model is – I mean, it’s just, it’s a better security practice. If you have a CI server like Jenkins that has the ability to create, update and delete whatever containerized services you're running, then if that CI server gets compromised, the person who pawned your CI server now has access to everything in production.
That kind of God-mode CI server is a really common security issue that you see in industry and lots of different companies. One of the things that’s nice about tools like Argo CD or flux is that because they’re running inside of the cluster already, you don’t have an external threat. You would have to already own your whole Kubernetes cluster in order to own Argo.
[0:04:58.2] MN: Right, rather than having like you mentioned, if you create a branch that will spin up the pipeline, it’s different than if you have Argo CD dealing, pulling for your GitHub because you would have to have access to Argo and the Kubernetes cluster versus your GitHub pipeline.
[0:05:23.5] WJ: Right, it’s like everybody in your org probably has at least limited access to your CI server, right? If I’m a developer and I can make a branch and push it up and have the CI server run some kind of pipeline and run some of the code I’ve written then I effectively have at least somewhat limited access to the CI server itself, right?
You could imagine if somebody’s GitHub credentials got compromised, an attacker could inject some malicious code, push up a branch and get the CI server to say, install a backdoor. With Argo CD, that threat is eliminated because it’s not a separate service and it’s not running scripts, it’s just applying for it but she’s just playing Kubernetes manifests based on what’s in version control.
[0:06:21.4] MN: Yeah, based on the information the Argo CD has. There’s like an additional layer of credentials you would need. Even then, try to install a backdoor for this particular Kubernetes cluster?
[0:06:36.4] WJ: Right, yes. The way it works is Argo CD allows you to target a revision and that could be a branch, it could be a tag, it could be a specific get shot, anything you could reference and get. Then, you can configure Argo CD to target that particular revision and make the environment match whatever Kubernetes manifests are defined, whether that’s like helm charts or like customized scripts or whatever.
[0:07:03.3] MN: Yeah, I’m looking at the GitOps.tech and another thing that I see that is a benefit to using GitOps would be faster recovery, given that you're able to see the history of your environment changes over time so then you’d be able to revert exactly to the get shop back had the infrastructure that was working at that time. I imagine just much easier than if you were to – I imagine in a push system, you won’t have to merge that change into master and then have that go up and then rerun the pipeline versus being able to have your, I guess, Argo CD point to the shot that you want to revert to and that’s just as fast.
[0:07:48.7] WJ: Yeah, it automates a bunch of stuff that otherwise you would need to do manually like you could build a Jenkins pipeline that allows you to do like rollbacks but if whatever’s in production, it’s just identical to whatever’s on your master branch then you don’t’ have to worry about implementing that because you just do a write command and merge it to master and it’s done. Faster, you don’t have to wait for a bunch of checks around. The checks are sort of independent, it’s like whatever checks needed to run, ran when you merge to master.
[0:08:22.7] MN: I guess the question that I have for you given that you’ve been working in this for some time, for any projects that are currently in the push that’s not using GitOps, would you recommend them to start moving in that direction or is this something that you would do for a brand-new project going forward?
[0:08:42.6] WJ: I think if you are operating a Kubernetes cluster as part of your infrastructure, I would definitely recommend switching to like a pull-based system using either Argo CD or flux. Probably Argo CD would be my recommendation. I don’t think there’s any reason to hold off on that for legacy projects and certainly if I were starting a new project, I would use that approach. I think that’s where the whole industry is kind of moving. Increasingly that will just be the standard.
[0:09:13.4] MN: Is Argo CD open source?
[0:09:15.5] WJ: Yeah, it’s open source, it’s under the CNCF foundation. Argo is like a family, they have four products, there’s like Argo CD, there’s Argo Workflows, that whole Argo family, all of those projects are under CNCF, which is a non-profit that cloud native computing foundation and they offer funding for open-source projects and incubation and stuff and the things that – The thing I‘m really excited about the things that are coming out of that foundation, they’re doing great work.
[0:09:52.0] MN: I see here the GitOps.tech. Should I hire GitOps engineers for my team now? The answer is no.
[0:10:00.0] WJ: No.
[0:10:00.5] MN: There are no GitOps engineers, it’s like how you need DevOps on your team, I guess. GitOps is not a role and neither is DevOps. GitOps is a set of practices. Are they best for clever for them to actually announce because the tech industry, they hear that we need this one thing and then we have to hire for those people all the time and just so you know, it’s a set of practice, it is not a role, right, I imagine William, you’re not just a DevOps procedures, you just know the set of practices to run the operations on your current client.
[0:10:35.7] WJ: Yeah, I am a GitOp if you are looking to hire a GitOp, please let me know.
[0:10:43.0] MN: Your one whole GitHub. Yeah. Awesome. Yeah, no, this is – there’s a whole lot of different tools that are mentioned on the gitops.tech and I was just – anyone who is interested in brushing up their infrastructure to check out gitops.tech.
[END OF INTERVIEW]
[0:11:03.7] MN: Follow us now on Twitter @radiofreerabbit so we can keep the conversation going. Like what you hear? Give us a five star review and help developers like you find their way into The Rabbit Hole and never miss an episode, subscribe now however you listen to your favorite podcast. On behalf of our producer extraordinaire, William Jeffries and my amazing co-host, Dave Anderson and me, your host, Michael Nunez, thanks for listening to The Rabbit Hole.
[END]
Links and Resources:
