Today we shine the spotlight on our very own William Jeffries to talk about the virtues of Terraforming and why it’s such a smart software tool for coding. Kicking things off we first dive into infrastructure as code and why we use it before we tackle the benefits of using Terraform. We find out that Terraform allows you to use a variety of languages designed for infrastructures like HCL or the HashiCorp language. We then ask William about CloudFormation and why Terraform can offer so much more. He continues by explaining how terraforming works with containers and Kubernetes, as well as explaining some of the associated limitations. Toward the end of the show, William then shares his experience with multi-clouds, what makes Terraform providers so versatile, and why the industry is moving toward Terraform. To find out more, be sure to join us today!
Key Points From This Episode:
{{addCallout()}}
Transcript for Episode 197. Terraform the Planet
[INTRODUCTION]
[0:00:00.5] MN: Hello, and welcome to The Rabbit Hole, the definitive developers podcast, living large in New York. I'm your host, Michael Nunez, our co-host today.
[0:00:09.0] DA: Dave Anderson.
[0:00:10.3] MN: And our producer.
[0:00:11.8] WJ: William Jeffries.
[EPISODE]
[0:00:13.1] MN: And today, we'll be talking about Terraform the Planet, Infrastructure as Code, and Why We Need it?
[0:00:18.7] DA: Yeah, that sounds awesome.
[0:00:21.1] MN: I imagine like word like geomancers, just like tearing up Earth and Terraforming things. When I hear Terraform always think about that. Like the Last Airbender or something like that.
[0:00:32.7] DA: Oh, wow.
[0:00:34.0] MN: Every time I think of Terraform, it’s that.
[0:00:36.6] DA: More code than in that show, like Hackers.
[0:00:40.7] MN: yeah. More like hackers.
[0:00:41.5] WJ: Hack the planet.
[0:00:42.7] MN: Hack the planet, Terraforming the planet, ladies and gentlemen. We’re talking about Terraform. We're going to talk about infrastructure as code. I got a lot of questions. I haven't been blessed enough to be sitting in front of doing things in Terraform, or in any infrastructure as code. We had a conversation about CI, recently, previous episodes, and I managed to do some deleting on my Travis YAML file. But I don't think, it's kind of similar, but not right, am I completely off?
[0:01:11.4] WJ: The idea with the infrastructure as code is that you want to be able to keep track of all of your infrastructure in a permanent way and version control much the way that you do your regular code. So, you don't have to worry about somebody going into your AWS web console, and deleting something important, and then nobody knowing how to fix it.
[0:01:33.5] DA: Just delete that one load balancer, and then we’re just, done. Which one was it? Bobby load balancer. He's like, on vacation in Hawaii.
[0:01:46.1] WJ: Or he doesn't work in a company anymore. Actually, nobody has touched that in years.
[0:01:51.8] DA: He’s completely going like, “Ah, ah, ah. No, no, no.” You don't want that. There's so much configuration that goes into infrastructure, and can be a real challenge to keep track of it. And also, to implement it, like, it's going to be very manual work if you're setting all of the set of configurations manually on every server, even if you're scripting it. But it is like pretty repeatable in a way that you can describe in a structured markup like, GAML.
[0:02:26.8] WJ: Yeah. And if you imagine, like, somebody tells you, “Oh, we need to set up another staging server. And if you're doing that manually, first, you got to set up your subnets, and your route tables and the associations.” You have to deal with all of your networking before you even start actually setting up the machines that are going to run the code. It's like a lot of work.
[0:02:54.6] MN: Right. This does not include, and when I hear manually, as well, I'm thinking like, “Oh, you want a new staging server, let me go to Best Buy real quick and buy your brand new desktop that we're going to host this out of.” But even then, not thinking of that as manual, but actually getting a new machine and setting up its networking, that is the thing that you can repeat with infrastructure as code.
[0:03:20.3] DA: I have a question though, does GAML count as code?
[0:03:25.3] MN: That is language, baby, I think so.
[0:03:31.6] WJ: We have another markdown language.
[0:03:34.2] DA: Yeah, exactly.
[0:03:34.7] WJ: So, with Terraform, you can use HCL, the HashiCorp language, which is designed for infrastructure. And it's a lot cleaner than YAML, or JSON, which is what we had to use before. And it feels a little bit more like a language because you know, you can have turn areas and stuff. But it's still very much not a full featured language. Like there's no list comprehensions and you're not scripting anything. It's just about defining resources.
[0:04:04.9] DA: Also determine if it's tearing a plea, I’m not really sure.
[0:04:08.7] MN: I mean JavaScript is a language, I think, that there's not much to code from that.
[0:04:14.8] DA: I mean, it's kind of nice to be able to do things like templating variables and interpolation and all that.
[0:04:21.8] WJ: Yeah, it’s got all that interpolation, all that.
[0:04:24.7] MN: And YAML is a little difficult to do that. I imagine. I think so. I don't know. Can you do variables in YAML? I think, I don’t know.
[0:04:30.3] WJ: We have to use like ERB or some kind of preprocessor to get the variables in there.
[0:04:35.4] MN: No, thanks. Thank you HashiCorp. Was like HashiCorp markup language? HCL or HTML?
[0:04:43.0] WJ: HCL.
[0:04:44.1] MN: Okay. HashiCorp Language. Thank you.
[0:04:46.2] WJ: Yeah, it really helps a lot. I'm migrating some CloudFormation Templates over to Terraform definitions, and the Terraform files are so much cleaner. It's way fewer lines of code and much more readable.
[0:05:00.3] MN: Is that the main reason why you're moving from CloudFormation to Terraform?
[0:05:05.1] WJ: I think there are a few reasons. I think, with CloudFormation, you can end up having to do a lot of bash scripting, because it's not super opinionated about how to plan and apply infrastructure changes, and how to set up a pipeline. People were just having a really hard time ramping up on how the infrastructure worked, because there was so much manual batch scripting.
[0:05:28.5] DA: It was like very custom in the end to your thing, even though it's maybe – your solution is not so custom. But you ended up having to implement a lot of things.
[0:05:41.3] WJ: Yeah. And Terraform is just nicer interface, like using the AWS Terraform COI, it's not a great experience. Terraform is much cleaner, you know, the way they do plan, diff, and you can easily see everything that's going to change. And you can save those plans and pipe them around and apply them
[0:06:05.1] DA: That’s pretty cool. I mean, like, it's interesting. I don't think we mentioned this before, but it's really like writing on migration for your code, where you have a schema that defines the shape of the target. And then I guess, HashiCorp is going to define what steps you need to take, like which columns you need to add or tables you need to create in database migration land, which would instead be servers migrating or setting changing. And then just applying the minimum amount of steps. That's pretty cool.
[0:06:36.6] WJ: I hadn't thought about it that way. But yeah, it does feel very much like our database, migrations.
[0:06:42.2] DA: And you can put it on version control. So, everybody loves version control.
[0:06:46.5] WJ: Yeah. And then you can roll back.
[0:06:46.6] MN: Is it terrifying work?
[0:06:48.3] WJ: Yeah, it terrifying. There’s a library called terrifying. It's just that it’s not good branding. [Crosstalk 0:07:01.8].
[0:07:05.3] MN: The terrified Terraform. But you can use, so you can version control the work that you do in the event, you know, the previous of the most latest build, I would say, is that the correct term when you're building these things in Terraform? What would you call that?
[0:07:19.9] WJ: What will you call what?
[0:07:22.6] MN: You're doing your HCLs, building up the server so that it can have X amount of space in the drive and network is set up correctly, what is that –
[0:07:33.1] WJ: So, all of your Terraform definitions, all of those TF files, those are in your code base, they're ready to be applied.
[0:07:38.0] MN: Right. Let's say what if the latest, you got a version of control, which is great, you just get pushed force master, of course, because that's your cowboy code over here. And then suppose that something, like if it breaks, for whatever reason, because you were trying to do things really fast, it's good with version control, you can roll it back if necessary, and then redeploy or re setup the server with the previous configurations if the newer ones were bad.
[0:08:06.7] WJ: Yeah. And to be clear, like, you could do that with CloudFormation Templates, too, or any other infrastructure as code solution. It's not unique to Terraform. But yeah, that is definitely a major selling point, for infrastructure as code in general. You can also do code reviews. So, if somebody wants to make an infrastructure change, instead of saying, “Okay, here are the prod keys, go do it.” You can say, “Okay, we can pull a request and we’ll apply your changes in a lower environment, and if nothing breaks, then we'll automatically apply those changes in prod.”
[0:08:37.7] DA: I guess also, like, if you are version controlling it, then you can go back in history to not only have the code as it was a year ago, but also the infrastructure as it was a year ago, because it's not always the case that the infrastructure and the code is closely tied. But sometimes there are breaking changes that the old infrastructure won't support the new code.
[0:08:59.4] WJ: Yeah. And you'll also get commit messages that explain why things changed. So, you can sort of document why you deleted that load balancer. So, that 10 years later, when somebody needs to fix it, it's clear.
[0:09:13.7] MN: So, just a friendly reminder, write decent and coherent commit messages, because that's important. The last thing you want to do is figure out why that happened and then you go on to get blame, and it's fix up and that’s it.
[0:09:27.9] DA: Whip effort.
[0:09:28.6] MN: Whip, whip, whip, whip. So that's important. Remember that. I mean, that's cool, though, the idea of being able to look back and see why these changes were made very similar to programming, but in an infrastructure sense, you understand why these things were built.
[0:09:45.0] DA: So how does this work with like containers and Kubernetes and the future of whatever? I feel like, we've talked a lot about Kubernetes and Docker in the past.
[0:09:56.2] WJ: Kubernetes is really hyper focused on containers. So, if you want infrastructure as code for containers, if you want container orchestration, and the ability to auto scale your containers and configure them every which way, then Kubernetes is the right tool for you. But Kubernetes is not going to help with any other kind of infrastructure. So, if you want to set up an RDS instance, if you have a permanent database, it's not running inside of a container, or if you want to set up a networking, if you want to set up a virtual private cloud that all of your stuff runs inside of, then you might need Terraform or CloudFormation or whatever, other infrastructures code solution. You don't have to have infrastructure as code for everything, you could say, “We only want infrastructure as code for the containers. And so, we're just going to use Kubernetes, and nothing else.”
[0:10:50.7] MN: What did that mean that you're just using Kubernetes, for this, but still doing things a little bit manually? Or is it that you're also writing scripts in Kubernetes, to then do the things that you need to do in Terraform or CloudFormation?
[0:11:05.0] WJ: It would mean that you just didn't have version control for a bunch of stuff that you maybe would want version control for, like all your networking, which seems like a bad idea, seems like you would want that it also in version control. I think, maybe if you have like a fully hosted – if you're just using EKS and you don't really have any infrastructure outside of Docker land, then maybe you don't need the EKS cluster itself to be in version control. You just have one resource and that's it. Simple enough you don't really need that documented.
[0:11:41.9] DA: But there is like a valid use case for like, “Why not both?” Use one for your databases and the other for all of your web containers and things like that?
[0:11:56.1] WJ: Yeah.
[0:11:57.8] MN: So, this came up on a conversation before we started recording, William, are you doing the multicloud?
[0:12:03.1] DA: All the cool kids are doing it. Come on.
[0:12:05.5] WJ: I don't think I've ever met anybody who is legit doing multicloud with a single app, like where one app needs to communicate to multiple clouds. The only time I've seen anybody actually using more than one cloud is when they work at a titanic sized company, you know, like a Fortune 50, where they're hundreds of engineering teams in hunting lead, like dozens of countries, and they all have their own thing. And some of them are less than some of them are on prem servers, and some of them are on GCP.
[0:12:32.3] MN: I asked that question to having no idea what it means to do that. Is it communicating across different cloud platforms? Or is it an application that's hosted in multiple servers or instances across the same platform or is it both?
[0:12:47.7] DA: I guess one of the cool things about Terraform, from my understanding, and you can correct me if I'm wrong, is that like Terraform is agnostic of the actual like platform that you're going to be hosting on. So, you can kind of plug it in to AWS as a back end, or you can plug it into GCP to have like Google hosted. So, you actually have somewhat platform agnostic description of how you would deploy your application.
[0:13:15.8] WJ: Yeah, anybody can write a Terraform provider that allows people to configure state for whatever. You could write a Terraform provider for your coffeemaker, if you wanted, assuming your coffee maker has an API.
[0:13:29.4] DA: I'm going to work on that.
[0:13:29.8] MN: My coffeemaker is me, so I guess, maybe, I don’t know. I got to get the hot water, maybe French press. I'll make an API for my phone and have someone do that. Don’t die bro. I can't Terraform just yet. I can’t Terraform you back into existence, we can't do that.
[0:13:51.7] DA: No. [Crosstalk 0:13:51.7].
[0:13:54.9] WJ: Still need to write that provider.
[0:13:55.7] DA: I’m going to make it.
[0:13:57.6] MN: So, you can create a provider is what you call it to communicate, so that whatever API that you have, understands how Terraform can communicate with this. Is that safe to say?
[0:14:09.5] WJ: I mean, you would only do that if you had a product that provided resources that somebody might want to use as part of their application, then you might want to write a Terraform provider for it. Most of the time, if you're a web developer, you just go and use the providers that are already created, and you're probably only going to be using one or two. So, you know, if you're hosted on AWS, you may only need the AWS provider, and that might be everything.
If you have infrastructure in both AWS and GCP, for whatever reason, like I don't know, maybe the Blob Storage is cheaper in GCP. And so, you're going to have most of your infrastructure on AWS. But in order to save some money, you're going to use GCP for this one thing, then you could do that. You could define the infrastructure and Terraform in both clouds and when you do a Terraform apply, would spin it up in two separate clouds, two separate cloud providers.
[0:15:00.0] DA: Yeah, or really like writing something that's like needs to be highly available as a platform itself for some other service, then maybe, go for some kind of more robust deployment multicloud.
[0:15:14.9] WJ: Yeah, you could set up failover, like in case all of Amazon goes down, then you're fine. You just fail over to GCP.
[0:15:23.2] DA: Yeah, in that case, that the S3 bucket is no longer available. It's nice to know that it's there, even if it's not needed right now.
[0:15:33.4] WJ: Yeah, I could see a use case if like, maybe you're doing neural networking, and you feel like Sage maker is not as good as the GCP offering. And maybe there are some specific services that you think are better in one cloud provider than another. You could mix and match. But I feel like multi cloud is one of those things that gets you know, highlighted is a really cool feature. But it's not a feature that anybody actually uses.
[0:15:58.6] MN: It does sound complicated, just like you're complicating your things a lot more than you really need to. But my question is, Dave, I mentioned that it is agnostic. Terraform is agnostic in the example of being able to choose whatever provider that you wish to use. Is it really then complicated? Does Terraform just make it that much easier for you at the end of the day? So, you can do the multicloud? Like, William, you set up the infrastructure at your workplace, bro, just do a multicloud. Stop playing games, you know, you can do it. Come on.
[0:16:30.9] DA: Hey, boss, I am ready to mosey cloud. I'm ready for my promotion. Thank you.
[0:16:34.8] MN: Yes, across all three platforms. Don't worry about it. We got this multicloud baby, what’s up?
[0:16:40.2] DA: But there's a tiny voice in your head. That's just like, “Yeah!”
[0:16:47.5] MN: Exactly.
[0:16:49.0] WJ: The thing is, like all of these cloud providers, they have multiple regions and multiple availability zones. And so, it's just you already get so much security by just setting up the ability to failover from even just US East one to US East two, will be fine. It doesn't even have to be a different region. And that already, what are the chances that the data center in Virginia goes down and the data center in Ohio goes down? Like the whole data center?
[0:17:21.1] MN: Yeah, I mean, that’s like the whole East Coast. All the data centers in the East Coast. You got a bigger problem.
[0:17:26.3] WJ: Yeah. If that happens, nobody's checking your website. They're checking the news.
[0:17:30.7] MN: Yeah, they're checking the news. They wonder why their doors doesn't open anymore. Because door locks are probably – the doorbell is broken. They can't get into that house. That coffeemaker isn't working. Their vacuum isn’t running.
[0:17:44.1] DA: Stop making their coffee for them.
[0:17:48.6] MN: Stay at home, boys. That’s it. I guess, to Dave's point YAGNI on the multicloud. Right now, William, my apologies. I was trying to get you to multicloud it out. But definitely –
[0:17:59.4] WJ: I'm going do it. You dared me. You double downed dared me. We’re going multicloud.
[0:18:07.2] MN: Just get it on your resume, bro. It’s going to look great. It’s going to look sweet. Whether you multicloud a single cloud, though, it seems like Terraform was the move that your organization ultimately made given that you just have more power in your infrastructure as code using Terraform versus a cloud formation?
[0:18:27.1] WJ: Yeah, I mean, I think the industry is just moving towards Terraform, because it's easier to recruit if your infrastructure as code is not written in a language that's tied to a particular cloud provider, because somebody else might have come from a company that uses Azure, or IBM Cloud or some other provider. And maybe they're not familiar with the CloudFormation templating system. And the CloudFormation templating system is also kind of hard to look at. It's just like the JSON for it is kind of gross looking. So, I think that more and more people are going to standardize around that tool.
[0:19:06.1] MN: Oh, that's awesome. I think that that is a hot tip for anyone who's listening interested in the infrastructure as code, like jump on that Terraform given that everyone slowly or is using Terraform as the de facto language, because it's agnostic across different providers.
[0:19:23.9] WJ: If you're already using CloudFormation, then maybe stop and think about it before you migrate. But if you haven't picked one yet, definitely don't pick CloudFormation, pick Terraform. You haven't started yet. There's no debt to migrate.
[0:19:39.1] DA: That goes to Terraform past or future.
[0:19:42.1] MN: I'm think I'm ready to start tearing up the ground with some Terraform.
[0:19:46.7] DA: Tearing up with Terraform.
[0:19:47.4] MN: Tearing it up, terrifying with Terraform, and hopefully building servers that could be multicloud. I think I'm going to do that. I have to figure out something maybe just like an API.
[0:19:57.0] DA: Just like internet.
[0:20:00.6] MN: Yeah, do the internet. Do a hello world that just goes to all the cloud. Say hello world and it will give you a random provider that it comes from. I think I'm going to do that just so I can have it on my resume, multicloud baby, right there at the top. Bobby multi cloud, that’s going to be me before you know it. Be ready. Bobby multicloud coming soon.
[0:20:25.0] DA: Leila Dallas multicloud.
[END OF EPISODE]
[0:20:26.9] MN: Follow us now on Twitter @radiofreerabbit so we can keep the conversation going. Like what you hear? Give us a five-star review and help developers like you find their way into The Rabbit Hole. Never miss an episode, subscribe now however you listen to your favorite podcast.
On behalf of our producer extraordinaire, William Jeffries, and my amazing co-host, Dave Anderson, and me, your host, Michael Nunez, thanks for listening to The Rabbit Hole.
[END]
Links and Resources:
