On today’s episode, we are joined by Chris Grande, a business analyst at 2U, an online higher education website that has partnered with some top universities to make higher education more accessible than ever. Also on the show is regular guest, Emmanuel Gernard, a software developer who has worked as a consultant at 2U. They discuss what it was like moving from a legacy system to Kubernetes, particularly what the experience was like for Chris, not having a technical background. He shares what working on the transition was like and how his role as a business analyst shifted and morphed during the project as well as how he made sense of this highly technical undertaking from a user’s perspective. Chris also shares what some of the difficulties and rewards of pushing through the transition were, both for the engineers and for the users. For all this and much more, join us today!
Key Points From This Episode:
Transcript for Episode 104. Steering a Kubernetes Migration: A Non Technical Team Member’s Voyage, w/ Chris Grande
[0:00:01.9] DA: Hello and welcome to The Rabbit Hole, the definitive developer’s podcast in fantabulous Chelsea, Manhattan. I’m your host, Dave Anderson, with me today we have our producer.
[0:00:11.0] WJ: William Jeffries.
[0:00:12.5] DA: And our once and future regular guest.
[0:00:14.7] EG: Emanuel Genard.
[0:00:15.8] DA: Today, we’re going to be talking about steering a Kubernetes migration, a non-technical team member voyage. Just generally being a non-techno person on a technical project. We have a special guest today.
[0:00:29.8] CG: Thank you very much for having me, I’m Chris Grande, I’m a business analyst at 2U, we’re an education technology company, powering graduate programs.
[0:00:38.4] DA: Hey, it’s great to have you on.
[0:00:40.0] CG: Thank you for having me.
[0:00:40.5] DA: I heard that you’re doing some cool stuff with Kubernetes?
[0:00:44.3] CG: Only the coolest stuff.
[0:00:45.5] DA: So exciting. Whenever I hear Kubernetes, I just I’m psyched.
[0:00:49.0] WJ: All the cool kids are doing it.
[0:00:52.1] CG: I did not originally know that going into this project.
[0:00:56.6] DA: When I tell you were in Kubernetes project, you weren’t just like, “yes.”
[0:01:00.9] WJ: So ready to put this on my resume.
[0:01:03.7] CG: I think it took me a little while to even realize that the little blue steering wheel that was an emoji in Slack actually meant Kubernetes, it took me a little bit to pick up on. If that shows you where I started with, yeah.
[0:01:17.4] DA: Yeah, why a steering wheel?
[0:01:19.1] WJ: Well, you know, because of the ship?
[0:01:20.8] EG: Kubernetes is steersman in Greek.
[0:01:23.6] DA: Wow.
[0:01:24.3] EG: Yeah.
[0:01:26.1] DA: So deep. I never knew that.
[0:01:28.0] EG: There are levels to this.
[0:01:31.9] DA: What even is Kubernetes? I think we talked about it briefly before but for people who are unfamiliar, be the technical or nontechnical people. May give them a little intro?
[0:01:43.0] CG: Okay yeah. I think my explanation will show you how nontechnical I am, not to try this at the example but it is essentially the containerization of applications using pod structures, so what that essentially means like in our context of the legacy system is when we move things into Kubernetes, it made the deployment of our technology that much faster.
[0:02:06.4] DA: The prior state for you guys was like actual physical machines or like AWS machines.
[0:02:12.5] CG: AWS hosted machines, yeah.
[0:02:13.7] DA: Okay. And just really white-knuckle deployments.
[0:02:17.7] CG: Yeah, usually they were about three hours biweekly on Thursdays.
[0:02:22.9] DA: That’s fun, that’s the train you don’t want to miss, I guess. Yeah, that’s a big undertaking to move all your infrastructure where to Kubernetes like how, had that come about, how did people decide that they wanted to do that? It seems like a difficult project to get onboard with.
[0:02:39.4] WJ: Yeah, hard to get enough buy in to move an entire organization with a massive software offering on to a completely new infrastructure tool like that.
[0:02:49.7] CG: Yeah, it was definitely quite a challenge and it was one of those things that it started out as a dream that for many years, even at the early onset of the early stages of the like systems, eventually this is where we will be or we could be and it’s going to be awesome.
[0:03:06.0] WJ: I have a dream.
[0:03:07.0] CG: Yeah, I have a dream. That’s where it was for the longest time and then it started to be prioritized, we knew it was going to happen and then we got to a point where some of the bones of the legacy systems started to have some like, aches and cracks and what not and it was at that point as some things started to come up that it was like, “wait, if we move this into Kubernetes now, this goes away? Okay, let’s really talk about this.”
[0:03:36.4] DA: That’s great, the stars really aligned about the business need for this thing, as well as like the fun technology challenge that on this side of the room are like really psyched about and I’m sure like initially, you were kind of a bit perplexed about it.
[0:03:53.8] CG: Well, I was always perplexed is a really good term. I was like, “I know it’s this thing, it’s a project that we’re embarking on.” But seeing the excitement of a lot of the engineers at work who had been with the company for a while, we’ve been talking about this and like, we’re finally able to do it and like, it’s a really new exploration and actually getting to research it.
I could tell how excited people were getting for that and then translating it into – from a user perspective of like, “wait, we don’t need downtime anymore? We’ll be up, that means our students can do the stuff that they need to do early in the morning.”
We don’t’ need all of this extra communication beforehand. Just things like that, that we were able to get – I was able to get excited about from a user perspective, but also our engineers, I could see it on the faces and be like yeah, this is going to be big.
[0:04:45.6] EG: Yeah, I remember, I’ve been a consultant at 2U for almost two years now and there are certain things we could only deploy every two weeks because they depended on the legacy system that can only be two weeks, on Thursday mornings with a three-hour downtime.
We got to always kind of coordinate our PRs to be merged into master a certain time in the new app we were working on because the old app, it depending on something the old app. The changes there had to be – we couldn’t tag and deploy that thing until the old app, the old legacy system had the didu code for – I would support it. It was really one.
[0:05:29.5] DA: That’s going to be challenging especially since you guys probably had your own deadlines that you were working on.
[0:05:35.3] WJ: Yes. I mean, there were times where it’s like, hey, we need to – we got a fix for this big bug, it needs to go out this Thursday. If it’s Tuesday, all that urgency starts to kind of fall on to you and you're like, okay, we have to totally shift our priorities now to make sure this gets done for Thursday.
[0:05:54.6] EG: Yeah, that’s disruptive.
[0:05:55.3] CG: Yes.
[0:05:57.5] DA: What was it like transitioning from being like you know, a domain expert who knows the business and like can really call the shots in some ways about like, you know, what’s important in different scenarios and then kind of move into a technical team where you’re kind of like the odd man out?
[0:06:16.0] CG: It was really tough. I think for two reasons, one, just not having the very specific technical expertise like my background isn’t even very technical. I’ve been in operations most of my career. To be able to get into something that was super technical and not be able to kind of explain it. For me, that was very tough. Like you kind of existed in the ether and I couldn’t visualize it the way, like you could a user interface of like, doing acceptance criteria’s, business analyst of saying, “okay, I can see this, I went through this whole test flow, this is good, I can accept this.”
There was a lot of having a lot of trust in the engineers who were having the discussions about what it would look like and saying, “okay, if you think this is what it should look like and this is how it should be done, great.” Then just giving the input of like, “how does this now affect our users? What does this ultimately mean? What are the consequences of this?”
[0:07:10.2] EG: Usually the business analyst would be the one who would accept stories, QA them, give the engineers feedback with things. How are you able to do that or were you able to do that in this project?
[0:07:21.5] CG: Yes and no, there were some tickets that I could actually accept because it did end up like affecting a user or I need to do regression on this one piece of functionality because we kind of – we have touched it or we’ve changed it as we have moved it over.
But then there are a lot of things that are just happening in the background and behind the scenes that I couldn’t test at all. Then a lot of trust went into the engineers of even just them working with one another to kind of say, “hey, no, maybe we should do it this way or –.” They had each other’s backs that way. I think we were able to function really well.
[0:08:01.4] WJ: It sounds like the role became less and more of like a BA and more like – I don’t know how to describe it, a glue guy? It’s a weird way to say it, right? Between sort of helping the team like you know, did you help with the backlog and I know, did you help with like communicating what was happening to other people without their other company and things like that?
[0:08:25.1] CG: Yeah, one of the big parts of my role was definitely the communications, both internally setting up success for our team to communicate even, if that was something simple like having the Slack channel created, saying like what is our cadence of standups and retros and stuff like that and making sure that we actually had a structure of communication that we could rely on. Also, communicating externally, that allowed me to give the team space because if I was doing my job well, letting other people know what was going on, they could call out things that we may not have caught early on, they could give their input that way.
Then also, trying to coordinate things like deployments and letting our support teams know what was going to be happening, so that we can manage their expectations and give ourselves the runway to deploy when we need it or even take extra downtime when necessary, which was something we did navigate.
[0:09:21.7] DA: Right, translating what the status of all these nerds are doing.
[0:09:27.8] CG: The download essentially of okay, “you have said all of these things, this sounds very good, I can see that the team is excited or maybe not excited.” Maybe some things like something’s up. Being able to pick up on that and dig into that and say, okay, what does this mean for our end user? What does this mean for other teams we’re working with? Who do I need to talk to?
[0:09:51.4] DA: Did you start to have a better understanding of Kubernetes and the technology that was like underlying the platform? What was something surprising that you learned about it along the way?
[0:10:01.2] CG: I think understanding what containerization meant and how pods were restarting, so that the application could keep running, but that pops could be restarting during that time meant that we could essentially keep our platform running without having to take it down. But we could do the deployments that we needed.
That was one of the technical things I picked up, I was able to understand and translate into layman’s terms and then ultimately with the big keyword being no down time now.
[0:10:33.7] WJ: The whole business value was no more, no downtime deployment.
[0:10:37.6] EG: No more downtime deployment.
[0:10:39.3] DA: That’s the dream of orchestration.
[0:10:41.7] CG: Even being able to go to the – “hey, we need extra downtime, we’re working on Kubernetes. What does that mean? Yeah, we’re not going to have downtime anymore.” They’re like, “okay, if you need anything else, let us know.” This sounds great, that wasn’t really a hard sell to our stakeholders once we had the resources to focus on it.
[0:11:00.8] DA: Just making friends.
[0:11:01.8] CG: Yeah, pretty much.
[0:11:02.7] WJ: were there any metaphors that you found helpful for understanding what was going on? You know, Kubernetes, they talk about the shipping as part of the core of the metaphor like we talked about what the word actually means.
[0:11:15.2] DA: Shipping, I love it. I didn’t get it.
[0:11:20.9] WJ: It sounded like the pod explanation was helpful, were there any other metaphors people used?
[0:11:25.2] CG: I don’t think there were too many other metaphors, which I probably could have done a better job to like boil it down and explain it. But I think, one of the things that I appreciated about Kubernetes and even about the engineers I was working with was it was simple enough to understand that concept and what that meant for our technology.
[0:11:44.7] DA: Yeah, I like front of the show, Dane’s explanation of just jars and boxes and I don’t know, he –
[0:11:54.6] EG: Jars and boxes?
[0:11:56.2] DA: The containers are boxes and you try to stack them up in the thing and I can’t do this.
[0:12:04.5] WJ: Yeah, there was a wonderful metaphor that a friend of the show, Dane O’Conner, provided for a thing that was working on Kubernetes and it goes like this. Imagine that there were a bunch of vases. You make a vase and I make a vase and Emanuel makes a vase and –
[0:12:20.2] DA: They’re all beautiful, in their own way.
[0:12:22.1] WJ: Totally unique and perfect and special, just little snowflake amazing vases. Then you need to get those vases to the store where you were going to sell them for example. If you try and stack them on top of each other because this one’s pointy and this one’s round and this one’s square and this one’s oval. There’s no way you’re ever going to get them to stay stacked so what do you do? You put them in containers.
[0:12:49.7] CG: I love this.
[0:12:50.4] WJ: Every single one goes into a little box and the boxes are all standard and then you can stack up the boxes. What Kubernetes helps you to do is stack and move your boxes.
[0:13:00.0] CG: That’s deep.
[0:13:01.7] DA: I don’t know where the boats come in.
[0:13:04.1] WJ: The boxes go on the boat. It’s the shipping containers.
[0:13:08.6] EG: Shipping container. Maybe the vase is just giant vases that need to be put in shipping containers.
[0:13:15.6] WJ: It’s really a better explanation of docker I think than it is of Kubernetes.
[0:13:22.7] DA: We’ll ask Dan if he’s figured out the next level of metaphor.
[0:13:26.6] WJ: For me, the metaphor for Kubernetes was the conductor. Kubernetes is the conductor of the orchestra and he’s orchestrating the cloud. He’s saying “okay, you know, the flutes in the back now is your time, you know, flute is number two, you’re going to take over for flutist number one, now we’re going to swap flutist number one out because he needs some maintenance.”
[0:13:47.6] EG: The music needs to keep playing.
[0:13:49.5] WJ: Right.
[0:13:51.0] EG: You can’t skip a beat.
[0:13:52.2] WJ: Right. It doesn’t really fit in with the whole shipping thing though.
[0:13:58.3] CG: Still works, I feel like these are much better metaphors than we originally conjured up. Keep the jars in the boxes especially. I remember one of the ways that our engineering lead put it, he’s like, “we have this legacy technology, it’s getting old, it’s kind of a pile of crap. We’re putting the pile of crap in a shiny new box; it’s going to look really good.”
[0:14:22.8] WJ: Yeah, that’s all you need to know.
[0:14:25.3] CG: You’re like, all right, cool. The ultimate point of view is getting to is once it’s in here, it doesn’t mean it’s done. Just an FYI, we’d still need to maintain this and all that. He was getting to a conclusion, but I was like “god, okay, cool.”
[0:14:38.5] WJ: At its core, it’s a pile of crap.
[0:14:42.8] DA: It will go very fast, you know?
[0:14:45.8] CG: It’s never been faster.
[0:14:47.9] DA: Yeah, as long as it’s doing what it needs to do, that’s what’s important.
[0:14:51.9] WJ: You put either the beautiful unique vase or the pile of crap in a shiny new box, was it all to go back to the sailing and shipping, was it all smooth sailing?
[0:15:02.7] CG: Definitely not. At the inception, that was the toughest time because a new team essentially had to come together to focus on this project and with this new team coming together, there were a lot of, it was new people working together which is always exciting, but also you need to kind of figure out the dynamics.
[0:15:20.5] DA: Yeah, storming. Part of the metaphor.
[0:15:25.7] EG: You were in a storm
[0:15:29.1] CG: There was a storm at the beginning. Very much so.
[0:15:31.1] DA: Odysseus
[0:15:32.3] WJ: Tempest.
[0:15:33.7] EG: A Tempest person.
[0:15:36.5] CG: I feel like there might even be an element for the Sirens come in there, trying to get people of like “hey, here’s this other thing, you want to come do this?”
[0:15:45.2] DA: The Sirens.
[0:15:45.7] CG: No, one of the engineers went to work for another company. It was not because of this project, but talking about Sirens getting people off of the boat.
[0:15:57.7] DA: Sure, yeah. It’s also like a big block of work, something that’ shard to visualize what done even looks like. I’m sure that was a challenge.
[0:16:08.3] CG: The visualization of it all was hard. It was more of a conceptual visualization of like we will get to this feature state that was the vision in the terms of like hey, what could the future look like? When we talk about actually visualizing something the way we would a UI, that didn’t exist.
We got glimpses of that through demos of our engineers pushing the button in Jenkins and then we see the pipeline go green and it was like, “this is awesome.” Yeah, buttons, that’s good. But then like, when you start to dig in to like what does that really mean? You're then showing something like, normally this took three hours, how long did this take? Two, three minutes, all right, cool.
You see at that one and you’re like man, this is awesome. That gets people real jazzed.
[0:16:53.1] DA: How’d you help like manage the work for a team like that where like, the back log is innately technical like it’s not immediately a user driven feature?
[0:17:03.2] CG: I think that started early on with planning, I think that some preliminary work had been done so we knew buckets of work and that was kind of went into their own epics. As we really like buckled down and said hey, if this is a deadline we’re driving to, what is the most important, it was kind of ruthless, like during that time, you throw out like documentation you were planning on doing.
This will come afterwards; this is not going to happen right now. Or what is anything that needs to happen by this date, that should be the most important stuff. Just making sure that the backlog only had bad in it and remove all other distractions. Sometimes you actually, an engineer might pick something up because they think it’s important and then we would have to revisit like in the engineering lead and I would coordinate a lot on this of like “okay, do they need to be working on that right now? Probably not.”
Let’s call that out before they spend too much more time on it. Always coming back.
[0:17:51.7] DA: What kind of things ended up being deprioritized there or –
[0:17:54.7] CG: Documentation was one of them, there were some pieces of alerting that we wanted to kid of take out, we got a lot of it in but then there was this like dream world of where it could all be, what it could all look like and that kind of got pushed aside, more technical aspects that I don’t even know if I could speak to.
[0:18:13.5] WJ: Yeah, it’s easy to get carried away with these things, when there isn’t really anything user facing for anybody to see. Like, wow, I could work on this indefinitely.
[0:18:25.5] CG: Yes. There was plenty of work to do definitely.
[0:18:29.7] DA: What are some big lessons learned from this, this was a major undertaking.
[0:18:35.0] CG: I think one of the biggest lessons learned and this might be just appreciating the importance of it, but trust. Trusting in people who aren’t two different sides of a spectrum of a very technical and technical work done and someone who is not technical who is responsible for protecting this team’s time and energy and handling all the outside stuff so that the engineers can actually get into what they need to do.
[0:19:01.9] DA: Trust that green button and Jenkins actually mean something good.
[0:19:07.7] CG: When you have the conversations of like, “hey, this is important, we should do this.” Actually, not being afraid to push back and say, “why do we need to do this? Is there another way we can do this?” And asking exploratory questions like that, even though I may not know the technicalities of it, that would at least facilitate more thought and different approaches collect like for our group to tackle collectively?
[0:19:31.6] DA: Stop people from getting like stuck in the weeds a little bit.
[0:19:34.3] CG: Right.
[0:19:35.0] DA: What about some things you’d like to avoid if you had to do it all again?
[0:19:37.8] CG: I think we would avoid or maybe kind of the inverse of that as almost we could facilitated knowledge share better off the bat and talked about that. I think engineers got into some silos and that kind of made conversations later on difficult and it kind of just said, “okay, this is your expertise, this is what you're going to work on.” Which might be tough when you’re going to a deadline but at the same time, if you get ahead of that and start knowledge sharing, I think that pays dividends later on.
[0:20:06.6] WJ: If you were advising a friend who is considering doing a Kubernetes project.
[0:20:13.3] DA: Is that you?
[0:20:15.6] WJ: No, not at the moment. Maybe later. What would be the things that you would ask them to consider or what would make you recommend to them, yes, now is the time to get in Kubernetes?
[0:20:25.1] CG: I mean, just seeing how much smoother our application is, a reduction in incidences and degradation of service and also the abilities like “hey, there’s a lot of use happening right now, we need to spin up more containers.” That’s a good thing. We are able to respond when it looks like the system is slowing down which is amazing and also just being able to deploy whenever we want makes our lives so much easier.
If there is a critical bug or something like that, we can put that out pretty quickly. It allows us to deliver value to our users more frequently and in smaller chunks and by doing that, we’re also reducing risk of deploying a lot of things at once that can kind of trip over one another, which is another issue we ran into when we were doing biweekly deployments.
[0:21:15.9] DA: Continuous delivery.
[0:21:18.6] WJ: If you had to do it over again, what would you do differently?
[0:21:21.6] CG: I think we would have managed expectations about, around what was expected of the team. Out to management, when this team came together, there were a couple of different assignments put on, it’s like, “okay, you need to move all of this over to Kubernetes for this one partner in particular, but we also want you to maintain the legacy platform.” We kind of sat on that for a while and we kind of danced around and tried to avoid it and I think, having had a conversation earlier on with the engineering lead to myself which we ultimately did and we had multiple conversations.
If we had gotten ahead of that, I think we would have saved ourselves a lot of time and a lot of unnecessary stress.
[0:21:59.3] DA: Cool, have you ever pushed the button yourself? The deployment?
[0:22:03.3] CG: No, that’s probably, I don’t even know if they would let me do it.
[0:22:09.0] DA: I don’t know.
[0:22:09.3] CG: It would have to be under strict supervision.
[0:22:11.3] DA: What do you think?
[0:22:12.5] EG: I think I can definitely grande. You’re definitely the Kubernetes of the ship and you’d be able to steer it wherever you want.
[0:22:22.9] CG: I don’t know about that. Now that it comes up though, I think I do want to press the button.
[0:22:29.0] DA: Awesome.
[0:22:30.7] WJ: Somebody needs to Arduino that and get an actual big red button, that would be so fun.
[0:22:37.0] CG: That would be amazing. I am sure we can swing that somehow, I’m sure the security teams would probably not like that, but we’ll see.
[0:22:46.6] WJ: Maybe it’s part of like a two-factor thing, you have to have the button and know the password.
[0:22:51.4] CG: Or maybe you need two keys like if it’s like you’re in a nuclear submarine, you need to turn the keys at the same time.
[0:22:58.7] DA: Wow, that’s not expected.
[0:23:01.1] WJ: You have to be playing welcome to the danger zone.
[0:23:04.4] CG: I think I have my next hackathon project. This sounds phenomenal.
[0:23:09.8] DA: Chris, it was great for you to take us along on this voyage, really appreciate it, once and future guest.
[0:23:17.3] EG: I was happy to be a passenger on the ship.
[0:23:22.1] DA: William, it was a pleasure.
[0:23:23.4] WJ: Yeah, great view from the crow’s nest. Just keep going. Does anybody else have anymore puns?
[0:23:31.4] DA: I’m just here swabbing the deck. Yeah, how can people get in touch with you?
[0:23:37.2] CG: You can find me on LinkedIn under Christopher Grande.
[0:23:40.6] DA: The one and only?
[0:23:41.4] CG: The one and only. I’m sure there are many of those.
[OUTRO]
Follow us now on Twitter @radiofreerabbit so we can keep the conversation going. Like what you hear? Give us a five star review and help developers just like you find their way into the rabbit hole and never miss an episode, subscribe now however you listen to your favorite podcast. On behalf of our producer extraordinaire, William Jeffries and our amazing host, Michael Nunez who is out being a dad and me your host, Dave Anderson, thanks for listening to the Rabbit hole.
[END]
Links and Resources:
