DevOps at Five: Where Are We Now? | GOTO Conference 2014
🚀 Dive into the fascinating journey of DevOps over five years! 🌐 Michael T. Nygard shares insights on the shift from risk management to continuous delivery, and the importance of quick recovery. 🛠️ #DevOps #ContinuousDelivery #RapidDeployment #Monitoring #QuickRecovery 🎥 Watch now: https://just3ws.github.io/interviews/michael-t-nygard-goto-conference-2014
The Interviewer
Mike Hall
Interviewer, UGtastic
The Guest
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
The Conversation
Mike Hall
Interviewer, UGtastic
Hi, it's Mike with Uptastic. I'm here again at GOTO Conference 2014. I'm standing here with Michael Nygaard, who gave a talk on DevOps at Five: Where Are We Now? Thank you very much for taking the time to speak with me. Can you tell me a little bit about DevOps at Five? What does that mean, and how did you come to that presentation?
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
Sure, I'd be happy to. So DevOps was actually coined by one individual. We know when the name was created. It was created by Patrick Dubois in 2009.
Mike Hall
Interviewer, UGtastic
Okay.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
So that was five years ago, and five years is an interesting time span. First of all, it's a round number and humans like round numbers.
Mike Hall
Interviewer, UGtastic
Yeah.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
But it's also sort of the time it takes any idea to go from radical to kind of entering the mainstream. It's about half the time it takes for an idea to become completely mainstream, and it's about one quarter of a normal career for a working programmer, or what you would call the time from entering the field to being a grizzled grape-beard type.
Mike Hall
Interviewer, UGtastic
Yeah.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
So, five years was an interesting time to sort of take stock, collect up many of the ideas that people have created, sort of provide a survey for folks who haven't been in it for that time, but maybe are hearing the term, are interested in what it's about, and frankly, also to give some cautions or some warnings.
Mike Hall
Interviewer, UGtastic
Right.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
And so I wanted to do all of that in kind of one 50-minute package.
Mike Hall
Interviewer, UGtastic
Yeah.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
And that was my talk. And I have to say, when I heard five, I was thinking more of, like, the volume that it's starting to get to the point where people are talking about DevOps more. And that's been kind of a theme a little bit here at GOTO Conference is about lean and lean enterprises and continuous delivery and taking this term DevOps and saying, "No, this is really something that we really need to think about. " Yeah, definitely. And we heard that theme echoed by Adrian Cockcroft in his keynote that opened the conference, where he talked about the trajectory at Netflix from 2009, where people were saying, "No, it'll never work," to somewhere around 2012, they say, "It only works for special unicorns like Netflix. " Right. To 2014, where, you know, a lot of companies are saying, "We're all unicorns. " Yeah, we need to do this, too. And that's definitely how DevOps has gone as well. It's gone from this idea that says, "Well, it's fine for web-native companies like Flickr and Etsy, but it doesn't do me any good in my, you know, retail organization. I've got brick-and-mortar stores to worry about. " Right. But as time has gone on, we've seen that the ideas are quite generally applicable.
Mike Hall
Interviewer, UGtastic
Right.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
In an earlier interview, we talked with somebody who talked about how they were using continuous delivery, working with the restaurant industry and able to deliver a change 10 hours before opening of a major bar. And that these concepts can be applied into serious product development that actually really touches an end user and what might have previously been considered a, "Oh, my God, we're in a freeze mode. " Don't touch anything. Yeah, definitely. When you look at the reason for freezes and the sort of traditional deployment process, a lot of it is around risk management. The idea of saying an event, a negative event, has a high cost, and so we need to avoid that cost.
Mike Hall
Interviewer, UGtastic
Right.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
I would turn that around these days and say preventing that event also has a cost. It's not as visible in the immediate sense, but it's a cost that you feel in the form of friction and delay in getting things done.
Mike Hall
Interviewer, UGtastic
Right.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
And so we've changed to a different style of risk management that says, "I'm going to try and reduce the chance of a bug slipping through to production, but I'm also going to make sure that I detect it immediately and can push a new release that eliminates the bug immediately. " Right. And so in some sense, the number of opportunities for error goes way up, but the cost of any individual error goes way down because you're limiting the scope. So is it trying to turn what is previously known as a risk period, like once we get into a certain period before it goes live, we have code freeze, to change is now not a risk. It's just part of the way we approach the process. Well, I'm a little cautious about that because there still is some risk associated with it. And if you do, if you don't adopt any of the other practices and you just go to fast deployments, you're going to incur some damage.
Mike Hall
Interviewer, UGtastic
Okay.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
As you speed up the deployments, you also need to speed up your monitoring and your metrics. You need to make sure that you can detect the problem very quickly. So focus on that mean time to detection. I've been in many places where the mean time to detection is measured in weeks. It needs to be seconds to really say you're doing the DevOps continuous delivery approach. And likewise, you need to be able to repair the damage very quickly. Adrian talked about detecting a bad deployment within five seconds of when it goes live.
Mike Hall
Interviewer, UGtastic
Right.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
And automatically switching back over to the old system that's still alive and running. So in that case, you say a 10 second error, how many users are going to hit that 10 second error?
Mike Hall
Interviewer, UGtastic
Right.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
So again, I'm trying to put it into dollar cost terms where I'm comparing the expected losses from bugs slipping through to production versus the expected losses of bugs that are already in production that you can't fix because you're in a freeze.
Mike Hall
Interviewer, UGtastic
Yeah.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
So to me, it's almost like you're trying to say we're going to reduce that surface area of that first group that hits a new change. And then once we know because we've kept that surface area small, we can grow it out and then be able to retract. Absolutely. I think of limiting it in space and time. So limiting the exposure in space is how broadly is my audience exposed to the new code right away. So we use techniques like feature flags and differential routing to let a few people in at a time. And then limiting the scope in time is how quickly can I fix the bug when it gets through.
Mike Hall
Interviewer, UGtastic
Right.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
So, yeah. So how the speed of, or even being able to turn it off because like when you strike the feature flag is, oh, that one didn't work. So that could be that instantaneous. Absolutely. Mitigation of risk. So the term DevOps, though, it's still been kind of a high level term. And we've talked a little bit about monitoring and continuous delivery. What, what, when you say the term DevOps, so can you kind of break that down into kind of pieces a little bit more? Yeah, I'll try. Actually, the schema that I would use comes from John Willis, who talked about DevOps as culture, automation, measurement, and sharing.
Mike Hall
Interviewer, UGtastic
Okay.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
And culture comes first for a good reason. So the idea with culture is that we want to create a high trust culture where we have strong and deep collaboration between development and operations. This is why, by the way, I consider it a fallacy to hire a new DevOps team that sits between Dev and OBS and is supposed to be a bridge. Because instead of creating a tight junction, what you've really done is now made two handoffs instead of just one. Yeah, and they're, they're alien and now there's friction. Yeah, absolutely. And both sides feel threatened by this new team. So the culture is one of enablement and mutual support. So we're previously, operations is frequently held accountable for code they didn't write, right, and the availability and performance of said code. Now we would close that feedback loop and say operations is going to enable development to move things into production as rapidly as possible and also give development the tools to see the effects of what they do.
Mike Hall
Interviewer, UGtastic
Right.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
And now the responsibility is on development to use that ability wisely.
Mike Hall
Interviewer, UGtastic
Right.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
So, instead of focusing on process and automation for self protection, you're focusing on automation to enable your partners and collaborators in doing this. So that's, that's kind of the culture that we want to create. All of the tools, all of the monitoring is in support of that culture. And, uh, you know, in what I'm thinking about when you're talking about the information and not looking at, um, those metrics is how do you kind of protect yourself, but we should say mitigate risk, but, uh, or find blame, I should say. That's what, that's what I was trying to look for is I think about if I'm running an app and I launch it on my, uh, on local machine, I'm watching my CPU and my RAM. I just want to know how it's doing. And that's a normal thing that I do when I'm writing code locally, but that we should also maybe look at our entire systems at that same level of, of not detachment from what it's, what it's, uh, saying that it's not something that we're looking for.
Mike Hall
Interviewer, UGtastic
To protect ourselves from, uh, blame or faults, but just how is this system working?
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
And yeah, in fact, I would go even farther and say, um, when I'm putting code into production, uh, I'm not just interested in how it's doing on RAM and CPU. I'm also looking at the effect on the users.
Mike Hall
Interviewer, UGtastic
What's the response time distribution? Uh, what have I done as far as average latency and the 99th percentile latency? What have I done in terms of conversion rates and revenue coming into my business?
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
You know, these are things that developers, uh, can care about and want to improve.
Mike Hall
Interviewer, UGtastic
Right.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
There's a much older idea that says, you know, developers don't care about the business. They don't understand money. They don't, uh, you know, as long as they get to play with their code, they don't care what happens to the business. Um, I found that to be, uh, a foolish stereotype. Um, and so now you'll have developers who will optimize a chunk of code in order to improve the response time and get the conversion rate up.
Mike Hall
Interviewer, UGtastic
Right.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
So we need to think more about the, when I'm having a profiler tool that it isn't just looking at bits and bytes, but it's looking at key transactions that are meaningful to, to the customer's experience and how they're going to interact. Absolutely. So all of these tools exist, um, but they're normally fragmented. So the operations group will have the system monitoring. Um, uh, you know, maybe you'll have a user experience group. That's got the, uh, real user monitoring system. Uh, and then you'll have the marketing group or the analytics group. That's got the, uh, business significant metrics. They're all being collected different ways. That's why you get 60 tracking pixels on a single webpage.
Mike Hall
Interviewer, UGtastic
Yeah.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
Uh, and they're all going into different databases. We actually need to bring them all together. So I can see a single graph that shows me, um, I don't know, IO saturation, a vertical line where a deployment went out and the conversion rate all overlaid on the same graph. So I can see, look, that, that went out and crap, the conversion rate went down.
Mike Hall
Interviewer, UGtastic
Yeah.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
So here's the next deployed went back to where it was supposed to be.
Mike Hall
Interviewer, UGtastic
Yeah.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
Or for sure the CPU or IO went up, but conversions went up higher. Yeah, exactly. And that's going to offset us buying more machines.
Mike Hall
Interviewer, UGtastic
Exactly.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
Those kinds of thoughts can be done instead of just, oh no, we've.
Mike Hall
Interviewer, UGtastic
Right.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
And so the lean thinkers would say that this is, uh, optimizing the whole value chain rather than locally optimizing pieces of the value chain.
Mike Hall
Interviewer, UGtastic
Right.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
And that's true for the, the actions we take to deliver the software as well as the value chain of the software executing in production.
Mike Hall
Interviewer, UGtastic
Yeah.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
So it's looking at changes in a holistic manner and just seeing how they reflect off of each other and whether or not, if there's an imbalance, whether that actually is okay.
Mike Hall
Interviewer, UGtastic
Okay. Yeah. Okay.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
Um, when somebody goes to your talk, if there's one thing that I, as a developer or a product manager, you'd like them to take away from that talk and go back to their, to their companies, to their projects and try to put some of these new ideas into practice.
Mike Hall
Interviewer, UGtastic
What would you like them to, to take away the most? Right.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
So I think the number one thing I would like people to take away is you can make these changes yourselves.
Mike Hall
Interviewer, UGtastic
Right.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
You don't need permission. You don't need, uh, approval from on high and you especially don't need to buy, you know, DevOps in a box.
Mike Hall
Interviewer, UGtastic
Right.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
Or buy someone's new tool or old tool that's been relabeled, you know, DevOps edition or something like that. Um, DevOps is a style of working and it can start at the grassroots. Um, and you know, if you show some results with it, uh, you'll get more support.
Mike Hall
Interviewer, UGtastic
Right. Yeah.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
And that's just how things go. What was it that, uh, I think it was, I'm going to say, I think it was kind of backed that said, don't tell them you're doing Agile.
Mike Hall
Interviewer, UGtastic
Yeah.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
Cause they don't understand what you do anyway.
Mike Hall
Interviewer, UGtastic
Right. Yeah.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
Cause the only answer is no.
Mike Hall
Interviewer, UGtastic
Right. Right. Yeah.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
Then they'll come and ask how you did it.
Mike Hall
Interviewer, UGtastic
Yeah.
Michael T. Nygard
five years of DevOps at GOTO Conf Chicago 2014
They'll want you to replicate it elsewhere. Or at least they'll very much get out of your way.
Mike Hall
Interviewer, UGtastic
Well, thank you very much for taking the time to speak with me. Glad to be here. [Music] tastic. com.