Episode #17 | What is DevSecOps? 

Announcement: Broadcasting from Fairfax, Virginia. You are now tuned in to The Highlight Cast with your hosts, Adam McNair and Kevin Long.

Adam McNair: Welcome to another episode of The Highlight Cast. I’m here, Adam McNair from Highlight, joined by Kevin Long. Hey, Kevin. Hey Adam, how’s it going? Good to see you again. Good, good to see you. Also joined by Kevin Milner, who is our northeast Kevin. It’s good to see you, man. How are you? 

Kevin Milner: I’m good. Thanks for asking. How are you? 

Adam McNair: Good. Um, alright, so what we were going to talk about today, we’ve talked about in a recent episode, we talked about a code challenge that we had done, and as we, uh, We look at the work that we do, those kinds of code challenges seem to fit into the DevOps or DevSecOps work domain. So we wanted to talk a little bit more about that. Because the last episode where we talked about code challenges, you could theoretically do a code challenge without code. 

Kevin Long: I Mean, we literally just did a style of code challenge for tool chain and a style of code challenge for knock, 

Adam McNair: right? Yeah. So we’ve, we’ve done infrastructure. We’ve done kind of COTS tool integration. We’ve done different kinds of code challenges. So it doesn’t necessarily need to be DevSecOps. And, um, We wanted to take the opportunity to delve a little bit more into that. I will lead off by giving you my deep expertise in DevSecOps. I know it’s development and security and operations integrated together. And, uh, I, I’m trying to remember the first time I ran across it. I, I think it might’ve been in DHS several years ago. It might’ve been at, uh, PBGC. I don’t recall exactly, but I know the idea that you could use tools to, you know, streamline the development code to delivery process and the way that I, I took it. Initially was if you if you started thinking about the way we used to build software or it was this waterfall type thing where you each step that you took in the process kind of took a long time and was super deliberate and, um. A lot of documents and a lot of signatures. Yeah. And, um, you know, I, I think that when, when Agile showed up, at least for me, from a non, you know, I am not a hardcore software developer. My lay person’s feeling about it was that you were kind of just speeding all those cycles up. And there might be some steps in the process where if for some software, where you really need a zillion page document and sign offs and everything. But a lot of things don’t require that and, and agile sped those cycles up, but also the idea, you know, the, the kind of catchphrase I used to hear was, you know, fail faster, you’ll find out if you’re going to have something don’t build. A million lines of code and then find out that one of them doesn’t work, right? Like 10 and see if 

Kevin Long: those 10 work because or, or worse, find out that it took you so long to write that code that it operates perfectly, but isn’t needed anymore. No longer needs the needs. 

Adam McNair: Yeah. And I will tell you, I, I had a project at departmental labor one time that it. It, uh, by the time we got ready to release, uh, like a year long software development cycle, uh, the system it was supposed to integrate with had been upgraded because that wasn’t under our contract. And so it didn’t work. And there wasn’t a mechanism in place to tell us, hey, uh, we’re going to turn system X off and launch a new upgraded version. All the back end is changing. We didn’t know. And it was a big problem. So, so those kinds of things, you know, okay, we’re going to, we’re going to do things faster. It’s just better real time visibility where it’s, I mean, nobody wants to do anything that you don’t really get a product out of for six months or a year, if you can avoid it. Um, it’s just a very long time to recover if you find out something is wrong and GSA, General Services Administration, what they call DevSecOps is safer software sooner. And so that’s kind of a positive angle on Fail faster, I think, so I, you know, kudos to them for coming up with something that doesn’t use failure in your, uh, in your value statement. Um, and I think GSA, they had an organization there for a while that, um, was really, I think, at the forefront of DevOps and DevSecOps, and, uh, 18F was an organization that was trying to jumpstart innovation and technology into the federal government. Because in the government, a lot of times, I think what happens is somebody asks the question, okay, well, this sounds like a really neat, cool, innovative thing. Where has it been done before? Which is not just a bureaucratic question. It’s a very valid one because the federal government has unique requirements from 

Kevin Long: the state. It’s just a different beast. 

Adam McNair: Yeah. I mean, I remember at one point when I was working on passport systems and I was talking to one of my customers and they said, you know, the thing is, There’s no COTS solution for issuing a passport. You’re like, oh yeah, that’s a fair point. It’s like, we have to have something that prints this highly specialized secure book and all of this stuff. You don’t find, uh, uh, an application where we can go borrow that from. 

Kevin Long: Right. 

Adam McNair: Okay. If 

Kevin Long: it was easy to find, it’d be easy to forge. 

Adam McNair: Yeah, absolutely. Right. Yeah. So that, you know, to me, DevOps or DevSecOps, it’s like, okay, we either development operations of software development, the security process and operations are integrated or mushed together in one word. And so it’s, it’s all one thing. You guys do this for a living. What do you, how would you describe it to a person such as, such as me that, uh, that doesn’t do it every day?

Kevin Milner: Well, what I would say is, is first of all, um, We when you think sort of you talked about the history of DevOps a little bit, and you know, they went from the waterfall to to the agile methodology and the idea of failing fast, what they noticed was that they were, you know, they would people would build a pipeline to take software from from code into a product. And then you would have your security people come in and say, well, can you also run these tests? Or can you also make sure that, you know, the container we’re releasing has the these particular versions of software on it? And so people would add in, you know, these, these Tests as they came up. And essentially, um, it was an afterthought. You’re, you’re tacking on security as an afterthought, uh, to the process. And that’s never particularly effective. So, so you could almost think of DevSecOps as DevOps 2. 0. Uh, it’s, it’s not so much different as, as improved with security built in from the beginning instead of, uh, Instead of added later on, 

Adam McNair: and so I’m guessing that when you do that, then are you avoiding the situation where you’ve written code? And then when it gets kind of further along in the process, it’s being security scanned. And they’re saying no, that’s incorrect, which means you have to go back and rework it. But it also means you could have other people doing kind of the same thing, making the same mistake. Right. So if you had some sort of the security litmus test in the middle of that, it kind of systemically stops it. Is that kind of the idea? 

Kevin Milner: Yeah. For instance, you might have a well meaning developer, you know, put a debug statement into the log file that says, you know, I tried this password. It didn’t work. Or I tried this password. It didn’t work, but then you’ve got your password in the log file. And so that’s a pretty secure, uh, pretty big security flaw right there. So, um, so the idea is, well, okay, let’s include tests to make sure that. That security, the passwords don’t go into log files and, you know, obviously that’s a mistake someone else might make later on down the road. So it’s a, it’s a chance to, to sort of make sure that doesn’t happen before you even get deeper into the testing phase.

Adam McNair: So here’s a question from a dumb person. How do you actually do that? So you want to make sure that there aren’t passwords in there. So what, what’s, is it a security tool? That’s just automatically out there. Sniff it. How does that, how does that happen? 

Kevin Milner: I mean, they, they have security tools that do that. Uh, Sono type. Um, so in our queue, uh, is one of those that has a suite of tests sort of to make sure that the security is stable, but it can also be done manually. You could write a unit test that that deliberately calls that API function and then after it’s called. scan the log file that was generated with, with, you know, a, uh, you know, just a simple text scanner to look for the password that you, you know, that you put it in the test.

Kevin Long: Yes. A lot like, uh, DevOps is based on tools that automate the integration and deployment of code. DevSecOps has tools that integrate into that whole tool chain that will do security checking on it. Like Kevin said, SonarQube will do a lot of static code scanning. There’s whatever Twistlock is called now, now that Palo Alto bought it. Um, You know, does container scanning and it does it’s things that that that integrate security pieces through the deployment and and development pieces of of of your software so that, you know, you don’t have a well meaning developer that that has this great idea, but only works if passwords are passed in plain text over. Non HTTPS and things like that, and then everything falls apart, right? Or security decisions that security decisions are made early and tested throughout each incremental deployment.

Adam McNair: So then one could think of the, that’s the security piece of the development and the ops piece on the development side. The, the idea of. Doing some coding and then doing some testing and looking at, you know, performance metrics and some of those kinds of things. So there are tools that are automating that, too, essentially. Is that right?

Kevin Long: Yeah, absolutely. It is part of this, uh, DevSecOps CI CD pipeline will include things like SonarQube and tools that do static and dynamic code scanning. Um, will also include things like unit tests. Built in also for, you know, that you would have in, in standard DevOps, right? If you’re doing TDD or TDD, uh, development where having the developers pay attention to all of the things that need to be tested throughout the life cycle of the, uh, of the development.

Kevin Milner: And then you might also have other, other things like, um, you know, tools that test to see if certain edge cases are, are covered in, in your, in your software. For instance, you might have a, a situation where somebody on submitting a form, uh, can actually include an SQL injection into, into the, the payload of the, the form. And you have to make sure that. You know, that’s sanitized and doesn’t let someone get control of your database just by, you 

Kevin Long: know, or drop your, drop your master database or little johnny drop tables. Yep. Little Johnny tables is what we call them.

Adam McNair: And I, so then on the upside, you know, I, I, I would think of that. Is that kind of similar to when you think of like the install shield, you know, wizard type thing that, you know, when you, when you buy professional COTS software, there’s usually a click here to install as opposed to, you know, I have done in the past where we were trying to deploy something and you had, Documentation about where all these files were supposed to go, and what was supposed to be where, and what was supposed to be configured in what way, and then once it went in, what was supposed to be changed on the server to allow it to work, and, you know, like, all this stuff, right? And it used to be I mostly did just relatively simple web systems, but it still might be a day or so, and depending on how good your documentation was, when somebody would forget to tell you that, oh, by the way, you have to either open this port, or it needs to have an empty folder or directory called such and such to drop log files in, and then if it isn’t there, the thing doesn’t work. thing will die or, you know, whatever. It was this big, long list of things you had to do. And so I’m assuming it kind of converts the custom software deployment aspect to more of a, here’s the install shield that has been told, go do all these things. Is that kind of the way that works? 

Kevin Milner: Yeah, with, uh, I’m sorry, Kevin, I didn’t mean to interrupt, but with, uh, especially if you have a system. Like, uh, infrastructure is code that allows you to be able to build, you know, the platform you’re deploying on the install shield would, would be more relevant for somebody installing the software locally on their, their workstation. But, um, one of the things you can do. with infrastructure as code is have it set up so that you build exactly the infrastructure, you know, the virtual machines have exactly the software that you know that you want them to have. They have the exact version, the exact ports open, all of that stuff. And it can be automated and therefore, um, you know, replicated quickly in a consistent and reproducible manner.

Adam McNair: Gotcha. So that makes a lot of sense. So that’s. You know, so we’ve automated essentially a lot of the manual steps throughout the process. And then there’s also a, um, um, kind of a mindset difference that, cause this takes the whole organization. Like you can’t just decide that your development contractor is going to do this. There’s a lot of. Overarching architecture and decisions and everything else that has to be made, um, when organizations start to go that direction, what kind of categories of benefits are they seeing? I mean, certainly it sounds like it could be faster or there could be less errors or there could be more secure. But what, like, what typically if somebody was going to say, Hey, we’re currently doing this, you know, we’re normal agile organization. And then we do whatever we do. Uh, what, what gets better?

Kevin Long: You’re finding security issues earlier in the development curve, where it’s much cheaper to fix. And if, and if you need to rearchitect something at that point, it’s with, with a million little deployments, instead of one monolithic thing going out, it is much. Cheaper to to take security into account as you’re going through without a doubt as as a manager. I know. I know. That’s the benefit that I see. 

Kevin Milner: I would say one of the main. Sort of, uh, I don’t want to say promises, but one of the, the early goals that you can present to, to your C level people, like your CTO and stuff is the ability to, um, consistently reproduce the environment. So one of the, One of the encounter problems you encounter as a developer is, is the works on my machine issue, um, you know, and that’s, there’s, there’s not a product manager in the world that will accept that as a, as an excuse, but, you know, I wrote this code. It works on my machine. Um, 

Kevin Long: all it has to do is run locally as admin and it’s fine. 

Kevin Milner: Yeah. So if you can say, Look, we’ll never have a works on my machine problem again, because the machine is always exactly what we expect it to be. Um, I think that would be a good starting place. You know, the deployment will always look like what we know it’ll look like and sort of sort of use that as your your motivator to start. Getting into the push to production phase of it, and then as they start to see the benefits of that, then you can include other things like code coverage. You know, I can guarantee that That in these known conditions, the program will operate in this way. I can prove it. I can show it to you or, um, you know, various things like that, that you can begin to, to add new features that bring new, uh, profit, not profitability, but, uh, benefit to, to the organization.

Adam McNair: And so that, you know, those things that I, I hear, so, you know, You don’t have to have rework as much, so if you’re finding things earlier, you’re not having to do as much rework maybe. You’re doing it, you know, you’re getting things done faster. That always equates to having it be done cheaper. That always equates to having it Deliver some value to the users faster, which is depending on the organization and what it is can be a really big deal. The other thing that it may be as it strikes me as I think about some of the other regular software projects I’ve been involved with over the years. When you’re going from a build 1. 0 to build 2. 0. The training curve and the usability curve for your users, where they, because you’ve lumped so much into that big deployment, right? And so you went from 1. 0 and you’re like, okay, now we also are going to add all these things in. We decided we’re going to reorganize these other things. We’re going to change the UI because we didn’t really look how it like how it looked that much. We’re going to add this search thing, but we’re going to have it work differently because we felt that there were some issues with it before. So And it’s almost like you have to relearn the application from one version to the next. I, I, I’ve been involved in a couple of those like that where, uh, you know, we had some big training room and you walk in and all these users are in there and the questions of like, where did my button go? And where did my this go? I mean, I, I would, I’d correlate it to the difference between like, if you look at the major versions of any of the Microsoft Office tools, if you find somebody that, that was using like 97 or 2003, And then they go into the current 365 version. It takes a couple of weeks, if not longer, to just get back to where the things that I use, like, I’m sure I can pull this. What the heck is this ribbon and where is my menu? Exactly, like, where’d the menu, you know, where did all that go? And yeah, versus, you know, like we’ve been on Microsoft Teams for a long time now. And every now and then there just happens to be a little tweak. 

Kevin Long: They just move things. 

Adam McNair: And you’re like, but it doesn’t completely destroy your ability to use the app. It’s just a momentary, like, wait a second. Didn’t there used to be. So where did this button go? Where did this button go? Now, the user communication would be, would be handy. Um, so, so certainly telling them what, what happened would be, would be outstanding, but from the idea of, you know, You think so many of our, our cloud based, you know, software as a service apps now, I’ve almost gotten used to, I think, going in and seeing like, oh, I guess that moved, it kind of gets you in a, in a slightly less kind of static role as a software user, where you’re not so accustomed, like, it’s not chiseled in stone, you accept the idea that this is not a typewriter, and for the last 117 years, all these keys have been the same place. It’s. It’s like, I’m going to look consistently somewhat aware of the fact that this is not a, a iron machine that has gears in it. It, things could move around and change. And so I think maybe as time goes on, users will get a little bit more used to, Oh, it moved from over here, not over there. You’re not going to be getting a bunch of help desk calls of like, where did the save function go? It’s probably still somewhere intuitive. It’s probably a major function. It just not, might not be the first, you know, floppy disk icon in, in the top left corner. Look around a little and you’ll probably find it. Um, so that might be a impact that I would think, you know, just from what I’ve seen of the way in organizations use, use proprietary software, certainly in the government.

Kevin Milner: Yeah, um, Adam, if I could address something that I sort of occurred to me as you were talking about that from from a developer’s perspective, uh, not necessarily, you know, the end user of the software, but from the developer’s perspective of someone working on an application, um, one of the one of the sort of moves that they’ve been going to for the last two years. 10 years or so has been micro micro services. Thank you. My brain just completely froze on me. Um, so as a developer, you’re, you’re frequently having to use external resources. Like, you know, I might need to reach out to the printer if you’ve got a situation. Like we had recently, where we had to have like a proprietary printer, um, that we needed to use, or, you know, we need to get information from this specific government website and bring that in and then format the data and present it. Um, one of the advantages of using microservices is that you can. You can do incremental changes to the various interfaces that those applications have so that the developer isn’t having to relearn like an entire monolithic, you know, front, uh, front end to communicate with back into communicate with, you can instead say, okay, well, I know, you know, the, the, uh, The print server has changed its format. So right now I can just worry about how to interact with the print server, then we can go and upgrade the criminal database server or whatever and and do incremental changes that way so that you can test them, make sure they’re good. Developers can get, you know, easy. Learning curve on it before having to move to, to, you know, having to learn everything just to make that one little change.

Adam McNair: So I guess that’s a concept that as a non developer doesn’t come along all that much, but I guess to your point is potentially daunting. So if you’re looking at software and it is written as these giant services, if you’re a new developer walking into that, before you’re an effective developer, you basically happen to just sit and learn the code base for some amount of time? 

Kevin Milner: Yes. Yes. The more complex the code base, the more time you have to spend learning it. Um, because, you know, most, most software that you encounter, at least from the code thing, is, is an evolved product over many years by many different people. So, um, You know, there’s, there’s obviously going to be different styles of coding, even code formatting if, if you want to, if you want to start a war, go on to Reddit and say, you know, open bracket on the same line is better than open bracket on a new line and then there’ll be hundreds of people to tell you why you’re wrong or right. Um, but yeah, so this, this makes it so that there’s a lot less of a learning curve. Um, if you have. If you have, you know, simple web interface, you can, you can even play around with a tool like curl on the command line to test it out before you go in to your, you know, uh, code base and try to try to make changes there. So definitely that has an advantage. 

Adam McNair: I think, you know, thinking back and with things I’ve been involved in, I, I know there was, uh, A team I used to work with and the lead developer, a fellow named Carl, was known as the guy that was kind of the godfather of all of their code. And I asked somebody at some point, like, well, could you do this differently or that way? He said, at this point, he’s had his hand in the thought process and the architecture of so many things that there is. The right way and the wrong way. And then there’s the way that he has done it, which is probably, you know, perfect in some places and might not be another, but you’re so far down that road that you’re like, everything is done this way. And so this services framework is what it is. And this is what we’re going to operate from because you just can’t, you know, you’ve, you’ve kind of, you know, you’ve already built the house, so to speak. You can’t. You know, could we knock half of it off? Yeah, but that’s not really how you do things. 

Kevin Long: Yeah, and if you’re exposing services like that, that are enormous to other applications, just like you were having earlier, where the application you’re integrating with changed when you’re. When you’re building, if you build giant services like that, you expose them out there. I mean, the backwards compatibility issues for, uh, for making, making the service do what it’s supposed to do going forward is, is, uh, it is a real hassle. Now, keeping them compact and finite like that instead of monolithic really makes, makes maintainability much, much better.

Kevin Milner: Yeah. And, and, uh, you can. You can do, this is the point I was trying to, I was struggling to recall later. You can, um, as part of the DevSecOps process, you can actually test at the same time you’re doing your other code tests, uh, you can test for syntactic style and various, uh, things in variable names, make sure that, you know, you have consistent variable names, make sure the code’s readable, formatted nicely. That’s really important in that it lets you have a more homogenous. Code base that, you know, looks the same, uh, and that makes it more readable. So you’re not having to translate everything as you go through, you know, Oh, well, this is, this is the code that Kevin wrote. You know, why would he have done it that way? Yeah. It, all the code looks like standard code for, for your organization. So that’s a, a really, another huge benefit of, of that. DevSecOps pipeline. 

Adam McNair: So does, does this help solve in some way, at least part of the problem where, I’ve walked into a lot of transitions where somebody would say, you got to try to make sure that you keep so and so because they are the, the lead developer on this system and there’s no chance that anybody else can ever maintain that thing. Which may or may not be 100 percent true, but I think what probably would be 100 percent true is, if anybody else is going to, going to handle this, they’re never going to be able to understand it the way that that person understood it. And they’re going to be largely ineffective for months as they try to figure out what this is. Uh, because I’ve had conversations in a transition before where they said if you can’t get This person to maintain this system, you are probably better off to start now rebuilding it because you’re going to have something at least serviceable in a couple of months, whereas you’re just going to. If you don’t, three months from now, you’re going to say, I still can’t figure this out. Does it help avoid that kind of, um, either through kind of immaturity of developer or I’ve certainly seen some developers before where I believed they did it because it was job security for the rest of their career that, you know, they had, they had built this Rube Goldberg type thing that nobody else was going to be able to handle. Does it help kind of protect against that? 

Kevin Milner: Yeah, what you’re talking about is, is called in the biz, the bus metric. Um, if you, if you’ve been hit by a bus, what is the number of days downtime before, before the code can be worked on it? So, so, um, You know, if I, if I wrote all the code and I’m the only one that understands it, if I get hit by a bus, you’re going to have to bring somebody on that can then, you know, pick up and learn everything. So ideally, everybody would have a really low bus metric and you would. You would do that through cross training, paired programming, so you have more than, than, you know, one creator of the software and stuff like that. Um, but yeah, that definitely is, is, is something that you want to reduce as much as possible.

Adam McNair: And so this might be a, how, how integrated into the DevOps mindset is paired programming? Is that a big, a big part of DevOps or is that just a development kind of theoretical type of thing? Activity that that can be used with anything.

Kevin Milner: That’s a good 

Adam McNair: question. 

Kevin Milner: Yeah, I would say it was probably be more on the developer side. Um, you know, you’re going to run into it a little bit when you’re merging the code, I guess. But, um, yeah, that wouldn’t be a DevOps policy. That would be that would fall more under the, like, development side of things. 

Kevin Long: It’s a convergence of popular, uh, popular development methodologies. Yeah. Yeah. I mean, because you could absolutely do, um, pair programming in Waterfall. Absolutely. Um, but, yeah, we’re seeing more and more, you know, extreme programming where pair programming is, is a core, core component of it, or just pair programming in general, uh, coming together as independently as DevSecOps is getting more and more popular. I wonder if they’ll merge. So let me, let me, 

Kevin Milner: uh, let me give you sort of. Just to clarify what, like, paired programming is, the idea is that you have two developers, let’s say, and they’re working on a new feature for this application. So you have one developer, maybe they’ll write a unit test and say, when, you know, when this button is pushed and this field is populated, I want to see this result. So they’ll write this unit test, then, you know, their, their co worker that they’re sitting next to or in a, in a virtual meeting with or whatever, we’ll then write code to make that test fail. Um, so then once it fails, then your other guy can go in and say, ha, now I’ll write the code to make it, I’ll fix the code to make it pass. And then, you know, the other guy’s like, well done, now, you know, here’s a unit test for you. And, and you do that and, and that not only has the benefit of letting everybody sort of get their hands dirty with a particular function of the code of the, of the application, but it also makes it a little more fun to program. It’s, it’s more like a game. So, you know, happy coders are. Are better coders, I think, and if you can keep people entertained while they’re while they’re working, even if even if it’s like a financial application, you know, calculating compound interest, it’s more fun if, you know. You are challenging your friend.

Adam McNair: So I will tell you, I, to just strongly agree with that statement, I mean, there’s a lot of things that we do with that idea, that people that are happy with what they’re doing do do better work. Um, the, you know, the whole idea, I I sat through a, kind of a seminar one time on gamification of, of work. And, Whether it be on a service desk where you are giving some incentive that whoever can close the tickets out, you know, they get something and you make the reward suit, you know, frequent enough that it’s not an annual award. Because then by 4 days, you kind of burn it or whatever. I’m behind it. It’s over something small. That’s on a, a daily basis. That makes it fun and it gives you something to focus on other than just, oh, all of these people’s, you know, the accounts didn’t work. I gotta go in and fix ’em. The, those kind of gamification aspects, something where paired programming where you, you know, you and somebody else are going back and forth. I, I think there’s a lot of that. That makes sense. And you know, I think also the, one of the real benefits of, of a lot of the, the automation that we see is. People just can’t operate where they, they’re just sitting somewhere for eight hours a day just waiting for something to happen. I’ve had, you know, watch desk programs or, um, you know, network monitoring programs where you’re just having somebody sit there waiting for something to happen and then expecting them. And a lot of those programs aren’t even eight hour shifts. I’ve had ones that were 12 hour shifts. You’re expecting somebody that’s been sitting there on hour number 11 to be just lightning quick and completely, you know, clear. Oh, this is exactly what we do in this situation. And you can drill and train and do all of that stuff. But, um, I think it makes a lot of sense what you’re saying. And the, the, the, the development methodologies that. Make the coders enjoy it more, and theoretically, I guess, probably also, are there tedious aspects of the work that DevOps or DevSecOps is removing? Are there some of these roles, like if you were going to have to do all this testing and the code quality and all those things, was there a point, I guess, where that was somebody’s job to sit there and read all the code with, uh, Kind of code style guide for lack of a better term.

Kevin Milner: Yeah, I mean Those have largely been replaced with yeah, you used to have to check code manually, you know, just hey Can you look at this now? The the code reviews are a little bit faster You just sort of make sure that there’s nothing really strange going on and Then if your tests fail Then you can go in and look at it. But, uh, various methodologies such as dependency injection, uh, where you, where you have pretend resources that you give the program for it to interact with, so you know how it’s going to respond, uh, you know, um, things like that. That that helps reduce that so you can automate your your testing and and as we mentioned, the code style, um, things like that. So, yeah, I mean, my goal is as a DevOps engineers to not have to do anything to get to a point where where I just sort of sit there and say, yep, it’s working. Um, obviously, I don’t want to tell my employers that. But, uh, um, but yeah, I mean, the goal is to, to automate everything that you can, so that not only is it more enjoyable to do, but you also want to make sure that there aren’t any tedious mistakes, um, you know, uh, You can solve that with checklists, but nobody wants to go through a checklist and say, uh, does this have a semicolon? Okay. Does this have, uh, you know, does this return a null value anytime? Stuff like that. You don’t want to have to do that. If, if, if you can automate the process itself, then you can spend more time doing things like adding cool features. Uh, instead of having to like make sure that every weird use case that you can’t envision, but your customer will immediately find, um, you can make sure that doesn’t happen. 

Kevin Long: Yeah, there’s nothing more demoralizing as a developer than having to go back into what was cool once, but now is done in your brain and rebuild it. Build it from the middle out because a security finding came in and said, no, no, good idea. Nice try. Not here. You can’t. 

Kevin Milner: Yeah. Right. First 80 percent of programming is great. The remaining 20 percent is tedious and not much fun. 

Kevin Long: Yeah. Getting that, getting that in your mindset while it’s still cool and exciting. And it’s another problem to solve while you’re still problem solving all of that. Like the initial stuff is, yeah. Makes it so, so much, so much better than, than squashing giant bugs at the end.

Adam McNair: Yeah. Well I will tell you, um, and this, this goes towards a lot of things that I have done, uh, whether it be software or when I worked, uh, you know, the infrastructure design or you know, whatever. I’m really, really good for your first 80%. Like that’s, that’s a wall you want me on. That last 20, I’m with you. Like the, the, Oh, I, I actually have to go up and line up all of the individual requirements and make sure that we completed all of them. I don’t, I don’t know if you want me to do that. I’m going to zone out in the middle of it. I’m going to miss one. I just know, I just know I am. Um, so, so DevOps, there’s a lot, certainly it sounds like there’s, there’s, there’s a lot to it. Cause it is, it is both an integration of tools and it, it requires a lot from the whole organization and you need your development and your whole technical team needs to be proficient in a whole set of tools and. Are there, are we really at a point now where there are, like you mentioned a couple of tools as we were talking, are you seeing a pretty standard set of tools at this point? Or are there, is it the app store and you just try one? How does that work? How does that work? 

Kevin Long: I would love to hear Milner’s take on this. 

Kevin Milner: It’s a little bit of both. Uh, there’s, there’s tools to do everything. Um, 

Kevin Long: and new tools every day. 

Kevin Milner: Yeah. New tools every day, being able to, uh, to, to learn how to use them and make them work with existing tools is, it’s another point entirely. But, um, it’s like any technology, really, if you look at JavaScript, there’s, There’s 8, 000 different JavaScript libraries. There’s Node, there’s Angular, you know, all of these things. So, um, you know, there’s, everybody wants to create their own flavor. And, and I think after a while, natural selection or artificial selection sort of whittles away, you know, the, the, the puppets and the, I know there’s going to be some haters that are like, No, not puppet. Um, and leaves you with chef, you know, uh, so, so it’s sort of a marketplace of ideas where, where you, you see when somebody identifies a problem, there’s lots of solutions to it. Over time, it, the, the better solutions usually tend to, to remain. Uh, the most robust, 

Adam McNair: so do we, does it look like it’s headed a direction where the, you know, Microsoft DevOps in a box thing shows up and it’s all of these integrated, already integrated tools, or does it seem like there’s enough out there that it’s just too diverse and you’re not going to see one product end to end that does everything?

Kevin Milner: Oh, it’s, it’s just like any. Program. Really? I mean, is there a Microsoft program that does everything? You know, um, 

Kevin Long: if you ask Adam that his answer is PowerPoint. 

Adam McNair: That’s correct. That’s that. If you look at the Venn diagram of things that can be done, uh, The part that can be done in PowerPoint is is overlaps many, many things, 

Kevin Milner: but you see my point. There’s, there’s, uh, you know, there’s, there’s not like, uh, you know, every organization is going to be slightly different. It’s going to have slightly different needs. Your, your D. O. D. Organizations are going to have. A much higher focus on security, uh, making sure that libraries that are used are not, not, you know, kind of dodgy, that sort of thing. Whereas as somebody that works at, you know, drizzly, the, the booze delivery service is going to, uh, You know, they’re going to be a little bit more concerned about making sure the maps are accurate or it’s right. So, um, you know, like, uh, it’s, it’s going to be very organization dependent. There’s large blocks of stuff. For instance, Jenkins is really good at coordinating all of this, all of the points between, uh, committing code to, to your repository. And. pushing a product out somewhere. But, you know, as far as monitoring, you’re going to need a different Probably a different tool for that. Yeah. And things like that. So there’s, I wouldn’t look at it so much as a, as a universal tool, as, as there’s a well rounded toolbox that you have available to you.

And I mean, 

Adam McNair: yeah, yeah. And I used to see some of that. So I probably could draw a correlation with the infrastructure side of things that. You know, there were, there would be a a service desk tool where you handled your, your incident management. There would be a tool for scanning the actual infrastructure to make sure that the point to point connections were up. There would be a tool for monitoring the end points to make sure that the end points were active. There would be a tool for capacity that was monitoring how much availability there was. And there’d be, you know, one for patching that would push patches out and you had all these different, different items. And so there started to be some consolidation where there were some tools that kind of did everything. But what I found was it really, truly, like you say, it depended on your requirements and what the priorities and areas of emphasis were. Because a tool that started as a service desk tool, Even if they bought or added on or started making something that would do patching it never did patching as well as your patching tool. And so when we used to use some of the earlier Microsoft tools there, there were there were ones that were. really enterprise monitoring and they started to build patching into it. And I, I worked with some wizards that they could deploy stuff all over the globe, kind of figuring out how to hijack some monitoring processes that the thing would do. Then other people said, well, why aren’t we just using these other patch management tools that are so much more robust and you have versioning of your patches and all of that. And so it probably at the same level, like when we look at recruiting systems, if you’re not a recruiting system company, you might have added recruiting into your HR system or your finance system. I’ve seen ones where they’ve added them into like your asset management system and that kind of suite. There’s just nuances that, depending on the organization, can be a huge deal. Um, so that, that makes a lot of, a lot of sense. Go ahead, Milner. 

Kevin Milner: Yeah, it’s, it’s sort of, Like all things, it’s a balancing act, um, and this sort of ties back into the microservices we were talking about earlier. You can, you can view the microservices, uh, concept as sort of the idea of a bunch of tools in the toolbox. And so, you know, yes, you can break them down and have like a bunch of little tools that do exactly one thing. And that’s super efficient, super, super good from that standpoint. But then you have the complexity of having to orchestrate all of those tools, whereas, you know, something that’s a little more. Integrated is a little easier to manage, but then you start having to have specific experts on that tool. Yeah. So then, so then your, your, your bus metric goes up. So it’s, it’s sort of a, a delicate balance. Um, I think that the, the project managers and other stakeholders need to be cognizant of. You can, you can go too far in either direction. You can have one tool to rule them all or, uh, you know, have a million tools to do little things, but you have to worry about orchestrating it. 

Adam McNair: It makes a tremendous amount of sense. Um, and I will tell you, I, I feel like as we’ve, you know, kind of been exploring DevOps here a little bit, it might make sense for us to do a part two on this and, you know, talk more about some of these. Development aspects because I think it’s certainly deep 

Kevin Long: in the weeds on this 

Adam McNair: stuff Yeah, it’s certainly been interesting to me and I I don’t I don’t know if anybody else is um has been around software as much and probably you know not not found an easy way to understand some of the You know, more complex aspects of it, but it certainly, I think, has been, um, has been interesting and enlightening in a lot of ways. So with that, I think we will go ahead and wrap up for today. Um, and. We will get back on a another highlight cast here sometime in the future and talk a little bit more about development and maybe some of the specific tools that we’ve, uh, that we’ve been using and, um, and so forth. Um, like to, uh, thank Kevin and Kevin for guys for taking your time to, uh, to, to, to work with us here today. Absolutely. Um, I appreciate all of that. So, uh, for more information about highlight, you can visit our website, highlight tech. com. Uh, there’s news. Uh, you can also follow us on LinkedIn where we have an active presence and, uh, thank you all very much. Appreciate your time. 

The views and opinions expressed in this episode are those of the hosts and do not necessarily reflect highlight technologies and or any agency of the U S government.