Episode #19: DevSecOps Part 2 

Announcement: Broadcasting from Fairfax, Virginia, you are now tuned in to the Highlight Cast with your hosts, Adam McNair and Kevin Long.

Adam McNair: Hey, hi, and welcome everybody to the Highlight Cast. My name’s Adam McNair, we have with us today Kevin Long and Kevin Milner. Hey guys, how are you? Good, so if, uh, now, first off, if you haven’t listened to our first episode that we did on DevSecOps, I would recommend that you pause here and go back and listen, uh, DevSecOps walkthrough, really what I would say is a walkthrough for a layperson, um, somebody that, you know, I’ve, like, like myself, I’ve, I’ve done some things in software and so forth in the past, but, uh, Definitely not a hands on, uh, technical architect or anything like that at this point. So I thought it was a very good conversation and we certainly hadn’t covered everything. So we had talked about the benefits to organizations of DevOps. We had talked about how we had seen some of our customers. Adopting DevSecOps because it is a transformational activity. It’s not like you just make a decision that you’re going to do it. And then all of a sudden, you know, you’re a DevSecOps organization, right? So we talked about that. And, um, Today, I think that the things we wanted to talk about were, uh, the toolbox that you need to implement DevSecOps and how to build a team, uh, to deliver in that way in, you know, in the, in the federal marketplace. So we talked about that. It’s, it’s a, it’s a tool heavy. Activity or you lean on tools to be able to automate things and so forth. So I think it begs the question, um, so what are the tools that you guys, you know, typically kind of are, are the major core go to go to tools? 

Kevin Milner: I would, I would say that like a lot of the tools that you’re going to lean on for DevSecOps are going to be, uh, you know, If you look at it holistically from, from the start of the pipeline on, you’ve got, you know, first you’re linting tools that make sure that the code is, is, is like, you know, properly formatted and stuff.

Then you go and you do your, Your unit tests, but then you also do you want to have a suite on your toolbox of like, um, security tests and code validators like sonar source and stuff like that so that you can really make sure that that’s part of your pipeline. And then, uh, other tools you might. You might use our, um, you know, diagnostic tools are really handy to have if you’re, if you’re looking at net security, you might use like Wireshark.

Um, my preference is to try to go for free and open source tools as much as possible, at least in this sort of space. Um, Just, just because generally they’re, they’re better vetted and people, you know, uh, are able to, um, you know, if you, if you have code that like everybody in the world can look at, then you can be, you can be pretty sure that if there were some problems with it, somebody would have pointed it out, uh, 

Kevin Long: When I think about the tools also, though, I think about it is sort of like turning software again into a factory and things that automate different processes, right? So, Kevin living in this every day has has has gone gone to the to the deep, some of the deep facts, which I love. But then there’s also been like the base automation tools out there that you get like, you know, Jenkins and SonarCube and, and then you end up and Maven and, and, and Gradle and tools like that are Ansible and, um, Chef, Puppet, uh, all of those really that allow you to, to push your code down the, it’s like making, making it a conveyor belt, you know, that which moves it, moves it. Moves it along through the process. 

Adam McNair: Now, where are we from the standpoint of interoperability? Are we, do they all kind of work with each other? So that if I want a automated unit test tool, will all of them work with the, you know, whatever the precursor tool would have been? Or is it some are better than others?

Or how, how does that, How good is your DevOps engineer? 

Kevin Milner: Generally, uh, generally the way it is, is set up, you have tools, you know, you’ve you’ve heard the phrase the proper tool for the proper job. So, so some tools are better at sort of generic analysis. But then other tools are specifically made for like the language and environment you’re working in. For instance, you can’t really easily use, um, say like Maven to compile NET code. It’s possible, um, and it comes out really messy. But, uh, you know, you can do that. But the better, the better. The better choice would probably be to find a specific tool for that particular implementation. Um, you know, in some cases it’s useful to have a tool that can reach out and talk to a Windows machine or a Linux machine or a Mac OS machine. But in other cases you’re going to really need to focus on what your specific application is in order to make sure that the tool properly works. 

Adam McNair: Okay, so it kind of goes into the category of the MacGyver type aspect of there are some things that will do what they are intended for all the time and then you can take something that’s not ideal and Maybe you can patch it around and rig it so that it will do something, but it is probably not ideal for the task. Yeah, 

Kevin Milner: and it’s sort of, there’s, there’s sort of two philosophies, I’d say, in computer science, or at least in, in this sort of thing, there’s either the idea that you have a whole bunch of little tools that does one thing really quickly. Uh, and, you know, you know, every time you type. Mv space file name space destination. It’s going to move the file. It’s not going to, you know, reset security, stuff like that. Uh, and interesting analog to this might be in chip design. Originally, the, the PC chips, uh, the IBM compatible chips for processors were designed using a complex instruction set. You have a rich, robust set of functions that don’t do each, each one does a specific thing and you have this, you know, really complex way of using it. Then, you know, you might, you might think of that as like trigonometry. Uh, you know, you’ve got all this, these complex equations to do things, but they only do the limited things that, that, that they do. Then the other option would be what Apple chose, um, which is the reduced instruction set code, which is. You have, you know, uh, basic, simple tasks that you do, and then you build up the more complex tasks from the simple set of tasks. So, uh, so you end up with like a much simpler, Design set a simpler tools that you have to be more creative to use, uh, to do what you want. So it’s sort of a, you know, a philosophy of do I want stuff that does, you know, these complex, actions, but, but that’s it? Or do I want to build up the complex actions from simpler sets of steps, which would be like, you know, arithmetic in a math example I was using earlier. 

Adam McNair: So one could also crosswalk it, like I think of, so when you look at a LEGO set, there are some of the LEGO sets these days that have customized pieces that It’s the top of the cantina in Tatooine or whatever and that’s but that’s really all That’s all it’s going to be because it was custom designed for specifically that purpose Whereas theoretically if you just had a large enough collection of one by one bricks, you could build anything you wanted 

Kevin Milner: Yeah, exactly 

Adam McNair: Got you.

Kevin Milner: It would be more effort, but right you you know, you have that one tool that you know I mean that one One by one brick, you know, it’s properties. You know exactly how it’s going to work all the time

Adam McNair: now. So really often in DevOps or DevSecOps, how frequently have you guys seen that you walk in and get to pick your own tool set versus that there’s an existing tool set or some kind of give and take? Where they’re wedded to some things, but they let you pick other things. 

Kevin Milner: Um, it, it happens both times, uh, obviously when you have like a, uh, um, a Greenfield project where you’re, you’re going in and starting from scratch, that’s obviously you have a much better chance of choosing your own tools. With Kessel Run, for instance, um, Nope, 

Adam McNair: had a little breeze on Kevin’s end. 

Kevin Milner: Oh, okay. Sorry. Um, at Kessel Run, you know, we had an existing environment that we were expected to integrate with. Uh, it was already a macOS environment. Everybody’s developing principally in Java, excuse me, JavaScript. Uh, so, so that sort of, you know, Dictated what tools we could look at when we went to say, okay, well, we need a specific tool to perform linting. Um, you know, the linter needs to know what javascript looks like as opposed to cobalt or something

Adam McNair: Okay and As a DevSecOps engineer, now, like I think back to when I’ve had development teams and you kind of either had a Java developer or you had a NET developer and occasionally you would have some real rock star expert Something that would work on both usually in an environment where they had both applications for some odd reason, and they needed somebody that was conversant in both. But typically I always saw you either had a dot net or Java team. That’s kind of how it. In the DevSecOps space, do you view yourself as a proficient user of a certain set of tools? Or is it more the theory of it, and you can sit down and use any tool once, whether it’s documentation or a little bit of just understanding of, you know, you’re watching YouTube on it, however.

Kevin Milner: Yeah, they call those full stack engineers, and that’s what everybody wants. Thanks. Now they want, they want full stack engineers, not, not a front end guy or a back end guy. They want somebody with full stack plus DevOps plus, you know, electrical engineering, uh, all that kind of stuff. Um, but in terms of, uh, Myself, I try to view myself more as a generalist that, um, you know, adapts sort of the basic theory to whatever the application is at the time. So, I mean, I’ve done DevOps for, for a software migration. A data migration platform, uh, and I’ve done, I’ve done a form of DevOps for the tool chain team. So, um, you know, at Kessel Run. So it, it really is a base theory that ideally you could apply to any sort of, 

Kevin Long: Programming is a mindset. The rest is syntax.

Kevin Milner: Yeah. Or, or my other, my other favorite phrase, you can, you can write Cobalt in any language. You can program in Cobalt in any language. I 

Kevin Long: mean, that said, I mean, everyone is going to have a strength in a different part of it. Right. And, and that’s really where you get the nuance around it. And so, uh, well, you’ll have someone that’s amazing at JavaScript, you know, they’re, they’re not going to be confused If they need to open up Eclipse and start writing a bit of Java, right, or they’ll understand a bit about, uh, you know, I don’t know, EKS maybe, or something like that. But when you get into something like EKS, then you’ll probably have someone that’s come into DevOps through the standard sysadmin infrastructure build out. And they wouldn’t necessarily be confused by, you know, some Java or some JavaScript, but it’s certainly not their strength. And so Kevin’s absolutely right. Everyone is is searching for the full stack everything. Nobody’s a full stack everything. There are people that understand the principles of it and are able to get smart on just about anything from what I’ve seen. It’s, it’s, it’s a lot about the mindset, willingness and because DevOps and DevSecOps is so new, it’s people that love the readme files. That, that do well with it? I think yeah. 

Kevin Milner: If you like to read documentation, this is, this is the position for you. 

Kevin Long: Yeah. ’cause the tool that’s amazing today is gonna be obsolete in, in, in a year. 

Adam McNair: Yeah. 

Kevin Long: And, 

Adam McNair: and I, I, I saw that. So when, when web apps started to become a thing. I feel like every little tool that existed that either allowed you to cache data so that you could have a disconnected user, or it allowed you to dynamically pull information from some place and display it on part of the screen, which all sounds pretty simple now, but there was a time where That wasn’t very simple, and you’d get this little plug in thing, and you’re right. It came from a random developer organization, it had a README file, and it was really about who could understand conceptually what it was trying to do and how you configured it. And half the time you’d have to look at what, what variables can I configure in the thing, and that would tell you what it was looking for. Um, so that, that makes sense, you know? And I think A lot of the, the emphasis, you know, like you say, on, on full stack engineering, I think it’s pretty common for staffing lists initially to be built with kind of some guesswork in mind. And if you go with somebody that can do everything, It’s, your staffing plan’s probably, probably accurate, you know, because if you went down the path of how many people do I need that know this or that or this other thing, you can be wrong, because I’ve had that happen to me before. I’ve, I’ve won programs and showed up and they said, And by the way, who’s going to manage the servers? Well, what servers? Well, there’s this one bullet in the Statement of Work that said, you know, and, and other systems support, but that means we have a data center and we need the servers maintained. Oh, and so you can get things wrong, whereas if you have people that are experts that know how to do everything, you have that comfort factor that if something goes sideways, if something new happens, if you want to do something new, that You put four or five of those people together and they’re probably going to be able to figure out what comes at them. Um, the question I guess I’d ask around that is, that’s mindset difference. That’s a different approach from instead of give me three of these people and four of those people and I need this many of that people and all these different work streams and swim lanes of people and different kinds of engineers go into, uh, Heavy full stack organization. What else when you go to DevSecOps, just as the typical, you know, I work in Agency X and I have this application that, you know, I’ve been responsible for managing as my team did, you know, a couple waterfall deliveries a year. What are some of the major changes that happen when you do that?

Kevin Milner: Well, um, you know, some of the major changes, I guess, If you, if you focus too heavily on generalists, then you lose out on some of the, the, the specialist skills and, or it costs you more, uh, it’s one of those, you know, powers of three type things like price, quality, and, and speed, you can have two, but not the other. So I look at it sort of in that sense that. You can get a lot done if you have some general, general skilled people, but then if you get into something like, you know, needing to, to efficiently move gigabytes of data from one place to another, you’re going to need a file system guy for that, because, because your generalist just isn’t going to know the, the, the details of how to really make it efficient. So, you know, I, you could even, even say that, you know, Even if you have like a large team of, of generalist engineers, you probably are going to want to have at least one or two specialists for your, for your field, uh, for the, the, the space that you’re in just Just to make sure that you get all the details covered. Was that an answer to your question? Yeah. Here’s another, 

Adam McNair: yeah. Here’s another question along that line. So let’s say I’ve got an organization, um, that we’ve still got legacy stuff, right? Mm-Hmm. . I’ve got, I’ve got a, a, a legacy system that is. feed some other systems. Can I go to a DevSecOps environment if I have some kind of old, creaky application? 

Kevin Milner: You Can. You can, you can do anything you want. Um, but you know, the effort involved might, might make it, uh, impractical to do so. I mean, if you’ve got a situation where you’ve got like this really old code that communicates with a a server somewhere over open text. Um, you can’t, you can’t really put that in a, in a DevSec environment, uh, you know, without, without some changes, making sure that you follow proper encrypting protocols and things like that. So, so again, it comes down to a balance between, um, uh, you know, practicality and, and, um, You know, benefit of, of doing so. Um, a lot of times people, especially like, you know, in the higher levels like to see the checkbox clicked. Okay. We’ve got, you know, this, this sort of, uh, compliance. With with this best practice, but if if it if it took you a long time to get there and and too much effort, then 

Kevin Long: yeah, but if they really want to cut over, I mean, let’s let’s walk down somewhat of the happy path because sure.

I mean, there are definitely old systems out there that it’s probably better to rewrite a lot of it. Yeah, but, but if you have some part of like a legacy system that that could be extracted, like, you know, like some sort of data sharing where it could be turned into a service, right, a web service out there, right? And with that, then you really could decide to start doing some, some DevOps stuff with that really to spin that up. Some of the major things that you’re going to have to change. Um, Yeah. Are you have to change the way your teams are put together. You’re not going to have a testing team, a development team, a production team and an infrastructure team.

Speaker 3: Yeah, 

Kevin Long: you put them all together and they’re all working for that. And so there’s a lot of times when you’re cutting over from old old in quotes to new in quotes. With that, there’s a lot of change management on the on the organizational and political side. Yeah. To get that put together, right? So where everybody is working together to push off is one of the core elements of, of DevOps is that you have everybody working together, right? You have development ended in years of security, all, all, all, all pushing, pushing it from the, from the same team. But then you also have all those tools that Kevin talked about in the beginning. It is a non non negligible amount of startup to, to put together a pipeline, uh, to have people that have done it before. If they’ve never done it before, to go through it, get to the read, get through the readmes, learn how to be able to, you know, connect your Git repositories through, through, uh, Jenkins to have thresholds put on a Sonar cube, uh, for, for code coverage as it’s running through your, your, your, uh, Selenium tests, right?

And things like that. So it, to, to spin it up, I mean, it is, it’s awesome. It will. Overtime save you time and it will allow you to deliver more value to your customer faster over time. But it is, it is an investment to, to, to cut that over for sure. To, you know, even on full and open, uh, uh, free open source software, you know, it’s not free engineering to, to put together and build, uh, a resilient pipeline. It is not, uh, without political capital to rearrange your software development and infrastructure shop, um, uh, to, to do all of those things. But where you have, where you have, uh, a legacy system that can, that can have incremental things, you know, sort of strangled off of it. Uh, uh, there’s whole design patterns around ways you can pick and choose small elements off of, uh, off of, uh, sort of larger monoliths to, uh, to, uh, make them more loosely coupled, which work really well in, in, in a DevSecOps environment. To make that shift. 

Adam McNair: So you’ve got two, two big categories of call it effort. You call it challenge might be some combination of both, but you’ve got a significant level of effort project to be done just to stand up the tool chain, the pipeline so that, you know, if you take it back to kind of a real world example of if you’ve, if you’ve had six people, You know, chess pieces and you decide that you’re going to make a factory that does this. You have to build the assembly line that makes those things. So you’ve got a significant undertaking to do that, which means you likely can’t use the same. People that you have doing your existing O and M because they are likely to either not have necessarily the skills or they are likely to not have the time. I mean, I guess it comes back to depending on how much, you know, what’s the stability of the system that they’re on and I guess I’ve been involved in, in software O& M projects where everybody seemed fine and it was kind of orderly and there was a backlog of, of enhancements is really what they were prioritizing around. So I guess if, if you were in that environment, that could certainly work. 

Kevin Long: If it’s not a bubble gum and bailing wire kind of place, uh, the existing team could almost certainly do it. And honestly, if you were to talk to them. I bet, I bet you 80 percent of the folks on the ground there would be excited about it because they’ve been reading about it, looking at it, and probably interviewing for jobs at places that do it. That makes sense. 

Adam McNair: Yes, I guess it’s also possible, you know, I think one of the things that, um, that we’ve done, you know, Kevin and I, For years is to bid jobs in places where they wanted to modernize software. So it’s possible that I’ve seen an overabundance of O and M organizations of the bubble gum bailing wire type where things are broken. Things are not backed up. You know, you’ve got. Developers that say, like, you know, I haven’t taken a day of vacation in four years because this thing goes down at least twice a week because as soon 

Kevin Long: as it sees I put in PTO, it starts to smoke. 

Adam McNair: Right? Right. So, so if you have a reasonably executing environment. Yeah, you could probably do it with your team. Uh, so that makes sense. Then from a, an organizational kind of thought process standpoint, you know, typically you have this, this Burndown list of user requirements where they’ve got some enhancements they’ve requested and so forth does DevOps dramatically impact that side of it. I mean, because you’re still you’re still getting your requirements from from the user base. Um, and I guess you’re not necessarily burning through them dramatically faster than you were before. It might just be a more efficient implementation. How do you see that work? 

Kevin Long: So let me give the management side on this first and then we’ll get to the tech side. What you see with the DevOps from a management side is you don’t see human error in deployments happen as much anymore. That’s really what you see. So the quality of what moves from dev to test to staging to prod is far more consistent because there’s literally no fat finger involved to fat fingers. Right. And so pipeline done right. You’ll catch those errors earlier on. That’s what I like about it. 

Kevin Milner: Yeah. Yeah. So, and also from, uh, you know, from the state, one of the ways that it can really impact is. Is you can set up a system if you have good management of your pipeline and your repository and stuff you can and you set it up with branches feature branches and stuff you can have it so people are working on parallel things without affecting each other so you might have one person working on you know the UI and another person working on the rest server in the background and anything that the UI person does. Isn’t going to affect that. And so you can build the product so that, okay, we can put in the new UI. Can’t call any of the new stuff in the backend yet, but it’s there for when we want to. And, you know, say the backend hits a major problem and they get delayed by a sprint. Well, you know, your UI guy can then go work on the next sprint worth of stuff while, while the backend guys fix it. And if you’re. If the pipeline is correct, then there’s no cross contamination of the branches. You don’t, you don’t end up with shipping, you know, a product that has the new UI, but has a UI that has this broken feature from, from the back end in it also. So it, it lets you be more granular about. Um, you know, requested new features or fixes or whatever, make it into a given, uh, distribution.

Adam McNair: And so you can definitely have what I’m hearing is more efficient use of resources, more efficient use of time. And it takes out some of the critical path areas, because I’ve definitely seen that before, too, where you have a development team. And, I mean, it happens on any team, but Out of 20 people, there’s a few of them doing nothing because it’s not time for them yet. And so because of that, um, You know, that’s, that’s the, a frustrating thing as a, as a, a being involved either as a user or as a customer where you’re there and you’re like, you’re both telling me that we’re, we can’t get this done until a certain date. And I see that we’ve got people that when you look at the, you know, the overall resource allocation, they’re not actively doing anything right. Exactly. Um, or they’re doing some task that’s way, way lower priority. But you say, why can’t they delay that? Like, well, but they can’t do that yet anyway, because they have to wait for these other things to be done. So, so that makes a lot of sense. Um, from a, a team construct idea, how different is it? Like, uh, are, are you used to as a developer? Are you used to not having any of the infrastructure security types talk to you when they do? Is that beneficial or does this get down to just it’s personality driven or how does that normally work? 

Kevin Milner: I mean, it’s, it’s a little of all of the above. Uh, all I can really speak from is, is, is through my work history, but I’ve worked in places where, um, You know, the release engineering would refuse to update the actual build environment that the code was built in, so we were still using Visual Studio 98. This is 2007 2008, somewhere around there, and 98. Only the most current. Yeah. Um, well, but, but the problem was the, the, the code we were working on in this case was medical device drivers. Um, it, it required windows, I mean, uh, Visual Studio 98, but the the main application, uh, that would take the data from the drivers was written in, uh, starting at a certain version number was written in dot net Visual Studio, and so the problem with that was, as a driver developer, I had to release two different applications Driver versions, uh, one for the, you know, pre version 4. 5 of the software that would work with visual studio drivers and one with the next version. So, so the inability to get, to get, um, release engineering to take that supported version. And update it to use this new development tool because it was completely incompatible, uh, meant that, that, yeah, it was a lot of extra work for us. Whereas had we had more of a team focus on agile DevOps and again, this was, This was early mid 2000s. So, you know, uh, things were different then. But, um, it made it so that there was a lot more work for, for my team, the, the, the driver team and, and, A lot more testing that had to be done for for the the QA part of that because they had to test both versions of the driver and stuff like that. So it would have been really effective if we had had managerial buy in in that case to be able to make changes. 

Kevin Long: Yeah, I mean, I can tell you from way back in the day when I was a developer, you know, back in the State Department, nothing made me happier than when security got involved. Earlier on, because I’ll tell you, nothing breaks software like applying Stig’s that you didn’t develop against.

Kevin Milner: Yep. 

Kevin Long: And literally, you just, I mean, you develop everything and then it’s like, Oh no, well, you can’t use this feature because.

Kevin Milner: We need encryption at rest.

Kevin Long: Well, I mean, or, I mean, or just, oh, you thought you were going to use this port. Oh, no, no, no, no, no, sorry. Right. Uh, just so much stuff. So getting, uh, the, the sec in DevSecOps, where you’re looping the security folks in earlier on is so much better.

Kevin Milner: Yeah. Yeah, absolutely. And, and that’s, I think what we talked about. In the last, in the last, uh, podcast was that, um, you have to, you have to build security and from the beginning, uh, it, it’s not something that you can reliably, uh, shoehorn in afterwards. 

Kevin Long: And it’s not something regular developers think of.

Kevin Milner: Yeah, yeah, it’s certainly not, not something I, I can consider when I’m, I’m hammering out code. Yeah. 

Adam McNair: Well, it’s something, you know, it comes back to a general idea that, that we talk about a lot, which is, it’s much more common for organizational challenges to be problems than it is technology. You know, and I, I think back to programs I’ve been involved in where really for just sometimes personality reasons, but a lot of times the alignment of priorities in one group versus the other. You know, the, the security office’s priorities were driven by somebody that had nothing to do with this app you were doing. And so you were over there kind of begging them to do security scans so you could move on with your deployment and it was officially not from a priority standpoint. It was not what they shouldn’t do be doing at that point. But in a big enough organization, sometimes Well, it’s never actually going to be your priority. This system does not have a high enough impact for you ever to prioritize this. And that’s where like, you know, sometimes there’s a contractor, almost the personal relationship of, you’d have to go over and be like, please, please. Yeah. Would you guys please just like, look at this real quick. I mean, we’re all going to get in trouble and it was, it would have to take some kind of human appeal. So I, I wonder. In the federal space, where there’s kind of the added complexity, you’ve got contractors in the mix and not only do you have the federal organization and there may be one or more organizations there with, with different charters or alignment or whatever, but you’ve also got potentially multiple contractors where because of scope and who’s allowed to do what and who’s supposed to do what and who’s being paid to do what, um, you know, I mean, you don’t, you don’t go to your house painter and ask them to.

You know, put a new microwave in because you, the appliance guy is late. Like, and that, that, that’s the kind of thing that, that happens on government contracts all the time is just because you’re there working on something. You’re like, that’s not my scope. I, I, I’m not supposed to work on that. So I wonder if we have seen organizations. Undertake change, try to transform the way the organization is set up and then go to DevOps or they go to DevOps and it helps them push the culture and organizational change in in the places where we are are working. You know, Kevin, I know we’ve talked about, about your project and, uh, Kevin Long, you know, there’s some that you’re setting up now for DHS. Do those feel like the organization was really on board with the concept of working together and the contract is just a mechanism to accomplish that? 

Kevin Long: That’s a great question. So, our DHS work is not what I think is standard because I, I, I, Absolutely think that, that, uh, they made the decision that this is the way it’s going to happen and, and be damned if you’re going to get in our way of it, right? That this is, this is what it’s going to be, um, that they are, I mean, and DHS and a lot of their subcomponents are, are newer, uh, in terms of how they’re organized. And so being newer, they’re not. Has saddled with, you know, 25 year old DOD legacy applications that, that if it goes down, awful things in the world happen, right? Um, or that they were able to, to decide much like Kevin was talking about much more earlier on much more greenfield, we’re going to stand up this. We want to do it this way. This is the right way to do it. And so they didn’t have as much legacy overhead, I think, to deal with. Um, I think that. A lot of, of, uh, of other places I’ve worked, it is really folks, uh, coming around saying, no, this really needs to happen. And then having to push the organization forward with it, uh, because, you know, everybody talks, people see the benefits or hear the benefits and, you know, it’s, And, you know, act IAC and government industry, uh, groups where it’s like, Hey, you know, this is how we’re doing it. And you have CMMI talk about even have PMI, uh, the project management folks coming in, getting in on, on Agile and DevOps being the natural, uh, natural progression on from that natural next step that they want the benefits. And then you have to say, okay, you don’t just get the benefits without the work without these fundamental changes to how you operate and are organized. And, um, uh, so I actually expected, uh, some of the Organizational pushback, uh, at our DHS stuff that we really haven’t gotten. It’s more like, no, this is how we’re going to do it. We, we believe it more so than at some of my other contracts where we’ve like helped literally design, you know, uh, agile and DevOps work and literally have to say, all right, when. Our customer doesn’t provide a product owner to sit in the meetings. You know, at this point, this is when we’re allowed to punch out and say, okay, you don’t get to do agile DevOps anymore. Now you have to do waterfall and your world’s going to slow down. 

Speaker 3: Yeah. Right. 

Kevin Long: Where I. We literally, you know, debated for two weeks about how much intransigence from our customers we could take before we had to, to, to bust back to the old way just to get something done. Um, so it’s, it’s very exciting on, to have. Customers in both sort of both mindsets. 

Kevin Milner: Yeah. Uh, Kessel Run, it’s almost, you know, sort of the, they were created as, as an agile shop for that purpose. That’s, that’s what they made for. Um, and since this DOD security right there from the beginning is, is. an integral part of it. So we were fortunate there in that they let us, they let my cyber security engineer essentially architect, uh, the entire approval process for the Kessel Run and potentially the Air Force and the DoD. So there’s been lots of interest from, from even higher up than the detachment for these, uh, this process that he developed. Uh, so we were really lucky with Kessel Run that they let us, you know, Make contributions towards the security posture of the development process.

Adam McNair: And I think that, you know, sometimes the comfort factor of how fast something is going has to be, has to be there quick. Is aligned with risky a lot of times, and it’s almost like if you got something done too quickly, if something got too easy, if a process got automated, are you sure that it’s, you know, still compliant with whatever? And realistically, it generally can be. So, like, we’ve talked a lot about. Some of the experiences we’ve had in organizations where this is kind of the way they do business in a very, like, it’s a cultural aspect of the organization at this point. Do you think we’re going to see a point? Because I would, I would translate this kind of to agile. But a little bit different, you know, the agile mindset, I think most federal agencies at this point would say that if they’re going to do development or in a lot of levels, just program management in general, that the agile terminology has really permeated, permeated the whole everything that you’re doing DevOps. Is some theory, practice, all of that, but it is, it carries around this kind of technical assembly line behind use and it has a lot of boundary. Kind of, you know, unifying a team all into one, you know, Agile doesn’t say you have to do that, like everybody can be in whatever team they want to be in and you just, you know, whatever. So, do you think most organizations, all organizations, like is the whole federal government headed for DevOps or 

Kevin Milner: Pockets or what do you think we’re going to see? I would say some form of it. We’re likely to see in the next 10 years, generally any, any software development shop perform like a, uh, a lot of, uh, at least mimicking, uh, DevOps. Uh, you know, I hate to use the term cargo cult programmer, but, uh, you know, there, there’s a lot of that sort of mentality. Yeah. Cargo cult. Is after World War two, they had a bunch of, uh, uh, you know, they had an army air bases out in, um, you know, the Pacific islands and some of the Islanders got so used to having troops stationed there. Um, they, after the, after the war was over and the planes left, they would, um, they would build mock ups of airplanes, hoping to lure back the soldiers with their bars of chocolate. And so. That’s called, that was called a cargo colt. Uh, then they came up with the term cargo cold programming, which is where you sort of, you know, imitate what you see online and hope that that, that works. And that’s not the ideal situation. So, so you might see a lot of people that are At least trying in name to, to adapt, uh, agile methodology and dev ops, but if they don’t actually believe in it and, you know, really sort of live by it. 

Kevin Long: So I think what we’re going to see a lot of with the bigger agencies now, there, there are a lot of smaller agencies out there that, that honestly, looking at the layout of, Of upfront cost to put DevOps in place is going to be problematic, um, until you get someone like GSA, like 18F did with, with a lot of their sort of their, uh, stuff, maybe someone like GSA might start selling the service, but I think things like platform one out of DOD. Uh, or are going to start having, uh, DevOps service that they can sell to the, to the different components within themselves, within themselves. And so, uh, I think that, that we are, we really are going to see it, uh, because it, it is, uh, It reduces risk. It includes security earlier on as a core component of software development. It reduces human error, and it speeds time to release as things go. And Where the larger sort of Uber agencies can, you know, as they’re creating platform as a service and things like that, DevOps as a service, I think is going to end up becoming more and more of a thing, you know, go buy yourself tool chain as a service.

Adam McNair: Yeah, and I think one other thing that I would say will probably help it is. As opposed to 20 years ago, there it is so much more common to see agencies working together, to see shared services come out, to have to see as a service offerings, um, and also for federal employees to move from agency to agency and come out into the industry and then go back into government. And I think, uh, You know, there’s been a lot of times where I was involved in programs where there was a lot of stasis and everybody that was working on the program, contractors and federal had all been there for 25 years. And when you have that, there’s not really a good opportunity and that the perception of risk from something different, if you’ve never seen it in person. I can absolutely understanding them say, like to Milner, to your point, when something infinitely bad could happen, whether that’s weapon related, whether that’s no, no, you know, no, no checks get out to, to, to Americans, the food stamp program stops working, like whatever that would be, I can completely understand if you’ve, if you’ve never seen it work before, you’re like, well, cool. Somebody else can try it first, and even if they have tried it first, every agency’s requirements and architecture are unique enough that you can still say, well, it might have worked for them, but it doesn’t feel like it feels like it’s risky here. Whereas I think if you have enough people that have moved around to enough organizations, um, and you have forums like Act IAC, where you can, uh, go You know, we were just talking today. We’re going to be one of the sponsors of the ELC conference in Hershey in November and several of us will be there. You can go there and talk to a dozen different agencies and a dozen different contractors that are have implemented it all in one day. And do much easier research than you ever could have just kind of, you know, sitting in your, in your one agency as a, as a federal employee or as a contractor, uh, without exposure to it. But so I, I do think, you know, it’s certainly something. Our staff, when you, as we hire employees, everybody wants to work on this kind of stuff. I think being in, in, in leading edge technology environments is exciting for everybody. And I think, uh, for people that are, that are federal employees that want to be able to more efficiently get, Change and value generated for their mission. It’s exciting for them too. So, um, yeah, so it certainly sounds like it’s, uh, it’s going to continue. Well, uh, with that, uh, we’re up on time here.

Thank you for listening to the highlight cast. Uh, thanks Kevin. Thanks Kevin. And, uh, to keep up to date with the news and activities, follow us on LinkedIn. You can visit our website, highlighttech. com and you can tune in to our next episode. Thanks, everybody.