David Geer recently spoke with Gene Kim, CTO of Tripwire and co-founder of the IT Process Institute, an independent research organization that exists to support the membership of IT audit, security, and operations professionals. Following is an edited transcript of that conversation. You may also listen to the original interview here, or visit our Podcast Center for more audio interviews.
Hello, I'm David Geer, and we're talking with Gene Kim about the
IT Controls Benchmark survey conducted by Kim and researchers from Carnegie Mellon, Florida State, and the University of Oregon, and published by the Institute. For this survey, 98 respondents from a wide range of company sizes and industries were queried with 97 questions related to IT controls. Forty-three percent of respondents were directors, vice presidents, or C-level managers. The purpose of the survey was to demonstrate how IT organizations can begin to move from good to great in handling IT control issues.
David Geer: In general, why is this survey important?
Gene Kim: This survey is important because I think we're starting to test the medication that we're doling out. In the manufacturing world, there was a big breakthrough in the decision sciences around automotive manufacturing when the lean manufacturing researchers out of MIT basically benchmarked every major automotive manufacturing plant in the world and they found out, wow, high performance exists. They have one-half the floor space, one-half the defects, one-half the inventory, one-half the cycle time, and they've called these the high performers. And they went out and captured and codified what they did. And so what the IT Controls Performance Study is all about is really trying to replicate that same methodology and figure out what is it exactly that the high-performing IT organizations are doing, and figure out more specifically what is it that the medium and low performers aren't doing that's keeping them from being high performers.
Geer: The two top controls in this survey that were most universally present in the high performers and yet virtually absent in everyone else, including 87% of the rest of the respondents, were monitoring systems for unauthorized changes and having defined consequences for intentional unauthorized changes. What surprised you about these results, Gene?
Kim: You just really put your finger on one of the two big surprises that came out of the study, which was that of the 63 controls and the six ITIL processes that we tested, what we were looking for was, is there a smoking gun that says there are a handful of things that management is focusing on in the high performers that none of the medium and low performers are. In other words, is there a small set of things that might be keeping the low and medium performers from becoming high performers?
High performers are doing two things that medium and low performers aren't, which is do you monitor a system for unauthorized change? And the second one is, do you have defined consequences for intentional unauthorized change? So the reason we think this is important is that when you take a look at ITIL process frameworks or COBIT control frameworks, they're full of many good ideas, but you can't do all of them. So if you can't do all of them, where do you start? Where do you have the highest rate of return? Where do you get the biggest bang for the buck? And these are things that these descriptive frameworks don't really give us a lot of guidance to management on.
So what we did in this survey was really tried to create the tool so that management can focus on what is most important -- which controls and behavior and processes really lead to the most improvements in IT effectiveness and IT efficiency.
The other big surprise that came out of the survey was how good the high performers were. In the lean manufacturing world, they found a factor of 2x difference in performance -- one-half the floor space, one-half the defects, one-half the cycle time -- so it's about 2x, 2x, 2x. In the IT world, we found a much larger discrepancy in performance. Literally high performers are outperforming everybody else by a factor of 5 to 8x. We found that high performers were completing eight times as many projects, were managing six times as many applications, they were implementing and authorizing 15 times as many changes, they had one-half the change failure rate. When changes fail and cause a server's impairment or a server's down, the fixes failed one-quarter as often. They had one-third the amount of unplanned work, they had five times higher services-to-administrator ratios, and amazingly enough, they had three times higher budgets as a function of operating expense. So it just says that the reward of being a high performer is, not only are you more effective and efficient, but also you're five to eight times more effective and efficient. So it was just these two surprises. This really hit us in the forehead and said this is something that every CIO and every IT executive needs to know about.
Geer: And was it surprising that the large percentage -- I understand 87% - were not using these two controls?
Kim: Oh, absolutely. I think the reason why we called this a smoking gun was that part of the goal of the study was really to help management to focus. COBIT has 318 COBIT controls, ITIL has 11 process areas. Which ones really are the most important? But we came up with 21 of the 63 that have the majority effective. In other words, it confirms management intuition that 20% of the controls are returning 80% of the benefits. - But what made this survey, I think, so astonishing was the fact that there were actually only two things that distinguished medium performers from high performers. And so what it says is that these are literally two things that if you do not do, you cannot be a high performer. So this is what the statisticians call discriminates, meaning that if you just look at how an IT organization answers those two questions, that is almost 100% accurate predictor of whether you're going to be a medium or low performer or a high performer. So that's why we think this is such a fantastic smoking gun in terms of saying that it really is a culture of change management that brings high performance IT.
Geer: And so why do you believe it's the case if only these two factors are necessary to move from moderate to high performers, that such a large percentage of the survey group doesn't implement these controls?
Kim: That's a great question. I think there [are] at least two reasons. One is that it's always easier to deploy a technology or draw pretty ITIL process diagrams on the whiteboard than it is to really tackle cultural issues. So I think many organizations will say we need to deploy technology, but when we talk about having defined consequence for intentional unauthorized change, what that's really measuring is tone at the top of the IT organization. It's saying, does the organization really care about change management and does it really dictate the way the work is done in the organization? So I think organizations will have an easier time monitoring a system for unauthorized change, but where organizations really stumble on is saying that it is a cultural imperative to make sure that we have a culture of change management. And I think the way you start that is by having whoever sets the tone in the organization, whether it's the CIO or the VP of operations or director of production support, having that person be able to email everybody in the IT organization and say, hey, no matter how we did things before, from now on the only acceptable number of unauthorized changes is zero, and here's what we're going to do when people circumvent the processes. When someone violates the change management process too many times, we have to put you in the role where you can't make changes anymore. That's how important it is to the mission of the IT organization.
Geer: So it sounds like the more that stakeholders near the top of the organization determine an attitude and policies and enforce those and are consistent with that, and that filters down through the rest of the organization, that's how they can begin to address this?
Kim: Absolutely right. So the first [step] is making that declaration and making that overt, obvious commitment that this is how we're going to run our IT shop. And then the second one is saying, hey, there's not going to be management by belief or management by faith. We're going to measure this and we're going to hold people accountable, and this is going to be reflected in daily procedures.
Geer: How do you see the future of this issue playing out over the next few years?
Kim: What an interesting question. I think our goal at the IT Process Institute is really to create the ecumenical movement in IT management. It's just like the lean manufacturing researchers did in the 1980s. Literally when they came out with these findings, it really changed the automotive landscape to the extent that it became not only a plant manager issue, but it became an executive issue at every major automotive manufacturer. And I think what the lean manufacturing researchers did so well was that they actually proactively negated almost every objection that in this case the domestic North American manufacturers put forth. For example, they said the Japanese labor cost base is lower. And so they went out and measured that and they said, wow, that's actually not true, the cost base is higher in Japan. Then another objection was, well, you're comparing their best cars with our worst cars. So they went out and said, all right, let's measure just the luxury and performance cars. And they found out, oh no, the plant measures for just those automotive lines, they're the same, the performance differential is the same.
So, we've really taken the playbook that the lean manufacturers used and are applying it in the IT space. So what we have done is assemble many of the same sort of ways to negate the objections that IT managers put forth about why they can't be high performers or shouldn't be high performers. And two of the ones that I think we've been able to prove out in the survey are that -- one misconception is that high performers are all the largest IT organizations, and we found out, no, medium performers and high performers are about the same size. In fact, medium performers tend to be a little bit larger than high performers.
The second objection that we've been able to negate is that you have to be in a certain industry to become a high performer, so you have to be a bank. Or actually people in the financial services industry say that you have to be a stock exchange, and people who are in the stock exchange business say that we don't have to be a high performer because we're not a Telco, and Telcos will say we don't have to be a high performer because we're not a bank. So what we found in the study was that demographically, high performers and medium performers are the same, so that it literally expands all regions, all company sizes, that there's something about IT that is universal and that these principles of how to become a high performer apply the same no matter what kind of IT organization you are.
Geer: Thank you, Gene. If you would like to learn more about Gene's work or the IT Process Institute, you can visit
http://www.itpi.org.
