Metrics by Maturity

It was one fine afternoon in the middle of October, 2012 when I was breaking my head with a tricky situation I was in. I had to execute a 600 person hour(s) worth of work in 21 business days with a 2 member team. I was supposed to share the plan with my stateside counterpart by end of the day.

So, I was putting in different permutations and combinations and suddenly my mobile rang. Usually, I don’t pick any calls if I am deeply engaged in something. I looked at the number and couldn’t help answering the call. It was from one of my college friends, Siva. We haven’t spoken in a while, so, I had to answer the call.

After exchanging the pleasantries, he asked me if I can spare 15-20 minutes. He wanted to talk about his new job and had few questions around that. I told him about the scenario I was trying to solve and asked him if we can catch up in the evening or the next day. Ultimately, we decided to meet the following day for lunch. Since we are anyway catching up for lunch, we decided to invite couple of our classmates, Esther and Master.

Next day, we caught up at this restaurant called BBQ Nation. We started discussing over some ridiculously delicious food.

“So, tell me Siva. What’s your new job?” I asked

“You know, I have been leading a product team for the last few years in my company. Our guys sold that product to a competitor for a great price and started deploying entire team to other projects. Part of that movement, I have been assigned to lead the Operational Excellence team” Siva took a pause.

“Product development to operational excellence…what a move!” Esther was surprised

“It doesn’t seem like a natural transition to me, buddy” pointed Master

“Guys, it wasn’t entirely in my control. Current lead is serving her notice period and there wasn’t much of a choice for my experience level. So, kind of had to accept it in” said Siva.

“Well, everything that happens, happens for good!” I said philosophically

“Guys, need some help. My boss wanted me to put together a proposal for improvising the metrics program in my organization and outside the product development space, I don’t really have much of an experience with other business lines. I don’t know where to start!” Siva stated his problem statement

“What’s the status quo?” I asked

“Well, we have few sets of metrics defined by functional lines such as product sales team, product development team, product maintenance team, product support team and product testing team. They have been in place for the last few years and majority of the teams look good with respect to these metrics. I did a quick search online and spoke to few seniors to determine if there are any gaps. All looks good. So, I don’t really know what else I can improve here” explained Siva.

“It’s similar situation in our company as well. Our operational excellence team defined metrics by functional lines and we provide data for those each month” said Master.

“Same or similar metrics in place for the last few years kind of tells you that there is a great opportunity to improvise” I quoted.

“Hey, don’t fall for this. Sekhar would again open up his project management philosophy” Esther said teasingly.

“Yeah, Esther is right. Don’t worry much about the program. I will send the stuff that we have in our company. Just change the names and present it at your place. Most of the folks do the same, anyway! They copy the templates and processes from their previous company and project it as theirs in their new company” Master added more salt.

“Guys, I am being serious here. I don’t need a solution or a readymade deck. Just help me find right direction and I will build on the top of it” said Siva.

What is a metric?

“Siva, here is the question for you. Define what a metric is for you” I asked

“Any measureable parameter of a project, process or a service” said Siva.

“Yupp...I agree” supported Esther.

“And we measure plenty of them such as SLA, customer service index, number of requests per hour, effort variance, average response per a ticket etc...” added Master

“So, why do you a measure a parameter” I asked

“Okay, buddy, stop treating us like kids.” Esther wasn’t excited to answer such basic questions

“I am sorry, I didn’t mean that way. I was just trying to start at ground zero” I said apologetically.

Why a metric?

“Never mind, Sekhar! She was just pulling you. Let’s continue the discussion. So, you were asking why we measure these parameters. In my view, these are the parameters that tell us how the project/process/service is doing. We would have a target for each of these parameters and metrics would help us determine if we are doing well or not with respect to these parameters.” said Siva

“That’s correct. In fact, they not only tell us if we are doing well or not, but also tell us if we are in the right direction or not” added Master.

Which metrics?

“Cool...now, how do you know which parameters to measure?” I asked again

“Well...it depends up on the goals of the project/process/service, I suppose. For example, if something is service driven, we would look at SLA as a metric and if something is project driven, we look at effort variance, schedule variance, rework etc...” Esther was not going to be left behind

“So, where do your project or process goals come from?” I pushed another button

“From the previous year’s goal sheet!” Master said with a big grin

“That’s funny! At least, my goals come from my business lines’ objectives” said Siva.

“And your goals and objectives remain same year on year” I asked

“Obviously not! For some of the goals, the targets shoot up, while some new goals get added. But for sure, they won’t be same year after year.” Siva replied

“In fact, the objectives of my business line have changed at least by 70% in last 2 to 3 years” supported Esther

“Then, how can you have similar or same metrics measured year after year? Shouldn’t the metrics program get enhanced as the objectives of the business lines change?” I asked

“That’s an interesting point. But my operational excellence team doesn’t even know my business line’s objectives” said Master

“Then, how can they define the metrics for your teams?” I asked again

“Sekhar, it doesn’t mean we should change metrics every year, right? I mean, you need to measure something consistently over a period of time to determine how things are” said Siva

“I am not saying change the metrics every time. I am just saying match the right metrics at the right time. Here is my definition for metrics. A metric is a measurable oriented decomposition of project/process/system/business objectives. So, you should continue to enhance your metrics program as your process/project’s objectives get enhanced” I replied

“What do you mean match the right metrics” asked Esther

“Let me give you an example. When you are setting up a new team for managing a process or an application or a level 1/level 2 support, your initial objective would be to meet the triple constraint which is on time, within budget and with decent quality. Right?” I asked

“I agree.” responded Master

“What about after a year or two? Would you still be interested in being measured for time, cost, and quality?” I inquired

“Of course, not! I think, I would have built enough confidence in my customers by then. If I am still being measured for the same parameters, then, I might have done a bad job” replied Siva

“I think, I am getting what Sekhar is pointing at. If I understand you right, you are saying what was being measured couple of years ago should be a given now and I should focus my attention and measurements on bigger and better objectives.” summarized Esther

“Right...that’s one point. There is another interesting aspect to the metrics program. Let me ask you a simple question. I am sure there would be a monthly or bi-monthly report that gets presented in your organization as part of metrics program.

How many times do you actually look at that report with genuine interest? How many times, does that report convey that there is a problem or a potential problem brewing in your project/process/application/program? On how many occasions have you ever opened a prior report for validating certain things?” I asked

“Never” said Master.

“Once or twice” Esther replied.

“Siva, that’s the improvement you could make in your metrics program. End of the day, if the output of metrics program is going to be an FYI, then, it’s not entirely serving the purpose. There should be no more than 50% FYI metrics in your report. At least, the other 50% should present you actionable information.” I summarized

“That’s a good input for me. Tell me more” Siva is curious now.

The Operational excellence philosophy

“Consider an application team that’s setup for the first time. Like we discussed earlier, their initial objective would be to deliver on time, within budget and with good quality. Now, to measure these objectives, I probably would use measurements such as effort variance (within budget), Schedule variance (on time), rework percentage and defect density (good quality). It’s typical that on a monthly or a bi-monthly basis, I would share these metrics with the customer or business.

Let’s say, 7 to 10 projects were executed in that application over a period 18 to 24 months meeting these objectives. By now, the customer or the business would be more than happy with the ability of the team. So, for them, getting a quality deliverable on a timely fashion is a ‘minimum’ expectation. They don’t hope for or anticipate a quality deliverable any more. They expect it by default. In other words, they won’t really care about these metrics anymore. So, what is the point in producing/sharing the same set of metrics again and again?” I elaborated

“That’s an interesting question. So, are you suggesting that, we shouldn’t capture/share those aforesaid metrics anymore? In case we don’t capture them anymore, don’t we run into the risk of not knowing when things actually go wrong from schedule and cost perspective? ” asked Siva

“No, I am not suggesting that. I was merely saying, sharing just those metrics won’t really help in anyway. They have now become hygiene metrics. These would be your FYI metrics now. You need to come up with next set of objectives and start measuring those. That way, you would still keep an on eye on your hygiene metrics but focus on next objectives.” I replied

“Sounds interesting enough...please elaborate more” asked master

“Continuing the same example that we discussed earlier, once the team is on the top of cost, time and schedule kind of metrics, you can turn focus on to Product Quality metrics or Process quality metrics or User experience metrics or Business quality metrics or even People quality metrics.

You could say, now that my delivery is continuously stable, let me focus my attention to improve my user community experience. As an example, if they are using the application to underwrite loans and if there was a series of 10 steps that take about 25 minutes to enter data, I could always look at ‘how can I reduce the time or automate some of those steps or reduce the steps so on and so forth’.

Or you could say, now that my delivery is continuously stable, let me focus my attention to improve the maintainability or testability of my application. As an example, if quality assurance activity, on an average, takes 3 weeks for projects, how can I bring that down to 2 weeks so that ‘Time to Market’ is improved for my business.” I elaborated

“Sounds like an idea. But, how do I know where to focus. Should I focus on product quality metrics or business quality metrics or user experience metrics? How do I determine that?” asked Siva

“Good question. I call this concept Metrics by Maturity. For any process/application/ system/service, spend time with its owner, business and technology teams and understand what the maturity cycle of that entity is. Maturity cycle can be derived out of the expectations of the business/customers. Answers to few simple questions, as listed below, might help derive the maturity cycle.

• What is your immediate need from this process/product/system?
•  How do you see this process/product/system shaping up two years from now?
• What is the roadmap of the business that this product is serving to?
• What role do you expect this process/product/system to play in the business? In other words, which objectives of your roadmap are expected to be addressed by this process/product/system

Again, as an example, you could have a four stage maturity cycle:

•  Stage 1 would be Stable output, where the business might say, ‘Hey guys...Don’t worry about anything else, just give me quality deliverable on time’. So your metrics would be effort variance, schedule variance, rework %, defect density, SLAs etc...Lot of times, standout performances from few team members would do the difference here.

• Stage 2 could be Stable product, where the technology team might say, ‘Hey guys...now that we are comfortable with the delivery, let’s focus on product quality metrics and turn our application into a well manageable, maintainable, configurable and testable application so that our stage 1 (stable output) won’t depend on individual brilliance anymore’. For a software product, metrics for this stage would be Cyclomatic complexity, code coverage, Maintainability metrics, Package metrics etc...

The point to be noted here is that if your project is to develop a product, you might combine both stage1 and stage 2

• Stage 3 could be Enhance business value, where the product team might ask the business or customer, ‘Hey guys...now that our delivery is quite stable and we have turned the application into better manageable one, please tell us how we can improvise this into more profitable product?’ For customer facing process/products, such metrics could be time it takes to add a customer, number of times a customer had to call customer service for a feature that’s already available in the product, % of reduction in customer calls or service requests, average time customer spends on a customer service call, % of effort a user spends to prepare the data to enter in to the product, analyze the data after downloading from the product etc...

• Stage 4 could be Lights-on Maintenance, where the business might say, ‘Hey guys...we are not going to enhance this product anymore, we just need its existing features to be available without any interruption. So, put just enough effort on this to resolve any tickets or queries as fast as you can’. So your metrics could be SLA, effort consumption per ticket, % of ticket reduction etc...

Note that these stages and their order wouldn’t have to be the same. You would have to derive the maturity cycle of your process/product/service based on the roadmap and expectations of various stakeholders.” I concluded my long speech

“I think I got few inputs that I was looking for. Let me put something on the paper and see how it comes out. If things go well, I will buy you lunch next time but for today, let’s go dutch” Siva ended the luncheon!

ఓం నమో ప్రాజెక్ట్ మనేజ్మెంతాయ నమః