Metrics Anyone?

Managers sometimes come to their employees requesting "metrics" data. What is Metrics Data, you might ask? Metrics data is a measurement that teams come up with to demonstrate "continuous improvement" in the form of a number or numbers that are supposed to go up or down, depending on what's being measured.

Who is the audience for metrics? Usually the boss's boss. He doesn't necessarily want to see specific information about the product to get a feel for the quality, he wants to see some sort of quantifiable number to show that his product is improving.


AN EXAMPLE

Suppose TEAMX produces widgets. They produce between 10 and 60 widgets a month, depending on demand.

TEAMX's boss's boss has requested metrics. Since TEAMX's boss feels the members of TEAMX are best familiar with their own process, he tells them THEY must come up with some sort of measurement he can show to his boss to indicate that TEAMX is dedicated to continuously improving their process in order to increase the quality of the widgets they produce.

TEAMX, a very busy team, couldn't think of a way one could really quantify how they're improving the quality of their production of widgets at first. They decided to use the number of widgets produced each month as their metric. Their goal was to increase production 10% per month. Here's their first 4 months' "metrics:"

Jan 50 widgets
Feb 40 widgets
Mar 30 widgets
Apr 20 widgets

When asked why the numbers were dropping, TEAMX said, "Well, there was decreasing DEMAND for our widgets. We only produce what's ordered by the customer."


SECOND TRY

TEAMX was asked to find another metric, so they decided to measure the number of defects closed per month. The goal was to close 10% more defects each month. Here's the next four months' "metrics:"

May 120 defects closed
Jun 40 defects closed
Jul 160 defects closed
Aug 80 defects closed

When asked why their numbers were all over the place, TEAMX said, "Well, June and August were our busiest months for orders, so we had less time we could devote to fixing defects. Also, there were fewer defects opened during those months."


THIRD TRY

Once again, TEAMX was asked to find another metric. They asked their boss, "What are you looking for?" He stated, "We just need something that shows a regularly improving number that indicates we are increasing our quality."

So TEAMX decided to measure the amount of time it took to produce a widget. Normally, it took about 2.5 person-hours to produce a widget. They decided their goal was to decrease the amount of time to produce a widget by 10% per month.

Over the next several months, they found many ways to cut costs, cut corners, and cut waste in order to reduce the length of time it took to produce a widget. Here's the next four months' "metrics:"

Sep 2.47 person-hours
Oct 2.44 person-hours
Nov 2.41 person-hours
Dec 2.38 person-hours

Their boss's boss was thrilled with the numbers. Each showed a decrease in the length of time it took to produce a widget. However, there was some fallout from this exercise.


FALLOUT

Here's TEAMX's next four months of defects opened:

Jan 80 defects opened
Feb 90 defects opened
Mar 100 defects opened
Apr 110 defects opened

The following month, management decided to just let TEAMX produce widgets and not try to make them think of some sort of measurement that showed continuous improvement in the product's quality.

Management learned three things:

1. Asking teams to come up with a measurement that can be quantified will always result in a quantifiable measurement.

2. There is not always a quantifiable trigger you can key off of to measure quality.

3. Showing "improvement" in the measurements that CAN be quantified is not always an indication of quality.

.