BA/QA Team
With a focus on requirements and quality management, Catapult’s business analysis and quality assurance teams utilize industry best practices and experience to deliver best-of-breed technology solutions. Their blog offers a range of insights including adapting Agile methodologies, requirements prioritization techniques, and testing strategies.

Metric Misuse – Quality Assurance Metrics Gone Awry   

Tags: General Quality Assurance Topics
Technorati Tags:

I was reading through some posts from Bob Sutton, one of my favorite management gurus, and I ran across a post that contains one of my favorite Dilbert comic strips.

Bob Sutton's post, as well as the comments that I made on his blog, reminded me of one of my favorite topics: misused Quality Assurance metrics.

Tying Quality Assurance Metrics to Financial Rewards – A Dangerous Game

"Treat monetary rewards like explosives, because they will have a powerful impact whether you intend it or not." -Mary and Tom Poppendieck, authors of Implementing Lean Software Development: From Concept to Cash

Over the years, many people have asked me what Quality Assurance metrics they should use to evaluate employee performance. My advice is that Quality Assurance metrics should not be used directly to evaluate employee performance. The Dilbert comic strip may seem a bit extreme, but it's exactly what happens when employee performance is based strictly on metrics. This is true regardless of whether monetary rewards are explicitly tied to the metrics or not.

In my comments on Bob Sutton's blog, I mentioned three specific metrics that had unintended effects when used for evaluating employee performance:

  1. rewarding testers for the number of test cases they wrote resulted in poorly written test cases;
  2. rewarding testers for the number of bugs they found resulted in a high number of unimportant or duplicate bugs reported; and
  3. penalizing testers for bugs rejected by the test lead or development staff resulted in important bugs going unreported.

Many people think that they have the ability to write a set of metrics that can be used to unequivocally gauge the performance of a Quality Assurance professional, but I have not yet encountered a metric that couldn't be manipulated to favor the employees.

(If the metric can't be gamed, it probably isn't under the control of the employees, so it wouldn't be effective at driving behavior anyhow.)

Are Metrics Worthless Then?

Actually, metrics are a great tool for identifying coaching opportunities and potential problems. However, in order to get honest metrics, they shouldn't be used directly for employee evaluations or employee rewards.

When I've looked at the metrics that I mentioned earlier with an eye towards coaching, I had excellent results.

  1. Reviewing the number of test cases written helped me identify a tester on my team who was putting much more detail than I wanted into his test cases. After some coaching, he was able to consistently meet my expectations.
  2. Reviewing the number of bugs found by each tester helped me identify a tester who was digging into the root cause of the most difficult to reproduce bugs. She didn't report as many bugs as others, but her work was critical to getting a great product out the door in a timely manner. It turned out that she was the most skilled tester even though she reported the least bugs.
  3. Reviewing the number of bugs rejected by the development staff helped me identify a manager who was evaluating his programmers based solely on the number of valid bugs found in their code. The developers were motivated to simply mark bugs as invalid rather than fix the bugs. This insight allowed me to address the problem directly with that manager.

Good Quality Assurance metrics provide powerful tools for managing a Quality Assurance team when used properly. However, they shouldn't be used in a vacuum. They should just be considered one data point among many.

I was only able to scratch the surface of this topic in this blog post. I plan to discuss specific metrics in future blog posts. 

Your Experiences

I know that a lot of people feel passionately about Quality Assurance metrics, both pro and con. I'm very interested to hear about your experiences with Quality Assurance metrics. Have you found any that were particularly useful? Have you found any that had unintended consequences?

 
Posted by  Kevin Rutkowski  on  4/25/2011
0  Comments  |  Trackback Url  | 1  Links to this post | Bookmark this post with:        
 

Links to this post


Trackback from  BA/QA Team  on  2/15/2011  5:48 AM
Testing for IE6 Compatibility – Microsoft Expression SuperPreview 

[...] Should Internet Explorer 6 (IE6) be supported?
Despite the f ... [...]



Trackback from  BA/QA Team  on  2/23/2011  8:24 AM
Testing for IE6 Compatibility – Internet Explorer Application Compatibility VPC Image 

[...] Although developers consistently tell me that IE6 should no ... [...]



Trackback from  BA/QA Team  on  3/9/2011  9:36 AM
Internet Explorer 6 Countdown 

[...] IE6 Deathwatch
I recently posted an article about why many c ... [...]



Trackback from  BA/QA Team  on  3/9/2011  9:36 AM
Internet Explorer 6 Countdown 

[...] IE6 Deathwatch
I recently posted an article about why many c ... [...]



Trackback from  BA/QA Team  on  4/4/2011  8:15 AM
Secure Password Policies – Protecting Yourself (and Your Company) on the Web 

[...] In my last post, I mentioned that I was joining my local cha ... [...]



Trackback from  BA/QA Team  on  4/12/2011  9:57 AM
Agile Test Automation – How Much is Enough? 

[...] I recently had the opportunity to sit on a panel that discus ... [...]



Trackback from  BA/QA Team  on  4/18/2011  8:26 AM
Testing Web Application Security – Role-Based Security Vulnerabilities 

[...] In an earlier post, I mentioned that I was going to learn mo ... [...]



Trackback from  BA/QA Team  on  5/20/2011  9:23 AM
Metric Misuse – Code Coverage 

[...] In my last post, I promised that I would discuss problems wi ... [...]



Trackback from  BA/QA Team  on  5/20/2011  9:23 AM
Metric Misuse – Code Coverage 

[...] In my last post, I promised that I would discuss problems wi ... [...]



Trackback from  BA/QA Team  on  5/25/2011  8:07 AM
Drama Addiction 

[...] In my last post, I lamented the common scenario of people be ... [...]



Trackback from  BA/QA Team  on  5/25/2011  8:07 AM
Drama Addiction 

[...] In my last post, I lamented the common scenario of people be ... [...]



Trackback from  BA/QA Team  on  6/2/2011  3:32 PM
Replicating Success 

[...] "Don't it always seem to go that you don't know what you got ... [...]



Trackback from  BA/QA Team  on  7/11/2011  5:14 AM
Effective Global Teams – Recognizing the Individual 

[...] In my last post, I discussed some techniques for making glob ... [...]



Trackback from  BA/QA Team  on  10/27/2011  7:16 AM
Too Many Cooks… 

[...] I recently read about the problems that Target's new website ... [...]



Comments

Name *:
URL:
Email:
Comment:


CAPTCHA Image Validation