Newsletters

qaSignature Newsletter

Volume 2, Issue 15 - August 2008

Ten Steps to Building Software Test Automation That Works

Step 9: Measuring Results

What Metrics are Important?

Whether you are on vacation relaxing or at work taking advantage of the relatively quiet time, we invite you to view actual results of a client who earned the prestigious CIO 100 Award issued annually by CIO Magazine.

We are going to share actual results of 6 important metrics from this automation project measured over 7 releases and 25 ½ months.

The first step in the "Ten Steps to Building Software Test Automation That Works" White Paper was to define goals and objectives before you begin automating your software testing. If you are a typical company, you probably included at least 3 of the goals below on your list.

Goals of Software Automating Software Testing:

  1. Expand the test coverage in order to improve product quality
  2. Reduce the release cycle time in order to get the products to market faster (Time to Market)
  3. Reduce the cost of quality by incorporating technology in the software quality assurance testing
  4. Build a solution that can be easily expanded to satisfy growing requirements

Measuring Performance - You Get What You Measure.

In my experience, there are 2 primary reasons for defining performance metrics and tracking the performance against the metrics.

  1. You direct attention and effort to the performance being measured
  2. You can review progress toward achieving the objectives

The Case Study - Exhibit A

So as promised, we will analyze the results of an actual project in Exhibit A below.

Exhibit A: Metrics Checklist for Software Test Automation
Release Number Total number of Resources Count of MRs* Duration (months) MRs per person per month MRs per Month
2.05 27 562 6 3.5 93.7
2.07 18 548 4 7.6 137.0
2.08 18 576 4.5 7.1 128.0
3.00 16 956 5 12.0 191.2
3.01 11 515 3 15.6 171.7
3.02 6 194 1.5 21.6 129.3
3.03 5 118 1.5 15.7 78.7

*Modification requests include defects, enhancements and new features.

The results in Exhibit A will be compared to the goals defined above to determine how well they were achieved.

Goals of Automation:

  1. Expand the test coverage in order to improve product quality

    Test coverage was not a specific metric tracked in this example and the actual coverage is unavailable. It can be inferred, however, because 1. MRs continued to increase, 2. the release duration decreased from release to release and 3, resources on the project continued to decline, the additional testing was not a result of adding more testers. In fact, the resources were declining while MRs increased. These results were due to expanded automated test coverage.

    Quality improved by catching more defects earlier in the process. As you can see this resulted in a declining number of defects in the dot releases, i.e. 3.01, 3.02, etc.

  2. Reduce the release cycle time in order to get the products to market faster (Time to Market)

    Cycle time was decreased by 4.5 months or 75% from 6 to 1.5 months as the automation test coverage increased.

  3. Reduce the cost of quality by incorporating technology in the software and hardware quality assurance testing

    Although this metric was not tracked in the chart below, the client has gone on record as having achieved a 400% improvement in the combined development and QA productivity.

  4. Build a solution that can be easily expanded to satisfy growing requirements

    The Achilles' Heel of automation is being able to develop test automation that is easy to maintain as the software and the test automation coverage continues to grow. It is evident by the productivity improvement that the automation passed the test of maintainability.

In summary, metrics tell the true story of the impact software test automation has on the development process. Test Automation can help you achieve your objectives much faster and at a lower total cost. A CIO 100 Award would be a nice bonus.

By Clayton Blaylock
VP, qaSignature

I hope you found this exercise of value. Please e-mail your comments to cblaylock@qasignature.com or call me at (617) 510-6545.

We have developed a Metrics Checklist that has expanded the above list of metrics from 6 to 10. You can get your copy by clicking on Free Metrics Checklist.

Click here for a PDF copy of the complete white paper

Proof of Concept Service

To Companies Who Want To Build Test Automation - But Don't Have The Time Or Resources To Get Started...

Take qaSignature For A Test Drive.

An effective approach to getting started with test automation is the qaSignature Proof of Concept Service. Standard test automation produces limited results and is difficult and costly to maintain. The qaSignature methodology is different. We'll prove it. Give us your most difficult automation challenge. We will:

  • Recommend the optimal automation tool (or use yours)
  • Build the automation as a proof of concept and show you how our methodology works
  • Let you change the code, we will update the test and rerun
  • Show you the results in our easy to follow test logs
  • Prioritize areas for automation based on results from a cost benefit analysis
  • Give you a fixed price estimate for a fast start program

Here's what our clients had to say.

"The amount of automation that they were able to develop in a couple of days including fact finding was impressive. Their competition could not even begin the process without first learning the application."
Keith Hillyard, Custom Engineering SQA Manager, Kronos

"The qaSignature proof of concept is a "no brainer." They delivered much more than we expected. The process helped us map out our QA Automation Framework."
Frank F. Frazier Jr., Senior Program Manager, ZANTAZ

"The dedication, enthusiasm and passion of the qaSignature team was refreshing. I would recommend them to anyone who needs to develop QA Automation."
Paul Bradley, Systems Consultant, DAFCA Inc.

"The proof of concept demonstrated the applicability of applying the qaSignature QA Automation Methodology to our Agile Development Process."
Keith Hillyard, Custom Engineering SQA Manager, Kronos

"I was impressed by qaSignature's objectivity. They recommended the right tool for the job even though it wasn't their standard offering. They really know QA Automation."
Larry Leonard, Director of Development, SmartTime

Call us now @ 617 510-6545

Click on the link below for details:

Proof of Concept

«« Back to qaSignature Newsletter Archive