NVP Software Solutions

Tag: Metrics

  • Using Metrics for Process Improvement

    There are three aspects to using metrics for Process Improvement:

    • Defining the correct metric to gather
    • Gathering the correct statistics to determine that metric
    • Making use of the results

    We have already discussed the first two in previous posts so we will make the assumptions that the correct metric has been defined and the correct statistics gathered to support that metric. We now need to make use of those metrics. We will start with an example related to Software Testing and then move to the general case.

    Assume we have an existing application to which we are making changes over time. It is a reasonably critical application for our company and we have built an extensive set of regression tests over the years. The regression tests have been augmented with new ones that represent implemented functionality. The regression tests are run with each release of code. We have been retaining the statistics on the number of tests that fail each time the regression tests are run. The failures are investigated, fixed if necessary (high enough priority), and the release implemented.

    Recently the number of failures for each run of the regression base has been increasing and it is becoming difficult to investigate and fix all the failures prior to implementation.

    The following need to be checked:

      1. Is the percentage of defects the same as before and the raw number is increasing solely as a result of the regression base getting bigger?

    This requires another measurement to determine the size of the regression base so we can complete the comparison.

      1. Is the number of defects increasing as a result of the increasing size of the application?

    This also requires another measurement that can determine the size of the application.

      1. Is the number of defects increasing as a result of making parts of the application active that were not used before?

    Again this requires a measurement that indicates application usage.

    The action to be taken depends on the answers to the above question and may vary from doing nothing to starting some refactoring and major rewrites.

    We have outlined three general principles with this example:

    1. Gather the statistics (both for the activity that we want to measure and for comparison purposes)
    2. Investigate the reasons for any change
    3. Take action based on that change.

    Contact us to talk about your metrics and how they can be used.

  • Examples of QC & QA Metrics

    It is reasonably easy to define measurements and metrics for any particular organization or project. The bigger question is whether those metrics are relevant to the project and work at hand.

    Metrics or Measurements Specific to Quality Control

    • Number of tests
    • Number of defects
    • Defects per module
    • Percent of tests completed

    The above are fairly obvious examples of metrics and are quite frequently gathered along with a number of others as Quality Control and Quality Assurance data. Many of the test tools supplied commercially or available as Open Source gather one or more of these measurements automatically and make them available on graphs and in spreadsheets for manipulation.

    Metrics or Measurements Specific to Quality Assurance

    • Number of processes in place
    • Degree of Acceptance of a process
    • Effectiveness of any process
    • Unmeasured processes

    The above metrics vary from easy-to-gather and maintain to almost impossible to accurately estimate. They usually do not depend on any particular project although they may be department dependent based on the level of detail. However, we find that most organizations, while using many processes with varying degrees of effectiveness rarely make a concerted attempt to measure them. It is very tempting to simply use the process without considering whether or not it could be improved.

    The key questions to ask are as follows

      1. Can the metric or measurement be gathered? (Preferably without impacting the test or quality effort.
      2. Are the figures accurate? (This needs to be tested just like the application.)

    & Most important

    1. Are we making use of the metric to provide information to the project or organization?

    The reason that we place such emphasis on the last point is that we encounter far too many instances where metrics and measurements are being gathered with no clear idea of why or how they can be used.

    • Do you want to know if your processes are effective?
    • Do you want to know if your processes can be improved? (And how to do it)
    • Do you want to know how you compare to other organizations?

    Contact us to see how your company can measure your metrics.

    Next week we will discuss using Metrics.

  • Quality Assurance Metrics

    In our last post we listed the need for some way of measuring Process Improvement without going into too much detail about how. What we were referencing was Quality Assurance Metrics.

    We define a metric similar to the way Wikipedia does in that it is the degree to which software possesses some desired characteristic. This feeds directly into the comments from our last post about needing to have something to measure in order to know if Process Improvement has occurred.

    More specifically, we can define a Measurement as something that has a single dimension: Height; Weight; Count of defects; Count of Requirements and many others.

    A Metric is derived from two or more Measurements and is often expressed as a ratio or a percentage. For example, we might count the number of defects but we might also count the number at each Severity level and then express that as a percentage of the total. This gives us a lot more information as the following example shows:

    If we have 100 defects and 20 of them are of the highest severity then we can state that 20% or one fifth are of the highest severity. We can watch that percentage over the duration of the project and track its movement up and down. We can also compare our percentage of highest severity defects to other projects directly using the percentage and see if our percentage is out of line with other projects. Note that this removes the bias that might occur with different project sizes. If someone has a much smaller project, they would expect commensurately fewer defects in total but might experience a similar percentage of high severity ones.

    In general metrics normalize the results to allow comparisons.

    As Peter Drucker said “you can’t manage what you can’t measure,”

    Give us a call to see what Metrics you need!