Category: QA

  • Implementing Metrics

    Implementing Metrics is the hardest aspect of the entire Process Improvement process. It is reasonably easy to determine a need for metrics – anything that can be measured and will lead to cost savings is a candidate for a metric. It is usually fairly straightforward to determine what to measure as long as the question as to what is needed has been answered correctly. Some effort may be required to get to the correct metric that measures the actual cost of any issue or failure but some experimentation can usually lead to a good approximation and it can be modified after if required. The actual measurement needs to have all the characteristics of a good measurement but that can be determined.

    Implementing the metric falls into two categories – one of which we will discuss in this post and following up with the other one in the coming weeks. The two categories are:

    1. Taking the measurements.
    2. Acting on the results.

    Taking the measurement

    Taking the measurement assumes we have defined the need for the metric, have researched what and how we want to measure and are ready and wiling to implement. Now comes the part that affects other people and projects. It is unlikely that it is just our project or our work that will need to be measured. If it was just ours, we could have completed the measurement or simply implemented the process improvement long ago and moved on. This is now going to impact other timelines and people.

    There are several steps:

    1. Determine who or what is going to be impacted
    2. Quantify the impact
    3. Anticipate the objections
    4. Prepare answers to the objections
    5. Demonstrate the benefits
    6. Win Support
    7. Start the measurement program
    8. Provide support to the people doing the measurement
    9. Publish the results
    10. Be honest about the results

    If the above process is carried out correctly, not only will we get valid and useful results, we will also have support for our next measurement program. If any of these look unfamiliar give us a call and we well talk about it.

    Next week: Acting on the results.

  • Using Metrics for Process Improvement

    There are three aspects to using metrics for Process Improvement:

    • Defining the correct metric to gather
    • Gathering the correct statistics to determine that metric
    • Making use of the results

    We have already discussed the first two in previous posts so we will make the assumptions that the correct metric has been defined and the correct statistics gathered to support that metric. We now need to make use of those metrics. We will start with an example related to Software Testing and then move to the general case.

    Assume we have an existing application to which we are making changes over time. It is a reasonably critical application for our company and we have built an extensive set of regression tests over the years. The regression tests have been augmented with new ones that represent implemented functionality. The regression tests are run with each release of code. We have been retaining the statistics on the number of tests that fail each time the regression tests are run. The failures are investigated, fixed if necessary (high enough priority), and the release implemented.

    Recently the number of failures for each run of the regression base has been increasing and it is becoming difficult to investigate and fix all the failures prior to implementation.

    The following need to be checked:

      1. Is the percentage of defects the same as before and the raw number is increasing solely as a result of the regression base getting bigger?

    This requires another measurement to determine the size of the regression base so we can complete the comparison.

      1. Is the number of defects increasing as a result of the increasing size of the application?

    This also requires another measurement that can determine the size of the application.

      1. Is the number of defects increasing as a result of making parts of the application active that were not used before?

    Again this requires a measurement that indicates application usage.

    The action to be taken depends on the answers to the above question and may vary from doing nothing to starting some refactoring and major rewrites.

    We have outlined three general principles with this example:

    1. Gather the statistics (both for the activity that we want to measure and for comparison purposes)
    2. Investigate the reasons for any change
    3. Take action based on that change.

    Contact us to talk about your metrics and how they can be used.

  • Examples of QC & QA Metrics

    It is reasonably easy to define measurements and metrics for any particular organization or project. The bigger question is whether those metrics are relevant to the project and work at hand.

    Metrics or Measurements Specific to Quality Control

    • Number of tests
    • Number of defects
    • Defects per module
    • Percent of tests completed

    The above are fairly obvious examples of metrics and are quite frequently gathered along with a number of others as Quality Control and Quality Assurance data. Many of the test tools supplied commercially or available as Open Source gather one or more of these measurements automatically and make them available on graphs and in spreadsheets for manipulation.

    Metrics or Measurements Specific to Quality Assurance

    • Number of processes in place
    • Degree of Acceptance of a process
    • Effectiveness of any process
    • Unmeasured processes

    The above metrics vary from easy-to-gather and maintain to almost impossible to accurately estimate. They usually do not depend on any particular project although they may be department dependent based on the level of detail. However, we find that most organizations, while using many processes with varying degrees of effectiveness rarely make a concerted attempt to measure them. It is very tempting to simply use the process without considering whether or not it could be improved.

    The key questions to ask are as follows

      1. Can the metric or measurement be gathered? (Preferably without impacting the test or quality effort.
      2. Are the figures accurate? (This needs to be tested just like the application.)

    & Most important

    1. Are we making use of the metric to provide information to the project or organization?

    The reason that we place such emphasis on the last point is that we encounter far too many instances where metrics and measurements are being gathered with no clear idea of why or how they can be used.

    • Do you want to know if your processes are effective?
    • Do you want to know if your processes can be improved? (And how to do it)
    • Do you want to know how you compare to other organizations?

    Contact us to see how your company can measure your metrics.

    Next week we will discuss using Metrics.

  • Quality Assurance Metrics

    In our last post we listed the need for some way of measuring Process Improvement without going into too much detail about how. What we were referencing was Quality Assurance Metrics.

    We define a metric similar to the way Wikipedia does in that it is the degree to which software possesses some desired characteristic. This feeds directly into the comments from our last post about needing to have something to measure in order to know if Process Improvement has occurred.

    More specifically, we can define a Measurement as something that has a single dimension: Height; Weight; Count of defects; Count of Requirements and many others.

    A Metric is derived from two or more Measurements and is often expressed as a ratio or a percentage. For example, we might count the number of defects but we might also count the number at each Severity level and then express that as a percentage of the total. This gives us a lot more information as the following example shows:

    If we have 100 defects and 20 of them are of the highest severity then we can state that 20% or one fifth are of the highest severity. We can watch that percentage over the duration of the project and track its movement up and down. We can also compare our percentage of highest severity defects to other projects directly using the percentage and see if our percentage is out of line with other projects. Note that this removes the bias that might occur with different project sizes. If someone has a much smaller project, they would expect commensurately fewer defects in total but might experience a similar percentage of high severity ones.

    In general metrics normalize the results to allow comparisons.

    As Peter Drucker said “you can’t manage what you can’t measure,”

    Give us a call to see what Metrics you need!

  • Quality Assurance Supports Process Improvement

    Quality Assurance Supports Process Improvement by showing us when the improvement has occurred. This is typically done through measuring the existing process; making the change and then measuring the results of the new process. In last week’s blog we discussed examples of Process Improvement. Now we need to ensure that the Improvement has actually occurred.

    The initial steps to tracking improvement are as follows:

    • Identify the process we want to improve
    • Identify the aspect of that process that needs to be improved
    • Identify a characteristic that will show if the improvement has taken place after the change
    • Find or create a measurement that will show change occurring in the characteristic
    • Measure the characteristic under the current process for a sufficient length of time to get a valid result
    • Make the change
    • Measure the characteristic under the new process for a sufficient length of time to get a valid result

    The above steps are not necessarily easy and will depend on what already exists in the place you are working. In order to do the first three steps you need to have processes in place or be able to identify them and their characteristics. The fourth step requires working out a measurement that validly checks the aspect that is planned for improvement. We then need to wait for while the measurement is taken with the current process. The change has to be implemented and used in the process without any modifications in order to allow us to make valid measurements of the impact of the change.

    Once all this is done, then we are in a position to measure the difference and decide on whether the process was actually improved. Quality assurance supports all of this by defining the necessary measurements, gathering the statistics and analyzing the results. NVP Software Solutions can complete all of the above steps for you.

  • Examples of Process Improvement

    Examples of Process Improvement are sometimes a little harder to find and measure than Product Improvement. As long as you have a standard to compare a product, any change can be determined and it is usually easier to determine if the quality of the product has improved or not. Also, we may not have to actually determine the measurement methodology prior to the product being produced. We can take an existing product, determine a standard and decide if it has met that standard. Future products can then be compared against that standard. A process, on the other hand, does not necessarily have an end product. It is part of what produces the end product and may not ‘exist in the literal sense’ after it is finished.

    There are two ways we can measure Process Improvement and then determine if the Process has improved.

    • Measure the created product. This becomes product measurement and that can be used as feedback to improve the process.
    • Measure the actual process.

    The first has already been covered so we will discuss the second.
    If we make a change to the process, the fundamental question is has the process improved.
    Examples:

    • The process continues to generate product after a change with no reduction in quality. Process Improvement.
    • The process generates the same product while using less materials. Process Improvement.
    • The process generates the same product with less waste. Process Improvement.
    • The process is halted or interrupted less often than before. Process Improvement.
    • Less resources are required for the same quality product. Process Improvement.
    • The process has reduced variation. Process Improvement.

    All of the above require a proper measurement process (another process) to be in place before the process is launched. It is very difficult to measure some of the above process improvements without having an existing measurement process in place. This may eventually remind you of The Siphonaptera!

    However, if you want to find out about your own processes, take a look at our Assessment process which identifies them.

  • Process Improvement

    Ongoing Process Improvement is the second critical aspect of process implementation. Last week we identified the components of a process. Once they are identified and it has been determined if they are applicable then the next step is Process Improvement.

    Many organizations identify the need for Process Improvement but they seem to say it as statement sliding the two words together as if it is something that will occur without effort and frequently without results. Philip Crosby’s book “Quality is Free” talks about this and in our last post we mentioned getting the process just right. Getting it just right is not a one step process. At a minimum it is a 4 step process and those steps contain a lot of subsidiary detail.

    Initially, as mentioned last week, we have to identify the processes that are in existence. Part of NVP’s Assessment process is to identify the existing processes. This can be done internally although that can be more difficult since people tend to live through the process and cannot see it ‘from the outside’.

    Once the processes are identified, then we need to see how they can be improved. This requires thinking ‘outside the box’ and coming with fresh insight to existing processes. A critical piece of this is to determine how the suggested processes can be measured. Without measurement it is impossible to know whether improvement has taken place. Peter Drucker – “you can’t manage what you can’t measure,” and extended to say “if you can’t measure it, you can’t improve it.”

    The next step then falls into place. We take the recommended improved process and try it out while taking measurements. Once the process has had some time to be tested, the measurements are collated and it is determined if the process has improved.
    There are two possible outcomes:

    • Process Improvement has occurred. In which case then we look for further process improvement.
    • Process Improvement has not occurred. In which case we discard the suggested process and look for some other way to obtain process improvement.

    Next week we will provide some examples.

  • Quality Assurance Processes

    Quality Assurance Processes are key to a successful project. Before we get into the where and why, here is  a brief overview of processes as we see them.

    A Quality Assurance Process has several components which are often described via a workbench.
    The following are common to most of the workbenches we have used:

    • Inputs are items that are provided to the workbench. Frequently they are the output from another workbench and they should be subject to entrance criteria.
    • Some Do Process to which the Inputs are subject
    • Some Check Process after the Do Process has completed
    • Tools may be used to assist in the Do and Check Processes
    • If the product or process passes then it is delivered to the next workbench or to the final customer
    • If the product or process fails then rework is undertaken to correct the errors
    • Standards should be applied to all the other components of the workbench

    The key point is to determine where to apply the processes for the most impact and best ROI.

    Too Much Process

    There are companies (some of them no longer in business) who applied processes to absolutely every step and every thing they did. They ‘bogged down in process’ to the detriment of the actual work they were supposed to be doing. There were too many processes to remember and people actually devoted time to working out how to avoid the process.

    Too Little Process

    The opposite of the above are the companies that avoid process entirely. Everything is left to current thought of the day and it can be changed tomorrow. These companies are at Level 1 (or less) of CMMi and very little actually moves each day since everyone knows it can change before. People resort to doing as a little as possible while waiting on the final decision.

    Just Right Process

    The best way is the Just Right Processes. We will talk about that next week.