NVP Software Solutions

Tag: Quality Control

  • Using Metrics for Process Improvement

    There are three aspects to using metrics for Process Improvement:

    • Defining the correct metric to gather
    • Gathering the correct statistics to determine that metric
    • Making use of the results

    We have already discussed the first two in previous posts so we will make the assumptions that the correct metric has been defined and the correct statistics gathered to support that metric. We now need to make use of those metrics. We will start with an example related to Software Testing and then move to the general case.

    Assume we have an existing application to which we are making changes over time. It is a reasonably critical application for our company and we have built an extensive set of regression tests over the years. The regression tests have been augmented with new ones that represent implemented functionality. The regression tests are run with each release of code. We have been retaining the statistics on the number of tests that fail each time the regression tests are run. The failures are investigated, fixed if necessary (high enough priority), and the release implemented.

    Recently the number of failures for each run of the regression base has been increasing and it is becoming difficult to investigate and fix all the failures prior to implementation.

    The following need to be checked:

      1. Is the percentage of defects the same as before and the raw number is increasing solely as a result of the regression base getting bigger?

    This requires another measurement to determine the size of the regression base so we can complete the comparison.

      1. Is the number of defects increasing as a result of the increasing size of the application?

    This also requires another measurement that can determine the size of the application.

      1. Is the number of defects increasing as a result of making parts of the application active that were not used before?

    Again this requires a measurement that indicates application usage.

    The action to be taken depends on the answers to the above question and may vary from doing nothing to starting some refactoring and major rewrites.

    We have outlined three general principles with this example:

    1. Gather the statistics (both for the activity that we want to measure and for comparison purposes)
    2. Investigate the reasons for any change
    3. Take action based on that change.

    Contact us to talk about your metrics and how they can be used.

  • Examples of QC & QA Metrics

    It is reasonably easy to define measurements and metrics for any particular organization or project. The bigger question is whether those metrics are relevant to the project and work at hand.

    Metrics or Measurements Specific to Quality Control

    • Number of tests
    • Number of defects
    • Defects per module
    • Percent of tests completed

    The above are fairly obvious examples of metrics and are quite frequently gathered along with a number of others as Quality Control and Quality Assurance data. Many of the test tools supplied commercially or available as Open Source gather one or more of these measurements automatically and make them available on graphs and in spreadsheets for manipulation.

    Metrics or Measurements Specific to Quality Assurance

    • Number of processes in place
    • Degree of Acceptance of a process
    • Effectiveness of any process
    • Unmeasured processes

    The above metrics vary from easy-to-gather and maintain to almost impossible to accurately estimate. They usually do not depend on any particular project although they may be department dependent based on the level of detail. However, we find that most organizations, while using many processes with varying degrees of effectiveness rarely make a concerted attempt to measure them. It is very tempting to simply use the process without considering whether or not it could be improved.

    The key questions to ask are as follows

      1. Can the metric or measurement be gathered? (Preferably without impacting the test or quality effort.
      2. Are the figures accurate? (This needs to be tested just like the application.)

    & Most important

    1. Are we making use of the metric to provide information to the project or organization?

    The reason that we place such emphasis on the last point is that we encounter far too many instances where metrics and measurements are being gathered with no clear idea of why or how they can be used.

    • Do you want to know if your processes are effective?
    • Do you want to know if your processes can be improved? (And how to do it)
    • Do you want to know how you compare to other organizations?

    Contact us to see how your company can measure your metrics.

    Next week we will discuss using Metrics.

  • Quality Assurance Metrics

    In our last post we listed the need for some way of measuring Process Improvement without going into too much detail about how. What we were referencing was Quality Assurance Metrics.

    We define a metric similar to the way Wikipedia does in that it is the degree to which software possesses some desired characteristic. This feeds directly into the comments from our last post about needing to have something to measure in order to know if Process Improvement has occurred.

    More specifically, we can define a Measurement as something that has a single dimension: Height; Weight; Count of defects; Count of Requirements and many others.

    A Metric is derived from two or more Measurements and is often expressed as a ratio or a percentage. For example, we might count the number of defects but we might also count the number at each Severity level and then express that as a percentage of the total. This gives us a lot more information as the following example shows:

    If we have 100 defects and 20 of them are of the highest severity then we can state that 20% or one fifth are of the highest severity. We can watch that percentage over the duration of the project and track its movement up and down. We can also compare our percentage of highest severity defects to other projects directly using the percentage and see if our percentage is out of line with other projects. Note that this removes the bias that might occur with different project sizes. If someone has a much smaller project, they would expect commensurately fewer defects in total but might experience a similar percentage of high severity ones.

    In general metrics normalize the results to allow comparisons.

    As Peter Drucker said “you can’t manage what you can’t measure,”

    Give us a call to see what Metrics you need!

  • Examples of Process Improvement

    Examples of Process Improvement are sometimes a little harder to find and measure than Product Improvement. As long as you have a standard to compare a product, any change can be determined and it is usually easier to determine if the quality of the product has improved or not. Also, we may not have to actually determine the measurement methodology prior to the product being produced. We can take an existing product, determine a standard and decide if it has met that standard. Future products can then be compared against that standard. A process, on the other hand, does not necessarily have an end product. It is part of what produces the end product and may not ‘exist in the literal sense’ after it is finished.

    There are two ways we can measure Process Improvement and then determine if the Process has improved.

    • Measure the created product. This becomes product measurement and that can be used as feedback to improve the process.
    • Measure the actual process.

    The first has already been covered so we will discuss the second.
    If we make a change to the process, the fundamental question is has the process improved.
    Examples:

    • The process continues to generate product after a change with no reduction in quality. Process Improvement.
    • The process generates the same product while using less materials. Process Improvement.
    • The process generates the same product with less waste. Process Improvement.
    • The process is halted or interrupted less often than before. Process Improvement.
    • Less resources are required for the same quality product. Process Improvement.
    • The process has reduced variation. Process Improvement.

    All of the above require a proper measurement process (another process) to be in place before the process is launched. It is very difficult to measure some of the above process improvements without having an existing measurement process in place. This may eventually remind you of The Siphonaptera!

    However, if you want to find out about your own processes, take a look at our Assessment process which identifies them.

  • Quality Assurance Supports Validation

    Quality Assurance supports validation for a number of reasons not least of which is the potential for cost savings for you! The key is to know where to apply the appropriate level of validation. It is quite possible to validate too many items and also to validate the wrong ones. There are some organizations who believe that if a little testing is good, then a medium amount of testing is better and a huge amount of testing is the best of all. The testers (and anyone else who can be brought into the process) test as much as they can or for as long as they can before the release date. The end result is a product that has been tested for some items but no one knows what was done or missed. As usual, these problems show up in production with depressing regularity and sometimes horrific consequences.

    Some projects have problems identifying the correct place to implement validation techniques and wait until it is too late in the project. The other mistake that some organizations make is matching the correct validation technique to their level of process maturity.

    Many people can test but not all can test effectively or efficiently.

    Both are necessary for a positive ROI.

    For all of the above reasons, it is critical that an assessment be carried out on the existing processes to determine the following:

    • The appropriate places to use validation techniques in your projects.
    • The appropriate types of validation to complete and when to do them.
    • The level of Validation to be carried out at each phase.
    • The relevant stakeholders who gain from the process and can see the benefits.
    • The use of the correct techniques based on SDLC maturity.

    Once all of the above are known; then an effective validation process can implemented for your benefit. NVP Software Solutions completes that assessment and provides Recommendations and a Roadmap to get you there. You realise instant benefits in reduced rework and reduced cost.

  • Examples of Validation

    Examples of Validation are one of the easier items to find as long as the definition referred to in the some of the earlier posts is used. Validation covers all active testing where code has been generated and we are able to run it actively.

    So it includes at least the following list of Test Phases:

    • Unit Testing
    • Integration Testing
    • System Testing
    • Acceptance Testing

    And the following list of Test Types:

    • Initial Testing
      1. Configuration Testing
      2. Compatibility/Conversion Testing
      3. Installability Testing
    • On-Going Testing
      1. Functionality Testing
      2. Facility Testing
      3. Security Testing
    • Performance Testing
      1. Volume Testing
      2. Stress Testing
      3. Load Testing
      4. Speed of Response Testing
      5. Storage Testing
    • Post Functional Testing
      1. Usability Testing
      2. Reliability Testing
      3. Recovery Testing
    • Post Installation Testing
      1. Serviceability Testing

    Despite the length of the two lists above, they are not that hard to deal with since they break the testing into several disparate groups and make it easier to ensure that everything is addressed and not omitted from consideration.

    The two key points are to determining into which phase the Types of Testing fall and whether they are worth doing. The second question is much harder to answer than the first.

    If you ask the stakeholders for most projects, they will simply answer that all Types of Testing should be full completed with all possible depth and speed. If you ask them to budget for it, you usually end up with a minimization statement asking for the cheapest testing. The best answer we have found to deal with this type of conundrum is to provide the following:

    1. Provide the above list of Validation testing types.
    2. Define each one briefly.
    3. List the recommended phase.
    4. Provide the recommendation for inclusion or exclusion.
    5. Provide the risk of failure to include them.

    This provides the stakeholders with all the information they need to make the correct decision and weigh the various merits of completing the various types of testing. In the event that they still cannot come to a decision, then add a recommendation based on the project.

  • Validation Applicability in Software Testing

    Many people believe that Validation is Software Testing and that is all there is to the profession. As was stated in our earlier blogs Validation is only half of it. It is the more expensive ‘half’ but has had far more effort put into it by many people.

    Validation is applicable in Software Testing as soon as you have any code at all to test. So it cannot launch until some development has been completed and some code generated. However, once that is completed, it is possible to launch Validation techniques and start to apply them.

    Validation can be applied at the Unit or Module level. This is the most cost effective place in which to use the technique since any error discovered at this stage is reasonably easy and cheap to fix and retest. A single module can be tested by the use of Drivers and Stubs and executed while the tester watches the progress of individual variables and conditions.

    Validation can also be applied at the Integration level where individual modules are strung together. There will still be a need for Drivers and Stubs to drive the calling and called modules but the effort will center on testing the interfaces between the modules.

    Validation can next be applied at the System and Acceptance Testing Levels although there will be more emphasis placed on some of the non-code aspects of the system.

    Lastly Validation can be applied to the Non-Functional testing aspects of the system. This requires some consideration as to what is important and what is cost-effective to test at this stage. A lot of funds can be expended with little results if due consideration is not given to the payback. If you have questions about this type of Validation Testing and would like to determine what to do, give us a call or email us at neil@nvp.ca.

  • Validation

    Quality Assurance has a number of subdivisions; one of those being Quality Control (sometimes referred to as Software Testing). We have been discussing Verification for the last four weeks. See our blog. Today we start into Validation

    We define Validation as being the active part of testing where completed, compiled, and promoted code is being run using data and generating results while Verification is the static part of testing. Note that promoted includes Unit testing on the developer’s workstation. So the promotion may not be a very formal one nor may the compile amount to a lot.

    Validation includes All aspects of active testing including the major divisions of White Box and Black Box (along with all the various Grey boxes in between). The fact that the code has to be completed means that it is difficult to launch into Validation before a lot of work has already been done. This is what adds to the cost of completing this type of testing and ensures that any defects that are found tend to be expensive to correct and retest. Hence our emphasis on Verification as being more generally cost effective.

    A lot of work has been completed to enable Validation to proceed smoothly and effectively. Many testers start their careers in this area and then move into various aspects of the process as they become more specialized. Some go towards Automation, others move into specialised areas like Performance or Security testing.

    While the concept is simple, the identification of the most cost-effective place to use Validation techniques is not always easy. There is no point in completing Validation without some end in view and a positive ROI. Give us a call at 416-927-0960 or visit our website at NVP.ca to find out where you would benefit from the implementation of Validation techniques in your organisation.