NVP Software Solutions

Tag: software testing

  • Acting on the Results

    This is the first of two blogs devoted to implementing metrics and discusses the piece most likely to cause failure. Assuming that the metrics were properly identified, measured and the results analyzed, we are now at the point of Acting on the Results of the metrics.

    There are two major ways in which a organizations tend to act on results.

    1. Making major changes all at once and cleaning up the results.
    2. Incremental changes

    Making major changes all at once

    In this approach the organization identifies where it wants to be, determines the changes and makes them all at once. This is sometimes called the Big Bang approach, often followed by the Gib Gnab which is a Big Bang in reverse
    There are several steps:

    1. Determine where the end point of the changes would be
    2. Make the changes to get there
    3. Look for what has gone wrong or is incomplete or has been resisted (this will likely be a long list)
    4. Fix the problems

    Incremental changes
    In this approach the organization identifies where it wants to be, determines the changes and decides on individual small steps that will end up getting to the correct point.
    There are several steps:

    1. Determine where the end point of the changes would be
    2. Determine individual small steps that will ultimately lead to the end result
    3. Coordinate the individual steps for each objective
    4. Obtain buy in from the affected parties for the individual steps
    5. Start making the steps
    6. Measure the progress continuously making changes if any measurements show adverse results
    7. Watch for progress towards the final goal

    If the above process is carried out correctly, not only will we get to the final point, we will also have support for our next measurement program. If any of these look unfamiliar give us a call and we will talk about it.

    Next week: Acting on the results – Redoing

  • Implementing Metrics

    Implementing Metrics is the hardest aspect of the entire Process Improvement process. It is reasonably easy to determine a need for metrics – anything that can be measured and will lead to cost savings is a candidate for a metric. It is usually fairly straightforward to determine what to measure as long as the question as to what is needed has been answered correctly. Some effort may be required to get to the correct metric that measures the actual cost of any issue or failure but some experimentation can usually lead to a good approximation and it can be modified after if required. The actual measurement needs to have all the characteristics of a good measurement but that can be determined.

    Implementing the metric falls into two categories – one of which we will discuss in this post and following up with the other one in the coming weeks. The two categories are:

    1. Taking the measurements.
    2. Acting on the results.

    Taking the measurement

    Taking the measurement assumes we have defined the need for the metric, have researched what and how we want to measure and are ready and wiling to implement. Now comes the part that affects other people and projects. It is unlikely that it is just our project or our work that will need to be measured. If it was just ours, we could have completed the measurement or simply implemented the process improvement long ago and moved on. This is now going to impact other timelines and people.

    There are several steps:

    1. Determine who or what is going to be impacted
    2. Quantify the impact
    3. Anticipate the objections
    4. Prepare answers to the objections
    5. Demonstrate the benefits
    6. Win Support
    7. Start the measurement program
    8. Provide support to the people doing the measurement
    9. Publish the results
    10. Be honest about the results

    If the above process is carried out correctly, not only will we get valid and useful results, we will also have support for our next measurement program. If any of these look unfamiliar give us a call and we well talk about it.

    Next week: Acting on the results.

  • Using Metrics for Process Improvement

    There are three aspects to using metrics for Process Improvement:

    • Defining the correct metric to gather
    • Gathering the correct statistics to determine that metric
    • Making use of the results

    We have already discussed the first two in previous posts so we will make the assumptions that the correct metric has been defined and the correct statistics gathered to support that metric. We now need to make use of those metrics. We will start with an example related to Software Testing and then move to the general case.

    Assume we have an existing application to which we are making changes over time. It is a reasonably critical application for our company and we have built an extensive set of regression tests over the years. The regression tests have been augmented with new ones that represent implemented functionality. The regression tests are run with each release of code. We have been retaining the statistics on the number of tests that fail each time the regression tests are run. The failures are investigated, fixed if necessary (high enough priority), and the release implemented.

    Recently the number of failures for each run of the regression base has been increasing and it is becoming difficult to investigate and fix all the failures prior to implementation.

    The following need to be checked:

      1. Is the percentage of defects the same as before and the raw number is increasing solely as a result of the regression base getting bigger?

    This requires another measurement to determine the size of the regression base so we can complete the comparison.

      1. Is the number of defects increasing as a result of the increasing size of the application?

    This also requires another measurement that can determine the size of the application.

      1. Is the number of defects increasing as a result of making parts of the application active that were not used before?

    Again this requires a measurement that indicates application usage.

    The action to be taken depends on the answers to the above question and may vary from doing nothing to starting some refactoring and major rewrites.

    We have outlined three general principles with this example:

    1. Gather the statistics (both for the activity that we want to measure and for comparison purposes)
    2. Investigate the reasons for any change
    3. Take action based on that change.

    Contact us to talk about your metrics and how they can be used.

  • Process Improvement

    Ongoing Process Improvement is the second critical aspect of process implementation. Last week we identified the components of a process. Once they are identified and it has been determined if they are applicable then the next step is Process Improvement.

    Many organizations identify the need for Process Improvement but they seem to say it as statement sliding the two words together as if it is something that will occur without effort and frequently without results. Philip Crosby’s book “Quality is Free” talks about this and in our last post we mentioned getting the process just right. Getting it just right is not a one step process. At a minimum it is a 4 step process and those steps contain a lot of subsidiary detail.

    Initially, as mentioned last week, we have to identify the processes that are in existence. Part of NVP’s Assessment process is to identify the existing processes. This can be done internally although that can be more difficult since people tend to live through the process and cannot see it ‘from the outside’.

    Once the processes are identified, then we need to see how they can be improved. This requires thinking ‘outside the box’ and coming with fresh insight to existing processes. A critical piece of this is to determine how the suggested processes can be measured. Without measurement it is impossible to know whether improvement has taken place. Peter Drucker – “you can’t manage what you can’t measure,” and extended to say “if you can’t measure it, you can’t improve it.”

    The next step then falls into place. We take the recommended improved process and try it out while taking measurements. Once the process has had some time to be tested, the measurements are collated and it is determined if the process has improved.
    There are two possible outcomes:

    • Process Improvement has occurred. In which case then we look for further process improvement.
    • Process Improvement has not occurred. In which case we discard the suggested process and look for some other way to obtain process improvement.

    Next week we will provide some examples.

  • Quality Assurance Processes

    Quality Assurance Processes are key to a successful project. Before we get into the where and why, here is  a brief overview of processes as we see them.

    A Quality Assurance Process has several components which are often described via a workbench.
    The following are common to most of the workbenches we have used:

    • Inputs are items that are provided to the workbench. Frequently they are the output from another workbench and they should be subject to entrance criteria.
    • Some Do Process to which the Inputs are subject
    • Some Check Process after the Do Process has completed
    • Tools may be used to assist in the Do and Check Processes
    • If the product or process passes then it is delivered to the next workbench or to the final customer
    • If the product or process fails then rework is undertaken to correct the errors
    • Standards should be applied to all the other components of the workbench

    The key point is to determine where to apply the processes for the most impact and best ROI.

    Too Much Process

    There are companies (some of them no longer in business) who applied processes to absolutely every step and every thing they did. They ‘bogged down in process’ to the detriment of the actual work they were supposed to be doing. There were too many processes to remember and people actually devoted time to working out how to avoid the process.

    Too Little Process

    The opposite of the above are the companies that avoid process entirely. Everything is left to current thought of the day and it can be changed tomorrow. These companies are at Level 1 (or less) of CMMi and very little actually moves each day since everyone knows it can change before. People resort to doing as a little as possible while waiting on the final decision.

    Just Right Process

    The best way is the Just Right Processes. We will talk about that next week.

  • Quality Assurance Supports Validation

    Quality Assurance supports validation for a number of reasons not least of which is the potential for cost savings for you! The key is to know where to apply the appropriate level of validation. It is quite possible to validate too many items and also to validate the wrong ones. There are some organizations who believe that if a little testing is good, then a medium amount of testing is better and a huge amount of testing is the best of all. The testers (and anyone else who can be brought into the process) test as much as they can or for as long as they can before the release date. The end result is a product that has been tested for some items but no one knows what was done or missed. As usual, these problems show up in production with depressing regularity and sometimes horrific consequences.

    Some projects have problems identifying the correct place to implement validation techniques and wait until it is too late in the project. The other mistake that some organizations make is matching the correct validation technique to their level of process maturity.

    Many people can test but not all can test effectively or efficiently.

    Both are necessary for a positive ROI.

    For all of the above reasons, it is critical that an assessment be carried out on the existing processes to determine the following:

    • The appropriate places to use validation techniques in your projects.
    • The appropriate types of validation to complete and when to do them.
    • The level of Validation to be carried out at each phase.
    • The relevant stakeholders who gain from the process and can see the benefits.
    • The use of the correct techniques based on SDLC maturity.

    Once all of the above are known; then an effective validation process can implemented for your benefit. NVP Software Solutions completes that assessment and provides Recommendations and a Roadmap to get you there. You realise instant benefits in reduced rework and reduced cost.

  • Examples of Validation

    Examples of Validation are one of the easier items to find as long as the definition referred to in the some of the earlier posts is used. Validation covers all active testing where code has been generated and we are able to run it actively.

    So it includes at least the following list of Test Phases:

    • Unit Testing
    • Integration Testing
    • System Testing
    • Acceptance Testing

    And the following list of Test Types:

    • Initial Testing
      1. Configuration Testing
      2. Compatibility/Conversion Testing
      3. Installability Testing
    • On-Going Testing
      1. Functionality Testing
      2. Facility Testing
      3. Security Testing
    • Performance Testing
      1. Volume Testing
      2. Stress Testing
      3. Load Testing
      4. Speed of Response Testing
      5. Storage Testing
    • Post Functional Testing
      1. Usability Testing
      2. Reliability Testing
      3. Recovery Testing
    • Post Installation Testing
      1. Serviceability Testing

    Despite the length of the two lists above, they are not that hard to deal with since they break the testing into several disparate groups and make it easier to ensure that everything is addressed and not omitted from consideration.

    The two key points are to determining into which phase the Types of Testing fall and whether they are worth doing. The second question is much harder to answer than the first.

    If you ask the stakeholders for most projects, they will simply answer that all Types of Testing should be full completed with all possible depth and speed. If you ask them to budget for it, you usually end up with a minimization statement asking for the cheapest testing. The best answer we have found to deal with this type of conundrum is to provide the following:

    1. Provide the above list of Validation testing types.
    2. Define each one briefly.
    3. List the recommended phase.
    4. Provide the recommendation for inclusion or exclusion.
    5. Provide the risk of failure to include them.

    This provides the stakeholders with all the information they need to make the correct decision and weigh the various merits of completing the various types of testing. In the event that they still cannot come to a decision, then add a recommendation based on the project.

  • Validation Applicability in Software Testing

    Many people believe that Validation is Software Testing and that is all there is to the profession. As was stated in our earlier blogs Validation is only half of it. It is the more expensive ‘half’ but has had far more effort put into it by many people.

    Validation is applicable in Software Testing as soon as you have any code at all to test. So it cannot launch until some development has been completed and some code generated. However, once that is completed, it is possible to launch Validation techniques and start to apply them.

    Validation can be applied at the Unit or Module level. This is the most cost effective place in which to use the technique since any error discovered at this stage is reasonably easy and cheap to fix and retest. A single module can be tested by the use of Drivers and Stubs and executed while the tester watches the progress of individual variables and conditions.

    Validation can also be applied at the Integration level where individual modules are strung together. There will still be a need for Drivers and Stubs to drive the calling and called modules but the effort will center on testing the interfaces between the modules.

    Validation can next be applied at the System and Acceptance Testing Levels although there will be more emphasis placed on some of the non-code aspects of the system.

    Lastly Validation can be applied to the Non-Functional testing aspects of the system. This requires some consideration as to what is important and what is cost-effective to test at this stage. A lot of funds can be expended with little results if due consideration is not given to the payback. If you have questions about this type of Validation Testing and would like to determine what to do, give us a call or email us at neil@nvp.ca.