NVP Software Solutions

Tag: Process Improvement

  • Malthus on Testing – A New Paradigm

    Last week we completed our discussion of how the proposed solutions to the Malthusian Catastrophe are only postponing the inevitable. Unlike development which can always go forward and create new code, testing always has a regression component for any code that integrates with existing applications. This adds  to the testing effort but doesn’t necessarily add value.

    The proposal we outline here is not entirely new although we are carrying it to another level. Portions of this are being done and have been done for years. However, we are suggesting a more intense and more rigorous application.

    Today, we find many people integrating applications to build something. They can borrow code or entire applications and make something new from it. The statement “there is an app for that” has never been more mainstream. However, everything that is borrowed or utilized may (not always) need to be tested to some level so we can trust that it does what it says it does and operates the way we need it to in our new configuration. Add to that Security concerns dealing with applications that are not under our control and may not have been built to the same level as we need and the chance of our new application failing increases substantially.

    We are recommending that a library of code pieces are created that have the following characteristics:

    1. They carry out a known function under known conditions.
    2. They have been tested thoroughly by an independent entity to prove number 1.
    3. They are secure from being compromised.

    Number 2 above may have levels of testing just like there are levels of code today depending on the risk of the application for which they are being used but the same process applies.

    People developing applications would be able to use this library secure in the knowledge that the three points above are guaranteed.

    Next series: Communication in Testing,

  • Quality Assurance – Implementing Right the First Time

    The last three NVP Blogs have discussed Doing it Right the First Time. The last one addresses the issue of Implementing Right the First Time. This is the part that usually trips up most people. They know they want to implement this in their organization. They know how or they can refer to explanations that will suggest how. The problem comes when we try to actually implement the process.

    As is usual with almost any new initiative, we want to follow three precepts:

    1. Start small.
    2. Pick off the easy items first.
    3. Publish the successes.

    Taking these in order:

    Start small
    Start with something over which we have full control or a limited number of stakeholders. Do not start with a mission critical project with numerous stakeholders; that will not work.

    Pick off the easy items first
    A quick analysis will reveal some small and easy to fix items that can be corrected. Sometimes it can be as small as adding a field to a defect report that avoids an extra step further down in the process or removing a step like having Quality Control test a defect as it arrives from the customer before passing it to development and then testing it on the way back out. The second example might have had use at one point but no longer makes sense. A third and frequently easy place to look for improvements is in items that have been made redundant by technology changes. These can include printing items that are now easily stored online or even still testing installation disks for a hosted system.

    Publish the successes
    Even though this is third, it requires some upfront thought. We need to know the current situation and statistics before we can determine whether improvement has occurred. So even though this is the ‘last’ step it must be considered early.

    Rule 3: Take a step back; look objectively at your process; pick off a few items that can be improved; check the current situation; make the change and measure again.

    This is the first step to implementing Right the First Time or Quality Improvement in your organization. Next week; an anonymous example of How Not to be Right the First Time.

  • Quality Assurance – Benefits of Right the First Time

    In the last couple of weeks, we have been discussing the first two steps of Doing it Right the First Time. This week, we will take brief side trip to the Benefits of Right the First Time. A lot of groups stumble over this point because they cannot articulate this well. They seem to use Descartes (as sometimes translated) “I Think therefore I Am” to state that Quality Assurance is “Good because it is” and leave it at that. That argument, in the Quality Assurance case, is circular. One might argue that Descartes is also circular but that is not something we want to take up here.

    There is effort involved in “Getting it Right” and the question is whether it pays back later. We argue that is does using the following reasons:

    1. Having people actually concentrate on each step individually means that they will fully understand that particular piece.
    2. They can project all the possible consequences of that step.
    3. They can determine all the possible precursors that might impact that step.
    4. They can determine all the possible outcomes from that step.

    There are two major counter-arguments to the above. One we are already addressing here. The other can be addressed using the argument we have already stated.

    The first major counter-argument is the lack of time to do this type of analysis. Few people will deny the necessity of doing the analysis so this is only a question of timing and doing it earlier usually is beneficial in informing people.

    The second major counter-argument is that things will change before we are done so why spend the time on something that we will need to redo later. This argument misses the point of the process. The analysis carried out in this process identifies that possibility for change, anticipates what is going to happen, addresses it and moves to the solution.

    Step 3: Make sure we understand the benefits of this process so we can explain it at any time.

  • Acting on the Results – Redoing

    Last week we discussed how to measure the results of a Process Improvement initiative and how to successfully implement the process. This week we have assumed that the Process Improvement Initiative has been completed and we are now faced with Acting on the Results.

    There are two possible outcomes from the results and both lead to further action.

    1. The measurements and metrics indicate a successful process improvement and we have achieved the results we want.
    2. The measurements and metrics indicate an unsuccessful process improvement and we have to decide what to do next.

    In the beginning it is usually suggested that we review the measurements and derived metrics to ensure that we are measuring what we thought we were measuring. A false positive, (we think our program was successful but it was not) can be caused by either measuring the wrong item, deriving the wrong metrics or drawing the wrong conclusion. If the analysis indicates that the conclusion is correct then the following steps are recommended:

        1. Determine a new process improvement initiative.
        2. Determine if our existing metrics will properly measure the new initiative.
        3. Define new metrics as required.
        4. Implement and measure as before.

    If, after due analysis, we determine that our process improvement initiative did not lead to the correct result, then we have to decide between the following two possibilities:

        1. Undo the changes made and revert to the previous method of doing the process.
        2. Initiate another process improvement session to change the process to something better.

    For either of the above, we still need to measure and make sure we are accomplishing what is desired in terms of Process Improvement or maintenance of the status prior to the initial change.

    Note that none of this is based on supposition or guessing. No changes are made without supporting statistics.

    Next week: Quality Assurance – Doing it right the first time.

  • Acting on the Results

    This is the first of two blogs devoted to implementing metrics and discusses the piece most likely to cause failure. Assuming that the metrics were properly identified, measured and the results analyzed, we are now at the point of Acting on the Results of the metrics.

    There are two major ways in which a organizations tend to act on results.

    1. Making major changes all at once and cleaning up the results.
    2. Incremental changes

    Making major changes all at once

    In this approach the organization identifies where it wants to be, determines the changes and makes them all at once. This is sometimes called the Big Bang approach, often followed by the Gib Gnab which is a Big Bang in reverse
    There are several steps:

    1. Determine where the end point of the changes would be
    2. Make the changes to get there
    3. Look for what has gone wrong or is incomplete or has been resisted (this will likely be a long list)
    4. Fix the problems

    Incremental changes
    In this approach the organization identifies where it wants to be, determines the changes and decides on individual small steps that will end up getting to the correct point.
    There are several steps:

    1. Determine where the end point of the changes would be
    2. Determine individual small steps that will ultimately lead to the end result
    3. Coordinate the individual steps for each objective
    4. Obtain buy in from the affected parties for the individual steps
    5. Start making the steps
    6. Measure the progress continuously making changes if any measurements show adverse results
    7. Watch for progress towards the final goal

    If the above process is carried out correctly, not only will we get to the final point, we will also have support for our next measurement program. If any of these look unfamiliar give us a call and we will talk about it.

    Next week: Acting on the results – Redoing

  • Implementing Metrics

    Implementing Metrics is the hardest aspect of the entire Process Improvement process. It is reasonably easy to determine a need for metrics – anything that can be measured and will lead to cost savings is a candidate for a metric. It is usually fairly straightforward to determine what to measure as long as the question as to what is needed has been answered correctly. Some effort may be required to get to the correct metric that measures the actual cost of any issue or failure but some experimentation can usually lead to a good approximation and it can be modified after if required. The actual measurement needs to have all the characteristics of a good measurement but that can be determined.

    Implementing the metric falls into two categories – one of which we will discuss in this post and following up with the other one in the coming weeks. The two categories are:

    1. Taking the measurements.
    2. Acting on the results.

    Taking the measurement

    Taking the measurement assumes we have defined the need for the metric, have researched what and how we want to measure and are ready and wiling to implement. Now comes the part that affects other people and projects. It is unlikely that it is just our project or our work that will need to be measured. If it was just ours, we could have completed the measurement or simply implemented the process improvement long ago and moved on. This is now going to impact other timelines and people.

    There are several steps:

    1. Determine who or what is going to be impacted
    2. Quantify the impact
    3. Anticipate the objections
    4. Prepare answers to the objections
    5. Demonstrate the benefits
    6. Win Support
    7. Start the measurement program
    8. Provide support to the people doing the measurement
    9. Publish the results
    10. Be honest about the results

    If the above process is carried out correctly, not only will we get valid and useful results, we will also have support for our next measurement program. If any of these look unfamiliar give us a call and we well talk about it.

    Next week: Acting on the results.

  • Using Metrics for Process Improvement

    There are three aspects to using metrics for Process Improvement:

    • Defining the correct metric to gather
    • Gathering the correct statistics to determine that metric
    • Making use of the results

    We have already discussed the first two in previous posts so we will make the assumptions that the correct metric has been defined and the correct statistics gathered to support that metric. We now need to make use of those metrics. We will start with an example related to Software Testing and then move to the general case.

    Assume we have an existing application to which we are making changes over time. It is a reasonably critical application for our company and we have built an extensive set of regression tests over the years. The regression tests have been augmented with new ones that represent implemented functionality. The regression tests are run with each release of code. We have been retaining the statistics on the number of tests that fail each time the regression tests are run. The failures are investigated, fixed if necessary (high enough priority), and the release implemented.

    Recently the number of failures for each run of the regression base has been increasing and it is becoming difficult to investigate and fix all the failures prior to implementation.

    The following need to be checked:

      1. Is the percentage of defects the same as before and the raw number is increasing solely as a result of the regression base getting bigger?

    This requires another measurement to determine the size of the regression base so we can complete the comparison.

      1. Is the number of defects increasing as a result of the increasing size of the application?

    This also requires another measurement that can determine the size of the application.

      1. Is the number of defects increasing as a result of making parts of the application active that were not used before?

    Again this requires a measurement that indicates application usage.

    The action to be taken depends on the answers to the above question and may vary from doing nothing to starting some refactoring and major rewrites.

    We have outlined three general principles with this example:

    1. Gather the statistics (both for the activity that we want to measure and for comparison purposes)
    2. Investigate the reasons for any change
    3. Take action based on that change.

    Contact us to talk about your metrics and how they can be used.

  • Examples of QC & QA Metrics

    It is reasonably easy to define measurements and metrics for any particular organization or project. The bigger question is whether those metrics are relevant to the project and work at hand.

    Metrics or Measurements Specific to Quality Control

    • Number of tests
    • Number of defects
    • Defects per module
    • Percent of tests completed

    The above are fairly obvious examples of metrics and are quite frequently gathered along with a number of others as Quality Control and Quality Assurance data. Many of the test tools supplied commercially or available as Open Source gather one or more of these measurements automatically and make them available on graphs and in spreadsheets for manipulation.

    Metrics or Measurements Specific to Quality Assurance

    • Number of processes in place
    • Degree of Acceptance of a process
    • Effectiveness of any process
    • Unmeasured processes

    The above metrics vary from easy-to-gather and maintain to almost impossible to accurately estimate. They usually do not depend on any particular project although they may be department dependent based on the level of detail. However, we find that most organizations, while using many processes with varying degrees of effectiveness rarely make a concerted attempt to measure them. It is very tempting to simply use the process without considering whether or not it could be improved.

    The key questions to ask are as follows

      1. Can the metric or measurement be gathered? (Preferably without impacting the test or quality effort.
      2. Are the figures accurate? (This needs to be tested just like the application.)

    & Most important

    1. Are we making use of the metric to provide information to the project or organization?

    The reason that we place such emphasis on the last point is that we encounter far too many instances where metrics and measurements are being gathered with no clear idea of why or how they can be used.

    • Do you want to know if your processes are effective?
    • Do you want to know if your processes can be improved? (And how to do it)
    • Do you want to know how you compare to other organizations?

    Contact us to see how your company can measure your metrics.

    Next week we will discuss using Metrics.