NVP Software Solutions

Tag: Metrics

  • RCA – Why – Part 1

    Why bother with Root Cause Analysis on Defects? We have enough to do without adding to the work!

    1. We found a defect.
    2. We fixed the defect.
    3. We retested the defect.
    4. We closed the defect (pending success on item 3).
    5. Enough said. Let’s move on to something else!

    We acknowledge that once we have a defect, all of the above have to occur (hopefully only once and not repeatedly). But Root Cause Analysis is intended to prevent 1 – 5 ever coming back to bother us again. The key word is ever! We never want to see that defect again. In order to do that, we need to find out where it occurred, why it occurred; and make sure that combination never comes up again. In the long term this saves us a lot of money and time and allows us to concentrate on more important items.

    October is Quality Month. Sign up for our newsletter, to see some things about Quality Month or request a copy if you are too late for this month.

    Photo by JJ Ying on Unsplash

  • What should be reported from your Testing Efforts – Part 3

    This seems like an obvious question and many people have successfully conquered it and set up a standard for reporting. Every once in a while it falls apart and people get blindsided by extra requests that they did not expect but for the most part it works fairly well.

    We first mentioned this question a few weeks ago. Two weeks ago we said we would make some suggestions as to what could be recorded. The following measurements are gathered by almost every project we meet.

    • Defects raised ranked by severity and priority.
    • Testcases completed, in progress, blocked, failed.
    • Number of times a testcase has been executed.
    • First time failures.
    • etc etc etc

    Almost all test management tools will supply all these measurements and many more besides. Sometimes the question is which ones to select. Just make sure that you are getting the measurements for your project and your time period (otherwise the figures are misleading).

    Metrics (combinations of two measurements usually by dividing one measurement by another) are also provided by almost any test tool. As long as you avoid dividing by zero, these are also quite common. Some examples include:

    • Testcases executed per week.
    • Defects generated per week.
    • High severity defects as a percentage of all defects
    • etc, etc, etc

    Again the test management tool supplies these and other metrics and the only concern is to make sure the measurements are for your project and time period (and not someone else’s).

    The items that we find that are missed on the first time around are the trend measurements. Since there are no trends in the first week (a single data point is not a trend) and pretty useless ones in weeks 2 and 3 of any project the trends become an extra calculation in the third or fourth week. At that point, they may supply some unpleasant information like:

    • High Severity defects are increasing in both number and percentage of all defects.
    • Defect fix time is increasing rapidly as the project progresses.
    • Testcase execution has slowed to a crawl.
    • etc, etc etc

    Usually, the test manager has a feel for this and probably knows that the testing is not going well but the trend analysis brings it out without question.

    The only caveat is to make sure you are comparing the same items from week to week (otherwise you might as well throw it out).

  • What should be reported from your Testing Efforts – Part 1?

    This seems like an obvious question and many people have successfully conquered it and set up a standard for reporting. Every once in a while it falls apart and people get blindsided by extra requests that they did not expect but for the most part it works fairly well.

    However, even when it is well planned, there is often a substantial difference in what people expect. Some of your stakeholders will want every detail of every test result, defect, and metric that you can generate. They will look and evaluate and ask questions. Others will want to concentrate on the highlights and only investigate when there is an obvious problem that is impacting the project. The last category might be summed up by a comment I heard at one Project Meeting addressed to the test manager and coming from the biggest stakeholder: “As long as you say it is okay; I don’t want to hear anymore”. We removed the actual names to protect the guilty!

    While we will come back to this question in a couple of weeks and welcome your input in the interim, one obvious comment comes to mind immediately. Test tools have had a huge impact on this aspect of testing. The ability to record almost everything, drill down, add comments, set statuses, and move items from user to user has facilitated reporting.

    In an interesting story from one client a long time ago, the layers of management made the whole process spin out over two weeks from the time the test was actually done to the time the report was consolidated for the last layer of maangement. We have moved a long way from that!

  • So you want to be a Software Test Manager?

    [et_pb_section fb_built=”1″ admin_label=”Section” _builder_version=”3.22″][et_pb_row admin_label=”Row” _builder_version=”3.25″ background_size=”initial” background_position=”top_left” background_repeat=”repeat”][et_pb_column type=”4_4″ _builder_version=”3.25″ custom_padding=”|||” custom_padding__hover=”|||”][et_pb_text admin_label=”Blog Post Content” _builder_version=”3.0.87″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” use_border_color=”off” border_color=”#ffffff” border_style=”solid”]

    So you want to be a software test manager and are wondering about your decision?

    There are plenty of ways to become a software test manager. If you ask current test managers how they got there, these are some of the answers you may get:

    • Traditional: They started as a tester; became a test lead and ultimately a manager.
    • They were a lead in some other area and a managerial role opened up which they took.
    • They have no background at all in any sort of role but were seconded to a Test Manger Role.
    • It is part of the career path (particularly applicable to larger organisations).

    Regardless of which path you used to get to Software Test Manager there are several common aspects once you are there:

    • Metrics are crucial at both the department and project level.
    • Much of the management is similar to other management roles.
    • Be prepared to communicate a lot and to explain over and over again why things are being done in a particular way.

    There is a lot of conferences, user groups or LinkedIn groups that cater to Test Management. One of the better known ones is the QUEST Test Manager’s Workshop.

    Due to ongoing evolution, it is critical not to lose your technical edge even as a manager. If you find yourself looking for a job later, it is helpful to be current with at least some aspects of technology. While it may not be an immediate concern it will certainly help..

    Some companies will provide career paths for personal and technical development and improvement.

    Make sure to join LinkedIn Groups and your local Quality Assurance group. These are great options for networking and finding out other information. Take a look also at straight management groups whether related to testing or not. They have lots of information for you.

    [/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section]

  • Quality Assurance – Implementing Right the First Time

    The last three NVP Blogs have discussed Doing it Right the First Time. The last one addresses the issue of Implementing Right the First Time. This is the part that usually trips up most people. They know they want to implement this in their organization. They know how or they can refer to explanations that will suggest how. The problem comes when we try to actually implement the process.

    As is usual with almost any new initiative, we want to follow three precepts:

    1. Start small.
    2. Pick off the easy items first.
    3. Publish the successes.

    Taking these in order:

    Start small
    Start with something over which we have full control or a limited number of stakeholders. Do not start with a mission critical project with numerous stakeholders; that will not work.

    Pick off the easy items first
    A quick analysis will reveal some small and easy to fix items that can be corrected. Sometimes it can be as small as adding a field to a defect report that avoids an extra step further down in the process or removing a step like having Quality Control test a defect as it arrives from the customer before passing it to development and then testing it on the way back out. The second example might have had use at one point but no longer makes sense. A third and frequently easy place to look for improvements is in items that have been made redundant by technology changes. These can include printing items that are now easily stored online or even still testing installation disks for a hosted system.

    Publish the successes
    Even though this is third, it requires some upfront thought. We need to know the current situation and statistics before we can determine whether improvement has occurred. So even though this is the ‘last’ step it must be considered early.

    Rule 3: Take a step back; look objectively at your process; pick off a few items that can be improved; check the current situation; make the change and measure again.

    This is the first step to implementing Right the First Time or Quality Improvement in your organization. Next week; an anonymous example of How Not to be Right the First Time.

  • Acting on the Results – Redoing

    Last week we discussed how to measure the results of a Process Improvement initiative and how to successfully implement the process. This week we have assumed that the Process Improvement Initiative has been completed and we are now faced with Acting on the Results.

    There are two possible outcomes from the results and both lead to further action.

    1. The measurements and metrics indicate a successful process improvement and we have achieved the results we want.
    2. The measurements and metrics indicate an unsuccessful process improvement and we have to decide what to do next.

    In the beginning it is usually suggested that we review the measurements and derived metrics to ensure that we are measuring what we thought we were measuring. A false positive, (we think our program was successful but it was not) can be caused by either measuring the wrong item, deriving the wrong metrics or drawing the wrong conclusion. If the analysis indicates that the conclusion is correct then the following steps are recommended:

        1. Determine a new process improvement initiative.
        2. Determine if our existing metrics will properly measure the new initiative.
        3. Define new metrics as required.
        4. Implement and measure as before.

    If, after due analysis, we determine that our process improvement initiative did not lead to the correct result, then we have to decide between the following two possibilities:

        1. Undo the changes made and revert to the previous method of doing the process.
        2. Initiate another process improvement session to change the process to something better.

    For either of the above, we still need to measure and make sure we are accomplishing what is desired in terms of Process Improvement or maintenance of the status prior to the initial change.

    Note that none of this is based on supposition or guessing. No changes are made without supporting statistics.

    Next week: Quality Assurance – Doing it right the first time.

  • Acting on the Results

    This is the first of two blogs devoted to implementing metrics and discusses the piece most likely to cause failure. Assuming that the metrics were properly identified, measured and the results analyzed, we are now at the point of Acting on the Results of the metrics.

    There are two major ways in which a organizations tend to act on results.

    1. Making major changes all at once and cleaning up the results.
    2. Incremental changes

    Making major changes all at once

    In this approach the organization identifies where it wants to be, determines the changes and makes them all at once. This is sometimes called the Big Bang approach, often followed by the Gib Gnab which is a Big Bang in reverse
    There are several steps:

    1. Determine where the end point of the changes would be
    2. Make the changes to get there
    3. Look for what has gone wrong or is incomplete or has been resisted (this will likely be a long list)
    4. Fix the problems

    Incremental changes
    In this approach the organization identifies where it wants to be, determines the changes and decides on individual small steps that will end up getting to the correct point.
    There are several steps:

    1. Determine where the end point of the changes would be
    2. Determine individual small steps that will ultimately lead to the end result
    3. Coordinate the individual steps for each objective
    4. Obtain buy in from the affected parties for the individual steps
    5. Start making the steps
    6. Measure the progress continuously making changes if any measurements show adverse results
    7. Watch for progress towards the final goal

    If the above process is carried out correctly, not only will we get to the final point, we will also have support for our next measurement program. If any of these look unfamiliar give us a call and we will talk about it.

    Next week: Acting on the results – Redoing

  • Implementing Metrics

    Implementing Metrics is the hardest aspect of the entire Process Improvement process. It is reasonably easy to determine a need for metrics – anything that can be measured and will lead to cost savings is a candidate for a metric. It is usually fairly straightforward to determine what to measure as long as the question as to what is needed has been answered correctly. Some effort may be required to get to the correct metric that measures the actual cost of any issue or failure but some experimentation can usually lead to a good approximation and it can be modified after if required. The actual measurement needs to have all the characteristics of a good measurement but that can be determined.

    Implementing the metric falls into two categories – one of which we will discuss in this post and following up with the other one in the coming weeks. The two categories are:

    1. Taking the measurements.
    2. Acting on the results.

    Taking the measurement

    Taking the measurement assumes we have defined the need for the metric, have researched what and how we want to measure and are ready and wiling to implement. Now comes the part that affects other people and projects. It is unlikely that it is just our project or our work that will need to be measured. If it was just ours, we could have completed the measurement or simply implemented the process improvement long ago and moved on. This is now going to impact other timelines and people.

    There are several steps:

    1. Determine who or what is going to be impacted
    2. Quantify the impact
    3. Anticipate the objections
    4. Prepare answers to the objections
    5. Demonstrate the benefits
    6. Win Support
    7. Start the measurement program
    8. Provide support to the people doing the measurement
    9. Publish the results
    10. Be honest about the results

    If the above process is carried out correctly, not only will we get valid and useful results, we will also have support for our next measurement program. If any of these look unfamiliar give us a call and we well talk about it.

    Next week: Acting on the results.