NVP Software Solutions

Category: Software Testing

  • What do we do with the information?

    We have spent the last couple of months asking what should be reported from testing and to whom it should be reported. Assuming that has all been decided, what do you do with the information? Unfortunately in the volume of information reported from many projects, it is easy to lose sight of the fact that this information needs to be acted on.

    One of the items that is often missed is calculation of a Risk Threshold or Trigger based on the testing. While some organisations are very good at dealing with results of testing, some ignore it. Calculation of a Risk Threshold or Trigger avoids that problem.

    A few of examples follow:

    1. Once the number of Critical or High Severity defects passes a pre-defined number, development resources must be re-deployed from new development to defect remediation.
    2. If the count of Critical or High Severity defects is not shrinking as the project approaches its completion date, a Project Meeting has to decide if the date can be met with the current scope and resources. This is trend analyis.
    3. If the time to execute the High and Medium Priority testcases exceeds the remaining time, then either the testing resources must be augmented, the testcases must be re-classified, or the scope reduced.

    Please note that all of these items are going to be preset in this case and as soon as the trigger or threshold is passed, then the listed actions are taken. No one has to consider whether or not they need to be done and the decision is not subjective.

     

  • Who do we tell – Part 2

    We feel that the question of to whom and what to report has to be answered by initially by Quality personnel. We know what information we have, we know what we can get reasonably easily, and how it can be presented. Thus it falls to us to present recommendations to each person and make sure they can be fulfilled throughout the project without issue.

    As mentioned two weeks ago, we encounter two major attitudes to this question:

    1. Everyone must know everything.
    2. How can I avoid being added to this list?

    We strongly recommend against the first one since if everyone gets an email no one ever looks at it. It becomes an SEP (someone else’s problem).

    The second one is not much better. People avoid being told so they can put the blame somewhere (even if they could have been told earlier) when things go wrong.

    So we need to present a hierarchy of reports that consolidate the information as they work their way up and recommend the correct report for each level.

    1. Developers, testers and their direct managers should receive all details of every report. They are immersed in the project and need the information.
    2. Next level up – counts of testcases completed and their priorities; counts defects raised and their priorities; any critical issues should be brought to their attention particularly if they are going to impact the project.
    3. Next level up – Weekly colour coded summaries showing the project status.
    4. Next level up (assuming there is one) – project completion

    We do expect that most of the online reports will be interactive and allow anyone to drill down as required.

    Next blog – How do you decide what to do with what has been reported

     

  • July Events in the GTA and beyond

    Well it is summer time and there are no meetings of TASSQ or KWSQA.

    However, the Board of TASSQ is busy preparing for the President’s Dinner meeting to be held in September. The speaker line up is almost completed and registration should open shortly. Stay tuned for further information or contact NVP to be added to the mailing list.

    Although we cannot prove it, we strongly suspect that the KWSQA Board is busy planning their Targeting Quality Conference. NVP has signed up as a sponsor and we look forward to seeing you there. Stop by our table for your free stress relieving bug.

    Planning education for the fall and want to know what is out there. Check our site and pick your training course.

    Want to keep up on your testing news. Subscribe to our newsletter.

  • Who do we tell?

    In light of some of the comments received, this blog is aimed at the ‘official’ list of people who should be told the results of testing. As per some of the comments, there are lots and lots of unofficial channels and these can be more useful and more informative than some of the official ones. However, (in order to stop the blog going on forever) we are looking at the ones who are listed as requiring the information.

    We encounter two major attitudes to this question:

    1. Everyone must know everything.
    2. How can I avoid being added to this list?

    The above may seem odd and probably depends a lot on the trust factor within the organisation. In some places, everyone gets added no matter how much the notifications and emails pile up. In others people avoid being told. There is an obvious dependency on the sophistication of the notification process and where the information is being stored. Less sophisticated processes will not have the granularity required or no-one to administer them. More sophisticated ones can do almost any desired work flow.

    One other comment that came back was that we need to recall that we have clients and they need to be kept informed of what is occurring (whether or not they look at it).

    In our next blog on this topic we will provide some specific suggestions.

  • June 2019 QA Events in the GTA and Beyond

    If you are in the Greater Toronto Area or Kitchener-Waterloo you might want to consider these events to network with other QA people or learn some of the new ideas in QA.

     

    NVP Software Solutions will be participating in the following software testing and quality assurance events happening this June in Ontario, Canada. The events are located in Toronto and Kitchener-Waterloo in the coming weeks. Check out the relevant websites for more information and to register. This is a great opportunity to connect with other software testing and quality assurance professionals. We hope to see you there!




    IS DIGITAL TRANSFORMATION A BUZZ WORD OR IS IT A BUSINESS EVOLUTION?

     June 25, 2019 6:00 p.m.  The Albany Club – 91 King Street East, Toronto, Ontario

    Presenters: Kyle Hulme

    See KWSQA.org for details

  • What should be reported from your Testing Efforts – Part 3

    This seems like an obvious question and many people have successfully conquered it and set up a standard for reporting. Every once in a while it falls apart and people get blindsided by extra requests that they did not expect but for the most part it works fairly well.

    We first mentioned this question a few weeks ago. Two weeks ago we said we would make some suggestions as to what could be recorded. The following measurements are gathered by almost every project we meet.

    • Defects raised ranked by severity and priority.
    • Testcases completed, in progress, blocked, failed.
    • Number of times a testcase has been executed.
    • First time failures.
    • etc etc etc

    Almost all test management tools will supply all these measurements and many more besides. Sometimes the question is which ones to select. Just make sure that you are getting the measurements for your project and your time period (otherwise the figures are misleading).

    Metrics (combinations of two measurements usually by dividing one measurement by another) are also provided by almost any test tool. As long as you avoid dividing by zero, these are also quite common. Some examples include:

    • Testcases executed per week.
    • Defects generated per week.
    • High severity defects as a percentage of all defects
    • etc, etc, etc

    Again the test management tool supplies these and other metrics and the only concern is to make sure the measurements are for your project and time period (and not someone else’s).

    The items that we find that are missed on the first time around are the trend measurements. Since there are no trends in the first week (a single data point is not a trend) and pretty useless ones in weeks 2 and 3 of any project the trends become an extra calculation in the third or fourth week. At that point, they may supply some unpleasant information like:

    • High Severity defects are increasing in both number and percentage of all defects.
    • Defect fix time is increasing rapidly as the project progresses.
    • Testcase execution has slowed to a crawl.
    • etc, etc etc

    Usually, the test manager has a feel for this and probably knows that the testing is not going well but the trend analysis brings it out without question.

    The only caveat is to make sure you are comparing the same items from week to week (otherwise you might as well throw it out).

  • What should be reported from your Testing Efforts – Part 2?

    This seems like an obvious question and many people have successfully conquered it and set up a standard for reporting. Every once in a while it falls apart and people get blindsided by extra requests that they did not expect but for the most part it works fairly well.

    We first mentioned this question a couple of weeks ago. The obvious and simple answer is to provide what your stakeholders want to have provided. The only issue with that statement is that we find a lot of stakeholders:

    1. Do not always know what they want at the beginning of a project.
    2. Change their minds as the project changes it risk profile (Stakeholders start taking more interest as the project risk increases and commensurately less interest as the project risk decreases)

    Since testers and Quality Assurance personnel are in the ‘Risk Reduction’ business we are in a good position to answer that question for them. As part of the anlysis of the project and the required testing, we will be judging the risk of the project and concentrating our testing on that. So we have a handle on the overall risk profile at the start. We can start with a level of reporting reflecting that risk profile.

    Since Software Testers and Quality Assurance personnel are also the ‘canary in mine’ when things start to go wrong, we can ramp up the reporting preemptively when things start to go wrong. For example, if there is a sudden surge in severe defects in a project, then we can start providing more information and recommendations.

    Most of the above seems obvious but when you are busy testing, the temptation is to leave the reporting at the same level (or even reduce it when we are busy). That is the exact opposite of what is needed.

    Next week – Some specifics for reporting.

  • What should be reported from your Testing Efforts – Part 1?

    This seems like an obvious question and many people have successfully conquered it and set up a standard for reporting. Every once in a while it falls apart and people get blindsided by extra requests that they did not expect but for the most part it works fairly well.

    However, even when it is well planned, there is often a substantial difference in what people expect. Some of your stakeholders will want every detail of every test result, defect, and metric that you can generate. They will look and evaluate and ask questions. Others will want to concentrate on the highlights and only investigate when there is an obvious problem that is impacting the project. The last category might be summed up by a comment I heard at one Project Meeting addressed to the test manager and coming from the biggest stakeholder: “As long as you say it is okay; I don’t want to hear anymore”. We removed the actual names to protect the guilty!

    While we will come back to this question in a couple of weeks and welcome your input in the interim, one obvious comment comes to mind immediately. Test tools have had a huge impact on this aspect of testing. The ability to record almost everything, drill down, add comments, set statuses, and move items from user to user has facilitated reporting.

    In an interesting story from one client a long time ago, the layers of management made the whole process spin out over two weeks from the time the test was actually done to the time the report was consolidated for the last layer of maangement. We have moved a long way from that!