NVP Software Solutions

Tag: #softwaretesting

  • What to do with the reported information?

    We have spent the last couple of months asking what should be reported from testing and to whom it should be reported. However, one of the major issues is that a lot of information and statistics are generated and often ignored until the end of the project. We end up with reams of results, counts, trends (sometimes), comments, analysis and opinions along with change documents and other project documentation with no obvious home and no-one looking at it.

    The end result is that decisions are taken without all the required information.

    This is obviously not just a QA issue but a project issue, however, our concern is with the QA information that has been generated. Although it may seem obvious, we need to make sure the items that need attention are brought up at the correct time and place for a decision. Many people will colour code their results to make sure that any critical issues are addressed but that does not help if no-one is paying attention. We tend to go back to the Quality Assurance aspect of determining the process that should be followed and of Process Improvement as we tweak that process to work for us. We need to identify who needs the information and from whom decisions must be obtained. Statistics and impact analysis also always help here to identify what could go wrong. A few weeks ago we mentioned trend analysis and thresholds. These are also critical for any of the information generated.

    Photo by Vladislav Babienko on Unsplash

  • What do we do with the information?

    We have spent the last couple of months asking what should be reported from testing and to whom it should be reported. Assuming that has all been decided, what do you do with the information? Unfortunately in the volume of information reported from many projects, it is easy to lose sight of the fact that this information needs to be acted on.

    One of the items that is often missed is calculation of a Risk Threshold or Trigger based on the testing. While some organisations are very good at dealing with results of testing, some ignore it. Calculation of a Risk Threshold or Trigger avoids that problem.

    A few of examples follow:

    1. Once the number of Critical or High Severity defects passes a pre-defined number, development resources must be re-deployed from new development to defect remediation.
    2. If the count of Critical or High Severity defects is not shrinking as the project approaches its completion date, a Project Meeting has to decide if the date can be met with the current scope and resources. This is trend analyis.
    3. If the time to execute the High and Medium Priority testcases exceeds the remaining time, then either the testing resources must be augmented, the testcases must be re-classified, or the scope reduced.

    Please note that all of these items are going to be preset in this case and as soon as the trigger or threshold is passed, then the listed actions are taken. No one has to consider whether or not they need to be done and the decision is not subjective.

     

  • Who do we tell – Part 2

    We feel that the question of to whom and what to report has to be answered by initially by Quality personnel. We know what information we have, we know what we can get reasonably easily, and how it can be presented. Thus it falls to us to present recommendations to each person and make sure they can be fulfilled throughout the project without issue.

    As mentioned two weeks ago, we encounter two major attitudes to this question:

    1. Everyone must know everything.
    2. How can I avoid being added to this list?

    We strongly recommend against the first one since if everyone gets an email no one ever looks at it. It becomes an SEP (someone else’s problem).

    The second one is not much better. People avoid being told so they can put the blame somewhere (even if they could have been told earlier) when things go wrong.

    So we need to present a hierarchy of reports that consolidate the information as they work their way up and recommend the correct report for each level.

    1. Developers, testers and their direct managers should receive all details of every report. They are immersed in the project and need the information.
    2. Next level up – counts of testcases completed and their priorities; counts defects raised and their priorities; any critical issues should be brought to their attention particularly if they are going to impact the project.
    3. Next level up – Weekly colour coded summaries showing the project status.
    4. Next level up (assuming there is one) – project completion

    We do expect that most of the online reports will be interactive and allow anyone to drill down as required.

    Next blog – How do you decide what to do with what has been reported

     

  • July Events in the GTA and beyond

    Well it is summer time and there are no meetings of TASSQ or KWSQA.

    However, the Board of TASSQ is busy preparing for the President’s Dinner meeting to be held in September. The speaker line up is almost completed and registration should open shortly. Stay tuned for further information or contact NVP to be added to the mailing list.

    Although we cannot prove it, we strongly suspect that the KWSQA Board is busy planning their Targeting Quality Conference. NVP has signed up as a sponsor and we look forward to seeing you there. Stop by our table for your free stress relieving bug.

    Planning education for the fall and want to know what is out there. Check our site and pick your training course.

    Want to keep up on your testing news. Subscribe to our newsletter.

  • Who do we tell?

    In light of some of the comments received, this blog is aimed at the ‘official’ list of people who should be told the results of testing. As per some of the comments, there are lots and lots of unofficial channels and these can be more useful and more informative than some of the official ones. However, (in order to stop the blog going on forever) we are looking at the ones who are listed as requiring the information.

    We encounter two major attitudes to this question:

    1. Everyone must know everything.
    2. How can I avoid being added to this list?

    The above may seem odd and probably depends a lot on the trust factor within the organisation. In some places, everyone gets added no matter how much the notifications and emails pile up. In others people avoid being told. There is an obvious dependency on the sophistication of the notification process and where the information is being stored. Less sophisticated processes will not have the granularity required or no-one to administer them. More sophisticated ones can do almost any desired work flow.

    One other comment that came back was that we need to recall that we have clients and they need to be kept informed of what is occurring (whether or not they look at it).

    In our next blog on this topic we will provide some specific suggestions.

  • What should be reported from your Testing Efforts – Part 4

    Feedback is always welcome and this week we are indebted to Paul Seaman for some very valid comments on the What should be reported from your testing – Part 2. He pointed out that the blog seemed to have a written bias, missed the value of verbal communication and seemed to silo testers.

    We were considering mainly final reports using an independent test team and with the potential need for an audit of the testing. So some bias may have crept in and Paul pointed that out. Let’s look at each of these.

    Siloed Test Team: The more you silo a test team, the less effective they are. Information does not flow to them or from them. So the opportunities to learn from each other is lost. In addition, any information that does flow back and forth will likely be somewhat distorted by the time it gets to the other party. Testers need to be included throughout the lifecycle embedded in the team.

    Verbal Communication: This is an obvious follow on from the previous point. If you are remote, or restricted in communication then the chance to provide and receive feedback is reduced. Non-verbal communication tends to be asynchronous – something is sent and there is a delay before there is a response. Verbal (throughout the project or testing effort) allows instant feedback and may speed up responses and reactions to changing events. The only thing you may lose is an audit trail. Anything crucial needs to be noted and put into a decision log for retention.

    Written bias: This comes down to the same thing as the last comment in the previous paragraph. Crucial information that needs to be retained should be documented and stored. If the report is simply a status that is being provided then it may not need to be fully documented. Point taken.

  • June 2019 QA Events in the GTA and Beyond

    If you are in the Greater Toronto Area or Kitchener-Waterloo you might want to consider these events to network with other QA people or learn some of the new ideas in QA.

     

    NVP Software Solutions will be participating in the following software testing and quality assurance events happening this June in Ontario, Canada. The events are located in Toronto and Kitchener-Waterloo in the coming weeks. Check out the relevant websites for more information and to register. This is a great opportunity to connect with other software testing and quality assurance professionals. We hope to see you there!




    IS DIGITAL TRANSFORMATION A BUZZ WORD OR IS IT A BUSINESS EVOLUTION?

     June 25, 2019 6:00 p.m.  The Albany Club – 91 King Street East, Toronto, Ontario

    Presenters: Kyle Hulme

    See KWSQA.org for details

  • What should be reported from your Testing Efforts – Part 3

    This seems like an obvious question and many people have successfully conquered it and set up a standard for reporting. Every once in a while it falls apart and people get blindsided by extra requests that they did not expect but for the most part it works fairly well.

    We first mentioned this question a few weeks ago. Two weeks ago we said we would make some suggestions as to what could be recorded. The following measurements are gathered by almost every project we meet.

    • Defects raised ranked by severity and priority.
    • Testcases completed, in progress, blocked, failed.
    • Number of times a testcase has been executed.
    • First time failures.
    • etc etc etc

    Almost all test management tools will supply all these measurements and many more besides. Sometimes the question is which ones to select. Just make sure that you are getting the measurements for your project and your time period (otherwise the figures are misleading).

    Metrics (combinations of two measurements usually by dividing one measurement by another) are also provided by almost any test tool. As long as you avoid dividing by zero, these are also quite common. Some examples include:

    • Testcases executed per week.
    • Defects generated per week.
    • High severity defects as a percentage of all defects
    • etc, etc, etc

    Again the test management tool supplies these and other metrics and the only concern is to make sure the measurements are for your project and time period (and not someone else’s).

    The items that we find that are missed on the first time around are the trend measurements. Since there are no trends in the first week (a single data point is not a trend) and pretty useless ones in weeks 2 and 3 of any project the trends become an extra calculation in the third or fourth week. At that point, they may supply some unpleasant information like:

    • High Severity defects are increasing in both number and percentage of all defects.
    • Defect fix time is increasing rapidly as the project progresses.
    • Testcase execution has slowed to a crawl.
    • etc, etc etc

    Usually, the test manager has a feel for this and probably knows that the testing is not going well but the trend analysis brings it out without question.

    The only caveat is to make sure you are comparing the same items from week to week (otherwise you might as well throw it out).