NVP Software Solutions

Category: Quality Assurance Consulting

  • Trend analysis on Defects – Part 1

    One of the items that is missed frequently in most organisations is Trend Analysis on Defects. This is the trend over time and over multiple projects. We are looking for whether things are improving or not as the case may be, over time. There are a number of prerequisites which we will discuss today prior to returning to this topic in a couple of weeks with further information.

    Prerequisites

    1. A consistent method of classifying defects with regard to Priority and Severity. I include definitions and standards for Priority and Severity either in the Test Strategy for an organisation or in every test plan for every project. Note that without this in place; it is impossible to do any trend analysis.
    2. A commitment to fairly recording defects in every phase of the project no matter what methodology you use. Otherwise it is possible to make certain phases look better than others by ignoring or failing to record the defects.
    3. A commitment from management to the years it will take to obtain sufficient statistics for trends to be meaningful.
    4. A commitment to and an understanding of the methodology used to reconcile projects that of different size and that are built in different ways. A very large project built in-house cannot be directly compared to a small project where the software was purchased.
    5. An understanding of the statistical processes that will be used to generate the trends. It is far too easy to draw the wrong conclusion
    If any of these prerequisites are not in place before anything starts, you are embarking on an expensive failure!

    Photo by Stephan Henning on Unsplash

  • What to do with the reported information?

    We have spent the last couple of months asking what should be reported from testing and to whom it should be reported. However, one of the major issues is that a lot of information and statistics are generated and often ignored until the end of the project. We end up with reams of results, counts, trends (sometimes), comments, analysis and opinions along with change documents and other project documentation with no obvious home and no-one looking at it.

    The end result is that decisions are taken without all the required information.

    This is obviously not just a QA issue but a project issue, however, our concern is with the QA information that has been generated. Although it may seem obvious, we need to make sure the items that need attention are brought up at the correct time and place for a decision. Many people will colour code their results to make sure that any critical issues are addressed but that does not help if no-one is paying attention. We tend to go back to the Quality Assurance aspect of determining the process that should be followed and of Process Improvement as we tweak that process to work for us. We need to identify who needs the information and from whom decisions must be obtained. Statistics and impact analysis also always help here to identify what could go wrong. A few weeks ago we mentioned trend analysis and thresholds. These are also critical for any of the information generated.

    Photo by Vladislav Babienko on Unsplash

  • What do we do with the information?

    We have spent the last couple of months asking what should be reported from testing and to whom it should be reported. Assuming that has all been decided, what do you do with the information? Unfortunately in the volume of information reported from many projects, it is easy to lose sight of the fact that this information needs to be acted on.

    One of the items that is often missed is calculation of a Risk Threshold or Trigger based on the testing. While some organisations are very good at dealing with results of testing, some ignore it. Calculation of a Risk Threshold or Trigger avoids that problem.

    A few of examples follow:

    1. Once the number of Critical or High Severity defects passes a pre-defined number, development resources must be re-deployed from new development to defect remediation.
    2. If the count of Critical or High Severity defects is not shrinking as the project approaches its completion date, a Project Meeting has to decide if the date can be met with the current scope and resources. This is trend analyis.
    3. If the time to execute the High and Medium Priority testcases exceeds the remaining time, then either the testing resources must be augmented, the testcases must be re-classified, or the scope reduced.

    Please note that all of these items are going to be preset in this case and as soon as the trigger or threshold is passed, then the listed actions are taken. No one has to consider whether or not they need to be done and the decision is not subjective.

     

  • Who do we tell – Part 2

    We feel that the question of to whom and what to report has to be answered by initially by Quality personnel. We know what information we have, we know what we can get reasonably easily, and how it can be presented. Thus it falls to us to present recommendations to each person and make sure they can be fulfilled throughout the project without issue.

    As mentioned two weeks ago, we encounter two major attitudes to this question:

    1. Everyone must know everything.
    2. How can I avoid being added to this list?

    We strongly recommend against the first one since if everyone gets an email no one ever looks at it. It becomes an SEP (someone else’s problem).

    The second one is not much better. People avoid being told so they can put the blame somewhere (even if they could have been told earlier) when things go wrong.

    So we need to present a hierarchy of reports that consolidate the information as they work their way up and recommend the correct report for each level.

    1. Developers, testers and their direct managers should receive all details of every report. They are immersed in the project and need the information.
    2. Next level up – counts of testcases completed and their priorities; counts defects raised and their priorities; any critical issues should be brought to their attention particularly if they are going to impact the project.
    3. Next level up – Weekly colour coded summaries showing the project status.
    4. Next level up (assuming there is one) – project completion

    We do expect that most of the online reports will be interactive and allow anyone to drill down as required.

    Next blog – How do you decide what to do with what has been reported

     

  • July Events in the GTA and beyond

    Well it is summer time and there are no meetings of TASSQ or KWSQA.

    However, the Board of TASSQ is busy preparing for the President’s Dinner meeting to be held in September. The speaker line up is almost completed and registration should open shortly. Stay tuned for further information or contact NVP to be added to the mailing list.

    Although we cannot prove it, we strongly suspect that the KWSQA Board is busy planning their Targeting Quality Conference. NVP has signed up as a sponsor and we look forward to seeing you there. Stop by our table for your free stress relieving bug.

    Planning education for the fall and want to know what is out there. Check our site and pick your training course.

    Want to keep up on your testing news. Subscribe to our newsletter.

  • Who do we tell?

    In light of some of the comments received, this blog is aimed at the ‘official’ list of people who should be told the results of testing. As per some of the comments, there are lots and lots of unofficial channels and these can be more useful and more informative than some of the official ones. However, (in order to stop the blog going on forever) we are looking at the ones who are listed as requiring the information.

    We encounter two major attitudes to this question:

    1. Everyone must know everything.
    2. How can I avoid being added to this list?

    The above may seem odd and probably depends a lot on the trust factor within the organisation. In some places, everyone gets added no matter how much the notifications and emails pile up. In others people avoid being told. There is an obvious dependency on the sophistication of the notification process and where the information is being stored. Less sophisticated processes will not have the granularity required or no-one to administer them. More sophisticated ones can do almost any desired work flow.

    One other comment that came back was that we need to recall that we have clients and they need to be kept informed of what is occurring (whether or not they look at it).

    In our next blog on this topic we will provide some specific suggestions.

  • What should be reported from your Testing Efforts – Part 4

    Feedback is always welcome and this week we are indebted to Paul Seaman for some very valid comments on the What should be reported from your testing – Part 2. He pointed out that the blog seemed to have a written bias, missed the value of verbal communication and seemed to silo testers.

    We were considering mainly final reports using an independent test team and with the potential need for an audit of the testing. So some bias may have crept in and Paul pointed that out. Let’s look at each of these.

    Siloed Test Team: The more you silo a test team, the less effective they are. Information does not flow to them or from them. So the opportunities to learn from each other is lost. In addition, any information that does flow back and forth will likely be somewhat distorted by the time it gets to the other party. Testers need to be included throughout the lifecycle embedded in the team.

    Verbal Communication: This is an obvious follow on from the previous point. If you are remote, or restricted in communication then the chance to provide and receive feedback is reduced. Non-verbal communication tends to be asynchronous – something is sent and there is a delay before there is a response. Verbal (throughout the project or testing effort) allows instant feedback and may speed up responses and reactions to changing events. The only thing you may lose is an audit trail. Anything crucial needs to be noted and put into a decision log for retention.

    Written bias: This comes down to the same thing as the last comment in the previous paragraph. Crucial information that needs to be retained should be documented and stored. If the report is simply a status that is being provided then it may not need to be fully documented. Point taken.

  • June 2019 QA Events in the GTA and Beyond

    If you are in the Greater Toronto Area or Kitchener-Waterloo you might want to consider these events to network with other QA people or learn some of the new ideas in QA.

     

    NVP Software Solutions will be participating in the following software testing and quality assurance events happening this June in Ontario, Canada. The events are located in Toronto and Kitchener-Waterloo in the coming weeks. Check out the relevant websites for more information and to register. This is a great opportunity to connect with other software testing and quality assurance professionals. We hope to see you there!




    IS DIGITAL TRANSFORMATION A BUZZ WORD OR IS IT A BUSINESS EVOLUTION?

     June 25, 2019 6:00 p.m.  The Albany Club – 91 King Street East, Toronto, Ontario

    Presenters: Kyle Hulme

    See KWSQA.org for details