NVP Software Solutions

Tag: #softwaretesting

  • Trend Analysis on Defects – Part 2

    Last week we discussed some prerequisites for Trend Analysis on defects. We are only discussing defects here but there are many other project metrics that could be analysed.

    Assuming that the prerequisites from the previous blog have been satisfied, what can we determine.

    1. The trend in the number of defects in each project (after normalization by some project size measurement).
    2. The trend in the number of defects in each Priority and Severity level (again after normalization).
    3. The phase or process that consistently generates the most defects.
    4. Any weak or defect prone supplier processes.
    5. Any product weaknesses that cause consistent customer complaints.

    Once we have the information listed above, we are in a strong position to fix any weak areas and correct any issues that may be costing us customers or internal funds. This is process improvement devoted to making sure that our internal processes are the best possible and we are generating the best possible product. There is more to trend analysis than we are mentioning here, but there are entire books and conferences devoted to this topic. We have only touched the surface of one item we can measure, address and improve.

    October is Quality Month. Sign up for our newsletter, coming out next week, to see some things about Quality Month.

    Photo by Stephan Henning on Unsplash

  • Trend analysis on Defects – Part 1

    One of the items that is missed frequently in most organisations is Trend Analysis on Defects. This is the trend over time and over multiple projects. We are looking for whether things are improving or not as the case may be, over time. There are a number of prerequisites which we will discuss today prior to returning to this topic in a couple of weeks with further information.

    Prerequisites

    1. A consistent method of classifying defects with regard to Priority and Severity. I include definitions and standards for Priority and Severity either in the Test Strategy for an organisation or in every test plan for every project. Note that without this in place; it is impossible to do any trend analysis.
    2. A commitment to fairly recording defects in every phase of the project no matter what methodology you use. Otherwise it is possible to make certain phases look better than others by ignoring or failing to record the defects.
    3. A commitment from management to the years it will take to obtain sufficient statistics for trends to be meaningful.
    4. A commitment to and an understanding of the methodology used to reconcile projects that of different size and that are built in different ways. A very large project built in-house cannot be directly compared to a small project where the software was purchased.
    5. An understanding of the statistical processes that will be used to generate the trends. It is far too easy to draw the wrong conclusion
    If any of these prerequisites are not in place before anything starts, you are embarking on an expensive failure!

    Photo by Stephan Henning on Unsplash

  • September Events in the GTA and Beyond

    If you are in the Greater Toronto Area or Kitchener-Waterloo you might want to consider these events to network with other QA people or learn some of the new ideas in QA.

    This image has an empty alt attribute; its file name is antenna-502680-unsplash-1024x683.jpg

    NVP Software Solutions will be participating in the following software testing and quality assurance events happening this September in Ontario, Canada. The events are located in Toronto and Kitchener-Waterloo in the coming weeks. Check out the relevant websites for more information and to register. This is a great opportunity to connect with other software testing and quality assurance professionals. We hope to see you there!


    Photo by Antenna on Unsplash




    PRESIDENT’S DINNER – V2X (VEHICLE TO EVERYTHING) COMMUNICATIONS

     September 24, 2019 6:00 p.m.  The Albany Club – 91 King Street East, Toronto, Ontario

    Presenters: Charles Finlay, Peter Watkins, Tony Hui, Ysni Semsedini Moderator: Colin Dhillon

    Targeting Quality Conference

     September 23 – 24, 2019 6:00 p.m.  Cambridge Hotel & Conference Centre easily located on Hespeler Road in Cambridge, just off of the 401.

  • What to do with the reported information?

    We have spent the last couple of months asking what should be reported from testing and to whom it should be reported. However, one of the major issues is that a lot of information and statistics are generated and often ignored until the end of the project. We end up with reams of results, counts, trends (sometimes), comments, analysis and opinions along with change documents and other project documentation with no obvious home and no-one looking at it.

    The end result is that decisions are taken without all the required information.

    This is obviously not just a QA issue but a project issue, however, our concern is with the QA information that has been generated. Although it may seem obvious, we need to make sure the items that need attention are brought up at the correct time and place for a decision. Many people will colour code their results to make sure that any critical issues are addressed but that does not help if no-one is paying attention. We tend to go back to the Quality Assurance aspect of determining the process that should be followed and of Process Improvement as we tweak that process to work for us. We need to identify who needs the information and from whom decisions must be obtained. Statistics and impact analysis also always help here to identify what could go wrong. A few weeks ago we mentioned trend analysis and thresholds. These are also critical for any of the information generated.

    Photo by Vladislav Babienko on Unsplash

  • What do we do with the information?

    We have spent the last couple of months asking what should be reported from testing and to whom it should be reported. Assuming that has all been decided, what do you do with the information? Unfortunately in the volume of information reported from many projects, it is easy to lose sight of the fact that this information needs to be acted on.

    One of the items that is often missed is calculation of a Risk Threshold or Trigger based on the testing. While some organisations are very good at dealing with results of testing, some ignore it. Calculation of a Risk Threshold or Trigger avoids that problem.

    A few of examples follow:

    1. Once the number of Critical or High Severity defects passes a pre-defined number, development resources must be re-deployed from new development to defect remediation.
    2. If the count of Critical or High Severity defects is not shrinking as the project approaches its completion date, a Project Meeting has to decide if the date can be met with the current scope and resources. This is trend analyis.
    3. If the time to execute the High and Medium Priority testcases exceeds the remaining time, then either the testing resources must be augmented, the testcases must be re-classified, or the scope reduced.

    Please note that all of these items are going to be preset in this case and as soon as the trigger or threshold is passed, then the listed actions are taken. No one has to consider whether or not they need to be done and the decision is not subjective.

     

  • Who do we tell – Part 2

    We feel that the question of to whom and what to report has to be answered by initially by Quality personnel. We know what information we have, we know what we can get reasonably easily, and how it can be presented. Thus it falls to us to present recommendations to each person and make sure they can be fulfilled throughout the project without issue.

    As mentioned two weeks ago, we encounter two major attitudes to this question:

    1. Everyone must know everything.
    2. How can I avoid being added to this list?

    We strongly recommend against the first one since if everyone gets an email no one ever looks at it. It becomes an SEP (someone else’s problem).

    The second one is not much better. People avoid being told so they can put the blame somewhere (even if they could have been told earlier) when things go wrong.

    So we need to present a hierarchy of reports that consolidate the information as they work their way up and recommend the correct report for each level.

    1. Developers, testers and their direct managers should receive all details of every report. They are immersed in the project and need the information.
    2. Next level up – counts of testcases completed and their priorities; counts defects raised and their priorities; any critical issues should be brought to their attention particularly if they are going to impact the project.
    3. Next level up – Weekly colour coded summaries showing the project status.
    4. Next level up (assuming there is one) – project completion

    We do expect that most of the online reports will be interactive and allow anyone to drill down as required.

    Next blog – How do you decide what to do with what has been reported

     

  • July Events in the GTA and beyond

    Well it is summer time and there are no meetings of TASSQ or KWSQA.

    However, the Board of TASSQ is busy preparing for the President’s Dinner meeting to be held in September. The speaker line up is almost completed and registration should open shortly. Stay tuned for further information or contact NVP to be added to the mailing list.

    Although we cannot prove it, we strongly suspect that the KWSQA Board is busy planning their Targeting Quality Conference. NVP has signed up as a sponsor and we look forward to seeing you there. Stop by our table for your free stress relieving bug.

    Planning education for the fall and want to know what is out there. Check our site and pick your training course.

    Want to keep up on your testing news. Subscribe to our newsletter.

  • Who do we tell?

    In light of some of the comments received, this blog is aimed at the ‘official’ list of people who should be told the results of testing. As per some of the comments, there are lots and lots of unofficial channels and these can be more useful and more informative than some of the official ones. However, (in order to stop the blog going on forever) we are looking at the ones who are listed as requiring the information.

    We encounter two major attitudes to this question:

    1. Everyone must know everything.
    2. How can I avoid being added to this list?

    The above may seem odd and probably depends a lot on the trust factor within the organisation. In some places, everyone gets added no matter how much the notifications and emails pile up. In others people avoid being told. There is an obvious dependency on the sophistication of the notification process and where the information is being stored. Less sophisticated processes will not have the granularity required or no-one to administer them. More sophisticated ones can do almost any desired work flow.

    One other comment that came back was that we need to recall that we have clients and they need to be kept informed of what is occurring (whether or not they look at it).

    In our next blog on this topic we will provide some specific suggestions.