NVP Software Solutions

Category: Quality Assurance Consulting

  • What should be reported from your Testing Efforts – Part 3

    This seems like an obvious question and many people have successfully conquered it and set up a standard for reporting. Every once in a while it falls apart and people get blindsided by extra requests that they did not expect but for the most part it works fairly well.

    We first mentioned this question a few weeks ago. Two weeks ago we said we would make some suggestions as to what could be recorded. The following measurements are gathered by almost every project we meet.

    • Defects raised ranked by severity and priority.
    • Testcases completed, in progress, blocked, failed.
    • Number of times a testcase has been executed.
    • First time failures.
    • etc etc etc

    Almost all test management tools will supply all these measurements and many more besides. Sometimes the question is which ones to select. Just make sure that you are getting the measurements for your project and your time period (otherwise the figures are misleading).

    Metrics (combinations of two measurements usually by dividing one measurement by another) are also provided by almost any test tool. As long as you avoid dividing by zero, these are also quite common. Some examples include:

    • Testcases executed per week.
    • Defects generated per week.
    • High severity defects as a percentage of all defects
    • etc, etc, etc

    Again the test management tool supplies these and other metrics and the only concern is to make sure the measurements are for your project and time period (and not someone else’s).

    The items that we find that are missed on the first time around are the trend measurements. Since there are no trends in the first week (a single data point is not a trend) and pretty useless ones in weeks 2 and 3 of any project the trends become an extra calculation in the third or fourth week. At that point, they may supply some unpleasant information like:

    • High Severity defects are increasing in both number and percentage of all defects.
    • Defect fix time is increasing rapidly as the project progresses.
    • Testcase execution has slowed to a crawl.
    • etc, etc etc

    Usually, the test manager has a feel for this and probably knows that the testing is not going well but the trend analysis brings it out without question.

    The only caveat is to make sure you are comparing the same items from week to week (otherwise you might as well throw it out).

  • What should be reported from your Testing Efforts – Part 2?

    This seems like an obvious question and many people have successfully conquered it and set up a standard for reporting. Every once in a while it falls apart and people get blindsided by extra requests that they did not expect but for the most part it works fairly well.

    We first mentioned this question a couple of weeks ago. The obvious and simple answer is to provide what your stakeholders want to have provided. The only issue with that statement is that we find a lot of stakeholders:

    1. Do not always know what they want at the beginning of a project.
    2. Change their minds as the project changes it risk profile (Stakeholders start taking more interest as the project risk increases and commensurately less interest as the project risk decreases)

    Since testers and Quality Assurance personnel are in the ‘Risk Reduction’ business we are in a good position to answer that question for them. As part of the anlysis of the project and the required testing, we will be judging the risk of the project and concentrating our testing on that. So we have a handle on the overall risk profile at the start. We can start with a level of reporting reflecting that risk profile.

    Since Software Testers and Quality Assurance personnel are also the ‘canary in mine’ when things start to go wrong, we can ramp up the reporting preemptively when things start to go wrong. For example, if there is a sudden surge in severe defects in a project, then we can start providing more information and recommendations.

    Most of the above seems obvious but when you are busy testing, the temptation is to leave the reporting at the same level (or even reduce it when we are busy). That is the exact opposite of what is needed.

    Next week – Some specifics for reporting.

  • What should be reported from your Testing Efforts – Part 1?

    This seems like an obvious question and many people have successfully conquered it and set up a standard for reporting. Every once in a while it falls apart and people get blindsided by extra requests that they did not expect but for the most part it works fairly well.

    However, even when it is well planned, there is often a substantial difference in what people expect. Some of your stakeholders will want every detail of every test result, defect, and metric that you can generate. They will look and evaluate and ask questions. Others will want to concentrate on the highlights and only investigate when there is an obvious problem that is impacting the project. The last category might be summed up by a comment I heard at one Project Meeting addressed to the test manager and coming from the biggest stakeholder: “As long as you say it is okay; I don’t want to hear anymore”. We removed the actual names to protect the guilty!

    While we will come back to this question in a couple of weeks and welcome your input in the interim, one obvious comment comes to mind immediately. Test tools have had a huge impact on this aspect of testing. The ability to record almost everything, drill down, add comments, set statuses, and move items from user to user has facilitated reporting.

    In an interesting story from one client a long time ago, the layers of management made the whole process spin out over two weeks from the time the test was actually done to the time the report was consolidated for the last layer of maangement. We have moved a long way from that!

  • Does your software Testing Pay – Part 4

    “Does your Software Testing Pay” is the question we posted six weeks ago. We posted a simple calculation for a ROI and then a few comments that were received as feedback. Those comments were appreciated since they extended the idea past the direct defect costing calculation. Today we want to wrap up this series with some general comments that extend the idea a little farther.

    For the most part we have concentrated on the Software Testing aspect of Quality Assurance. Almost everyone we speak with is familiar with this aspect. Most software companies or those who depend on software complete some form of testing and the concept of testing can be generalised to cover a multitude of activities. However, it is not the entire picture.

    So, at the risk of making the title slightly inaccurate, what other aspects of Quality Assurance can be considered:

    1. Process improvement for the entire SDLC.
    2. Timely intervention at the Root Cause of many of the defects to prevent them from occuring in the first place.
    3. Concentration on the required end-result to make sure everyone is working towards that end. It is suprising how often this is obscured in large organisations. We attended a course a couple of weeks ago and spoke to the instructor afterwards and apparently many people on the course have only been told a very small piece of what they need to do (and part of that was to attend the course although the reason why was not provided). Not only is that against the concepts of keeping people informed of why they are doing something but it is also very de-motivating.

    The above list (particularly number 1) can cover a lot items that are very detailed and can add a lot of value to the Quality Assurance efforts. We may take this up in the fall.

  • Does your software Testing Pay – Part 3

    “Does your Software Testing Pay” is the question we posted four weeks ago. We then posted a simple calculation for a ROI two weeks ago.
    After we posted the second blog we had a couple of comments to the effect that concentrating on defects was a very narrow focus in terms of cost recovery and benefits. This was certainly a valid criticism; we had concentrated on something that could be answered and calculated without too much effort.

    Some of the suggestions that came back were as follows:

    1. Contributing to a better product using the information gleaned from testing
    2. Enhanced knowledge of the product for both testers and developers.
    3. Future design improvements.
    4. A better quality product.

    No doubt more items could be added to the above list

    Now we just need to cost them

    Since some of these are subjective benefits, it is suggested that they be documented and then all stakeholders can assign them to a value bucket (independently). As a starting point we can use an averaged value for each benefit and then convert that into a dollar value to determine the benefit.

  • Does your software Testing Pay – Part 2

    “Does your Software Testing Pay” is the question we posted two weeks ago.
    There are obviously two parts to this question, one is how much it costs and the other is how much it saves.

    The cost portion is reasonably easy:

    1. Chargeback rate on resources (means hours per release must be recorded).
    2. Equipment and space usage (if not included in the above).
    3. Test Tool cost (amortized over all the projects)
    4. Opportunity cost if the resources should be engaged in something else.

    The savings portion is somewhat harder:

    What would each found defect have cost to fix in production? This is obviously an estimate.  One calculation is supplied below.

    1. Look at each defect found by testing.
    2. Estimate the probability of it occuring after release of the code to the users. You might want to take estimates from several people and average them. Either the probabilities must be between 0 and 1 or else you need to convert to a figure between 0 and 1 before using it.
    3. Estimate the cost to the organisation under the assumption that the defect does occur in production. This cost includes the direct costs to the company (fixing, testing and deploying); the indirect costs (administration etc) that are often hidden (Iceberg – 9/10 of the costs are hidden)
    4. The cost of the customers in rework, lost data, inability to respond proerly.
    5. Add up the costs per defect and multiply by the percentage.
    6. Add up the resulting figure for all defects found by testing for a release.

    If Savings > Costs your software testing is paying for itself.

  • March 2019 QA Events in the GTA and Beyond

    If you are in the Greater Toronto Area or Kitchener-Waterloo you might want to consider these events to network with other QA people or learn some of the new ideas in QA.


    NVP Software Solutions will be participating in the following software testing and quality assurance events happening this March or April in Ontario, Canada. The events are located in Toronto and Kitchener-Waterloo in the coming weeks. Check out the relevant websites for more information and to register. This is a great opportunity to connect with other software testing and quality assurance professionals. We hope to see you there!

    This image has an empty alt attribute; its file name is TASSQ-1.png



    WHY AND HOW TO ADOPT BDD IN SOFTWARE DEVELOPMENT – Rob Ashall- 26 March, 2019
    See https://www.toronto-assq.com/ to register
    This image has an empty alt attribute; its file name is cropped-kwsqa_withicon_small.png


    KWSQA – Kitchener Waterloo Software Quality Association – Wednesday, April 3, 2019 – Making the Web a Little More Accessible – Samantha Campbell  – See https//www.kwsqa.org/kwalitytalks  to register

    Courses

    Plan your 2019 Courses today. See nvp.ca/services/training for a complete schedule.

  • Does your Software Testing Pay?

    “Does your Software Testing Pay” is another question (in the series we are addressing here) that comes up quite frequently. Although it may not be stated directly it can manifest itself in the following actions:
    1. Lack of resources for testing.
    2. Lack of time provided for testing.
    3. Lack of suitable hardware on which to conduct testing
    4. Lack of software testing tools.
    5. …
    This is not a comprehensive list but it covers many of the major manifestations of not having an answer as to whether the testing pays. If the question cannot be answered, then it is difficult to justify expenses or investment in testing. In addition, some Quality Assurance people go to great lengths to avoid answering this question.
    Come back in two weeks to see one way of addressing this question. It will require research, metrics, and statistics.

    Contact us or join the discussion.
    Past Blogs
    Monthly Newsletter