NVP Software Solutions

Tag: Quality Assurance

  • Does your software Testing Pay – Part 3

    “Does your Software Testing Pay” is the question we posted four weeks ago. We then posted a simple calculation for a ROI two weeks ago.
    After we posted the second blog we had a couple of comments to the effect that concentrating on defects was a very narrow focus in terms of cost recovery and benefits. This was certainly a valid criticism; we had concentrated on something that could be answered and calculated without too much effort.

    Some of the suggestions that came back were as follows:

    1. Contributing to a better product using the information gleaned from testing
    2. Enhanced knowledge of the product for both testers and developers.
    3. Future design improvements.
    4. A better quality product.

    No doubt more items could be added to the above list

    Now we just need to cost them

    Since some of these are subjective benefits, it is suggested that they be documented and then all stakeholders can assign them to a value bucket (independently). As a starting point we can use an averaged value for each benefit and then convert that into a dollar value to determine the benefit.

  • Does your software Testing Pay – Part 2

    “Does your Software Testing Pay” is the question we posted two weeks ago.
    There are obviously two parts to this question, one is how much it costs and the other is how much it saves.

    The cost portion is reasonably easy:

    1. Chargeback rate on resources (means hours per release must be recorded).
    2. Equipment and space usage (if not included in the above).
    3. Test Tool cost (amortized over all the projects)
    4. Opportunity cost if the resources should be engaged in something else.

    The savings portion is somewhat harder:

    What would each found defect have cost to fix in production? This is obviously an estimate.  One calculation is supplied below.

    1. Look at each defect found by testing.
    2. Estimate the probability of it occuring after release of the code to the users. You might want to take estimates from several people and average them. Either the probabilities must be between 0 and 1 or else you need to convert to a figure between 0 and 1 before using it.
    3. Estimate the cost to the organisation under the assumption that the defect does occur in production. This cost includes the direct costs to the company (fixing, testing and deploying); the indirect costs (administration etc) that are often hidden (Iceberg – 9/10 of the costs are hidden)
    4. The cost of the customers in rework, lost data, inability to respond proerly.
    5. Add up the costs per defect and multiply by the percentage.
    6. Add up the resulting figure for all defects found by testing for a release.

    If Savings > Costs your software testing is paying for itself.

  • Are you satisfied with your testing – Part 2

    You may recall from the blog of two weeks ago that may organisations end up dissatisfied with their testing.

    The key to resolving this, from Quality Assurance, is to plan your testing before your start. Decide on what must be done, what should be done and what need not be done before the project gets very far.

    Sometimes people call these decisions ‘tradeoffs‘ since they imply that something is being traded off against something else and someone is losing out. Tradeoffs are different and do have the characteristics mentioned in the previous sentence. Here we are planning for what needs to be done.

    Other people claim they need to see the software in order to know what to test. At the detail level this can be true but it is not true at the upper levels.

    Still others will claim that they will think of all the testing that needs to occur while they are doing it. This is not a bad method as long as the tester fully understands all the business, technical, and software requirements and can handle all of this. Small projects with little risk can be done this way. Larger projects with higher risks are not so easy.

    A Quality Assurance process considers all the relevant items at the start and does not wait for a crisis to occur or for management to worry about what has been completed or not completed. It is determined at the beginning and the decisions taken at that point, not in the last 10% of the project with a huge amount of the work to complete. Ongoing reporting and process improvement ensures this works properly.

    Contact us or join the discussion.
    Past Blogs
    Monthly Newsletter

  • Are you satisfied with your testing?

    Following on from the theme of last month, many organisations are dissatisfied with their testing. They feel it is incomplete, or ineffective or costs too much. At the end of the testing effort they are left with a vague feeling of unease. Often it is difficult for people to quantify their concerns but they are very real and lead to delays and ongoing expensive testing in an effort to remove this feeling.

    The trouble is that the more they do in testing, the more they may realise what they have not done. This does not increase the confidence level! Furthermore, if they do find problems during this ‘extra’ testing effort the level of confidence drops commensurately and even more testing is required until there is no energy, budget or time left.

    Come back on February 25 to see how a Quality Assurance process can address these concerns long before they become issues.

    Contact us or join the discussion.
    Past Blogs
    Monthly Newsletter

  • Review of the Year – Automation

    Review of the Year – Automation
    Automation of software testing is something that seems to be on the minds of many Quality Assurance Managers and Test Leads. It has been a popular topic for many years.

    Currently we get requests for particular tools and knowledge of their attributes in particular environments; these are usually serviceable. Current status seems to be a separate tool for every need and environment and sometimes every organisation. Based on past experience, in a few years, someone will consolidate all the disparate technologies under one umbrella tool. Then the cycle will start again with people inventing new tools for specific niches and products.

    We also receive requests for people to “automate our testing” with no decision on the tool attached. This is a completely different question and requires some discussion to occur before the attempt to automate even starts. We need to know the what; when and Why the company wants to automate their testing. The thing we want to avoid are the actions below.

    1. Purchase Automated Test Tool.
    2. Install Tool.
    3. Wait for successful automation to save all the cost of the tool and cost of manual testing.
    4. Become disillusioned.
    5. Go to 1; Repeat ad infinitum.

    This has occurred over and over in different organisations. A lot of money gets used up with no progress and eventually the organisation gives up on the cycle and continues manual testing (see last week’s blog).

    Our best recommendations are as follows:

    Look up where you are on the technology maturity level with your current technology.

    Decide what you need (criteria are available) in terms of automation and what you are capable of handling based on maturity level.

    Then do a Plan to implement your automation. Never assume it will just occur. It won’t.

    Want to discuss how to automate effectively? Contact us.

  • Review of the Year – Manual Testing

    Review of the Year – Manual Testing

    Manual Testing is something that seems to be on the minds of many Quality Assurance Managers and Test Leads. Usually they want out of the Manual Testing and view Automated Testing (Blog planned for next week) as the saviour of their budget and time constraints.

    However, judging by the vacant positions we get requests to fill, there is still no shortage of Manual Testing positions at least in our area. There are still a lot of requests for Manual Testers with business knowledge preferred and new software and startups still start with manual testing. We get requests for Automated Testing with specific tools usually requested and will discuss this next week.

    The part that seems to be missing from many of the requests and the subsequent position is any discussion of the How; What; Why; When; and If; of the manual testing.

    There seems to be limited thought given to How the testing is to be done apart from some vague request to build testcases and execute them.

    Little consideration is given to What to test and Why beyond the statement: “We need to test the software”.

    When and If are not such an issue: Yesterday and definitely are the one word answers to those questions.

    These answers certainly provide freedom for the tester to do what they want but that may not always align with all the stakeholder’s wishes and may be 180 degrees off in some cases.

    This leads to a poor ROI and a large waste of time and money.

    There will continue to be a market for manual testers for new changes and new applications that are not yet mainstream. We expect automation to take over many of the repetitive tasks (as has always been the case) The only open question at this stage might be what AI will do to the industry. That we cannot predict.

    Want to discuss the effectiveness of your Manual Testing further? Contact us.

  • QA with No time

    QA with No Time

    One of the constant issues that comes up in discussions or classes is the lack of time for Quality Assurance and Quality Control. We seem to be under constant time pressure and with Agile and DevOps it has become worse rather than better.

    While Quality Assurance cannot anticipate everything, the Continuous Improvement aspect can help improve the time considerations. The key is to prioritise and plan for the high priority items to be completed.

    Some people will say (with a fair amount of truth) that even planning for the priorities and only doing those high priority items will still not be possible in the time allotted for Quality Assurance and Quality Control. With development reacting to changing requirements from the users, constant upgrades in technology (both hardware and software) and impacts from other projects, the time scale can shrink radically.

    However, without a definitive list of what must be done and the risk of not completing attached, it is very hard to push back against the time pressure. With the list in hand, it is easier to quantify the risk of not doing something.

    Some people will say that they do not have the time to even create the list. They are under extreme pressure to start testing immediately and provide issues so the developers have something to handle now that they have finally delivered all the changes requested by the user. Our recommendation, in this case, is to simply make a list of the functions or requirements, assign High, Medium or Low to the risk of not testing that function or requirement and send it around for review. This will at least alert people to the challenges faced by Quality Assurance and Quality Control.

    Want to discuss your list further? Contact us.

  • A Better Way – Case Study 5 – Distributed with Poor Testing

    A Better Way – Case Study 5 – Distributed with Poor Testing

    In our last several blogs we have discussed ‘A Better Way to Test”.

    The issue is to apply this to actual situations. We have 5 Case Studies we plan to use over the next several weeks to address this. The fifth case study might be called “Distributed with Poor Testing”.

    In this case the developers were geographically remote, the Quality Assurance (such as it was) was local and the clients were geographically remote. This was a somewhat unusual situation since it is frequently the Quality Assurance that is remote and the other two local. However, some of the considerations had to be the same. We needed a way of communicating things without losing anything in transit.

    Two solutions came to mind immediately:

    1. Make sure there are stated expectations regarding the flow of information. This applies to any artifact like a test case, a requirement, a defect, or a design document. We did this via some flow charts we placed in the Master Test Plan although there are certainly other places they could have been recorded.
    2. Acquire some tools that will support this information store. There are many free or very cheap cloud-based tools that support these processes.

    Once we had the framework in place, we set up the appropriate loops and feedback mechanisms to ensure information flow and good quality. As time went on we expanded the groups who had access to the systems ensuring that the information flowed in the correct way and with sufficient security to the concerned parties. The two way flow of information allowed us to eliminate a backlog of defects that had accumulated and start addressing customer concerns in a timely manner.

    The investment was not large when compared to the improvements realised.

    If you want to discuss this further contact us.