NVP Software Solutions

Tag: Quality Control

  • Does your software Testing Pay – Part 2

    “Does your Software Testing Pay” is the question we posted two weeks ago.
    There are obviously two parts to this question, one is how much it costs and the other is how much it saves.

    The cost portion is reasonably easy:

    1. Chargeback rate on resources (means hours per release must be recorded).
    2. Equipment and space usage (if not included in the above).
    3. Test Tool cost (amortized over all the projects)
    4. Opportunity cost if the resources should be engaged in something else.

    The savings portion is somewhat harder:

    What would each found defect have cost to fix in production? This is obviously an estimate.  One calculation is supplied below.

    1. Look at each defect found by testing.
    2. Estimate the probability of it occuring after release of the code to the users. You might want to take estimates from several people and average them. Either the probabilities must be between 0 and 1 or else you need to convert to a figure between 0 and 1 before using it.
    3. Estimate the cost to the organisation under the assumption that the defect does occur in production. This cost includes the direct costs to the company (fixing, testing and deploying); the indirect costs (administration etc) that are often hidden (Iceberg – 9/10 of the costs are hidden)
    4. The cost of the customers in rework, lost data, inability to respond proerly.
    5. Add up the costs per defect and multiply by the percentage.
    6. Add up the resulting figure for all defects found by testing for a release.

    If Savings > Costs your software testing is paying for itself.

  • Review of the Year – Automation

    Review of the Year – Automation
    Automation of software testing is something that seems to be on the minds of many Quality Assurance Managers and Test Leads. It has been a popular topic for many years.

    Currently we get requests for particular tools and knowledge of their attributes in particular environments; these are usually serviceable. Current status seems to be a separate tool for every need and environment and sometimes every organisation. Based on past experience, in a few years, someone will consolidate all the disparate technologies under one umbrella tool. Then the cycle will start again with people inventing new tools for specific niches and products.

    We also receive requests for people to “automate our testing” with no decision on the tool attached. This is a completely different question and requires some discussion to occur before the attempt to automate even starts. We need to know the what; when and Why the company wants to automate their testing. The thing we want to avoid are the actions below.

    1. Purchase Automated Test Tool.
    2. Install Tool.
    3. Wait for successful automation to save all the cost of the tool and cost of manual testing.
    4. Become disillusioned.
    5. Go to 1; Repeat ad infinitum.

    This has occurred over and over in different organisations. A lot of money gets used up with no progress and eventually the organisation gives up on the cycle and continues manual testing (see last week’s blog).

    Our best recommendations are as follows:

    Look up where you are on the technology maturity level with your current technology.

    Decide what you need (criteria are available) in terms of automation and what you are capable of handling based on maturity level.

    Then do a Plan to implement your automation. Never assume it will just occur. It won’t.

    Want to discuss how to automate effectively? Contact us.

  • Review of the Year – Manual Testing

    Review of the Year – Manual Testing

    Manual Testing is something that seems to be on the minds of many Quality Assurance Managers and Test Leads. Usually they want out of the Manual Testing and view Automated Testing (Blog planned for next week) as the saviour of their budget and time constraints.

    However, judging by the vacant positions we get requests to fill, there is still no shortage of Manual Testing positions at least in our area. There are still a lot of requests for Manual Testers with business knowledge preferred and new software and startups still start with manual testing. We get requests for Automated Testing with specific tools usually requested and will discuss this next week.

    The part that seems to be missing from many of the requests and the subsequent position is any discussion of the How; What; Why; When; and If; of the manual testing.

    There seems to be limited thought given to How the testing is to be done apart from some vague request to build testcases and execute them.

    Little consideration is given to What to test and Why beyond the statement: “We need to test the software”.

    When and If are not such an issue: Yesterday and definitely are the one word answers to those questions.

    These answers certainly provide freedom for the tester to do what they want but that may not always align with all the stakeholder’s wishes and may be 180 degrees off in some cases.

    This leads to a poor ROI and a large waste of time and money.

    There will continue to be a market for manual testers for new changes and new applications that are not yet mainstream. We expect automation to take over many of the repetitive tasks (as has always been the case) The only open question at this stage might be what AI will do to the industry. That we cannot predict.

    Want to discuss the effectiveness of your Manual Testing further? Contact us.

  • A Better Way – Case Study 4 – Caught Between Vendor and Client!

    A Better Way – Case Study 3 – Test Plan in a Hurry!

    In our last several blogs we have discussed ‘A Better Way to Test”.

    The issue is to apply this to actual situations. We have 5 Case Studies we plan to use over the next several weeks to address this. The fourth case study might be called “Caught between Vendor and Client”.

    Although one could argue that Quality Assurance has always been caught between a Vendor or Vendors (developers) and possibly multiple clients (users), it has become a little more obvious and formal with the purchase of software from outside groups. This is not something that is going to go away. Assembly of a solution rather than building it has been around for quite a while and will probably become more frequent rather than less. The question is what Quality Assurance does to “Bridge the Gap” between what the client wants (‘perfection!’) and what the vendors are willing to supply in terms of proof given competitive secrets and possibly some Non-disclosure requirements.

    In our case (as discussed here) we built a plan that provided the final client with what they wanted. We then mapped what was supplied by the vendors against what was required and filled in the rest. Not surprisingly, the main items that were missing were Integration testing between what the various suppliers had provided and the type of testing that needed the entire system to be there including Performance, Security and Usability (to name a few).

    Bridging the gap in this fashion satisfied everyone and made use of everything that was already in place. That saved us a lot of time and allowed us to concentrate the tests that were critical to the client.

    If you want to discuss this further contact us.

  • A Better Way – Case Study 3 – Test Plan in a Hurry!

    A Better Way – Case Study 3 – Test Plan in a Hurry!

    In our last several blogs we have discussed ‘A Better Way to Test”.

    The issue is to apply this to actual situations. We have 5 Case Studies we plan to use over the next several weeks to address this. The third case study might be called “Test Plan in a Hurry!”.

    This issue came up from an organisation that was part way through (as usual!) an engagement and suddenly required a test plan to satisfy the client. The request came from the Project Manager on Friday with a deadline of Monday afternoon. There was no prior exposure to the project; no knowledge that the request was coming; and very little in the way of Project Documentation (with certainly no time to review it). There was a temptation to ignore the request and had they not been an existing client, we might have been tempted to point out that this was not really an effective use of time. However, given the nature of the request and the people from whom it came, we went ahead with the attempt.

    Clearly, we were not going to get detailed testcases or test objectives based on what we had been given. So we opted for a process based Master Test Plan. In other words, we put together a statement of how the project would be tackled from a Quality Assurance point of view when the information became available. We put in processes for all the Quality Assurance items and highlighted the risks inherent in testing under these conditions (including the lack of any understanding of the project) and went forward with that. We put a strong emphasis on what was required now in order to make this work.

    If you want to discuss this further contact us.

  • A Better Way – Case Study 1 – Thinking Outside the Box

    A Better Way – Case Study 1 – Thinking Outside the Box

    In our last several blogs we have discussed ‘A Better Way to Test”.

    The issue is to apply this to actual situations. We have 5 Case Studies we plan to use over the next several weeks to address this. The first one revolves around a junior QA, a very successful (small) company and a need to test effectively and quickly for a large final client. The issue that came up was the ability to “Think outside the box”.

    We received a call from the President of the company indicating that the new junior QA was having trouble considering things “Outside the box”. They were good with what was presented in the Use Cases or User Stories. Most people can generate testcases based on what is provided and the happy path from the Use Cases and User Stories. Experienced testers will apply other techniques and may specifically try Exploratory testing. However, you cannot explore or test what you don’t know or don’t think of at the time.

    Rather than dictating “Outside the box” which is a contradiction in itself, we decided to go with more of a checklist approach listing some of the areas that would be considered to be non-Happy Path and see if it could lead to further extensions. We did not dictate everything but started with more of a charter and guidance list to see what would come of it. The process had two advantages:

    1. We would have some coverage of everything “inside” and “outside” the box.
    2. We could use the results to evaluate the suitability of the junior QA for further roles and projects.

    The President came up with the basic list and ran it from there.

    If you want to discuss this further contact us.

    Take a look at some of the seminars that we offer that address this situation and see if they apply to you. Testing can be better.
    Contact us for further information.

  • October QA Events – Toronto and GTA

    NVP Software Solutions will be participating in the following software testing and quality assurance events happening this October in Ontario, Canada. The events are located in Toronto, Kitchener-Waterloo and London in the coming two weeks. Check out the relevant websites for more information and to register. This is a great opportunity to connect with other software testing and quality assurance professionals. We hope to see you there! (more…)

  • A better way to test – 6

    A better way to Test – 6

    In our last blog we discussed, as the last point in our list, testing to risk.

    If you have the risks listed, explained, classified, calculated, ordered and entered into a database, it may be easy to attach testcases to them and determine when you have addressed the risk with testing. Not all risks can be efficiently addressed by testing, it is simply one of the possible techniques to be used.

    However, people rarely agree on the risks and, as mentioned above, not all risks can be addressed by testing. So this needs to be flexible.

    If you don’t have your risk recorded then the testcases may still be built to address perceived risks but they may be difficult to justify. Having both sides of the need (the risk and the testcase) can make each justify the other.

    Once we have this for one release of the software, the only question is whether we can continue using the same testcases from release to release. Risks will change, some will disappear or be downgraded, new ones will appear or the ones we missed last time will be included for future releases. Like all project requirements, nothing is ever perfect and we progress (sometimes unevenly) to an end result of coverage that is sufficient for the project. Of course something will always surprise us.

    The key consideration is not to become inured to the other methods of addressing risk – testing is not the end solution for everything.

    Take a look at some of the seminars that we offer that address this situation and see if they apply to your situation. Testing can be better.
    Contact us for further information.