NVP Software Solutions

Category: Uncategorized

  • Test Run

    Our latest blog will discuss the Test Run. For today’s purpose, NVP considers a Test Run to be one, single execution of a testcase. This could mean that the testcase ran to completion and the expected AND actual results were identical, or that the case the testcase did not have actual results that equalled the expected. We have stayed away from the words ‘successful’ and ‘unsuccessful’ since some may feel a testcase is only successful if it uncovers a problem and is unsuccessful if it does not.

    We are interested in this statistic of test runs for a number of reasons:

    1. It helps in estimation
    2. It helps justify the time taken to test
    3. It provides a measure of code stability

    Estimation

    Knowing the number of Runs of a testcase helps determines how long the cycles and the whole test effort will take next time. If we know we had to run each testcase an average of 5 or 6 times before it ran to completion without raising an issue then we know how many times we may need to run it next time. Note that unsuccessful runs may include attempts that lead to fixing the testcase or relevant test data. Once we have ‘debugged’ the testcase, these runs may not recur.

    Justification

    If we only report the count of completed testcases with actual results equalling expected results, then each testcase might only show a single execution. This would hide a lot of work and effort and make the testers appear very unproductive. Showing that each testcase was executed 6 or 7 times before we were satisfied gives a much better idea of the effort involved.

    Code Stability

    If a testcase is run a dozen times and only on the last time does it run to completion with Expected Results equal to Actual Results, then we may have a concern with code stability or whether that final run was really correct. Something that fails a dozen times and then is successful is highly suspect. Maybe the conditions changed, maybe we missed something, maybe the issue was finally fixed. Whatever the case, we are not sure of the stability.

    Discussion Questions

    1. Do you have defined test Runs?
    2. What is the worst case for number of times they had to be run?
    3. What is your least number of runs

    Next Week: Process Improvement

  • Upcoming Software Testing & Quality Assurance Events – May 2016

     

    NVP Software Solutions will be participating in the following two software testing and quality assurance events happening this May in Ontario, Canada. The events are located in Toronto, and London in the coming two weeks. Check out the relevant websites for more information and to register. This is a great opportunity to connect with other software testing and quality assurance professionals. We hope to see you there!

     

    Toronto Association of Systems & Software Quality

    TASSQ – Toronto Association of System and Software Quality – Are you using your wireless devices safely? – Frank Clegg & One Digital Lab in the Cloud : A Key to Digital Engagement Success – Karla Toyloy
    See http://www.tassq.org/

     

    &

    London Quality Assurance Peer-to-Peer Contact – The Future of Software Testing – Neil Price-Jones, President – NVP Software Solutions. Contact neil@nvp.ca for more details

  • Quality Assurance Process Improvement – Part 1

    Quality Assurance Process Improvement is the next topic in our Blog Series. We completed a series of 4 on Assessments and at the end of the Assessment process a lot of organizations tend not to act on the results if they do not have a plan for moving forward. This is particularly true if the Assessment has not been tailored to the particular company. A standardised Assessment process generates standard recommendations which may not be applicable. Make sure you detail your expectations at the beginning of the Assessment so you get value from the process and your expenditure of time.

    Assuming that you have the results of an Assessment available, the next step is often some sort of Process Improvement. So what is Process Improvement?

    We define Process Improvement as a planned and measured activity that leads to a better process.

    Planned – Process Improvement carried out without a plan is simply change for the sake of change with no knowledge of whether this is an improvement or not.

    Measured – Measurements need to be taken both before and after the process improvement activity is completed. Before measurements provide a current baseline of how the process is currently operating. After measurements determine whether the process improvement activity has had the desired effect. Without both, there is no point to the process improvement.

    Better Process – The measurement process tells us whether the process has improved at all. There is a number of things that could be measured. Maybe the process has got quicker; maybe it is cheaper in some way; maybe it is more consistent so the results are more similar; or maybe the end product or service generated from the process is better. It would be nice to get all four with one process improvement activity but frequently it will require multiple iterations of the process improvement cycle to start realising gains on all four items.

    Next Week: Coming Meetings

  • Test Cycles

    NVP considers a Test Cycle to be one complete execution of a group of test cases. The reason we’re interested in this particular item is that it leads to estimation. The first questions in any testing project are:

    1. How long is it going to take?
    2. How much is it going to cost?
    3. When will you be done?

    These questions can be difficult to answer when starting a project as a new tester or test manager or with limited experience in the software one has been asked to test. Having test cycles helps solve that issue.

    In order to answer those questions we need to:

    1. Define the contents of the group of tests constituting the cycle
    2. Get an estimate of how long each test will take
    3. Add up the resultant times
    4. Build in some contingency
    5. Use that as an estimate for the length of the cycle

    The above gives us an estimate for the length of a single cycle.

    The next question is how many cycles will be run. Our answer is usually three at a minimum on the grounds that there are two debug cycles and hopefully a clean run. In our experience we have managed to get away with two cycles but that’s unusual. Many times it’s many more than three especially if the code is weak or the full requirements are still being worked out. Usually you will have an idea after your first test cycle as to how many will have to be run.

    In order to answer the question of when you will be done, you then need to multiply the number of projected cycles by their individual lengths, add in time for the fixes to be made and promoted and use that as an estimate of the completion date (and the cost by using the chargeback rate).

    1. Do you have defined test cycles?
    2. What is the worst case for number of times they had to be run?
    3. What is your least number of runs

    Next Week: Process Improvement

  • Quality Assurance Assessments – Part 4

    Quality Assurance Assessments take a variety of forms in an IT project and can range from very informal to very formal. This week we will discuss What to do with the results of a QA Assessment now that we have completed an explanation of HOW to do Assessment in a past blog.

    What to with the results of a QA Assessment

    There is a strong temptation to (facetiously) say Do Nothing with the Results since that happens so frequently. The Assessment is completed and everyone just wants to forget about it. Not only is that a direct waste of the effort and time included in the assessment, it also sends the signal to everyone that their effort was unnecessary and their thoughts unappreciated. Don’t expect a lot of effort next time under this scenario.

    If we use the example from the last blog (referenced above) of the questionnaire or in person interviews to elicit the information using open-ended questions, then we will end up with a lot of disparate information that may not be readily parsed.

    The steps are as follows:

    1. Review all the provided answers.
    2. During the review write down some general categories for the answers (i.e. insufficient testing; requirements issues; development issues; testing issues). If these categories were predetermined then this step does not apply.
    3. Allocate the answers into the categories.
    4. Allocate the answers that fit into more than one category (put them into both).
    5. Allocate the answers that only occur once and do not fit into any category (make a category of Other and put them there).
    6. Extract a common consensus from each category (there is a lot of work in this step)
    7. Start a process of finding the root cause of the common problems.

    Now we have to act on the root causes and resolve them. This could be a whole series of blogs but we will leave that for the process improvement cycle.

    If you are having trouble working this out, contact us and we can help guide you and your team in the right direction.

    Finally, we’ll leave you with a few questions and ask you to post your answers.

    1. Have you participated in a Test Process Assessment?
    2. Has anyone acted on the results?
    3. Were the results used for Process Improvement?

    Next Week: Vocabulary

  • Software Testing Strategy – The Test Strategy

     

    We discussed Test Plans and their contents in the earlier blog and today we will discuss the Test Strategy. Some people will look at the title above and think we are just repeating our blog from three weeks ago. Luckily, that’s not the case. We differentiate between Test Plans and Test Strategies.

    NVP considers a Test Strategy to be a document that outlines a long term direction for testing. It’s possible that the Test Plans for each project will borrow heavily from the Test Strategy and may even be based on it. We’re not necessarily considering that level of connection in this scenario.

    The following is a set of contents for a Test Strategy

    1. Introduction – outlines the purpose of the document
    2. Corporate Business Strategy – Copy and paste from the relevant business document
    3. A high level outline of the Test Strategy – See below
    4. Alignment with corporate strategy and identification of the gaps
    5. Baseline of current testing (where we are today)
    6. Gaps between the baseline and Test Strategy
    7. Goals or methodologies to close the gaps
    8. Strategy assessment process
    9. Continuous improvement

    The Test Strategy section would include answers to the following:

    1. Where is the best place to complete testing in the life cycle?
    2. Who are the best resources for each level and type of testing?
    3. What is the long term development process for the identified resources?
    4. From where are the resources going to be sourced?
    5. What is the long term automation process or direction?
    6. Where is the greatest ROI for testing?

    Some questions to assess your Test Strategy readiness:

    1. Do you have a Test Strategy?
    2. If you have a Test Strategy; has it been reviewed recently?
    3. Is there a person or a group responsible for the Test Strategy?

    If you answered No to any of the above questions, send us an email to see if a Test Strategy would be a fit for your organization.

    Next Week: Assessments – Deal with Results

  • Upcoming Software Testing & Quality Assurance Events – April 2016

    NVP Software Solutions will be participating in the following three software testing and quality assurance events happening this April in Ontario, Canada. The events are located in Toronto, Kitchener-Waterloo and London in the coming two weeks. Check out the relevant websites for more information and to register. This is a great opportunity to connect with other software testing and quality assurance professionals. We hope to see you there!

     

    Toronto Association of Systems & Software Quality

    TASSQ – Toronto Association of System and Software Quality – Everything you Wanted to Know about the CSQE! – Brenda Fisk, Director,
    ASQ Canada Deputy Regional Director 2014-2016
    Software Division, Division Executive Team – April 26, 2016 – See http://www.tassq.org/

     

     

    Software Testing in Kitchener Waterloo

     

    KWSQA – Kitchener Waterloo Software Quality Association – the bare minimum you need to know about web application security in 2016 – Ken De Souza – April 27, 2016 – See www.kwsqa.org

     

     MANHATTAN

    London Quality Assurance Peer-to-Peer Contact neil@nvp.ca for more details

  • Quality Assurance Assessments – Part 3

    Quality Assurance Assessments take a variety of forms in an IT project and can range from very informal to very formal in nature. This week we will discuss HOW to do a QA Assessment now that we have completed an explanation of WHY to do Assessment in a past blog.

    HOW Quality Assurance Assessments are Done:

    The following needs to be done in order to complete a Quality Assurance Assessment:

    1. Determine the objective of the assessment. (Refer to Why Quality Assurance Assessments are Done in a past blog).
    2. Set up a team (may be a team of 1) to do the assessment.
    3. Determine the targeted group who are going to provide information to the team.
    4. Determine the method of getting the information (questionnaire; in person interviews; survey).
    5. Using the method selected above, carry out the assessment.
    6. Collate the results.
    7. Provide a report.

    The above is a general methodology. The following is a short example of a very basic QA Assessment.

      1. The objective is to determine how well the Software Testing Process worked on the last project.
      2. The team will be the Quality Assurance department (not involved in the particular project).
      3. Target audience: Software Testers, Developers, Project Manager(s), End Users, Management, all other interested stakeholders.
      4. Methodology: Individual interviews using a questionnaire. Some sample questions follow
        • What went right with the project in terms of Software Testing?
        • What went wrong with the project in terms of Software Testing?
        • What expectations were and were not met?
        • How could the process be improved?

    This is a very small sample of questions to be answered. From there:

    1. Compile the answers into a report removing the names and any identifying comments.
    2. Create a set of recommendations based on the results.

    If you are having trouble working this out, contact us and we can help guide you and your team in the right direction.

    Finally, we’ll leave you with a few questions and ask you to post your answers.

    1. Have you participated in a Test Process Assessment?
    2. What was the justification for the Assessment?
    3. Were the results used for Process Improvement?

    Next Week: KWSQA, TASSQ and London Peer-to-Peer