Tag: Project Management

  • The Out-of-Scope of Testing

    The Out-of-Scope of Testing has only one definition (unlike the Scope of Testing) but it is much harder to document. We want to list everything we are not planning to test! However, the inevitable question is how far to go with our list. Obviously we are not going to list everything we are not going to test and we cannot simply say that everything that is not in-scope is out-of-scope (tempting as that may be).

    The (not entirely rhetorical) questions you will get asked include the following:

    1. Are you planning to omit that from your testing? (should be included)
    2. What made you think that isn’t part of this project? (should also be included)
    3. There is too much risk to avoid testing that.

    The danger with listing out-of-scope items is that they remind people of other items and they then start to think about the risk of avoiding testing that item. You start to get scope creep on your testing and since it is not part of the plan you start to get schedule slippage and cost escalation. Very few people will tell you to test less – it is too risky to state that.

    Whatever your policy; the following will guide the creation of the out-of-scope list:

    1. What can we avoid testing?
    2. What will simply not be able to be completed within budget and resource constraints?
    3. What is the best use of the resources?

    Make sure to document the out-of-scope up front and get it signed off. That will reduce the problems later on and create a much more harmonious working relationship.

    One of the best methods is to build a System Boundary Diagram and let that drive the Scope of testing. Not only does it clarify the thoughts of everyone in the project, it is very effective way of conveying the scope and out-of-scope pieces of the project.

    In general we try to test the interfaces of the System Boundary diagram but avoid full testing of the items outside the System Boundary Diagram. Those are out-of-scope.

    Discussion Questions

    1. Do you define your Out-of-Scope of Testing?
    2. Has it been disputed?
    3. What would you have done differently based on what you know now?

    Next Week: Training

  • Quality Assurance Assessments – Part 4

    Quality Assurance Assessments take a variety of forms in an IT project and can range from very informal to very formal. This week we will discuss What to do with the results of a QA Assessment now that we have completed an explanation of HOW to do Assessment in a past blog.

    What to with the results of a QA Assessment

    There is a strong temptation to (facetiously) say Do Nothing with the Results since that happens so frequently. The Assessment is completed and everyone just wants to forget about it. Not only is that a direct waste of the effort and time included in the assessment, it also sends the signal to everyone that their effort was unnecessary and their thoughts unappreciated. Don’t expect a lot of effort next time under this scenario.

    If we use the example from the last blog (referenced above) of the questionnaire or in person interviews to elicit the information using open-ended questions, then we will end up with a lot of disparate information that may not be readily parsed.

    The steps are as follows:

    1. Review all the provided answers.
    2. During the review write down some general categories for the answers (i.e. insufficient testing; requirements issues; development issues; testing issues). If these categories were predetermined then this step does not apply.
    3. Allocate the answers into the categories.
    4. Allocate the answers that fit into more than one category (put them into both).
    5. Allocate the answers that only occur once and do not fit into any category (make a category of Other and put them there).
    6. Extract a common consensus from each category (there is a lot of work in this step)
    7. Start a process of finding the root cause of the common problems.

    Now we have to act on the root causes and resolve them. This could be a whole series of blogs but we will leave that for the process improvement cycle.

    If you are having trouble working this out, contact us and we can help guide you and your team in the right direction.

    Finally, we’ll leave you with a few questions and ask you to post your answers.

    1. Have you participated in a Test Process Assessment?
    2. Has anyone acted on the results?
    3. Were the results used for Process Improvement?

    Next Week: Vocabulary

  • Quality Assurance Assessments – Part 3

    Quality Assurance Assessments take a variety of forms in an IT project and can range from very informal to very formal in nature. This week we will discuss HOW to do a QA Assessment now that we have completed an explanation of WHY to do Assessment in a past blog.

    HOW Quality Assurance Assessments are Done:

    The following needs to be done in order to complete a Quality Assurance Assessment:

    1. Determine the objective of the assessment. (Refer to Why Quality Assurance Assessments are Done in a past blog).
    2. Set up a team (may be a team of 1) to do the assessment.
    3. Determine the targeted group who are going to provide information to the team.
    4. Determine the method of getting the information (questionnaire; in person interviews; survey).
    5. Using the method selected above, carry out the assessment.
    6. Collate the results.
    7. Provide a report.

    The above is a general methodology. The following is a short example of a very basic QA Assessment.

      1. The objective is to determine how well the Software Testing Process worked on the last project.
      2. The team will be the Quality Assurance department (not involved in the particular project).
      3. Target audience: Software Testers, Developers, Project Manager(s), End Users, Management, all other interested stakeholders.
      4. Methodology: Individual interviews using a questionnaire. Some sample questions follow
        • What went right with the project in terms of Software Testing?
        • What went wrong with the project in terms of Software Testing?
        • What expectations were and were not met?
        • How could the process be improved?

    This is a very small sample of questions to be answered. From there:

    1. Compile the answers into a report removing the names and any identifying comments.
    2. Create a set of recommendations based on the results.

    If you are having trouble working this out, contact us and we can help guide you and your team in the right direction.

    Finally, we’ll leave you with a few questions and ask you to post your answers.

    1. Have you participated in a Test Process Assessment?
    2. What was the justification for the Assessment?
    3. Were the results used for Process Improvement?

    Next Week: KWSQA, TASSQ and London Peer-to-Peer

  • Quality Assurance Assessments – Part 2

    Quality Assurance Assessments take a variety of forms in an IT project and can range from very informal to very formal in nature. This week we will discuss WHY we want to do a QA assessment now that we have completed an explanation of WHAT an Assessment is in a past blog.

    WHY Quality Assurance Assessments are Done:

    1. They’re Mandated

    If it is mandated but not accepted then it is an exercise in futility. This is quite possibly the worst reason to start a QA Assessment, especially from the viewpoint of those carrying out the assessment. The process will be completed with little effort given, in the shortest time possible and the results will end up on a shelf, never to be discussed again.

    2. To Find Out What Went Wrong

    In this case, there will be an effort to find out what went wrong in the just-completed project and even if no further action is taken, at least people are aware of what everyone considered to be problems (which may not have been obvious to other people in the project). The danger is that this session and the final report can easily turn into an exercise of complaint and self justification.

    3. To Improve the Process

    This is by far the best justification for a Test Process Assessment. If parties are focused on what needs to be done to make the next effort easier, better, or more effective, then the discussion will focus on finding solutions to problems encountered. The participants cannot simply list their problems and concerns and then walk away.

    We want to finish with a few questions and ask you to post your answers.

    1. Have you participated in a Test Process Assessment?
    2. What was the justification for the Assessment?
    3. Were the results used for Process Improvement?

    Next Week: Test Plans

  • Quality Assurance Assessments – Part 1

    Quality Assurance Assessments take a variety of forms in an IT project and can range from very informal to very formal in nature. No matter where each assessment lands on the spectrum, they are intended to do only one thing: Improve the Processes that are used. This week we will discuss the various forms of assessments.

    Post Project

    Post project assessments have had a variety of names throughout the years:

    1. Post-Implementation Review
    2. Lessons Learned
    3. Post Mortem

    Regardless of what they are called, they all have one purpose. The people on the project are asked to recall what went right and what went wrong in a project. This could mean trying to remember things that are a few years old, which isn’t always ideal. The person conducting the assessment then provides the information to a recorder. The resulting information is compiled somewhere and (sometimes) used for process improvement. (More on that in our next blog series).

    In-Project Assessments

    In-Project Assessments can have a variety of formats:

    1. Walkthroughs
    2. Reviews (checkpoint and in process)
    3. Inspections

    Sometimes the above are referred to as Reviews and not as Assessments.

    Test Process Assessment

    The last type of Assessment we want to discuss is the Test Process Assessment. This one is aimed solely at assessing the Test Process used for any particular project. It is often carried out by an independent group although it can be done by a Quality Assurance person or a test manager. The assessment looks for places where the process is not efficient or not effective. It looks for redundancies and repetition. Frequently the assessment starts with the Quality Assurance aspect of the project but finds itself extending back into the Development piece and forward into Implementation. It’s usually difficult to carry out an assessment just of the Quality Assurance piece when there are so many other interlocking pieces that impact Quality Assurance and Quality Control. See nvp.ca/services/assessments/ for further details.

    Next Week: Test Cases

  • Early vs Late Testing

    Early versus Late Testing within a project is an ongoing debate within the software testing industry. With the Waterfall methodology, active testing was not possible until late in the development cycle when the project was ‘thrown over the wall’ to testing and the testers started what some called the ‘final exam’. A long test-fix cycle often ensued. Many newer methodologies promote earlier involvement and many of the recent ones promote continuous testing of multiple builds using test automation. With a broad definition of testing that encompasses more than just direct Validation it is quite possible to start some form of testing very early in the project. However, some feel that we have obtained as many benefits as we possibly can from early testing and we need to bring back an emphasis on late cycle testing when the system is fully developed in order to prove everything works together.

    This debate is missing the point of the testing process. The only question is when is it appropriate to do testing and this can be answered at the time of planning and with the help of Quality Assurance. Testing has to be completed when it is appropriate and when the necessary items are in place to support that endeavor. During the test planning process, it is necessary to identify what testing needs to be completed. Once we know what has to be completed, then we can identify where in the process it can be completed. Quality Assurance processes allow that identification to take place easily. With some overarching process in place that indicate what the organisation requires in the way of Risk reduction, it is easy to identify what testing is required and then map that to the appropriate place in the Software Development Life Cycle.

    While Early or Late testing may not always be appropriate, early planning is always necessary and Quality Assurance guides that planning.