Blog

One of the most common misconceptions is that Quality Assurance is Unmoving or at least Perceived as Unmoving.  Many people believe that once Quality Assurance  has set out a process it cannot be changed (ever).  This came about from the approval process inherent in some companies. By the time a process or standard (for example) had been

• Identified as needed.
• Created.
• Made the rounds for approvals.
• Gone up the ladder for approval (and back down) several times.
• Been enshrined in the documentation.
• Shipped out to all the divisions.
  and
• Finally used.

there was little incentive to start on a new process or modify the existing one that had finally come into existence and was still working its way through the company.

However, nothing could be further from what was intended. Quality Assurance  advocates continuous improvement and nothing is static if it can be improved. Furthermore a review and revision process is automatically built in for all processes (including the process for building processes!). It is assumed that improvement is always possible and refinement through several versions is expected, particularly for widely used standards or processes.

The problems arose as a result of the approval and promulgation process that was mandated after Quality Assurance had done its work. The improvement had already been identified and was accepted. However, the process to get it out, accepted and used was long and arduous. Most of the dissemination process has been fixed recently with the use of various communication methods  but Quality Assurance is still labouring under the misconception that it is static.

In some ways, Quality Assurance  anticipated many of the iterative methodologies that are now so popular.  There is always room for improvement and the feedback loop runs continually.  Nothing is ever really final until it is shipped and even then we can continue to make improvements for next time.

A colleague pointed out recently that we are back in System Maintenance mode; a lot of systems are being patched and upgraded instead of being replaced. Not everyone is in this mode; there is still a replacement and upgrade occurring but others are hoping that their existing systems along with a little tender loving system maintenance will last them awhile longer.

Some of this reluctance to invest is driven by budget concerns; there is a general unwillingness to spend substantial funds if the business direction is not clear or certain. There is no point in embarking on a large expenditure if there may be a large change in the business need.

Some of this reluctance is driven by an uncertainty of what technology can or will provide in the near future. Systems that were built a few years ago may be outdated but with a little judicious tweaking they will accommodate connections to the new technology and allow the organisation to get partial benefit while still utlising their existing investment in technology.

A substantial amount of this is driven by a lack of knowledge of the existing systems; how they work and what they actually do. It is very hard to embark on a major system replacement when no one knows how the existing systems work and what they do. There is a fear of losing functionality on which the company depends for basic business activity.

And lastly, there is strong component of corporate memory which recalls how badly it went last time a mass IT upgrade was attempted. Nothing stops a new initiative faster than someone saying “The last time we tried this, it went x% over budget, y% over schedule and we ended up with less than we had before.”

Quality Assurance can alleviate all these problems. QA is not just testing; it is process improvement, rationalisation, and organisation.

QA techniques can save on the budget and ensure that it is monitored and controlled.
QA can determine how the design fits with future trend and add the proper consideration of emerging trends at the correct points in the SDLC.
QA can provide techniques for determining the activity of existing systems.
QA can utilise past experiences to improve future outcomes.

Last week we talked about the common comment ‘always having time to do it over but never having the time to do it right in the first place’. Doing it Right has that famous answer: “If there are n people at a meeting, then there are n opinions on what is the right way and the n+1 st is the correct one”.

To some people doing it right means following the process. This makes the prior assumption that the process is right and provides the correct result. If you are reasonably sure that the process is stable and generates at least close to what is needed as an end result then you might get something that is reasonable at the end. It would be necessary to ensure those two items but it is a start. If the process is wrong, then no amount of work will get you the correct result. If the process is unstable then you may get something useful or you may get something completely wrong (and not know it).

To other people doing it right means doing it their way or no way. If you are absolutely sure that your way is the right way, then that may also suffice. However, if you are in the least bit unsure then the end result may bear no resemblance to the initial requirements.

Some believe that doing it right means supplying the customer with what they want no matter the effort or the convolutions in the process along the way.

Quality Assurance takes all of these into consideration and melds them together to ensure we are doing it right. A QA Assessment reviews the entire process with the following considerations as initial guidance:

1. Is the customer on track to get what they want?
2. Does the process support that end result at present?
3. Are there parts of the process which are not supporting the end requirements?
4. Are there parts of the process which are not being done even though they would support the end requirement?
5. Are there efficiencies that could be realised by process improvement?

NVP Software Solutions specialises in QA Assessments. Give us a call.

Just this morning, the common lament appeared on a LinkedIn post: We always have time to do it over but never time to do it right in the first place. This is by no means new, there are people who have been saying this or similar comments for years and they long predate software.
If you read some of the original Quality Pioneers (who were looking at manufacturing and not at software) they had tried to calculate the cost of not creating the correct part in the first place. While the companies they studied realised some of the costs there were a lot of hidden costs that had not been considered. The obvious and calculated ones were: Rework, Extra testing, Inspections, and Recalls. However, there were all the costs that were considered to be ‘part of doing business’ such as: Excessive Field Service Expenses, Complaint Handling, Late paperwork and Time with a Dissatisfied Customer. The average cost was considered to be in the neighbourhood of 30% of Gross Profits. Suffice to say, with Net Profit being somewhat smaller, regaining that 30% would easily make the difference between a Net profit and Loss in any given year.
Coming back to Software we have a particular case which occurred recently. (The names have been withheld to protect the guilty).

The product was for internal use to support service for an external customer. The external customer had made some changes and required updates to the provided service. The system was updated and testing commenced:

• Unit testing was done well.
• Integration Testing was a little constrained.
• System Testing was done well.
• Acceptance Testing was a little tight for time but was completed.
• Other types of testing were completed as required.

The problem came as a result of corrections and changes being made to each environment as the program moved through – the corresponding changes were not made to the previous environments. No problems with the current release. the problem arose the following year when a few further changes were required. It took about 6 months to fully reconstruct the development environment so that the work could commence. We never have time to do it right the first time but we always have time to do it over. We never realise the benefits of doing it right the first time.

To complete our trilogy we need to discuss Standards Not (just) Measurement. While Measurement is generally resisted for fear that it will be used for Personal Performance Reviews, there are organisations that Measure everything but then do nothing with the measurements they have accumulated. They have an increasing heap of measurement data being carefully recorded and stored by either some recording tools or some person who has had the task for years.

The Measurement data needs to be reviewed at least once a year to see if it is relevant to the needs of the business and whether it is even correct. There are too many cases of wasting time measuring either the wrong item, or at the wrong calibration, or at the wrong level. We may be measuring a total we think is related to our project only to discover that the count includes other projects that have nothing to do with us. We may be measuring something in metres when a measurement in millimetres is required to display any concerns. And we may be measuring individual occurrences of defect classifications and generating tables of data whereas we really only need the accumulated count and the trend to tell us how the project is progressing. The accumulated figures are worse than useless, they are misleading and decisions based upon them will be incorrect.

The second issue is that whatever Measurement we are taking must relate back to the Standards we set. if we cannot relate the Measurement to some Standard which we need to attain for business purposes then why bother taking the Measurement. So the Standards are just as important as the items used to check whether they have been attained. This means that the Standard must be set in a way that lends itself to a measurement indicating the Standard has been met. We could set a Standard that says that only so many Defects should be Critical. This is measurable provided we have a set (and Measurable) Standard for Critical and a set (and Measurable) Standard for a Defect.

It is not for nothing that Standards, Processes and Measurements all have to be reviewed.

Last week we talked about having a Process to achieve a Standard. This week we wanted to add the concept of Measurement and start with a quote from Tom DeMarco. “If you don’t measure, then you’re left with one reason to believe you are still in control: hysterical optimism.” This came from his book ‘Controlling Software Projects: Management, Measurement, and Estimation’ New York, NY,; Yourdon Press 1982.

Many people will designate a standard and design a Process to achieve that Standard but will never put in the Measurement. The Measurement sees whether the Process has achieved the Standard and whether there is further room for improvement.

There are two reasons for measuring.

Even the most stable of Processes may deviate slightly from the mean now and then (or all the time!) but if is in statistical control that should be all. It is when it starts to drift higher or lower on a continuous basis that we need to check for problems.

The second reason for measurement is to determine where and when further improvement is necessary. Just because we have achieved the initial standard does not mean that there is not room for another and higher standard or a completely different standard now that we know more about the process. Inaction is a recipe for stagnation and being bypassed by the competition.

So we need Measurements to ensure that we have achieved what we set out to do and to identify opportunities for further improvement. Measurements may take many formats and not always obvious ones. We may have objective (countable or able to be calculated) measurements or we may have subjective measurements based on how people feel about what has occurred and been achieved. Either way we need to be able to state that we have or have not successfully used our process to reach our standard.

Many consultants will provide the Standards (what needs to be done) but not the process to be followed. People have their list or toolbox of of standards that they apply in every engagement. At the end a report is provided saying what standards need to be achieved but rarely is a process road map provided that allows the organisation to achieve those goals. Standards without processes to achieve them are not of any use to an organisation already running at high speed in an attempt to keep ahead of the competition. What is even more critical is the fact that the process is never finished. Once the new level of standards are achieved (which takes time) then there needs to be a review process built in to ensure continued improvement over time (Deming Wheel or Shewhart Cycle).

Two other key considerations are to ensure that we are really fixing on-going problems and not just reacting to the inherent system variability. It is very easy to fall into the trap of simply reacting to every little change and trying to fix that. Doing that leads to more instability as we adjust the system back and forth chasing the process that will provide perfect stability while simply adding more wild swings to the result. It is very similar to oversteering when the car is sliding on ice. Minor changes in the steering result in a controlled car moving in a straight line. Major corrections usually result in the ditch!

The other major item to watch is the one time causes of issues or those external to the system that impact it but are rare and or out of our control. Trying to build solutions for these into the system adds unnecessarily to the complication and may never address the next one time error that occurs. These need to be addressed by contingency plans that can be implemented when necessary but are not otherwise used.

Make sure you are concentrating on the critical issues; make sure you have a process for getting to where you want and then see if you have attained the standard you want.

Recently a talk was given at a local association and clearly the presenter felt a need to be dramatic (at the very least). The initial statement was that Regression Testing was bad now and would only get worse. Under two current methodologies (listed below) that is the case:
1. Continually add to the regression test base and retain all that is there. The ‘more is better’ approach. Eventually the regression base grows too large (as happened to a client of ours where the regression test for the release every quarter occupied several people testing for the entire quarter and took more than a year of resource time to complete) and thus takes an increasing share of the testing budget to the exclusion of useful testing.
2. Ditch the entire base when it is felt that it is no longer providing any benefit (has not found anything new for several releases) and either start with a new base or forget about regression testing.

Either of the above approaches are expensive in either resources or risk – take your pick.

The correct way is to make sure that the regression is planned and organised to serve its purpose which is Risk Reduction commensurate with outlay.