Category: Uncategorized

Just this morning, the common lament appeared on a LinkedIn post: We always have time to do it over but never time to do it right in the first place. This is by no means new, there are people who have been saying this or similar comments for years and they long predate software.
If you read some of the original Quality Pioneers (who were looking at manufacturing and not at software) they had tried to calculate the cost of not creating the correct part in the first place. While the companies they studied realised some of the costs there were a lot of hidden costs that had not been considered. The obvious and calculated ones were: Rework, Extra testing, Inspections, and Recalls. However, there were all the costs that were considered to be ‘part of doing business’ such as: Excessive Field Service Expenses, Complaint Handling, Late paperwork and Time with a Dissatisfied Customer. The average cost was considered to be in the neighbourhood of 30% of Gross Profits. Suffice to say, with Net Profit being somewhat smaller, regaining that 30% would easily make the difference between a Net profit and Loss in any given year.
Coming back to Software we have a particular case which occurred recently. (The names have been withheld to protect the guilty).

The product was for internal use to support service for an external customer. The external customer had made some changes and required updates to the provided service. The system was updated and testing commenced:

• Unit testing was done well.
• Integration Testing was a little constrained.
• System Testing was done well.
• Acceptance Testing was a little tight for time but was completed.
• Other types of testing were completed as required.

The problem came as a result of corrections and changes being made to each environment as the program moved through – the corresponding changes were not made to the previous environments. No problems with the current release. the problem arose the following year when a few further changes were required. It took about 6 months to fully reconstruct the development environment so that the work could commence. We never have time to do it right the first time but we always have time to do it over. We never realise the benefits of doing it right the first time.

To complete our trilogy we need to discuss Standards Not (just) Measurement. While Measurement is generally resisted for fear that it will be used for Personal Performance Reviews, there are organisations that Measure everything but then do nothing with the measurements they have accumulated. They have an increasing heap of measurement data being carefully recorded and stored by either some recording tools or some person who has had the task for years.

The Measurement data needs to be reviewed at least once a year to see if it is relevant to the needs of the business and whether it is even correct. There are too many cases of wasting time measuring either the wrong item, or at the wrong calibration, or at the wrong level. We may be measuring a total we think is related to our project only to discover that the count includes other projects that have nothing to do with us. We may be measuring something in metres when a measurement in millimetres is required to display any concerns. And we may be measuring individual occurrences of defect classifications and generating tables of data whereas we really only need the accumulated count and the trend to tell us how the project is progressing. The accumulated figures are worse than useless, they are misleading and decisions based upon them will be incorrect.

The second issue is that whatever Measurement we are taking must relate back to the Standards we set. if we cannot relate the Measurement to some Standard which we need to attain for business purposes then why bother taking the Measurement. So the Standards are just as important as the items used to check whether they have been attained. This means that the Standard must be set in a way that lends itself to a measurement indicating the Standard has been met. We could set a Standard that says that only so many Defects should be Critical. This is measurable provided we have a set (and Measurable) Standard for Critical and a set (and Measurable) Standard for a Defect.

It is not for nothing that Standards, Processes and Measurements all have to be reviewed.

Last week we talked about having a Process to achieve a Standard. This week we wanted to add the concept of Measurement and start with a quote from Tom DeMarco. “If you don’t measure, then you’re left with one reason to believe you are still in control: hysterical optimism.” This came from his book ‘Controlling Software Projects: Management, Measurement, and Estimation’ New York, NY,; Yourdon Press 1982.

Many people will designate a standard and design a Process to achieve that Standard but will never put in the Measurement. The Measurement sees whether the Process has achieved the Standard and whether there is further room for improvement.

There are two reasons for measuring.

Even the most stable of Processes may deviate slightly from the mean now and then (or all the time!) but if is in statistical control that should be all. It is when it starts to drift higher or lower on a continuous basis that we need to check for problems.

The second reason for measurement is to determine where and when further improvement is necessary. Just because we have achieved the initial standard does not mean that there is not room for another and higher standard or a completely different standard now that we know more about the process. Inaction is a recipe for stagnation and being bypassed by the competition.

So we need Measurements to ensure that we have achieved what we set out to do and to identify opportunities for further improvement. Measurements may take many formats and not always obvious ones. We may have objective (countable or able to be calculated) measurements or we may have subjective measurements based on how people feel about what has occurred and been achieved. Either way we need to be able to state that we have or have not successfully used our process to reach our standard.

Many consultants will provide the Standards (what needs to be done) but not the process to be followed. People have their list or toolbox of of standards that they apply in every engagement. At the end a report is provided saying what standards need to be achieved but rarely is a process road map provided that allows the organisation to achieve those goals. Standards without processes to achieve them are not of any use to an organisation already running at high speed in an attempt to keep ahead of the competition. What is even more critical is the fact that the process is never finished. Once the new level of standards are achieved (which takes time) then there needs to be a review process built in to ensure continued improvement over time (Deming Wheel or Shewhart Cycle).

Two other key considerations are to ensure that we are really fixing on-going problems and not just reacting to the inherent system variability. It is very easy to fall into the trap of simply reacting to every little change and trying to fix that. Doing that leads to more instability as we adjust the system back and forth chasing the process that will provide perfect stability while simply adding more wild swings to the result. It is very similar to oversteering when the car is sliding on ice. Minor changes in the steering result in a controlled car moving in a straight line. Major corrections usually result in the ditch!

The other major item to watch is the one time causes of issues or those external to the system that impact it but are rare and or out of our control. Trying to build solutions for these into the system adds unnecessarily to the complication and may never address the next one time error that occurs. These need to be addressed by contingency plans that can be implemented when necessary but are not otherwise used.

Make sure you are concentrating on the critical issues; make sure you have a process for getting to where you want and then see if you have attained the standard you want.

Recently a talk was given at a local association and clearly the presenter felt a need to be dramatic (at the very least). The initial statement was that Regression Testing was bad now and would only get worse. Under two current methodologies (listed below) that is the case:
1. Continually add to the regression test base and retain all that is there. The ‘more is better’ approach. Eventually the regression base grows too large (as happened to a client of ours where the regression test for the release every quarter occupied several people testing for the entire quarter and took more than a year of resource time to complete) and thus takes an increasing share of the testing budget to the exclusion of useful testing.
2. Ditch the entire base when it is felt that it is no longer providing any benefit (has not found anything new for several releases) and either start with a new base or forget about regression testing.

Either of the above approaches are expensive in either resources or risk – take your pick.

The correct way is to make sure that the regression is planned and organised to serve its purpose which is Risk Reduction commensurate with outlay.

Recently we found ourselves unexpectedly with a group of people who had been told what they needed without being asked what the fundamental problem was or how it could be solved. No needs assessment had occurred. Instead a quick solution had been sold to them and we had been asked to deliver it on behalf of someone else. A brief conversation at the beginning of the meeting elicited that the requirements were completely different and needed to be delivered in a different way. It pays to know the needs first.

One of the concerns with Process Improvement is the tendency to throw out all the old processes and start anew as if nothing had ever existed. This is an understandable reaction if the existing processes have never been organised, recorded, and classified. The problem is compounded if they are scattered throughout the organisation, exist only in unwritten format, and vary from department to department. However, the effort to create net new processes far exceeds that of recording the old ones. It is worthwhile to inventory the existing processes first before emabarking on wholesale creation.

“Bouncebacks’ is not a technical term in common use. In our usage here we are counting the number of times a defect (or issue) bounces back and forth between tester and developer (or other personnel). It is common for a defect to have a count of at least 2 and maybe 4 if some clarification is required. Here we are looking for defects or issues that are bouncing back and forth between development and testers too many times. One of our clients recently did this and then concentrated on the items that had high counts. Not surprisingly they discovered that the majority of the issues with high bounceback counts revolved around poorly written requirements. Instead of getting the requirement correct in the first place, the concept was being clarified by continuous back and forth communication late in the project with a high overhead and testing cost. Call us for more information.