Since at least April, IBM has been looking at the issues surrounding the shocking TSB software rollout that cause so much banking pain for customers.
Big Blue has discovered that the main issues seem to stem from a lack of testing and control surrounding software changes. All in all, it looks like the War Room’s would find a problem which was then changed without checking with other War rooms. This sort of practice tends to compound errors due to a lack of understanding of the impact of change.
Together with the overconfident rollout on mass rather than several stages of testing and roll back to rectify issues. It seems shocking to this writer that any software organization with so many customers and especially in the finance sector would not have a plan to roll back if there was a failure, never mind the simple-mindedness of TSB actually believing that they had tested everything prior to putting the system “live” for the world to see (crashed and burned).
The IBM presentation document makes interesting reading with several hints for software houses on IBM’s own practices. All of the stages they mention and highlight as TSB failures along with the recommendations should help anyone write a comprehensive playbook for testing and rolling out new software.
This ex-software test engineer is pleased to see IBM pointing out so many errors that can be learned from by others. I seem to recall the MD of a company I used to work for stating “We don’t need a test department, that’s what customers are for”. As shocking as that is to anyone in the sector it seems that TSB’s test team have forgotten some simple lessons and should go back to the drawing board.
At some point you have to realize that clear communication between the Software Team, the Test Team and Management are essential, in the case of TSB I would have to guess at a communications breakdown.
For TSB customers, the nightmare has been very real, for TSB it has led to more bad banking press and that has to generate trust issues. Let us remember that publishing faulty code that causes systems to grind to a halt is never great, but think of the potential for vulnerabilities in the system and suddenly I’m glad I don’t bank with TSB!
There is an interesting article on The Register with a link to the IBM PDF version of the slide show
About the Author: Stephen J. Richards has 25 years experience in Data Management and Information Technology. This information is provided as a public service by Neon Enterprise Software, a leading provider of mainframe disaster recovery and data retention technology.