Friday, June 29, 2012

There but for the grace of the angels go I.

So if you have a RBS, NatWest or Ulster Bank account you probably don't need telling that something went very, very badly titsup last week. Stories of people being stuck in jail as their bail could not be paid, people temporarily homeless as solicitors couldn't hand over keys on house transactions and so on and so forth.

It appears the cause was a screw up in operating the software which schedules the processing of the various software programs that makes up the bank's "books" as it were. When I first started in IT I was a junior programmer writing those kinds of programs on a Burroughs mainframe for what was then The Midland Bank. The overnight suite must have been over 150 discrete programs all of which had dependencies on others and the RBS suite is almost certainly no different and probably much more complex and relies on a lump of code called "CA7" to ensure everything runs when it should. Now it seems some change was made within CA7 that didn't work and needed to be "backed out"; this sort of thing happens all the time in software releases and before releasing any change to a production environment you always, without fail, have a backout plan. Regrettably for RBS a junior numpty, hired on the cheap and operating out of Hyderabad  (RBS having saved money by sacking all the UK based very experienced staff) rather than removing the change removed everything, completely wiping the schedule database.

Oops.

This fiasco will probably end up costing RBS, in other words me and you the taxpayers, upwards of 500 million quid when everything is taken into account. How's that cost saving working out for you RBS?

I do feel sorry for the junior numpty though; that must have been the longest stomach-churning "Oh fuck. What have I just done" moment in history. And I defy anyone in IT to say they've never had one.

Mine was back in those junior coder days, wrangling COBOL on the overnight batch systems. Late on friday there was some urgent patch needed to a part of code that handled automated cheque book issuing (back when people used cheques our code worked out when you were running low and ordered a new book to be printed and sent). I made the change but didn't really have time to test it but it was a two line change and I told my supervisor this.

"Are you sure it's OK? You sure you put that full stop in the right place?" she said whilst putting her coat on. Clearly she didn't intend to check it.

"Yes, definitely" said a supremely confident dragon and uploaded the code and clocked off.

I woke on Saturday morning with the gnawing feeling something was wrong. I could see the code in my head. I could see that full stop. It was in the WRONG FUCKING PLACE!

I'd terminated an "If... Then... Else" structure too early.

I have ordered every single Midland Bank current account customer a new sodding cheque book at a cost to the bank of tens of thousands of pounds.

After regaining control of my stomach I started to contemplate what my new career would be because it certainly wasn't going to be computer programming. Maybe herding yak in Tibet would be more my line.

Monday came and I went in to face the music. All the team looked at me as I walked in. I waited for the hammer to fall.

It never did. In inadvertently screwing up that if/else structure I'd caused some other part of the subroutine to fail and so no books got ordered at all. The code could be patched (and I did the patch) so the missing sends would get picked up on the next run. I'd dodged a bullet and, 25 years later, I'm still wrangling code and not yak.

Don't think that chap in Hyderabad is going to get that chance.



1 comment:

Anonymous said...

Great post! I recently re-discovered your blog after having lost it for about 7 seven years. I remember emailing you when I was looking my first job and you send back a kind and thoughtful reply. I'm sad to see you're not blogging anymore!