Monday, February 15, 2010

System upgrades and why they should be avoided

Well folks it’s been a real Monday. One of the great joys of life when you work in IT is what is commonly referred to as a systems upgrade. It’s when you leave on Friday with an ordered system full of carefully filed documents and return to find the sort of order that occurs when one armed men juggle kittens and chainsaws in high powered wind tunnels.
In some ways I feel oddly to blame, I awoke this morning to find myself in considerable pain from an old leg injury that to the best of my knowledge I hadn’t antagonized in any way whatsoever. After careful reflection I cannot think of a single thing I’ve done to offend said leg, but hurt it does. Maybe I’ve moved from sleepwalking to sleep full contact shoot wrestling? Who can tell? But given the state of work over the last few weeks has been akin to bedlam after a tanker of cheap gin has crashed into the cell block housing the most notorious crazies, I stupidly thought to myself, “well, at least this week must be a little better.” Shame on me. I should know better. That sort of idea has a deadly tendency to come back on you. Its like wearing underpants fashioned from particularly angry ferrets. It may feel warm and fuzzy, but its going to come back to bite you on the butt!
I arrived at work and made a bee line from the bike parking lot to the designated smoking area and then to the nearest supply of tolerable coffee. After all, it is a work day and no work day should be attempted without a double shot of caffeinated optimism.
Since I’m well known on campus (cant imagine why) and had a rather pronounced limp in the time it took me to go from A to B to C I fielded no less than 8 repetitions of same question, “what happened to your leg?”
The first few people got a brief summary, the next few an even briefer one and the last a plain faced and calmly delivered “oh I transgressed tribal law and was speared in the leg by my elders as a warning to others.” I managed to utter this while ducking through the office door into our air-conditioned haven for lost souls and coffee addicts.
I’d come in early, as experience has taught me that systems upgrades can cause difficulties in proportion to the size of the upgrade and the number of users involved. This was a campus wide roll out so potentially could have the sort of productivity implications that Godzilla brings to Tokyo.
And it did. Our new passwords didn’t work, our networks were no longer synchronized and our outlook mailboxes seemed to have undertaken their own digital version of the popular US teamwork approach known as “going postal.”
My day was spent reassuring clients while rushing from one tech issue to the next. Thankfully our tech support hotline had been pre warned and gave the distinct impression that many of our problems they had seen before. Maybe they recruit them from Beirut?
But by lunch time even they were obviously reeling under the workload and a call meant sitting on hold listening to soothing music and an automated message that assured us that the first available customer representative would answer our call. It was very reassuring, I mean otherwise I might have thought I’d rung into some radio station that played only old and soppy tunes. Being on hold does wonders for my ability to rationally deal with my fellow bipedal primates. In fact so fast does it erode my general sense of calm I long ago decided on a little personal rule that I tend to use in such situations. Well not so much a rule as a formula. 20 divided by X = Y, where 20 is the number of minutes, X is the quotient of my general discontent and Y is the likelihood that I’m going to have some fun with the person on the other end of the phone when I finally reach one. In short if I’m on hold for any lengthy period of time the likelihood that I’m going be sensible dramatically decreases. Thankfully they answered before things got truly crazy. He asked if he could have my name and I said only if I could have his, in fact maybe switching persona’s could be fun. He took it well, and dutifully recoded my name in his logging software.
I think the tech had a rough idea something was up when he asked what room number I was in and I replied I was somewhere between purgatory and the residence of Cerberus. “Would that be c3?” He asked.
“Impressive” I conceded, “your system obviously works very well,” wondering if my details had come up when my name was entered, or if perhaps they had a large map of Dante’s inferno on the wall of their call center. (Don’t rule it out, when I worked in a call center we did, and it was referred to often.)
I outlined the liturgy of errors and malfunctions we were experiencing and in a truly professional manner he noted “yes we seem to be having a few teething problems.”
“We are indeed” I concurred. “The sort of teething problems saber tooth lion cubs have in the land of sugary gazelles!”
“Oh,” he says “that bad?”
“Well let me put it this my friend almost everything I’ve touched this morning has fallen over, failed miserably or burst into flames. Its so bad I’m frightened to scratch my head for fear it will fall off, and am terrified of taking a loo break in case the law of unspeakable averages transfers to that field of operations and I’m left a scorched Eunuch.”
“Ahh,” he says. And we begin the process of slowly resolving the problems one by one. Or rather some of the problems. Some require a person on campus with administrator level access. Which of course mere mortals are not allowed in case they mess with the system. And if we let just any old IT professional do that what would we need consultative bodies for?
So we toiled, we logged, we resolved and escalated to tier two. And then when we had finished, a new batch of problems had arrived so we started again.
At one point my regular calling brought me back to the same tech.
“Oh hi,” he cheerfully intoned “I think we spoke earlier” in a voice that suggested whatever he was imbibing was a damn site stronger than the coffee I was using.
“Indeed,” I said agreeably. “I’m working up on my frequent caller points in the hope of saving up enough to get some padded wallpaper or one of those neat white huggy shirts with the brass restraining straps.”
“really?” he asked (now I’m sure he’s gone way past coffee)
“No Michael” I reply tolerantly “it s just your voice is so sexy I think I’m enamored with you.” Pregnant pause while the poor chap considered this.
“Do you realize these calls are recorded for training purposes…”
“I do,” I replied, “but I don’t have time to discuss the ethics of offshoring your work onto harmless aquatic sea mammals, I’m dealing with the biggest plague of bugs since the pharaoh told Moses to go stick it in the desert.”
“Yep,” he concludes “your defiantly the bloke I was talking to earlier….”

You see this is what an upgrade is, its where you remove old bugs while introducing a whole load of new ones. And like any invasive introduced species let loose into a pristine environment they multiply and wreak havoc. Though of course if you ask the vendors they will inevitably tell you “its not a bug, it’s a feature!”
So we logged our problems, assigned them numbers and cast them into the abyss of the technical support customer service software. (Note, service in this sense is apparently used the same way it is by vets when they use IVF techniques to “service” cattle.)
So I logged the jobs and realizing it was well past lunch time and the point of return to hope, I grabbed my filthy lung defilers and headed to the leper zone. There were a few other Tech chaps there. I decided you can tell who really cares at times like this. They are the ones steadfastly trying to kill themselves by any legal means. I smile at one who has the dull vacant lifeless eyes one associates with prisoners of war. It’s a mistake, he obviously thinks I might know something he doesn’t and like a shipwrecked man trying to clamber onto a tiny raft asks, “how’s the systems upgrade going”
“Oh its challenging” I say. We both know full well that this is tech speak for totally balls’ed up
“What’s your plan?” He asks
We’ll I was going to hide under my desk and hope no one noticed, or sacrifice a goat to the elder gods in the hope they would intervene, but the first was undignified (and my leg hurts) and the second would probably contravene a HR policy. So I’m going for the traditional one. Head for the nearest source of water armed with mankind’s oldest brewed beverage and not come out till the bottles empty and I’m full.”
He nodded thoughtfully “I think that’s the best plan I’ve heard all day”
So at least I can say my day wasn’t totally wasted. I brought some order to chaos, hope to the bereft and beer to the proximity of a large pool.
But the next time they want to upgrade my systems, I think I’ll plan some recreational leave. My sanity is fragile, and like windows, I fear each new upgrade will bring it crashing down.