Disaster Recovery

Far far away, and a few thousand light beers ago, there lived a novice System Administrator named Baja Dey. Baja was in the process of being initated to the world of System Administratorgiri by Dr. Root, his guru and mentor, and an expert from the world of Kyu-nix systems. One fine autumn afternoon, a not-so-innocent bystander evesdropping upon our Kyu-nix expert and wannabe, would have observed Dr. Root educating Baja on the intricacies of Disaster Recovery.

Dr. Root: So Baja, I’ve run you through the works. Now you tell me… how do we setup a DR plan for this critical development server?

Baja: Guruji, first we’ll do a weekly tape backup of the server, followed by daily incremental backups. Then we ship the weekly tapes offsite, and the daily tapes to a remote lab onsite… blah blah blah…

Well, don’t have to bore you will all the details. You probably are aware of how a Disaster Recovery plan works.

Anyway, Dr. Root liked Baja’s plan. Even though a rookie, this boy showed promise, Dr. Root thought to himself. Satisfied that the Disaster Recovery Plan was in good hands, Dr. Root walked away and buried himself into the world of core dumps and kernels that needed a debugging.

Things were peaceful in System Administrator world until one day, when the development server finally crashed after months of abuse. This was it… this was the moment when the Rookie’s Disaster Recovery Plan would be put to test!

Dr. Root: Baja! This is your day of reckoning! Out with those tapes my son. Show them your magic.

Baja: Roger that Guruji! Should I call for the offsite tapes right away?

Dr. Root: Absolutely! And, while we are waiting for them tapes, let’s backup all the disks that are still accessible and send them offsite. Just so we have redundant backup.

Baja: Aye Aye Sir!

One hour later… A petrified Baja is seen running to Dr. Root’s desk.

Baja: Er, Guruji, we have a problem.

Dr. Root: Out with it my boy!

Baja: Er, the van carrying today’s backup offsite crashed with the van that was shipping the offsite tapes back!

Dr. Root: Oops… we seem to have got ourselves in a bit of a tangle.

Baja: er, yes sir… to be precise, two magnetic tape lengths of a tangle…

—————————————————————-

p.s. While this is not relevant to the story, a conversation that Dr. Root had with the lead developer later that day went like this:

Dr. Root: Er, we lost the backups. Can’t restore your code.

Lead Developer: Thanks!

Dr. Root: Yeah sure! Be sarcastic!

Lead Developer: No Really! I mean it!  Did you really think we wanted all those disasters we’d coded in recovered! Thank god for small mercies! Phew!

Murumookan Rajave, Mooku Pokki Nadakalle

Came into office after a nice week-long onam vacation. The good food and payasam hangover is going to last a few days… but, the hangover does not account for what happened today.

Turned up at office at around 1130 hours, grabbed a dose of caffeine, settled my tashreef on the cold seat, stretched my legs, faced the hibernating PC and brought it back to life with a tap on the power button. A few whirs and hiccups later, the agitated machine roared to life and greeted me with a cold hello in the form of the login screen. Remembering the password after a week-long binge hiatus was tough, but a few attempts later the machine relented and offered me passage into its subconscious. Here, I was supposed to be connected not just to the entity of the machine itself, but also to its mother switch attached at the other end of an RJ45 umbilical cord, and thereon to a world of millions of equally frigid bit crunchers.

And then the problem… misunderstanding the week’s absence to be a notice to refrain from paying rents in future, the DHCP landlord had terminated my machine’s lease, and erased all traces of its existence by taking away its christian name – its IP address. With no valid name, and hence no acceptable identity, the PC was immediately rejected by its peers, who would not communicate with it anymore. The machine had to be re-baptized, and alas, the customary offering of an ‘ipconfig /renew’ was not accepted by the authorities.

Thankfully, Bell’s talking machine was still around, and I picked it up and punched the few keys that led to the IT Helpdesk, where I was greeted by a phlegmatic lady, who didn’t seem very impressed with people calling up on a Monday morning to say hello. Here’s how the conversation went:

Helpdesk Lady: HELL-o

Me: Er, I’m so-n-so… have a problem with my network connection. Not getting a valid IP from DHCP.

HL: You need to raise a Helpdesk ticket (grrr… what do these nerds think? Did we put up our web-portal just for aesthetic amusement??!!)

Me: Er, can’t do that… remember, I don’t have a network connection (grrr… she should realize that she’s talking to a geek, who would’ve exhausted all other options before resorting to the phone)

HL: Oh, right! (thanks for being so rude, wise guy…). I’ll send someone over.

Now, while I’m waiting for the good Samaritan IT rescuer to show up, I take another swig at the caffeine, let out a feeble yawn and stretch my limbs once again… and then, almost instantly spot the problem. During the first of my stretching exercises, I’d managed to kick the RJ45 out of its socket, thereby cutting off my poor machine’s umbilical cord. The poor baby was now lost in an isolated world of its own, trying to figure out why the idiot sitting up front insisted that it tried to call mother switch, after sewing its lips shut. Well, anyway, quickly plugged the cable back in, and the world was a happier place again… the circuits were chirping, and my PC almost seemed to be humming a happy tune…

So I quickly called up IT again…

Me: (sheepishly)… Sorry about the earlier call. My lan cable was plugged out… hee hee… You don’t need to send anyone over now…

HL: (Grrr… ) Thanks (for nothing!)

Well, don’t be surprised if you don’t see me in office for a couple of days… I don’t plan to exhibit my face around for sometime.

Happy SAAD

Belated happy SAAD really! That’s right, System Administrator Appreciation Day just went by… and I wonder if it was just a coincidence that HJ and I were called in to look at two crashed servers just recently. Now, I haven’t been an SA in a long long time, but for those of you who are wiring networks and reconfiguring clusters around the world till date, thanks on behalf of all the clueless nerds working in your labs - thanks cos I know they won’t even recognize you are around, till something goes wrong.

In celebration, here’s a mail someone had fwded to me, on SAAD a few years back:

 ——————————clip ——————————

The System Administrator – User relationship is like that of a doctor and a patient… this is not a happy relationship, cos sadly, things often go this way:

 (At doctor’s office, phone rings, doctor answers)Dr.: Yes, how can I help you?

patient: Doc, I have a headache

Dr.: OK, take two asprins and call me in the morning

(Hour later)

Pa: Doc, I still have a headache

Dr.: OK, take a stronger tablet, and put some cold compresses on your forehead.

(hour later)

Pa: Doc, the headache wont go away!

Dr.: Strange. OK, come here to my clinic, let me have a look…

A few minutes later, the door opens and the patient walks in. He is a gruesome sight – someone had rammed a woodcutter’s axe into the top of his head, and he was bleeding profusely.

Dr.: What on earth? Why didnt you tell me what the real problem was?????

Pa: But I did, Doc! I *told* you I have a headache!

 ——————————clip ——————————

http://en.wikipedia.org/wiki/System_Administrator_Appreciation_Day

Cluster Maps

Took a cue from karram’s blog and added a Cluster Map here (check bottom – right panel). Well, one might argue how useful hit stats are on a blog that is not even on ad-sense… but must admit, this adds a nice touch to the presentation.

Want your own hit map? This should help:

http://clustrmaps.com/