The worst DevOps mistake I ever made:

Assignment: On ~1,000 -physical- RHEL systems, change the default run level from command line to GUI (don't ask).

Solution: Hey, all our config files are controlled by Puppet, so this'll be easy!

(If you don't know what Puppet does, it enforces file configurations, so if you change a single file on the Puppetmaster, that change is pushed out to all servers running Puppet)

Ok, all I need to do it edit a single file, change a single number in said file and issue a single command: reboot. Easy-fuckin'-peasy.

The file I need to change is /etc/inittab -- this file tells a Linux system which "run level" it should initiate upon booting up. runlevel 3 is command line and runlevel 5 is a GUI like Gnome or some other tragic perversion of the whole reason you run Linux in the first place. All I had to do was change from runlevel 3 to runlevel 5. And reboot.

So simple; so stupidly simple.

So stupidly simple at 3:00am. When I hadn't slept all night. On a production network. When I'm working from home away from the office. On a Saturday when no one is in said office.

I make my change and save it, then push it to the version control system. Puppet picks it up and pushes the change to ~1,000 physical computers.

Done and done!

Remember I mentioned that I had to change a single file AND execute a single command: reboot?

Here's where things go tragically wrong.

My changes worked PERFECTLY. Everything did exactly what I told it to: Puppet changed the file, and rebooted the servers.

Only they keep rebooting. They keep rebooting over and over and over and over. I can't access any server on the network. Worse, while I'm trying to figure out WTF I did wrong, the 30 minute time-out I'd set on our alerting system, Nagios, expires.

Did I mention that I pushed this change to ~1,000 servers? ~1,000 servers that won't stop rebooting and aren't reporting into Nagios, thus being marked as down?

At 3:31am, on Saturday morning, the pages to ALL the on-call engineers began. One page per engineer per machine. About one every two seconds. And I'm getting paged, too -- except some of the pages are Nagios and some are utterly irate engineers who want to know exactly WTF is going on and I can't tell which is which because I'm getting text-spammed like crazy.

And those servers? They just keep right on rebooting.

At that point, I felt the kind of existential dread that only people who work in IT know -- the kind of dread that arises a picosecond after you've hit ENTER and realized you've type 'rm -rf /' or some-such -- because I knew at that very second exactly what I'd done wrong.

I'd typo'd "5" and made it "6" in the runlevel. And pushed it to ~1,000 -physical- servers. And then rebooted them ALL.

"So," you're asking, "Whyfor is runlevel 6 a big deal?"

Because of this:

runlevel 3: command line.
runlevel 5: GUI
runlevel 6: REBOOT THE FUCKING COMPUTER.

What I'd done was told every production server on our network to reboot as soon as it rebooted, which leads to another reboot, which leads to another reboot, lather rinse repeat.

At 3:45am on Saturday morning, I knew that every person in IT would have to drive into the office, visit every production server with a bootable USB key, change the BIOS to boot off the key, boot the server into Single User Mode, change the damned file by hand, then reboot the server. This takes about 10 minutes per server -- times ~1,000.

I learned a number of valuable lessons that day:

1. DOUBLE CHECK YOUR FUCKING WORK.
2. See lesson #1
addendum: filing for unemployment insurance in Washington state is amazingly easy.

And that was the very last time I ever worked on physical hardware. To this day, if it's not in the cloud, I ain't fucking touching it.

Here endth the lesson.

An old EQ tale:

I was invisible and running through the Karanas one day when I noticed a young gnome near the gypsy camp. He was fighting a lion and though it looked like he would win the battle, being a fellow gnome, I decided to help the guy out.

I targeted the lion, clicked on my mesmerize spell, then started to type: "I'm mesmerizing the lion for you." I got as far as: "I'm " when I remembered that I had replaced my mesmerize spell with an Area of Effect mesmerize spell... and that I was standing next to an NPC enchantress. Gulp.

My movement keys are mapped to "w a s d" so I frantically stabbed at my keyboard, trying to MOVE and interrupt the spell.

I forgot that I was in typing mode.

The gypsy enchantress didn't like my attempt to mezz her so she promptly charmed me [IE Mind control] and made me go after the gnome I had been trying to save. I watched in horror as my peace-loving character, knife flailing like a crazed sushi chef, chased the little guy down and stabbed him to death.

I found my victim later and apologized profusely... I even gave him a nice weapon and a piece of armor. He was great about it, and laughed when I told him what happened.

He said he didn't know WHAT was going on. One minute he was fighting a lion, the next minute a strange gnome appeared out of NOWHERE, announced: "I'm wwwaaaddd", then sliced him up like Freddy Krueger.

What a great discussion, but those fucking dogs. I hate small dogs. Not all of those small dogs are bad breeds, but collectively they're like a gang of snappy, squeaky, barking leper gnomes with sharp teeth. AHHHHHHHHHHHHHHHH!!! Interviewer is like JESUS.

Anyway. lol -- concise theory on the current state of the presidency -- and the American class war.

Oh, and the gnome was always on the chimney about half way up.

And at that moment, he realized he had outed himself as a magical gnome masquerading among the humans.

I can't fathom the number of animals I've murdered in video games...

I believe I've killed at least 1000 boars in one sitting. We're digital mass murderers! But that's ok, I have also been digitally sentenced, imprisoned, digitally raped and murdered, and digitally risen from the grave to have my vengeance on the Tolkien universe --

The numbers of elves and dwarves and humans and gnomes that have met their untimely death at my undead hands, wrapped in leather, cloaked in shadows -- another very large number of killings.

You may refer to me as a High Warlord of Blackrock Mountain. Rawr!!!!!!!!!!!!!!! Death to ALL life! May the black dragonflight rise again! May they roast endless numbers of sheep.

When Morgoth in that day of doom
had slain the trees and filled with gloom
the shining land of Valinor,
there Fëanor and his sons then swore
the mighty oath upon the hill
of tower-crownéd Tún, that still
wrought wars and sorrow in the world.
From darkling seas the fogs unfurled
their blinding shadows grey and cold
where Glingal once had bloomed with gold
and Belthil bore its silver flowers.
The mists were mantled round the towers
of the Elves' white city by the sea.
There countless torches fitfully
did start and twinkle, as the Gnomes
were gathered to their fading homes,
and thronged the long and winding stair
that led to the wide echoing square.

There Fëanor mourned his jewels divine
the Silmarils he made. Like wine
his wild and potent words them fill;
a great host harkens deathly still.
But all he said both wild and wise,
half truth and half the fruit of lies
that Morgoth sowed in Valinor,
in other songs and other lore
recorded is. He bade them flee
from lands divine, to cross the sea,
the pathless plains, the perilous shores
where ice-infested water roars;
to follow Morgoth to the unlit earth
leaving their dwellings and olden mirth;
to go back to the Outer Lands
to wars and weeping. There their hands
they joined in vows, those kinsmen seven,
swearing beneath the stars of Heaven,
by Varda the Holy that them wrought
and bore them each with radiance fraught
and set them in the deeps to flame.
Timbrenting's holy height they name,
whereon are built the timeless halls
of Manwë Lord of Gods. Who calls
these names in witness may not break
his oath, though earth and heaven shake.

Curufin, and Celegorm the fair,
Damrod and Díriel were there,
and Cranthir dark, and Maidros tall
(whom after torment should befall),
and Maglor the mighty who like the sea
with deep voice sings yet mournfully.
'Be he friend or foe, or seed defiled
of Morgoth Bauglir, or mortal child
that in after days on earth shall dwell,
no law, nor love, nor league of hell,
nor might of Gods, nor moveless fate
shall defend him from wrath and hate
of Fëanor's sons, who takes and steals
or finding keeps the Silmarils,
the thrice-enchanted globes of light
that shine until the final night.'

QM, Mossadegh's effort to nationalize his nation's oil was not a threat to the United States. For one, they were not doing it to make a political point against any government; so, the Iranians had no interest in withholding oil from any customers. What they wanted were the revenues and profits from the oil coming out of their own land; which would have hurt British interests, or more specifically, the precursor to BP. The US got involved because of cold war ideology. Also, our governments were not in conflict at the time; Mossadegh even visited the states at least once.

Your proposition that they would have eventually overthrown Mossadegh is faulty on a number of points. First of all, Mossadegh was an elderly man who could not have lasted a couple of decades. Secondly, he was a prime minister in a democracy with many thriving factions. Like in all democracies, his administration would have eventually been voted out. Also, the Islamic fundies in Iran developed as a direct result of the Shah's brutality. The Persians had one of the richest, intellectual and tolerant cultures in the region, beforehand, IMO. If their democracy had had a chance to mature and thrive, they could have been a major positive force in the region. Why would people in possession of wealth and democracy overthrow their own government?

It's a very interesting story, documented in the book All the Shah's Men, if you are interested in learning more.

>> ^quantumushroom:

@ChaosEngine Thank you for your more civil tone of late. Am I surprised that someone reached for the mouse to downvote MY comment while ignoring THIS?

Stukafox: There's only two kinds of Republicans: Corporate tools and complete psychopaths.

The Blame America First mindset is very real. It's taught in our public schools government indoctrination centers from K thru kollij. "Anarchist" Gnome Chompsky has made millions off this bizarre worldview, which glibly ignores the 100 million murdered by communist regimes and the defeat of fascism (and rebuilding of Europe) by the United States.
As for the charge of US meddling in Iran, the reality is we have interests around the world, things we want to buy and nations that want to sell. Glancing at wikipedia: when the elected government nationalized the Iranian oil industry, that was a threat to both Britain and the US.
Yeah, the Shah was an a-hole, but he was replaced with an even bigger a-hole, an islamic fundie. So instead of utopian perfection, we had an evil replaced with a greater evil. (And who's to say had Prime Minister Mosaddegh kept power through the 1970s, he wouldn't have been overthrown by Khomeini anyway)?
There is not any one era when international relations was superior and reasonable, just brief burps where there was an odd peace.
If you want to celebrate red china for "putting America in its place" like our idiot excuse of a president does, you better damned well understand what you're favoring: a ruthless communist regime that kills people as easily as you throw away coat hangers.

@ChaosEngine Thank you for your more civil tone of late. Am I surprised that someone reached for the mouse to downvote MY comment while ignoring THIS?

Stukafox: There's only two kinds of Republicans: Corporate tools and complete psychopaths.

The Blame America First mindset is very real. It's taught in our public schools government indoctrination centers from K thru kollij. "Anarchist" Gnome Chompsky has made millions off this bizarre worldview, which glibly ignores the 100 million murdered by communist regimes and the defeat of fascism (and rebuilding of Europe) by the United States.

As for the charge of US meddling in Iran, the reality is we have interests around the world, things we want to buy and nations that want to sell. Glancing at wikipedia: when the elected government nationalized the Iranian oil industry, that was a threat to both Britain and the US.

Yeah, the Shah was an a-hole, but he was replaced with an even bigger a-hole, an islamic fundie. So instead of utopian perfection, we had an evil replaced with a greater evil. (And who's to say had Prime Minister Mosaddegh kept power through the 1970s, he wouldn't have been overthrown by Khomeini anyway)?

There is not any one era when international relations was superior and reasonable, just brief burps where there was an odd peace.

If you want to celebrate red china for "putting America in its place" like our idiot excuse of a president does, you better damned well understand what you're favoring: a ruthless communist regime that kills people as easily as you throw away coat hangers.

@Grimm - At the risk of stating the obvious, don't you think it's more logical to believe that Reagan's loyalty to big money (and ALL of his predecessors) might have played a dominant role in the degradation of American public education, rather than the fact that a Department of Education exists? Boise laid out a number of deliberate poison pills in his comment. You've merely asserted your claim without any rational other than an arbitrary number of trips around the sun.

Let's say you buy a new car, and I tell you I hate it and intend to pop the tires, break the windows and light it on fire after you go to sleep. If the next morning you wake up to find your car on fire, with popped tires and broken windows, would you take it back to the dealer and claim the car was faulty? This, in essence, is what you are doing here.

If I were you, the logical counter argument would be, "well there you go, you've made my case, a malicious or subservient (take your pick) president was able to have a hugely negative effect on education nationally. Had it been left to the states, our educational system would be a utopic wonderland."

To which I would respond, "If big money can compromise a huge government, what makes you think they couldn't eat a state house for champagne brunch?"

The problem with libertarians is that they are unwitting allies of the corporate state. They believe that getting rid of government would end authoritarianism, completely failing to understand that the kind of authoritarianism that haunts our country would prefer to be unrestrained by government too. Right libertarianism, if enacted, would indeed provide more liberty to a handful of wealthy and powerful people, but it would come at the cost of liberty to the vast majority. 1% vs 99% if you will. Sound familiar? I see no clear difference between libertarianism and social Darwinism. If you respond to any of this, I'd most like to know how you differentiate libertarianism from social Darwinism.

I think a vast amount of people would prefer the liberty of healthcare, education, roads, fire departments, police departments, schools and libraries to the liberty to dominate a labor force, the liberty to pollute the environment with impunity, the liberty to manipulate the banking system or the liberty to build bloody corporate empires on foreign shores. What makes you think the business men that took us to war in the middle east wouldn't be twice as brutal without a single shred of oversight or transparency? What makes you think deregulated labor markets wouldn't revert back to pre-regulation era slavery if given the option?

If social Darwinism is what you truly desire, then we have nothing more to say to each other. However, if you want to stop authoritarianism, then stop trying to make it easier for authoritarians to thrive. Ron Paul is a nice fella and all - an adorable little grandfatherly gnome even - and I take him at his word when he says he believes his economic hypothesis would create liberty. Unfortunately, reality begs to differ. And, sincerity is no excuse for bad ideas.

Good debate. Peace.

You can join the convo too if you like @GeeSussFreeK

Dude kinda sounds like the Travelocity Gnome...

You have been awarded 1 Power Point for fixing the embed code for Dead Pool video Pink Floyd -- The Gnome. Thank you for helping maintain VideoSift's reliability.

Comment hidden because you are ignoring dag. (show it anyway)

I agree with most of your points. I would like to make a small defense of the inability to change things in OS X. With mutability can come a lot of overhead and chaos. There is something to be said for an iron hand on the tiller of user interfaces - but only if you trust the group making decisions.

I am not a UX expert. Up until Lion I trusted the UX people at Apple to have a better idea about how humans can optimally interact with a computer. For the most part, I think they were right. Up until Lion - now I think I'm starting to be sold a crock. The decisions they have made don't seem to be based on making efficient interactions happen - but instead about some grand unified melding of Macs and iOS devices. It's bullshit.

The mandatory click to focus thing is really a taste thing. For me, personally it drives me batty. I don't want focus until I've clicked.

Bouncy in your face icons - agreed, annoying - but not as bad modal windows you have to dismiss.


>> ^srd:

>> ^dag:
Up until Lion I would completely disagree with you and say the UX of OS X is simply the best. Yes, I'm talking against Windows 7, Gnome, KDE et al. Now however, I'm starting to cast a wandering eye back towards Linux.
Windows 7 however, is a frigging awful experience any way you slice it. It's stupid little things like the alt-tab selecting whatever window is in the background when really you just want to cycle through the icons. Also, I can't believe they still haven't killed the dysfunctional bloatware ridden system tray. The retarded nanny-ware labyrinth that has to be navigated to connect to a wireless network makes my eyes bleed.
The way I'm feeling now is that all operating systems suck hard, but OS X sucks a little less, at least until Lion - which, again, is starting to suck much harder for all the reasons outlined in this video - and more.


Gnome, KDE, Windows et al have been scampering after the OSX UX for some years now, and I agreee have been doing it rather badly. And this is a trend I'm very skeptical of. However, if you like the workflow that OSX/Quarz imposes, I'm sure you can be happy with it. Where I take exception is having no choice except for what some people in a meeting in Cupertino decide is how I should do my work.
Things that really put me off:
- Menu bar at the top of the screen instead of attached to the individual application... Sure, thats traditional on apple computers and that made sense back in the days when the Mac didn't have real multitasking. But nowadays it's just terribly confusing and imposes longer mouse travel distances.
- Mandatory click-to-focus, which can be seen as a neccessary corrolary of the previous point. I've been using the focus-follows-mouse model (without raise-on-focus) for 15 years now and the difference is jarring. Imagine having to click away an overlay on each and every page you go to in your browser.
- Bouncy in-your-face animations and notification boxes that are reminiscent of Paperclip. Shut up already and get out of my face, I'm trying to work, not playing a game of whack-an-icon.
- Apple marketing OSX as 64 bit but delivering it in 32 bit mode and not telling you until you a) find out by accident and then b) spend 10 minutes gooling around until you find the command to switch it to 64bit default mode (no GUI level preference here for whatever reason).
I'd be a lot happier if I had a choice. Either by having real preferences that goes beyond what color scheme do I want and in what way do I want to stroke my touchpad to do what. Or open up the possibility for alternative window managers.
For all the "think different" attitude that Apple likes to spread, the OSX ecosystem seems to be hard at work to remove individual preferences. Apple turned into the opposite of what the 1984 commercial implied.
Dag, if you're looking at linux again, both KDE and Gnome (especially Gnome 3) are IMO horrible too. If you don't like them, give XFCE a go. I've been using it since '03 IIRC, when I grew tired of Blackbox. And you'd be in good company too

>> ^dag:

Up until Lion I would completely disagree with you and say the UX of OS X is simply the best. Yes, I'm talking against Windows 7, Gnome, KDE et al. Now however, I'm starting to cast a wandering eye back towards Linux.
Windows 7 however, is a frigging awful experience any way you slice it. It's stupid little things like the alt-tab selecting whatever window is in the background when really you just want to cycle through the icons. Also, I can't believe they still haven't killed the dysfunctional bloatware ridden system tray. The retarded nanny-ware labyrinth that has to be navigated to connect to a wireless network makes my eyes bleed.
The way I'm feeling now is that all operating systems suck hard, but OS X sucks a little less, at least until Lion - which, again, is starting to suck much harder for all the reasons outlined in this video - and more.



Gnome, KDE, Windows et al have been scampering after the OSX UX for some years now, and I agreee have been doing it rather badly. And this is a trend I'm very skeptical of. However, if you like the workflow that OSX/Quarz imposes, I'm sure you can be happy with it. Where I take exception is having no choice except for what some people in a meeting in Cupertino decide is how I should do my work.

Things that really put me off:

- Menu bar at the top of the screen instead of attached to the individual application... Sure, thats traditional on apple computers and that made sense back in the days when the Mac didn't have real multitasking. But nowadays it's just terribly confusing and imposes longer mouse travel distances.

- Mandatory click-to-focus, which can be seen as a neccessary corrolary of the previous point. I've been using the focus-follows-mouse model (without raise-on-focus) for 15 years now and the difference is jarring. Imagine having to click away an overlay on each and every page you go to in your browser.

- Bouncy in-your-face animations and notification boxes that are reminiscent of Paperclip. Shut up already and get out of my face, I'm trying to work, not playing a game of whack-an-icon.

- Apple marketing OSX as 64 bit but delivering it in 32 bit mode and not telling you until you a) find out by accident and then b) spend 10 minutes gooling around until you find the command to switch it to 64bit default mode (no GUI level preference here for whatever reason).

I'd be a lot happier if I had a choice. Either by having real preferences that goes beyond what color scheme do I want and in what way do I want to stroke my touchpad to do what. Or open up the possibility for alternative window managers.

For all the "think different" attitude that Apple likes to spread, the OSX ecosystem seems to be hard at work to remove individual preferences. Apple turned into the opposite of what the 1984 commercial implied.

Dag, if you're looking at linux again, both KDE and Gnome (especially Gnome 3) are IMO horrible too. If you don't like them, give XFCE a go. I've been using it since '03 IIRC, when I grew tired of Blackbox. And you'd be in good company too