search results matching tag: puppetmaster

» channel: learn

go advanced with your query
Search took 0.000 seconds

  • 1
    Videos (3)     Sift Talk (0)     Blogs (0)     Comments (10)   

The Worst Typo I Ever Made

StukaFox says...

The worst DevOps mistake I ever made:

Assignment: On ~1,000 -physical- RHEL systems, change the default run level from command line to GUI (don't ask).

Solution: Hey, all our config files are controlled by Puppet, so this'll be easy!

(If you don't know what Puppet does, it enforces file configurations, so if you change a single file on the Puppetmaster, that change is pushed out to all servers running Puppet)

Ok, all I need to do it edit a single file, change a single number in said file and issue a single command: reboot. Easy-fuckin'-peasy.

The file I need to change is /etc/inittab -- this file tells a Linux system which "run level" it should initiate upon booting up. runlevel 3 is command line and runlevel 5 is a GUI like Gnome or some other tragic perversion of the whole reason you run Linux in the first place. All I had to do was change from runlevel 3 to runlevel 5. And reboot.

So simple; so stupidly simple.

So stupidly simple at 3:00am. When I hadn't slept all night. On a production network. When I'm working from home away from the office. On a Saturday when no one is in said office.

I make my change and save it, then push it to the version control system. Puppet picks it up and pushes the change to ~1,000 physical computers.

Done and done!

Remember I mentioned that I had to change a single file AND execute a single command: reboot?

Here's where things go tragically wrong.

My changes worked PERFECTLY. Everything did exactly what I told it to: Puppet changed the file, and rebooted the servers.

Only they keep rebooting. They keep rebooting over and over and over and over. I can't access any server on the network. Worse, while I'm trying to figure out WTF I did wrong, the 30 minute time-out I'd set on our alerting system, Nagios, expires.

Did I mention that I pushed this change to ~1,000 servers? ~1,000 servers that won't stop rebooting and aren't reporting into Nagios, thus being marked as down?

At 3:31am, on Saturday morning, the pages to ALL the on-call engineers began. One page per engineer per machine. About one every two seconds. And I'm getting paged, too -- except some of the pages are Nagios and some are utterly irate engineers who want to know exactly WTF is going on and I can't tell which is which because I'm getting text-spammed like crazy.

And those servers? They just keep right on rebooting.

At that point, I felt the kind of existential dread that only people who work in IT know -- the kind of dread that arises a picosecond after you've hit ENTER and realized you've type 'rm -rf /' or some-such -- because I knew at that very second exactly what I'd done wrong.

I'd typo'd "5" and made it "6" in the runlevel. And pushed it to ~1,000 -physical- servers. And then rebooted them ALL.

"So," you're asking, "Whyfor is runlevel 6 a big deal?"

Because of this:

runlevel 3: command line.
runlevel 5: GUI
runlevel 6: REBOOT THE FUCKING COMPUTER.

What I'd done was told every production server on our network to reboot as soon as it rebooted, which leads to another reboot, which leads to another reboot, lather rinse repeat.

At 3:45am on Saturday morning, I knew that every person in IT would have to drive into the office, visit every production server with a bootable USB key, change the BIOS to boot off the key, boot the server into Single User Mode, change the damned file by hand, then reboot the server. This takes about 10 minutes per server -- times ~1,000.

I learned a number of valuable lessons that day:

1. DOUBLE CHECK YOUR FUCKING WORK.
2. See lesson #1
addendum: filing for unemployment insurance in Washington state is amazingly easy.

And that was the very last time I ever worked on physical hardware. To this day, if it's not in the cloud, I ain't fucking touching it.

Here endth the lesson.

War with Iran

Ron Paul "When...TRUTH Becomes Treasonous!"

VoodooV says...

Watched the Chomsky video and yeah, I think it sums it up nicely. It's not that that they don't have legitimate complaints. The problem is:

1. Their solution (answer as Chomsky put it) is not grounded in education or reality.
2. Because of their origins, their credibility is almost nil. They pretend they are grass roots but there is evidence of corporate influence to maintain the status quo. So who is really talking here? Angry People with legitimate complaints or their corporate puppetmasters? The timing of their introduction is suspect. They hate gov't spending, but they didn't seem to mind when Bush took us to TWO unnecessary wars and they didn't come on the scene until immediately after the black democrat took office. We get it that they don't like Obama, but when you start complaining and blaming him before the guy even has a chance to get settled into office, it's a little suspicious and makes you look dishonest.

Scary dinosaur prefers the escalator

Elijah Wood Shows Kids How to Do a Trippy Knee Raising Dance

Elijah Wood Shows Kids How to Do a Trippy Knee Raising Dance

Olbermann: Mr. President, you are wrong!

Xax says...

So these evil people get a pass for the benefit of the unity of the country? I don't buy it. That's a lie. Are all these guys really just an extension of the same puppetmaster?

In the beginning, God created injustice

BicycleRepairMan says...

>> ^ctrlaltbleach:
To me life would be too depressing if there was not one. To me without a god life becomes meaningless.


Why? To me, I'd much rather be in a universe where I am an evolved creature capable of pondering its own existence. In a way, we humans are the universe understanding itself. its such a beautiful concept to me. Beats being at the mercy of some invisible mysterious puppetmaster's plan of some kind

More importantly, the fact that you feel this way is no argument for it being true. The universe does not owe you meaning or purpose, How does your bad feelings about a godless universe make God more LIKELY to exist?

I really cant stand seeing children getting their limbs ripped off by bombs, that doesnt really change anything in Gaza, does it?

Librarian with "McCain=Bush" Sign Charged with Tresspassing

NetRunner says...

@choggie

I agree with you on virtually all of what you said.

I'm not a huge fan of the political game, and I'm way less of a fan of the media war that's waged for the profit of a few, the benefit of none, and likely does have some sinister puppetmasters trying to tilt things towards their desires.

I haven't seen much good come from the political process in my lifetime, though the history books never made it sound fast or easy, but it did seem like over time, committed people can make a difference. Problem is, most people I've known haven't given a fuck either way -- until now.

I don't know that it'll amount to anything, and I'm worried it won't come to anything, but I'm giving it a try, because the only other alternatives seem to be fight or flight at this point.

As for "some people" getting everything from TV, or "some people" needing to understand history without parroting it, I agree, though I can't think of anyone around here for who that's true. This is essentially an anti-TV shrine, with quite a few intellectual views of history. It's certainly not true of me, if that's what you're implying.

I'm sure watching a lot of TV these days, but it's generally accompanied with me yelling at the TV like a madman as they lie and distort virtually everything they cover. I'm watching it because what the mind-control minions of the cable news networks say ends up tilting elections.

I'm still not willing to believe all elections are shams, even if the process by which we narrow 300 million people to 2 bothers me a lot, so I try to keep up on the "narrative" of the day, and do my best to debunk it where I can.

I'm rather torn about my particular horse in this race now, as he just voted to eliminate the 4th amendment yesterday, and there's no good explanation for why, except he thinks it'll help him win, I suppose.

Things like this little old lady getting a ticket for political expression just reminds me what's at stake, and that even if Obama's been duped into ceding part of the constitution to try to stop these idiots from spinning another lie in the noise machine, he at least has to rely on a base of support that wants this shit to stop. Even if he's just another power-hungry fool, he's one who's got to make a good show of preserving our rights, or he'll be eaten alive by the entire political spectrum.

You and I shouldn't be at odds with each other, because we are both crazy about the same problem. We differ on solutions, in that you don't see the point in mine, and yours is too radical for me...for now.

We'd probably disagree on the exact form we'd want a new paradigm to take, but we can worry about that after we've won.

Dick Cheney: Creep

  • 1


Send this Article to a Friend



Separate multiple emails with a comma (,); limit 5 recipients






Your email has been sent successfully!

Manage this Video in Your Playlists

Beggar's Canyon