The worst DevOps mistake I ever made:

Assignment: On ~1,000 -physical- RHEL systems, change the default run level from command line to GUI (don't ask).

Solution: Hey, all our config files are controlled by Puppet, so this'll be easy!

(If you don't know what Puppet does, it enforces file configurations, so if you change a single file on the Puppetmaster, that change is pushed out to all servers running Puppet)

Ok, all I need to do it edit a single file, change a single number in said file and issue a single command: reboot. Easy-fuckin'-peasy.

The file I need to change is /etc/inittab -- this file tells a Linux system which "run level" it should initiate upon booting up. runlevel 3 is command line and runlevel 5 is a GUI like Gnome or some other tragic perversion of the whole reason you run Linux in the first place. All I had to do was change from runlevel 3 to runlevel 5. And reboot.

So simple; so stupidly simple.

So stupidly simple at 3:00am. When I hadn't slept all night. On a production network. When I'm working from home away from the office. On a Saturday when no one is in said office.

I make my change and save it, then push it to the version control system. Puppet picks it up and pushes the change to ~1,000 physical computers.

Done and done!

Remember I mentioned that I had to change a single file AND execute a single command: reboot?

Here's where things go tragically wrong.

My changes worked PERFECTLY. Everything did exactly what I told it to: Puppet changed the file, and rebooted the servers.

Only they keep rebooting. They keep rebooting over and over and over and over. I can't access any server on the network. Worse, while I'm trying to figure out WTF I did wrong, the 30 minute time-out I'd set on our alerting system, Nagios, expires.

Did I mention that I pushed this change to ~1,000 servers? ~1,000 servers that won't stop rebooting and aren't reporting into Nagios, thus being marked as down?

At 3:31am, on Saturday morning, the pages to ALL the on-call engineers began. One page per engineer per machine. About one every two seconds. And I'm getting paged, too -- except some of the pages are Nagios and some are utterly irate engineers who want to know exactly WTF is going on and I can't tell which is which because I'm getting text-spammed like crazy.

And those servers? They just keep right on rebooting.

At that point, I felt the kind of existential dread that only people who work in IT know -- the kind of dread that arises a picosecond after you've hit ENTER and realized you've type 'rm -rf /' or some-such -- because I knew at that very second exactly what I'd done wrong.

I'd typo'd "5" and made it "6" in the runlevel. And pushed it to ~1,000 -physical- servers. And then rebooted them ALL.

"So," you're asking, "Whyfor is runlevel 6 a big deal?"

Because of this:

runlevel 3: command line.
runlevel 5: GUI
runlevel 6: REBOOT THE FUCKING COMPUTER.

What I'd done was told every production server on our network to reboot as soon as it rebooted, which leads to another reboot, which leads to another reboot, lather rinse repeat.

At 3:45am on Saturday morning, I knew that every person in IT would have to drive into the office, visit every production server with a bootable USB key, change the BIOS to boot off the key, boot the server into Single User Mode, change the damned file by hand, then reboot the server. This takes about 10 minutes per server -- times ~1,000.

I learned a number of valuable lessons that day:

1. DOUBLE CHECK YOUR FUCKING WORK.
2. See lesson #1
addendum: filing for unemployment insurance in Washington state is amazingly easy.

And that was the very last time I ever worked on physical hardware. To this day, if it's not in the cloud, I ain't fucking touching it.

Here endth the lesson.

The actual letter-

Dear Senator Schumer:

Thank you for your Democrat public relations letter and incorrect sound bites, which are wrong in every way.

As you are aware, Vice President Pence is in charge of the Task Force. By almost all accounts, he has done a spectacular job.

The Defense Production Act (DPA) has been consistently used by my team and me for the purchase of billions of dollars’ worth of equipment, medical supplies, ventilators, and other related items. It has been powerful leverage, so powerful that companies generally do whatever we are asking, without even a formal notice. They know something is coming, and that’s all they need to know.
A “senior military officer” is in charge of purchasing, distributing, etc. His name is Rear Admiral John Polowczyk. He is working 24 hours a day, and is highly respected by everyone. If you remember, my team gave you this information, but for public relations purposes, you choose to ignore it.

We have given New York many things, including hospitals, medical centers, medical supplies, record numbers of ventilators, and more. You should have had New York much better prepared than you did, and as Dr. Fauci and Dr. Birx said yesterday, New York was very late in its fight against the virus. As you are aware, the Federal Government is merely a back-up for state governments. Unfortunately, your state needed far more of a back-up than most others.

If you spent less time on your ridiculous impeachment hoax, which went haplessly on forever and ended up going nowhere (except increasing my poll numbers), and instead focused on helping the people of New York, then New York would not have been so completely unprepared for the “invisible enemy.” No wonder AOC and others are thinking about running against you in the primary. If they did, they would likely win.

Fortunately, we have been working with your state and city governments, Governor Andrew Cuomo and Mayor Bill DeBlasio, to get the job done. You have been missing in action, except when it comes to the “press.” While you have stated that you don’t like Andrew Cuomo, you ought to start working alongside him for the good of all New Yorkers.

I’ve known you for many years, but I never knew how bad a Senator you are for the state of New York, until I became President.

If you have any questions, please do not hesitate to call. Or, in the alternative, call Rear Admiral Polowczyk.

Sincerely yours,

Donald J. Trump


So, not the time to play politics, eh? Hilarious letter, eh? Hilarious in it's self centered ignorance, self congratulatory message, complete lack of self awareness, and total lack of anything helpful or useful, just more Don the Con whining that bad man make little Donny cry hurt and firing tactless political shots. It would be hilarious if tens of thousands of lives weren't at risk because of this narcissistic idiocy.
Remember, Trump said those who aren't nice enough to him won't get their calls answered, and publicly instructed Pence to follow suit.
Con man Don told Schumer that he had written this "very nasty letter" (Donny's words) but would try to keep it from being sent and publicly apologize if it was, then Trump's office released it to the media.
Not the time for politics....yep. *facepalm

So delusional.
You asked why, which was answered in the video clearly....I guess reality means nothing to you. An impossible dream of some magically universally enriching personal responsibility (your plan-all you need to do is save that money you aren't earning for hard(er) times) means less to me than hundreds of thousands of deaths and trillions in socialist handouts and losses.

I live in the real world, not the fantasy where being undereducated, under employed, and destitute is a choice people make. This fictional reality where everyone has the means and opportunity to succeed. If working hard and staying clean was a guarantee of financial success, you might have 1/2 a point to make, but it's not.

Yep, typical....berate me for being Santa by suggesting paid leave is smart, then pat orange Santa on the back for enacting a more expensive 1/2 ass version of that plan but wasting and grifting hundreds of billions in the process. Having it be federal law before it was too late could have slowed the spread, flattening the curve....1/2 measures long after it's an epidemic won't.

Economy, in the toilet. Unemployment, jumping faster than the great depression. Health and safety, a memory. Infrastructure, completely forgotten. Debt and deficit, skyrocketing like never before in history. How exactly are you winning today?

Come on, where's my "red tsunami 2020"?

I want technology that will allow 1G of acceleration constantly for about 3 days. That's all you need to get to Mars.

Spring Break in an electric dunebuggy on Olympic Mons.

Damn straight.

Considering that religion is a spinoff from the bible and the bible was created by artists, then their remains a puzzle.

Simply: God is love
God would be the character name of the person reading the book.

You are the creator of your world. Believe in yourself.

All you need is love. -Beatles

Love: Ingredients - Brain. Heart. Courage. -Wizard of Oz

Love with purpose.

So, instead of calling yourself, "GOD", I think "newt" would be perfect! After all you created it.

Be the star that others wish upon.
https://www.youtube.com/watch?v=pguMUFyJ3_U




All that you touch
And all that you see
All that you taste
All you feel
And all that you love
And all that you hate
All you distrust
All you save
And all that you give
And all that you deal
And all that you buy
Beg, borrow or steal
And all you create
And all you destroy
And all that you do
And all that you say
And all that you eat
And everyone you meet (everyone you meet)
And all that you slight
And everyone you fight
And all that is now
And all that is gone
And all that's to come
And everything under the sun is in tune
But the sun is eclipsed by the moon -Pink Floyd

Meh, all you need to do is think of Old Town Road.





Aw shit...

You accepted my challenge, so it's only fair that I accept yours.

1) If love is all you need then it follows (or precedes) that all you need is love. However, I believe in life after love, therefore I cannot accept the original premise. My answer is no.

2) Well, I may know the undercover agents, and therefore I possibly know who they are, but I'm not aware that I'm aware who the undercover agents are. I'll give this one a 'probably not'.

3) Communicating the entirety of my knowledge on any subject via text is impractical, therefore I will assume that an attempt at quantification will suffice. I'll say... six.

As for the code, it's been a long time since I watched ST4, but a short time since I listened to The Wall, so I'll go with my gut answer. Is there anybody out there?

I like a nice challenge. Thanks.

"The Synagogue of Satan' is not playing games."

Poor chap. Never heard (of) the Beatles.

If you want to win, you can to learn to play the game. To play the game you've got know whose game your playing. It's easy.

How many times do they need to say it?

Love, love, love
Love, love, love
Love, love, love
There's nothing you can do that can't be done
Nothing you can sing that can't be sung
Nothing you can say, but you can learn how to play the game
It's easy
Nothing you can make that can't be made
No one you can save that can't be saved
Nothing you can do, but you can learn how to be you in time
It's easy
All you need is love
All you need is love
All you need is love, love
Love is all you need
All you need is love
All you need is love
All you need is love, love
Love is all you need
There's nothing you can know that isn't known
Nothing you can see that isn't shown
There's nowhere you can be that isn't where you're meant to be
It's easy
All you need is love
All you need is love
All you need is love, love
Love is all you need
All you need is love (all together now)
All you need is love (everybody)
All you need is love, love
Love is all you need
Love is all you need
(Love is all you need)
Love is all you need
(Love is all you need)
Love is all you need
(Love is all you need)
Love is all you need
(Love is all you need)
Love is all you need
(Love is all you need)
Love is all you need
(Love is all you need)
Love is all you need
(Love is all you need)
Love is all you need
(Love is all you need)
Love is all you need
(Love is all you need)
Love is all you need
(Love is all you need)
Love is all you need
(Love is all you need)
(Love is all you need)
(Love is all you need)
(Love is all you need)
Yesterday ---------------------https://www.youtube.com/watch?v=6uqvgPm8U4c
(Love is all you need)
Oh
Love is all you need
Love is all you need
Oh yeah
Love is all you need
(She love you, yeah, yeah, yeah)
(She love you, yeah, yeah, yeah)
(Love is all you need)
(Love is all you need)
Songwriters: John Lennon / Paul Mccartney

I'll interject.

I accept your challenge.

1) Do you believe love is all you need?

2) Are you aware who the undercover agents are?

3) What do you know about acting?

Can you crack this code?

Star Trek IV: The Voyage Home*

GILLIAN: What you're hearing is recorded whale song. It is sung by the male. He'll sing anywhere from six to as long as thirty minutes, and then, start again. In the ocean, the other whales will pick up the song, and pass it on.

(Spock is seen swimming in the underwater tank)

GILLIAN: The songs change every year, but we still don't know what purpose they serve. Are they some kind of navigational signal? Could they be part of the mating ritual? Or is it pure communication beyond our comprehension? Frankly we just don't know.

--------------------------------------------

Hey you, out there in the cold
Getting lonely, getting old
Can you feel me?
Hey you, standing in the aisles
With itchy feet and fading smiles
Can you feel me?
Hey you, don't help them to bury the light
Don't give in without a fight
Hey you out there on your own
Sitting naked by the phone
Would you touch me?
Hey you with you ear against the wall
Waiting for someone to call out
Would you touch me?
Hey you, would you help me to carry the stone?
Open your heart, I'm coming home*
But it was only fantasy
The wall was too high
As you can see
No matter how he tried
He could not break free
And the worms ate into his brain
Hey you, out there on the road
Always doing what you're told
Can you help me?
Hey you, out there beyond the wall
Breaking bottles in the hall
Can you help me?

Hey you, don't tell me there's no hope at all
Together we stand, divided we fall

Songwriters: Roger Waters

https://www.youtube.com/watch?v=ymgYEQgSqLI

No...the gift of being violated is not precious. The gift of mistrust might have kept me out of trouble to some extent, but I could and would have learned that lesson without a gun to my head and knee in my neck, and without the threats to my freedom if I told on him.

I don't KNOW....I said I would bet. Please read more carefully before making assumptions and accusations. It's far more likely he's not remorseful, I said why I think that.

They wouldn't take a report or complaint, and I don't remember his name. That makes checking in on the thug a bit difficult. Your suggestion is like asking someone to look up their uncaught mugger to see if they feel bad decades later. I would love to face him as an adult, but he's not my white whale like you want him to be....he's just one more dangerous whale in a sea full of them, he just happens to be the one that bit me, they all bite. Nothing special about him.

I have zero need to give him forgiveness nor to take away the mistrust. I don't care a whit about him. Hate implies he's important to me, he's just not except as an example, an instance where I learned some harsh reality, that police are not there to serve and protect citizens as I had been taught.

Please don't presume to tell me what I need or how I feel. I find it insulting. You obviously don't know me a bit or you would never take that path. Assuming I'll react, think, or feel like an average, normal person is about as big an insult as I can think of.

I'm no cop. I'm not assaulting those I'm charged with protecting and serving. I'm not abusing anything by having an opinion.
Just stop, please, you're becoming dismissively insulting with your naive assumptions. Love isn't all you need...obviously....or you would be in line for your lobotomy right now and happily spend your life loving whatever you see without thinking. Remember, the group that told you that lie was broken up by love.

If you need to give love a character name, I can live with that. Artists do it all the time.

Truth is, love is all you need. It's the highway to hell. If you want to get to heaven, you've got to go through hell.
How can there be such a sinister plan
That could hide such a lamb, such a caring young man

I was wrong once.

1) You only need to be correct about ONE thing 100% of the time.

2) When you lose the one you love and suffer their loss, you will feel you did the wrong thing. Feeling tricked. You will feel VERY ALONE. This is not where love was supposed to take you. You will be left with one decision. Do you continue to love knowing you will be sending the people you've loved and who love you, into the same hell you are stuck in or do you take the chance that those you love can band together with all those who chose to love you and them?

3) You will only need to make one decision to get beyond the hell and grief that love delivered you into.

The bleeding hearts and artists are waiting for you.

Love is all you need. -Beatles

The blacksmith and the artist
Reflect it in their art
They forge their creativity
Closer to the heart
Yeah, it's closer to the heart -Rush

--------

Thank you for breaking my heart
Thank you for tearing me apart
Now I've a strong, strong heart
Thank you for breaking my heart -John Charles Reynolds / Sinead O'connor



As far as the guy with the poop, yeah, I think you're right.