History of VideoSift Part III
I'm writing a series of blog posts about the history of VideoSift. This is Part III, but you can go back and read Prelude to a Sift , Part I and then Part II.
2007 was my 20th high school reunion. I went up to my old home town in Alaska and stayed with @deathcow. We both went to the first night of the reunion in the old local bowling alley lounge. Afterwards, nursing cheap beer hangovers, we vowed to skip all future reunions.
Why is this relevant to VideoSift? On the way back from Alaska, I had an appointment with a man in San Francisco - let's call him "K".
At this time, in late 2007 - VideoSift was still growing like the Blob. To be honest- I was a little worried that it was too much for two people to manage. We kept buying servers - and some months, the money was pretty tight for just paying our bills. I had the thought that we could be taken under the wing of a benevolent corporate benefactor, either through an acquisition or a partnership that would allow us to keep growing and keep paying our bills.
K was a vice president of Conde Nast and managed the acquisition of both Wired and Reddit. I honestly don't remember how I got the meeting. I might have just emailed the guy or connected through LinkedIn.
I walked up to the Wired Digital HQ on Market in downtown San Francisco, ten minutes early for the meeting. I was led in to meet K - he was sitting in the Wired cafeteria (wow - company chef, exposed ducting and soft lighting). I showed him some of the traffic stats for VideoSift and then the conversation went something like this:
At that point, the meeting was pretty much over. He took me back to meet the Reddit team. They handed out their business cards to me and we chatted awkwardly for a bit. K said "Hey, do we have any of those t-shirts left?" - the Reddit guys looked around sheepishly - they did not. The meeting was over and I was shown to the exit. Later that day I got back on a plane for Australia.
*************
In 2009 VideoSift suffered a mortal wound - we recovered, but the scars remain even today. I'm going to let @lucky760 tell the story of the Siftpocalypse as he remembers it best:
I was working on one of our servers around midnight, Mar 12, 2009 when I was contacted via chat by @dag mere minutes after the first bomb of the Siftpocalypse was dropped. This is from those actual transcripts:
It was at this point I first started to realize the horrifying extent of my blunder. I had been working on setting up a development environment for another project and I meant to wipe the database clean for that project, but I was mistakenly still talking to the VideoSift database when I issued the command.
I immediately went to our backups since we back everything up nightly for just such an emergency, but... gulp... several database tables were completely missing. Specifically, none of our backups had records for: user ignores, user profiles, user stars, user statistics, users, video hosts, views, and votes.
You may notice that those are all in alphabetical order starting at "U". It turned out the backups were missing because we had a table called "user channel filters" which was triggering an error during every nightly backup, causing all subsequent tables to be skipped altogether.
Most of the missing tables weren't overly important, but two of them were huge: We had no users and no votes. (Thank goodness the table of videos is named "posts" otherwise all our videos would have been lost as well!)
There was a small ray of sunshine through the mushroom cloud when I found that our development server for VideoSift had a full database import including the missing tables, except that was created a year earlier. This means we had no member and no video voting information between March 2008 and March 2009.
Once we finally accepted the fact there was nothing else we could do, we addressed the community.
To this day there are still lots and lots of members without a username and video votes by unknown voters (which have since been converted into siftbot votes that can be reclaimed), and after almost 4 years most people have thankfully forgotten the horror of that day... I wish I was so lucky.
The amazing thing is that we did recover, and quickly - we "deputized" high-level Sifters, members of the community who knew where to fix things - and it worked. A powerful community can overcome technical setbacks. If all the content of VideoSift was deleted today, VideoSift would rebuild and come back as strong as ever. (knock on wood)
Coming soon, the next and last chapter (for a while):
The best cheese festival ever!
VideoSift 4.0
VideoSift 5.0 and VideoSift turns 7
2007 was my 20th high school reunion. I went up to my old home town in Alaska and stayed with @deathcow. We both went to the first night of the reunion in the old local bowling alley lounge. Afterwards, nursing cheap beer hangovers, we vowed to skip all future reunions.
Why is this relevant to VideoSift? On the way back from Alaska, I had an appointment with a man in San Francisco - let's call him "K".
At this time, in late 2007 - VideoSift was still growing like the Blob. To be honest- I was a little worried that it was too much for two people to manage. We kept buying servers - and some months, the money was pretty tight for just paying our bills. I had the thought that we could be taken under the wing of a benevolent corporate benefactor, either through an acquisition or a partnership that would allow us to keep growing and keep paying our bills.
K was a vice president of Conde Nast and managed the acquisition of both Wired and Reddit. I honestly don't remember how I got the meeting. I might have just emailed the guy or connected through LinkedIn.
I walked up to the Wired Digital HQ on Market in downtown San Francisco, ten minutes early for the meeting. I was led in to meet K - he was sitting in the Wired cafeteria (wow - company chef, exposed ducting and soft lighting). I showed him some of the traffic stats for VideoSift and then the conversation went something like this:
K: So what are you after, an acquisition, an exit?
Me: Well, we like doing what we're doing - we're looking for a way to keep doing it and get paid for it. I've noticed that Reddit doesn't host embedded videos. VideoSift could be a good complement - fill in that gap.
K: Why would we acquire another site for that? If we wanted to turn on embedded videos we'd just flip the switch.
Me: (stuttering mumble)
K: Sigh. Look, how about I introduce you to John Battelle, if he can't figure out a way for you guys to make a profit with this - no one can.
At that point, the meeting was pretty much over. He took me back to meet the Reddit team. They handed out their business cards to me and we chatted awkwardly for a bit. K said "Hey, do we have any of those t-shirts left?" - the Reddit guys looked around sheepishly - they did not. The meeting was over and I was shown to the exit. Later that day I got back on a plane for Australia.
*************
In 2009 VideoSift suffered a mortal wound - we recovered, but the scars remain even today. I'm going to let @lucky760 tell the story of the Siftpocalypse as he remembers it best:
I was working on one of our servers around midnight, Mar 12, 2009 when I was contacted via chat by @dag mere minutes after the first bomb of the Siftpocalypse was dropped. This is from those actual transcripts:
12:08AM dag: Holy fuck. What happened?
12:09AM lucky760: i fucked us big time. i wiped the db. i restored our backup from last night but it doesn't have the Users database at all!!! i'm going thru any backup i can find but they are all missing the Users db!
It was at this point I first started to realize the horrifying extent of my blunder. I had been working on setting up a development environment for another project and I meant to wipe the database clean for that project, but I was mistakenly still talking to the VideoSift database when I issued the command.
I immediately went to our backups since we back everything up nightly for just such an emergency, but... gulp... several database tables were completely missing. Specifically, none of our backups had records for: user ignores, user profiles, user stars, user statistics, users, video hosts, views, and votes.
You may notice that those are all in alphabetical order starting at "U". It turned out the backups were missing because we had a table called "user channel filters" which was triggering an error during every nightly backup, causing all subsequent tables to be skipped altogether.
Most of the missing tables weren't overly important, but two of them were huge: We had no users and no votes. (Thank goodness the table of videos is named "posts" otherwise all our videos would have been lost as well!)
There was a small ray of sunshine through the mushroom cloud when I found that our development server for VideoSift had a full database import including the missing tables, except that was created a year earlier. This means we had no member and no video voting information between March 2008 and March 2009.
Once we finally accepted the fact there was nothing else we could do, we addressed the community.
To this day there are still lots and lots of members without a username and video votes by unknown voters (which have since been converted into siftbot votes that can be reclaimed), and after almost 4 years most people have thankfully forgotten the horror of that day... I wish I was so lucky.
The amazing thing is that we did recover, and quickly - we "deputized" high-level Sifters, members of the community who knew where to fix things - and it worked. A powerful community can overcome technical setbacks. If all the content of VideoSift was deleted today, VideoSift would rebuild and come back as strong as ever. (knock on wood)
Coming soon, the next and last chapter (for a while):
24 Comments
dagComment hidden because you are ignoring dag. (show it anyway)
Make haste Siftbot *frontpage
siftbotPrinting this post atop the VideoSift homepage - frontpage requested by dag.
ant*quality
Test those backups and recoveries!!!!!!
chingaleraOOOPS! Wasted a quality!
I remember not taking it as hard as you guys when the siftpocalypso dance started....
I don't remember exactly why I asked RS for a transcript of all my chatter here but he obliged. Whew, trying to read that shit!??
*quality
Test those backups and recoveries!!!!!!
bareboards2The Sift meltdown was the death of bareboards and the birth of bareboards2. I regret to this day not trying to reclaim my old user file.... I was but a lurker in those days and it didn't seem worth the farting around to figure out how to reclaim my stats.
Never dreamed I'd start Sifting....
deathcowDuring the Siftpocalypse, I wrote a Google crawler to retrieve all the users who voted on every video. It would pull down the google page for each Sifted video and then parse out all the users who had voted on it. I ran it through about 1000 videos before Google thought I was attacking them, or rather, thought the network of the entire big will-remain-unnamed telecom company I work for was attacking them. As a consequence google blocked all access to their search engine from the company. I shit big golden bricks. People notice FAST when Google disappears. Luckily for me... Google automatedly reenabled access within ?an hour? OMG
jonny"siftbot votes that can be reclaimed"
The ability to reclaim votes seems to have gone away. I was actually about to post to sifttalk about it yesterday, but didn't get around to it. And since it specifically came up here, I thought I'd mention it.
Great work again, @dag. Though I admit I was hoping to read more about "K" and Battelle. Did that introduction ever happen? Did Battelle have any good insights for you? Did you get anything other than a bad taste in your mouth from that meeting?
dagComment hidden because you are ignoring dag. (show it anyway)
The introduction was more of an email hand-off, I think Battelle and I exchanged a couple of mails and that was it.
A bad taste is the right way to characterise that session in San Francisco. I was left feeling that I was completely out of my league in the world of Silicon Valley business.
"siftbot votes that can be reclaimed"
The ability to reclaim votes seems to have gone away. I was actually about to post to sifttalk about it yesterday, but didn't get around to it. And since it specifically came up here, I thought I'd mention it.
Great work again, @dag. Though I admit I was hoping to read more about "K" and Battelle. Did that introduction ever happen? Did Battelle have any good insights for you? Did you get anything other than a bad taste in your mouth from that meeting?
SagemindI remember it well, Crisis always brings people together.
Although the Sift team was "sifting bricks" on the back end, I though the job you guys did on the recovery was amazing. I've been through a few complete database crashes at various places I've worked and I know they aren't pretty.
From my side, sure there was some confusion and a little panic, which is to be expected, but I found the recovery, from an outside point of view, went "tickity-boo." I especially liked that you guys came forward and just stated it as it was so we all knew what we were dealing with. We all went through it together. It could have been handled by shutting down the site, nursing the wounds, and then popping back up online a week later, but that's not what happened.
By coming forward and keeping us all in the loop with both the negatives and positives, we all went through it together. Not only that but all who had the capabilities pitched in and worked together.
That's how communities are born!
antWhen is bareboards3 coming?
The Sift meltdown was the death of bareboards and the birth of bareboards2. I regret to this day not trying to reclaim my old user file.... I was but a lurker in those days and it didn't seem worth the farting around to figure out how to reclaim my stats.
Never dreamed I'd start Sifting....
lucky760I'm keeping secret the date of the next Siftpocalypse.
(I'd like for it to be a surprise.)
When is bareboards3 coming?
dingensDecember 21, 2012 would have been a nice date for Siftpocalypse Reloaded
I'm keeping secret the date of the next Siftpocalypse.
FantomasThe Siftocalypse managed to occur at a time I was pondering whether I wanted to stay on the sift or not, so I took it as an opportunity for a break.
Videosift being the place it is, I obviously couldn't stay away. Although my presence is now much more paid back than it once was. I still have my T-Shirt from the server fundraiser.
dagComment hidden because you are ignoring dag. (show it anyway)
My dream is to one day see a VideoSift t-shirt in the wild.
The Siftocalypse managed to occur at a time I was pondering whether I wanted to stay on the sift or not, so I took it as an opportunity for a break.
Videosift being the place it is, I obviously couldn't stay away. Although my presence is now much more paid back than it once was. I still have my T-Shirt from the server fundraiser.
antWait, the last/previous one was on purpose? WHY?!?!?!
I'm keeping secret the date of the next Siftpocalypse.
DeanoYe gods that would be so cool.
Keep the posts coming in, thoroughly enjoying them.
Quote hidden because you are ignoring dag. (show it anyway)
My dream is to one day see a VideoSift t-shirt in the wild.
antThere should be a poll on who wear VS shirts in the wild.
Ye gods that would be so cool.
Keep the posts coming, in thoroughly enjoying them.
gwiz665I do.
There should be a poll on who wear VS shirts in the wild.
dagComment hidden because you are ignoring dag. (show it anyway)
I do too. I have seen them at Sift Ups, but that doesn't count.
spoco2Great reading dag, memory lane travelling to be sure.
I've dropped off my frequenting of the sift of late, haven't felt the real want to watch videos as much as I used to. It's still one of my homepage tabs though, and still the place on the web that I have by far posted the most content to.
antonyeJeez, I must have slept through that one...
rasch187All this talk of the siftocalypse and t-shirts reminds me of this.
dagComment hidden because you are ignoring dag. (show it anyway)
Small chance I might be moving to Melbourne next year- if so, Sift Up is on - must go to Gaylords again.
Great reading dag, memory lane travelling to be sure.
I've dropped off my frequenting of the sift of late, haven't felt the real want to watch videos as much as I used to. It's still one of my homepage tabs though, and still the place on the web that I have by far posted the most content to.
spoco2Definitely. Plus so many other eateries to try.
Quote hidden because you are ignoring dag. (show it anyway)
Small chance I might be moving to Melbourne next year- if so, Sift Up is on - must go to Gaylords again.
Discuss...
Enable JavaScript to submit a comment.