History of VideoSift Part III
I'm writing a series of blog posts about the history of VideoSift. This is Part III, but you can go back and read Prelude to a Sift , Part I and then Part II.
2007 was my 20th high school reunion. I went up to my old home town in Alaska and stayed with @deathcow. We both went to the first night of the reunion in the old local bowling alley lounge. Afterwards, nursing cheap beer hangovers, we vowed to skip all future reunions.
Why is this relevant to VideoSift? On the way back from Alaska, I had an appointment with a man in San Francisco - let's call him "K".
At this time, in late 2007 - VideoSift was still growing like the Blob. To be honest- I was a little worried that it was too much for two people to manage. We kept buying servers - and some months, the money was pretty tight for just paying our bills. I had the thought that we could be taken under the wing of a benevolent corporate benefactor, either through an acquisition or a partnership that would allow us to keep growing and keep paying our bills.
K was a vice president of Conde Nast and managed the acquisition of both Wired and Reddit. I honestly don't remember how I got the meeting. I might have just emailed the guy or connected through LinkedIn.
I walked up to the Wired Digital HQ on Market in downtown San Francisco, ten minutes early for the meeting. I was led in to meet K - he was sitting in the Wired cafeteria (wow - company chef, exposed ducting and soft lighting). I showed him some of the traffic stats for VideoSift and then the conversation went something like this:
At that point, the meeting was pretty much over. He took me back to meet the Reddit team. They handed out their business cards to me and we chatted awkwardly for a bit. K said "Hey, do we have any of those t-shirts left?" - the Reddit guys looked around sheepishly - they did not. The meeting was over and I was shown to the exit. Later that day I got back on a plane for Australia.
*************
In 2009 VideoSift suffered a mortal wound - we recovered, but the scars remain even today. I'm going to let @lucky760 tell the story of the Siftpocalypse as he remembers it best:
I was working on one of our servers around midnight, Mar 12, 2009 when I was contacted via chat by @dag mere minutes after the first bomb of the Siftpocalypse was dropped. This is from those actual transcripts:
It was at this point I first started to realize the horrifying extent of my blunder. I had been working on setting up a development environment for another project and I meant to wipe the database clean for that project, but I was mistakenly still talking to the VideoSift database when I issued the command.
I immediately went to our backups since we back everything up nightly for just such an emergency, but... gulp... several database tables were completely missing. Specifically, none of our backups had records for: user ignores, user profiles, user stars, user statistics, users, video hosts, views, and votes.
You may notice that those are all in alphabetical order starting at "U". It turned out the backups were missing because we had a table called "user channel filters" which was triggering an error during every nightly backup, causing all subsequent tables to be skipped altogether.
Most of the missing tables weren't overly important, but two of them were huge: We had no users and no votes. (Thank goodness the table of videos is named "posts" otherwise all our videos would have been lost as well!)
There was a small ray of sunshine through the mushroom cloud when I found that our development server for VideoSift had a full database import including the missing tables, except that was created a year earlier. This means we had no member and no video voting information between March 2008 and March 2009.
Once we finally accepted the fact there was nothing else we could do, we addressed the community.
To this day there are still lots and lots of members without a username and video votes by unknown voters (which have since been converted into siftbot votes that can be reclaimed), and after almost 4 years most people have thankfully forgotten the horror of that day... I wish I was so lucky.
The amazing thing is that we did recover, and quickly - we "deputized" high-level Sifters, members of the community who knew where to fix things - and it worked. A powerful community can overcome technical setbacks. If all the content of VideoSift was deleted today, VideoSift would rebuild and come back as strong as ever. (knock on wood)
Coming soon, the next and last chapter (for a while):
The best cheese festival ever!
VideoSift 4.0
VideoSift 5.0 and VideoSift turns 7
2007 was my 20th high school reunion. I went up to my old home town in Alaska and stayed with @deathcow. We both went to the first night of the reunion in the old local bowling alley lounge. Afterwards, nursing cheap beer hangovers, we vowed to skip all future reunions.
Why is this relevant to VideoSift? On the way back from Alaska, I had an appointment with a man in San Francisco - let's call him "K".
At this time, in late 2007 - VideoSift was still growing like the Blob. To be honest- I was a little worried that it was too much for two people to manage. We kept buying servers - and some months, the money was pretty tight for just paying our bills. I had the thought that we could be taken under the wing of a benevolent corporate benefactor, either through an acquisition or a partnership that would allow us to keep growing and keep paying our bills.
K was a vice president of Conde Nast and managed the acquisition of both Wired and Reddit. I honestly don't remember how I got the meeting. I might have just emailed the guy or connected through LinkedIn.
I walked up to the Wired Digital HQ on Market in downtown San Francisco, ten minutes early for the meeting. I was led in to meet K - he was sitting in the Wired cafeteria (wow - company chef, exposed ducting and soft lighting). I showed him some of the traffic stats for VideoSift and then the conversation went something like this:
K: So what are you after, an acquisition, an exit?
Me: Well, we like doing what we're doing - we're looking for a way to keep doing it and get paid for it. I've noticed that Reddit doesn't host embedded videos. VideoSift could be a good complement - fill in that gap.
K: Why would we acquire another site for that? If we wanted to turn on embedded videos we'd just flip the switch.
Me: (stuttering mumble)
K: Sigh. Look, how about I introduce you to John Battelle, if he can't figure out a way for you guys to make a profit with this - no one can.
At that point, the meeting was pretty much over. He took me back to meet the Reddit team. They handed out their business cards to me and we chatted awkwardly for a bit. K said "Hey, do we have any of those t-shirts left?" - the Reddit guys looked around sheepishly - they did not. The meeting was over and I was shown to the exit. Later that day I got back on a plane for Australia.
*************
In 2009 VideoSift suffered a mortal wound - we recovered, but the scars remain even today. I'm going to let @lucky760 tell the story of the Siftpocalypse as he remembers it best:
I was working on one of our servers around midnight, Mar 12, 2009 when I was contacted via chat by @dag mere minutes after the first bomb of the Siftpocalypse was dropped. This is from those actual transcripts:
12:08AM dag: Holy fuck. What happened?
12:09AM lucky760: i fucked us big time. i wiped the db. i restored our backup from last night but it doesn't have the Users database at all!!! i'm going thru any backup i can find but they are all missing the Users db!
It was at this point I first started to realize the horrifying extent of my blunder. I had been working on setting up a development environment for another project and I meant to wipe the database clean for that project, but I was mistakenly still talking to the VideoSift database when I issued the command.
I immediately went to our backups since we back everything up nightly for just such an emergency, but... gulp... several database tables were completely missing. Specifically, none of our backups had records for: user ignores, user profiles, user stars, user statistics, users, video hosts, views, and votes.
You may notice that those are all in alphabetical order starting at "U". It turned out the backups were missing because we had a table called "user channel filters" which was triggering an error during every nightly backup, causing all subsequent tables to be skipped altogether.
Most of the missing tables weren't overly important, but two of them were huge: We had no users and no votes. (Thank goodness the table of videos is named "posts" otherwise all our videos would have been lost as well!)
There was a small ray of sunshine through the mushroom cloud when I found that our development server for VideoSift had a full database import including the missing tables, except that was created a year earlier. This means we had no member and no video voting information between March 2008 and March 2009.
Once we finally accepted the fact there was nothing else we could do, we addressed the community.
To this day there are still lots and lots of members without a username and video votes by unknown voters (which have since been converted into siftbot votes that can be reclaimed), and after almost 4 years most people have thankfully forgotten the horror of that day... I wish I was so lucky.
The amazing thing is that we did recover, and quickly - we "deputized" high-level Sifters, members of the community who knew where to fix things - and it worked. A powerful community can overcome technical setbacks. If all the content of VideoSift was deleted today, VideoSift would rebuild and come back as strong as ever. (knock on wood)
Coming soon, the next and last chapter (for a while):
Load Comments...
Discuss...
Enable JavaScript to submit a comment.