CloudFlare and an Apology for Recent Down Time
Hi everyone. I wanted to apologise for the down periods we've had over the last couple of weeks and explain what's been going on. A couple of weeks ago we signed up with CloudFlare, a CDN and distributed domain name server service. It's informally billed as "Akamai for the little guy". On the surface it sounds fantastic. They cache content, protect against hack attempts and route requests to DNS servers close to your geographic location. When it was working, there were definitely great speed benefits - pages were loading in a jiffy - but when it was not working - everything fell apart.
They seem to be a frequent recipient to DDOS attacks - the very thing that they are supposed to help mitigate - and this brings their nodes down in various locations at different times. See their status page. This means that sometimes sifters from Los Angeles might not be able to access the Sift while people from New York are just fine. It's an untenable situation for us - monitoring our own servers and the 20 or so nodes that CloudFlare has as well.
So we quit. We've had a very bad experience with CloudFlare and are moving back to our own server and a standard CDN service for static files. If you are looking at CloudFlare for your own website, I don't recommend it.
Even though we shouldn't be having any more CloudFlare outages into the future - we're also struggling with one of our own servers that doesn't want to play nice with its friends. We're hoping to have that resolved soon.
So sorry again for the outages - should be a lot less from here on out.
They seem to be a frequent recipient to DDOS attacks - the very thing that they are supposed to help mitigate - and this brings their nodes down in various locations at different times. See their status page. This means that sometimes sifters from Los Angeles might not be able to access the Sift while people from New York are just fine. It's an untenable situation for us - monitoring our own servers and the 20 or so nodes that CloudFlare has as well.
So we quit. We've had a very bad experience with CloudFlare and are moving back to our own server and a standard CDN service for static files. If you are looking at CloudFlare for your own website, I don't recommend it.
Even though we shouldn't be having any more CloudFlare outages into the future - we're also struggling with one of our own servers that doesn't want to play nice with its friends. We're hoping to have that resolved soon.
So sorry again for the outages - should be a lot less from here on out.
12 Comments
dagComment hidden because you are ignoring dag. (show it anyway)
Guys - although we are moving off of CloudFlare immediately - it may take a day or more for all the changes to be propagated out through the net.
darkrowanOh thank the gods. So we loungers are not totally going insane here. Thank you for the update, makes sense of it all now
critical_dToo much cloud...not enough flare.
BoneRemakefuck'em.
dagComment hidden because you are ignoring dag. (show it anyway)
According to our Facebook group - we still have people who can't get to the Sift, @dotdude and others. As the DNS changes propagate, expect everyone to be back on board.
darkrowanIf memory serves, and the actual logistics haven't changed, it takes roughly 48 hours for the world's DNS servers to fully update a major change like this. So yeah, @dag and @lucky760 killed our support it still take a while for it to process.
ctrlaltbleach@dag <<< he said nodes
antAh, that explains the problems and light traffics here. I didn't notice DNS problems at work and home (OpenDNS).
antIt's sure quiet on VS lately. Are people still having problems with DNS?
eric3579It seems site traffic has been down quite a bit this week. Was curious if the CloudFare was responsible and if you think we will be getting it back. From the one month traffic chart on Alexa, it looks like we have lost 40% to 50% of our avg. monthly traffic this week. Just very curious. @lucky760 @dag
dagComment hidden because you are ignoring dag. (show it anyway)
We definitely lost some traffic during that failed experiment, but not permanently. Alexa is a bit delayed and kind of inaccurate IMHO. We seem to be firing on all cylinders again now though. CloudFlare is definitely on my shit list. Reminds me I must go tweet about this. >> ^eric3579:
It seems site traffic has been down quite a bit this week. Was curious if the CloudFare was responsible and if you think we will be getting it back. From the one month traffic chart on Alexa, it looks like we have lost 40% to 50% of our avg. monthly traffic this week. Just very curious. @lucky760 @dag
damoncloudflareHi,
Sorry about the bad experience. I did want to advise that the attacks are generally not against CloudFlare proper, but are generally attacks directed against client sites, and we are working on some network solutions to greatly reduce this problem in the very near future (testing and building out solutions now). While you are indeed correct in stating that the rest of the network is up when one datacenter is having an issue, we most certainly do realize that it is little consolation when any of your visitors hitting that datacenter are having a problem accessing your site.
Discuss...
Enable JavaScript to submit a comment.