tag:blogger.com,1999:blog-5819005.post7178948667390505748..comments2024-03-16T02:17:55.433-07:00Comments on Programming and politics: Why Johnny Can't ScaleAnonymoushttp://www.blogger.com/profile/08698131967747444207noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-5819005.post-87965538546622607522008-04-25T23:17:00.000-07:002008-04-25T23:17:00.000-07:00Oh wow, thanks. Hmm let's see though... I'm not su...Oh wow, thanks. Hmm let's see though... I'm not sure if I would <A HREF="http://www.alexa.com/data/details/traffic_details/moli.com?site0=yfly.com&y=p&z=3&h=400&w=700&u%5B%5D=yfly.com&x=2008-04-26T06%3A07%3A58.000Z&check=www.alexa.com&signature=YYsi%2F82%2BVpLZEsMDUcxUzIx3%2FlE%3D&range=3m&size=Large" REL="nofollow">qualify yfly as web scale</A>. Still that's great that they have a federated DB and Rails working together. I'd love to read about it, heck I'd love to write about it.<BR/><BR/>I guess one possible explanation for why there are no large scale sites that use Rails and have a federated DB is that there are few large scale sites. Another possible explanation is that those sites choose not to use Rails for whatever reasons. Web scale is hard enough that it is not going to be done by a code generation script or by some method_missing cleverness. But it's not that hard. After all it's been done by numerous sites in all kinds of languages/frameworks: PHP, Java, Perl, ASP.NET, C++, Coldfusion, etc. It's a long list, just not a list that includes Rails. Twitter has the best (currently only?) shot of adding Rails to that list. I hope they make it, but I don't think I would bet on it.Anonymoushttps://www.blogger.com/profile/08698131967747444207noreply@blogger.comtag:blogger.com,1999:blog-5819005.post-61439116572536872782008-04-25T19:28:00.000-07:002008-04-25T19:28:00.000-07:00From a quick question on IRC, I got that http://yf...From a quick question on IRC, I got that http://yfly.com/ is running a sharded setup with Rails. I talked to a few other folks doing similar setups and one guy readying up a reusable plugin for this (who mentioned that the implementation for his particular approach was pretty trivial).<BR/><BR/>But that doesn't even matter. Most people don't shard because it's a pain in the ass (as a technique, not as an implementation) and increases the complexity of your setup considerably. Thus, you usually only take this road as a last resort when you can't go further with your replicated database setup.<BR/><BR/>Therefore, the fact that not more people are doing sharding is probably more easily explained by the fact that most people don't need sharding. That says absolutely zero about the capabilities of the tools these people use to do sharding.<BR/><BR/>So it seems we're down to the fact that your "empirical evidence" was one conversation with one team some time in the past who thought another technique looked easier to them. Now there's an iron-clad argument you can take to the bank :).Unknownhttps://www.blogger.com/profile/03745311034806593667noreply@blogger.comtag:blogger.com,1999:blog-5819005.post-15729416966613946282008-04-25T16:42:00.000-07:002008-04-25T16:42:00.000-07:00I am sure you are much more of a Rails expert than...I am sure you are much more of a Rails expert than, at least I would hope so. Thus perhaps I am entirely mistaken. Please point me to web scale site that is running Rails and has federated their database. I don't know of any, but perhaps such a site exists and I am just ignorant.<BR/><BR/>Oh, and you asked "Have you talked to people who've tried to do sharding with Active Record and given up because it was too hard?" Actually my statements were based off my conversations with the Twitter engineering team last fall. Now I hate to quote other people, I only mention it because you asked. They mentioned several Gems that had been created to address this, but that they had all fallen short. They also mentioned trying to do it themselves, but decided that it would be easier to build some messaging and caching infrastructure instead.Anonymoushttps://www.blogger.com/profile/08698131967747444207noreply@blogger.comtag:blogger.com,1999:blog-5819005.post-6512942952361554772008-04-25T14:45:00.000-07:002008-04-25T14:45:00.000-07:00What empirical evidence? Have you talked to people...What empirical evidence? Have you talked to people who've tried to do sharding with Active Record and given up because it was too hard? <BR/><BR/>I know of a good number of startups that run one-database-per-customer setups (which is similar to sharing, albeit at a different tier).<BR/><BR/>It's entirely possible to run sharding setups with Rails. And it's definitely not something that I run away from. We're going to do sharding the day we have to with Basecamp. <BR/><BR/>For now, we don't need to, so it would amount to premature optimization to do so, but that doesn't mean there's anything inherent in Rails that makes it impossible. That's preposterous.<BR/><BR/>I fail to see any substance to this rambling (other than it's easier than ever to build big sites, but scaling big sites is always hard regardless of the technology -- I certainly agree to that).Unknownhttps://www.blogger.com/profile/03745311034806593667noreply@blogger.comtag:blogger.com,1999:blog-5819005.post-17472620863575150172008-04-25T10:41:00.000-07:002008-04-25T10:41:00.000-07:00Perhaps I rambled a bit too much in that post. I l...Perhaps I rambled a bit too much in that post. I lost part of it because of a (scheduled) Blogger outage. <BR/><BR/>My point is that people think it is easier than ever to build a site. That may be true for toy sites, but all of the brilliant work of many technologists has not made it much easier to build a web scale site. I don't think many people realize this, and that includes both programmers and "technologists" like Michael Arrington. <BR/><BR/>I find it funny that you run away from the problem of federating your database. Master-master and master-slaves are just band-aids. Do you know how many databases we have at eBay? Or how many they have at Yahoo, Amazon, or Facebook? Now there are a lot more Basecamps in this world than there are Facebooks, granted. But if you are an entrepreneur, aren't you hoping to be a Facebook rather than a Basecamp? I'm pretty sure I know how the guys at Twitter would answer that question.<BR/><BR/>So sure, Rails is great for sites where 1 Database Server is all you will ever need. The problem is when you need more. It's not that hard to rip up the right parts of Java, PHP, or C# to get them to scale, but empirical evidence seems to indicate that it is not so easy with ActiveRecord.Anonymoushttps://www.blogger.com/profile/08698131967747444207noreply@blogger.comtag:blogger.com,1999:blog-5819005.post-54005322327101206312008-04-25T09:19:00.000-07:002008-04-25T09:19:00.000-07:00I don't think I understand your argument. Rails is...I don't think I understand your argument. Rails is flawed because it doesn't ship with default features to do sharding, but neither does any other frameworks, and btw you can't easily shard the Twitter model. That's doesn't seem to compute into a logical argument.<BR/><BR/>BTW, you can certainly scale the database a fair bit before you need to consider sharding. You can do master-master and you can do master-slave setups. If Twitter is still on a single database, it sounds like they haven't gone down that path yet.<BR/><BR/>When you do go for those setups, there is often database technology to solve the problem rather than framework technology. For MySQL, I've heard good things about mysql-proxy.<BR/><BR/>Finally, I call bullshit on "when you like beautiful code you can't write ugly code when you have to". At 37signals, we even dropped down to the ugliest of all code (C) when we felt that was a good way around a scaling issue.<BR/><BR/>You can still enjoy beautiful code for the 95% of the application that's unlikely to be the bottleneck, then get your work trousers on and do the last 5% in whatever way you need to make it fast enough. (For the majority of sites out there, you can even go 100% beauty as you're unlikely to be the next Twitter).Unknownhttps://www.blogger.com/profile/03745311034806593667noreply@blogger.com