Flutterby™! : Web scale databases 2011-11-06 15:54:35.484487+00

Web scale databases

2011-11-06 15:54:35.484487+00 by Dan Lyke 4 comments

Don't use MongoDB, a rant on why you should use real databases for "web scale" applications.

#Comment Re: made: 2011-11-07 05:41:37.744268+00 by: igor' [edit history]

This is a somewhat misleading rant. Its unclear where it came from (supposedly 10gen) but some 10gen folks are saying it didn't. 10gen CTO + others refute a lot of those points here http://news.ycombinator.com/item?id=3202081

edit: after digging around it seem that this is a rant based on on older version of mongo (1-2 years) and is written by someone that didn't properly use it (i.e its not meant for heavy writes, etc). I am going to continue to use it where appropriate (mostly for schema-less things).

It seems to me echoing a lot of Philip Greenspun's "Use a database, not MySQL" rants of the late '90s. Which is a lot of why I chose the PostgreSQL path, and, of course, now MySQL actually is a real database.

It seems to me like every generation has to reinvent the database, usually because they're trying to overcome hardware limitations, and that there may be a few profitable years in doing that, but shortly the hardware catches up, so thinking long-term with a real relational database is the way to build products that'll really last.

Not that anybody cares too much about that, but I think about the first big system I worked on, where we did amazing things on a 4.77MHz PC XT with a highly tuned database, and how that system could have migrated on to become a web backend and didn't. The business processes underlying that system could have evolved to exist now if we hadn't put so much energy into running on even then questionable hardware. To take shortcuts like that to save hardware costs if you're deploying on anything less than ten thousand units seems like it's a bad basis for business decisions.

I mean, heck, even now I've moved from tied hashes on BerkeleyDB to basic ORM on SQLite for my simple one-offs, just because it saves coding time.

The question is what "web scale" means. You can get away with all sorts of things if yours is, oh, I'd say, within the lower 80% of the "average daily web traffic" number line. Hell, I can make you a website with flat-file data that will stay robust for a few hundred page hits a day - and let's face it, a lot of sites are lucky to see that much.

The reason my personal site is MediaWiki - which means it is MySQL - is not because of the performance advantage of using a database, which at my traffic levels I don't need. It's because of MediaWiki's other features.

I built a commerce site a few years ago for a company that rents and sells training videos. It's still working just fine, although I gather the code has been overhauled by someone else since then because I no longer have time to maintain it. I doubt they have switched databases. That was MySQL. But the site got a few hundred inquiries, and under a hundred orders, a day. If I'd said, "You need a much larger and more expensive database, because you're doing web commerce," I would have been misleading the customer.

I've found that if you are sane with how you use MySQL, it is absolutely rock solid and amazingly fast.

Clustering and "HA" might have us go to PostgreSQL, but Maria is not out of the question either.