Kristina Chodorow's Blog
With a name like Mongo, it has to be good
I just got back from MongoSF, which was awesome. Over 200 Mongo geeks, three tracks, and language-specific workshops all day.
I got to the conference early and took a picture of the venue... an hour later it was packed.
The highlight, for me, was Eliot Horowitz’s talk on sharding. He set up a MongoDB cluster of 25 large EC2 instances and started hammering them.
He pulled up an incredibly snazzy sharding GUI (okay, I wrote it, it’s not actually that snazzy) that displayed a row of stats for each shard. Each shard had about the same level of operations per second, so you could see that Mongo was doing pretty well distributing the data across the shards.
Eliot scrolled down to the bottom of the stats table where it showed the total number of operations per second across all the shards. The cluster was doing 8 million operations per second.
8,000,000 operations per second.
The whole audience burst into applause.
That’s about 320,000 operations per second per box, which is about what would be expected for Mongo on a powerful server (like a large EC2 instance).
Cool things about this:
- If your app needs more than 8 million ops/sec, you can just add more machines and Mongo will take care of redistributing the load.
- Your app doesn’t need to know if it’s talking to 1, 25, or 7,000 servers. You can focus on writing your app and let Mongo focus on scaling it.
Speaker's dinner the night before
If you missed out on MongoSF, there are a bunch of other Mongo conferences coming up: MongoNY at the end of May and MongoUK and MongoFR in June.
| This entry was posted by kristina on May 5, 2010 at 2:58 pm, and is filed under MongoDB. Follow any responses to this post through RSS 2.0. You can leave a response or trackback from your own site. |
No trackbacks yet.
Return of the Mongo Mailbag
about 4 days ago - View Comments
On the mongodb-user mailing list last week, someone asked (basically): I have 4 servers and I want two shards. How do I set it up? A lot of people have been asking questions about configuring replica sets and sharding, so here’s how to do it in nitty-gritty detail. The Architecture Prerequisites: if you aren’t too
Sharding and Replica Sets Illustrated
about 3 weeks ago - View Comments
This post assumes you know what replica sets and sharding are. Step 1: Don’t use sharding Seriously. Almost no one needs it. If you were at the point where you needed to partition your MySQL database, you’ve probably got a long ways to go before you’ll need to partition MongoDB (we scoff at billions of
There must be 50 ways to start your Mongo
about 4 months ago - View Comments
This blog post covers four major ones: Single server Master-slave Replica pairs Sharding Feel free to jump to the ones that interest you (for instance, sharding). Just start up a database, Ace Starting up a vanilla MongoDB instance is super easy, it just needs a port it can listen on and a directory where it
Once and Future Presentations
about 4 months ago - View Comments
On Monday, I gave a presentation on MongoDB to the San Francisco MySQL user group. It was a lot of fun, you can watch the recording on ustream: Streaming Video by Ustream.TV Apparently the audio was buzzy (I haven’t actually listened to it myself yet). The audience especially enjoyed this slide about MySQL’s current situation:
Sharding with the Fishes
about 5 months ago - View Comments
Sharding is the not-so-revolutionary way that MongoDB scales writes (it’s very similar to techniques described in the Big Table paper and by PNUTS) but many people are unfamiliar with what it is and how it works. If you’ve seen a talk on MongoDB or looked at the website, you’ve probably seen a diagram of sharding
Sleepy.Mongoose: A MongoDB REST Interface
about 6 months ago - View Comments
The first half of the MongoDB book is due this week, so I wrote a REST interface for Mongo (I’m a prolific procrastinator). Anyway, it’s called Sleepy.Mongoose and it’s available at http://github.com/kchodorow/sleepy.mongoose. Installing Sleepy.Mongoose Install MongoDB. Install the Python driver: $ easy_install pymongo Download Sleepy.Mongoose. From the mongoose directory, run: $ python httpd.py You’ll see
MongoDB Survey Results
about 6 months ago - View Comments
A few weeks ago we did a survey and I thought people might be interested in the results. Check out my post on how people are using Mongo on the official MongoDB blog.
“Introduction to MongoDB” Video
about 6 months ago - View Comments
This is the video of the talk I gave last Sunday at the NoSQL Devroom at FOSDEM. It’s about why MongoDB was created, what it’s good at (and a bit about what it’s not good for), the basic syntax for it and how sharding and replication work (it covers a lot of ground). You can
Mongo Mailbag #2: Updating GridFS Files
about 6 months ago - View Comments
Welcome to week two of Mongo Mailbag, where I take a question from the Mongo mailing list and answer it in more detail. If you have a question you’d like to see answered in excruciating detail, feel free to email it to me. Is it possible (with the PHP driver) to storeBytes into GridFS (for
FOSDEM
about 6 months ago - View Comments
I gave a talk at FOSDEM (Free and Open Source Developers European Meetup) this morning: “Introduction to MongoDB”. It went pretty well, I think. Slides are up at scribd.com and it was recorded, so the video for it should be somewhere soon (I’ll update when I find out where). The trip across the Atlantic was
Comments are closed.
about 4 months ago
The sharding talk sounds interesting. Will any videos be posted?
about 4 months ago
Nice! was this achieved using the experimental auto-sharding functionality or is this readily available in the stable version?
about 4 months ago
@Chris there was video taken, I’m not sure if/when/where it’ll be up, but I’ll link to it if it is. There are slides at http://www.slideshare.net/mongosf, but these don’t include the demo, of course.
@Slimo this was done with the latest 1.5 branch build, which is what I’d recommend using for anyone trying sharding.
about 3 months ago
Looks like videos are up at http://www.10gen.com/event_mongosf_10apr30
about 3 months ago
Any idea on what was the setup of the cluster and what kind of workload it was?
320k ops seems awesome but if the dataset was small (like fewer than 1 millions documents) and all in memory with only reads, the results are not that surprising.
about 3 months ago
IIRC, it was ~20,000 each of writes, deletes, updates, inserts, commands, & get mores and ~200,000 reads. I think that it was a quite large data set, but I’ll check with Eliot tomorrow.
about 3 months ago
I agree – an absolutely awesome demo!
@Nicolas, I think it was five shards of five replica-set servers each. I might misremember.
about 3 months ago
was at MongoSF.. the sharding demo was definitely cool, a lot of the other talks/demos were very good as well. overall a great conference, esp. for the price. thanks again
about 3 months ago
@Noah unfortunately, replica sets aren’t ready yet. So, it was just 25 shards with no replication, which was actually kind of a problem because Amazon thought that we were DDOSing ourselves, so kept shutting down our instances, thus breaking the demo.
About replica sets: I am totally in love with them, so if you’re interested subscribe to my RSS feed because I’ll be doing a blog post on how to use them ~3 seconds after they’re available in the nightly build.
about 3 months ago
8 million ops/s is 320000 ops/s per instance.
As you wrote this is far above a local Mongo, Redis or OrientDB instance.
And this on EC2 with more network letency??
Sounsd impossible. What’s behind this magic?
Regards
Stefan Edlich
about 3 months ago
As I wrote, ~320,000 is reasonable for a powerful server. There were lots of clients sending requests and MongoDB can do reads in parallel, which is what there were the most of.
about 3 months ago
Kristina,
Those numbers sound amazing, but there are a lot of missing details
.
1. what was the concurrency level?
2. where there concurrent CRUD operations? (basically I expect things to behave quite differently when sets of READS are (almost completely) separated from sets of WRITES)
3. what was the average data size/op?
You know such benchmarks can be “fabricated” and missing information tends to encourage that kind of thought.
about 3 months ago
1. I have no idea.
2. Yes.
3. I have no idea.
I think people are taking this post a bit too seriously… I just thought it was a really cool demo of what Mongo’s going to be able to do: scale horizontally to practically infinite capacity. It’s certainly not meant to be a benchmark.
I think we’re working on a more interesting application (one has actual functionality other than hammering the database) that we’ll open source and use as a demo in the future. We’ll also do some real benchmarks against sharding before releasing 1.6, obviously.
about 3 months ago
i think it's blog class ho, meet girls on webcams for free, %O,
about 3 months ago
Do you have any more details on the “incredibly snazzy sharding GUI” that you mentioned in the post? I'm interested in doing some load testing to see how MongoDB will perform for my system and it sounds like that GUI was giving some pretty valuable data — anyway to get access to that program? Thanks!
about 3 months ago
Unfortunately, it's not publicly available yet, but we'll probably release it when 1.6 comes out. It doesn't give any information you can't get from the shell (it basically runs the serverStatus command once per second for each machine), it just presents it in a pretty way. I'm hoping to add a chunk viewer and stuff, too, but it's very, very, very alpha at the moment.