Debunking Myth of RAID 10 as Best Practice on AWS

For the most amazing performance on AWS, use RAID 10 with provisioned i/ops (henceforth piops).  A combination of RAID 10 and piops achieved read performance equaling the sum of all provisioned i/ops. Thus, 4 x 1000 piops disks achieved 4000 read i/ops.

However, we live in a world that optimizes more than one variable. There are always costs.

On AWS, we find:

  • RAID 10 improves performance on non-optimized queries and large write volumes.
  • RAID 10 is prohibitively expensive on a cost-to-performance ratio.
  • Optimizing schema yields exponentially better return on investment than RAID 10.
  • RAID 10 makes EBS snapshots impossible — disabling simple disk backups.

Given time, always optimize database usage, and ditch RAID 10 performance requirement (see MongoDB Performance & Scalability — Schema design is more important than anything else).  Every checklist for AWS and MongoDB includes RAID 10.  This inclusion is a relic of the past and oversells the ability of RAID 10’s usefulness on the AWS platform.

We benchmark RAID 10, table scans, and optimization strategies.  The optimization strategy returns 50x the performance gains.

Drive Configurations

  • RAID 10 with 4 x 1000 piops drives – $446.80
  • Single drive with 4000 piops – $446.80
  • RAID 10 with 4 x 4000 piops drives – $1,976.48

The Gnarly Test

We are doing some truly gnarly things to MongoDB – like 66% table scans and 33% updates.  Using YCSB for our benchmarking tool, our configuration looks like:

readproportion=0.0
updateproportion=0.5
scanproportion=1.0
insertproportion=0

With this test, the limiting factor will be drive performance.  To ensure we stretch the disk, we are using a 22GB dataset, which exceeds the RAM capacity on the server of 7.5 GB.  Table scans on this test will churn the disk.

Results

  • RAID 10 w/ 4 x 1000 piops drives averaged 13.4 ops/sec, with 16.1 ops/sec max
  • Single 4000 piops drive averaged 21.4 ops/sec, with 23.4 ops/sec max
  • RAID 10 w/ 4 x 4000 piops drives averaged 38.3 ops/sec, with 43 ops/sec max

These results are awful, awful, awful! We are paying $1,976.48 per month for 38.3 ops/sec?  On unrestricted SSD backed physical hardware, we did not achieve better performance.

To prove our point that optimization trumps RAID 10, we will run 2 more tests.

A Reasonable Mix

With the “Gnarly Tests”, we performed 100% table scans. Now, to swing in the other direction and perform 50% inserts, 25% updates, and 24% optimized reads, and 1% table scans.

  • RAID 10 w/ 4 x 1000 piops drives averaged 282 ops/sec; maxed at 349 ops/sec
  • Single 4000 piops drive averaged 310 ops/sec; maxed at 398 ops/sec
  • RAID 10 w/ 4×4000 piops drives averaged 290 ops/sec; maxed at 338 ops/sec

Reducing table scans to 1% table scans yielded between 9x and 21x return on performance.

An Optimized Mix

What difference is an addition of 1% table scans?

  • RAID 10 w/ 4 x 1000 piops drives averaged 1291 ops/sec; maxed at 2123 ops/sec
  • Single 4000 piops drive averaged 1327 ops/sec; maxed at 2114 ops/sec
  • RAID 10 w/ 4 x 4000 piops drives averaged 1988 ops/sec; maxed 2212 ops/sec

Removing all table scan yields between 58x and 113x performance over the Gnarly results.  Wow!  We knew table scans were significant, but this is amazing.

Straight Optimized Reads

Want to create the headline grabbing benchmarks?  Use 100% optimized reads; or 50% optimized reads and 50% optimized updates.  Below is 100% reads:

  • RAID 10 w/ 4 x 1000 piops drives averaged 6741 ops/sec; maxed 6940 ops/sec
  • Single 4000 piops drive averaged 6566 ops/sec; maxed 6816 ops/sec
  • RAID 10 w/ 4 x 4000 piops drives averaged 6471 ops/sec; maxed 6757 ops/sec

When performing the straight optimized reads, the limitation was still the disk.  To get to the next level of performance, order more RAM.  By having a dataset fully in RAM, we could hit 15,000 – 25,000 reads per second.

The Recommendations

RAID 10 is for table scans.  RAID 10 with provisioned i/ops is expensive. If you are running a database requiring table scans, RAID 10 will be your expensive savior.  If you have very deep pockets, or a small dataset, covering your data size with RAM is a faster alternative.

Deep pockets are a terrible answer.  Get a better return on investment by running an optimized database.

Moral of the Story

Optimize your MongoDB, save money and increase performance. Relying on RAID 10 to save you from unoptimized queries is an expensive path.  It will also provide with less than stellar results.

MongoHQ has the Slow Query Tracker and Profiler,  which logs and records all slow queries.  The profiler comes with a 1-click index create to help you optimized and often.  Get started optimizing early and often.