Follow Datanami:
December 10, 2020

NoSQL Database Smackdown Takes to the Clouds

via Shutterstock

Customers want to get the most bang for their database buck, no matter whether they live on premise or in the cloud. However, as databases move to the cloud, performance and cost can be impacted in different ways. Now a new benchmark performed by Altoros attempts to shed some light on the matter.

Altoros was paid by Couchbase to compare how its cloud database offering compares to other cloud database offerings, including MongoDB Atlas and Amazon Web Services’ Amazon DynamoDB. That’s worth keeping in mind as you read the results, although Altoros said that payment did not impact the integrity of its test.

The goal of the test, which was a follow-up to a similar NoSQL database test Altoros performed about two year ago, was to measure the relative performance of the three NoSQL databases in terms of latency and throughput across three different cluster configurations.

The databases were loaded on clusters with six, nine, and 18 nodes, and subjected to queries from four different workloads: an update-heavy job with an even amount of reads and updates (Workload A); a short-range scan job with 95% scans and 5% updates (Workload E); a query with a single filtering option to which an offset and a limit are applied (Pagination); and a JOIN job with grouping and ordering applied (JOIN).

Altoros’ performed the tests on Amazon Web Services’ Amazon Elastic Compute Cloud (EC2) instances utilized the Yahoo Cloud Serving Benchmark. The company tried to approximate the Couchbase Cloud and MongoDB Atlas setups.

Results from the mixed read and update workload (Courtesy Altoros)

The MongoDB Atlas environment utilized M60 compute types with eight CPUS; 64 GB of RAM; 1,900 GB of SSD storage; with an IOPS rating of 5,700. The Couchbase Cloud setup utilized EC2 R5 Double Extra Large/r5.2xlarge instances with eight vCPUs; 64 GB of RAM, 200 GB of Amazon Elastic Block Storage (EBS); and up to 10 Gigabit of bandwidth. DynamoDB is a database service offered by Amazon, so there were few configuration options available; Altoros selected the default capacity mode.

In terms of cost, a number of variables were at play in Altoros’ comparison, including data transfer fees, gateway fees, load balancing fees, and storage fees. The relative costs for running the clusters on the different databases appear the most useful. For Couchbase, those monthly costs came out to be about $5,500, $8,300, and $16,600, respectively. For MongoDB, those monthly costs came out to be about $6,100, $9,000, and $17,600, respectively. For DynamoDB, those monthly costs came out to be about $6,100, $9,000, and $17,500, respectively. (Computers are expensive, no matter where you run them.)

Test Results

Couchbase Cloud outperformed the other two cloud offerings on the update-heavy job on nine- and 18-node clusters. It also beat the others in terms of latency, with 1 millisecond latency across all three cluster sizes, compared to 50 ms for MongoDB and 100 ms for DynamoDB.

MongoDB outperformed Couchbase and DynamoDB on the short-range scan job for the six-node cluster, but Couchbase had greater throughput for the nine-node and 18-node tests. Latency for Couchbase was also lower, according to Altoros’ results.

Results from short-range scan workload (Courtesy Altoros)

On the pagination workload, which was a read-only job with OFFSET and LIMIT filters, Couchbase Cloud also came out on top. DynamoDB was not able to compete in this test because it “requires every attribute in the index key schema to be a top-level attribute of a string, number, or binary type,” Altoros stated in its report. Since nested attributes and multivalued sets are not allowed in indexes, DynamoDB would have to perform a full table scan, which eliminated it from contention.

Couchbase Cloud dominated the read-only JOIN workload with 3x higher throughout and lower latencies on all cluster sizes relative to MongoDB Atlas. DynamoDB, once again, was AWOL from the results.

All told, Couchbase Cloud proved the most versatile database, according to Altoros. “In case of queries, Couchbase Cloud provides sufficient functionality to handle the deployed workloads,” the company wrote. “Furthermore, the query engine of Couchbase Cloud supports aggregation, filtering, and JOIN operations on large data sets without the need to model data for each specific query.”

“MongoDB Atlas produced comparatively decent results,” the company stated. “MongoDB is scalable enough to handle increasing amounts of data and cluster extension. Under this benchmark, the only issue we observed was that MongoDB did not support JOIN operations on sharded collections out of the box. This resulted in a negative impact and poor performance in JOIN Workload.”

Results from the pagination workload (Courtesy Altoros)

With two failed tests, DynamoDB was a different story. “Amazon DynamoDB is significantly different from the other databases, because it looks like a pure service without proper tuning,” Altoros states. “Only two parameters can be changed: read and write capacities. In this case, read and write capacities have been calculated depending on the cost of other databases for each workload…. DynamoDB did not take part in several workloads, because the data model had to be changed.”

Altoros data engineer, Artsiom Yudovin, who was the tech lead on the project, said that one of the reasons Couchbase fared so well is its multidimensional scaling capability.’

“While all the cloud databases offer optimization options, Couchbase Cloud demonstrates more flexible configuration possibilities, which has enabled engineers to fine-tune its engine for achieving the highest performance across several workloads,” Yudovin tells Datanami. “However, digging into the results of the report, one may find that MongoDB Atlas performed well, too, in many scenarios—for instance, on a large cluster (Workload E – short-scan ranges). At the same time, it’s worth noting that there are scenarios where all of the evaluated databases could have performed better.”

While Couchbase compensated Altoros for the report, it didn’t have control over the results, Altoros says.

Results from the JOIN workload (Courtesy Altoros)

“To make the benchmark study happen, Couchbase compensated for the computing resources used, as well as for the time spent on the evaluation,” an Altoros spokesperson said. “At the same time, we make sure that Couchbase—as a company—influences neither the final results of the comparison, nor its conclusions, so that the research stays vendor-neutral to provide confidence in the integrity of the results.”

You can download a copy of Altoros’ full report, titled “Performance Evaluation of NoSQL Databases as a Service: Couchbase Cloud, MongoDB Atlas, and Amazon DynamoDB,” at this link

Related Items:

Who’s Winning the Cloud Database War

Benchmarking NoSQL Databases

Three NoSQL Databases You’ve Never Heard Of

Datanami