10x CouchDB Performance Gains for a AAA Game Launch

One of our CouchDB Support customers, a AAA game developer, has used CouchDB for over ten years. They contacted us ahead of launching the newest edition of a major sports game with some performance concerns.

Setup Changes for Increased Performance#

• a six-node CouchDB cluster
• hosted on AWS EC2, 192 cores each
• with 768 GB RAM, NVME storage

Their goal was to achieve a 3x +1 node performance gain. However, their benchmark showed an increase of only 50%. At the same time, request latency increased tenfold.

Investigation#

First off, we benchmarked the whole system. Then, our team took a systematic approach and investigated each variable looking for bottlenecks, continuing to the next candidate as we went.

This is what we looked at:

1. Network Throughput
   1. Instinctually, we thought there may be an issue here. We discovered one in the AWS configurations.
   2. While this improved some benchmarks, the low overall system performance still held, so we dug deeper, but kept returning to this metric each time we made changes.
2. Input/Output
   1. We observed and manipulated the Erlang process message queue, which demonstrated it wasn’t a bottleneck.
3. HTTP Request Tracing
   1. By monitoring request and response completion times on each cluster, we identified two bottlenecks:
      1. One when accepting new requests.
      2. Another when collecting response body data from many shards across the cluster.
4. TCP Accept
   1. CouchDB uses mochiweb to listen to HTTP requests over a TCP socket, so we investigated this too and found a bottleneck:
      1. We increased the acceptor pool size to 32 from 16 and saw an unexpected change.
         1. Since reporting the issue this cause, this is now a default in CouchDB. Hooray!
      2. This revealed the system was sending more concurrent requests than the acceptor pool could handle.
5. CPU Utilisation & Process Scheduling
   1. We took another look at CPU utilisation on each CPU core using Erlang’s microstate accounting. We didn’t find a bottleneck.
6. Erlang Network Messaging
   1. Thanks to help from more of the CouchDB project members, we were able to track down message buffering settings and adjust these to bring request times down even further.

Solving Bottlenecks#

By using distinct metric tooling and revisiting key parts of the system as we made changes, we ended up with three bottlenecks to solve: total networking throughput, socket accepts and inter-cluster networking bandwidth. Our team was able to solve all three.

Customer Gains#

We’re very proud to have helped our client exceed the performance gains they had set out to achieve. In addition:

• Requests per second went from ~6000 req/s to 25,000 req/s, a more than 4x improvement in request throughput at minimum.
• Some benchmark variations allowed us to reach as much as 60,000 req/s, a 10x performance gain.

If you would like to learn more about CouchDB benchmarking and what we did to help our AAA game publisher, we’ve written about it extensively on our blog.

And if you would like us to help with your CouchDB performance, do get in touch. Most customers get started with an Architecture Review.