Case Study
Ooma Architecture Review: Fault Tolerance Without the Performance Compromise
Ooma is a US-based telecommunications company known for their VoIP services in both homes and businesses across North America. Their CouchDB team approached us for help when their self-implemented, real-time monitoring revealed they were experiencing uneven loads across clusters. Their team also had questions about View performance and resource use.
Ooma was working with CouchDB defaults when it came to replication and conflict resolution, and custom scripts for compaction and server-side Erlang apps for call control and management.
Challenges
In addition to concerns the Ooma team had, we uncovered some bigger challenges getting in the way of their desired behaviour and performance. Here’s a snapshot we discuss in more detail further down:
- Uneven cluster loads.
- High latency between nodes, with nodes in one cluster experiencing frequent fabric worker timeouts.
- Remote nodes were often inaccessible.
What We Found
Because our Architecture Reviews are so extensive, these are just a fraction of the issues we were able to find and report on. We’re highlighting some of the things we see fairly regularly, so you may recognise issues you’ve experienced or see behaviour you’re after too.
View Performance
By taking a close look at how the Ooma team had structured their Views, we found opportunities to merge some and eliminate redundancy using the key structure Ooma already had in place. We also advised where to opt for Mango indexes instead of Views where they would offer speed advantages.
Getting toward the architecture side of things, we saw some great opportunities for how CouchDB databases and Views storage could be better arranged for the Ooma team. In particular, we suggested storing databases and view indexes on separate drives.
Server Architecture
By splitting different kinds of network traffic using virtual network interface cards (NICs), we saw a chance for the Ooma system to both reduce bandwidth consumption of each load, and to better understand their respective proportions in the virtual environment. One reason this is important is to have a grasp on whether you might be encountering artificial cloud-provider limits when systems are oversubscribed.
Probably the most important observations we made of Ooma’s setup revolved around cluster distribution, or, in their case, how consolidation could enhance performance.
We often see teams aim for fault tolerance by distributing CouchDB clusters across data centers in different regions. This significantly increases latency in a cloud scenario, especially when the system is oversubscribed as we saw above. The problem is more common than you would think! Depending on CouchDB’s replication behaviour, in a cloud scenario the system operates at the mercy of its slowest-responding node. We shared a strategy with Ooma to help them achieve the fault tolerance they were after while consolidating for superior speeds either with a cloud or off-site approach.
Serious Issues
This section is where the Architecture Review shines! Here we often turn up things that teams may not yet have on their radar, or where we’re able to help distinguish more benign log messages from the ones that need attention and give a breakdown of how to respond to them in different scenarios.
For Ooma, a dig into their couch.log helped us give the team a rundown for identifying and resolving which nodes are struggling to respond quickly and which operations induce problems. In Ooma’s case, we uncovered both fabric worker timeouts and remote node failures and could provide playthroughs for both.
These are just a couple of the serious issues we found, we’re very happy to have found around a dozen more ways to immediately address issues for improved performance.
About the Architecture Review
All our Architecture Reviews come with a 5-business-day turnaround once we’ve received your details. It’s our most popular product, and we’ve only ever received accolades for the things we’ve been able to uncover — for example, we’re usually able to help teams achieve 3–4x CouchDB performance quite easily.
Not ready for the Architecture Review yet?
All our Architecture Reviews make use of Opservatory. It’s a CouchDB diagnostics tool we use to learn more about your database without ever exposing or touching your actual data. While the Architecture Review provides a much deeper analysis, our team hand-crafted Opservatory to help get started fixing issues and better tailoring CouchDB to your use-case.