Apache CouchDB 3.4.1 has been released, here is what this means for Neighbourhoodie Customers posted Monday, September 23, 2024 by The Neighbourhoodie CouchDB Team

The Apache CouchDB team has released CouchDB 3.4.1 a combined feature and bugfix release that includes, among a number of high-profile features, a slew of performance enhancements and bugfixes. This blog post goes into detail about what this means for Neighbourhoodie’s customers with concrete recommendations and tips for upgrades.

Overview#

The previous CouchDB release version was 3.3.3. By incrementing the middle digit in the version, the CouchDB team is signalling that this release includes features and bugfixes, but that changes that are incompatible with 3.3.3. are kept to a minimum and most applications are safe to adopt the new version right away.

What happened to 3.4.0?#

The eagle-eyed among you might realise that the next feature release after 3.3.3 should have been 3.4.0. You are correct, so what happened? — CouchDB 3.4.0 was almost ready to be released. At the last minute, while writing this very blog post, Neighbourhoodie realised that there might be an undesirable upgrade behaviour in 3.4.0. We worked closely with the CouchDB team and jointly decided to not announce the release of 3.4.0 and instead fast-track the release of 3.4.1 that addressed the behaviour.

What exactly was the problem? — CouchDB 3.4.0 (and 3.4.1) come with a feature that upgrades user’s password hashes to a stronger algorithm automatically. In 3.4.0 this feature was enabled by default. This meant that if you upgraded and tested the new version and then for whatever reason decided you have to downgrade again, some of your user’s password hashes would already have been upgraded. Those users would be locked out of their accounts until an admin would restore the old password hash from a backup.

Needless to say, this is an undesirable behaviour. The CouchDB team prides itself in producing upgrades that are safe to adopt. So while CouchDB 3.4.1 retains the feature, it is disabled by default. Both the CouchDB team and Neighbourhoodie recommend to enable it as soon as you are safely running on 3.4.1 and no longer have a need to downgrade.

Upgrading#

As always when upgrading CouchDB, do take a backup of each node you are upgrading right before you do the upgrade.

If you are running version 3.3.0 or later, we recommend upgrading each node in-place. That means stopping CouchDB on the node, installing the new version, and starting it again before moving on to the next node. Wait a bit in between each node upgrade to have internal replication catch up the new node. If your upgrade process takes some time, consider putting the node into maintenance mode until it is caught up with the rest.

If you are still using an earlier version of CouchDB, or if you have any other questions about upgrading, do get in touch, we can help

Breaking Changes#

As outlined above, incompatible changes are kept to a minimum, but there are two small items you need to be aware of to make sure you are not relying on at the moment:

1. JSON Web Tokens (JWT) now require a valid exp claim by default. Auth tokens should expire by default and the new default configuration reflects this security best-practice. Administrators can disable the check by changing the required_claims configuration back to the empty string. The CouchDB team also recommends adding nbf to the required claims if you know your JWTs include them.
2. The _changes feed option combination of feed=continuous&descending=true&limit=N produced nonsensical behaviour. Instead of waiting for more updates and then sending the first few updates over and over, the request will now terminate after all existing update sequences have been sent to the client. At this point CouchDB can be sure that no new rows will occur at the beginning of the _changes feed.
3. If your application creates new entries in the _users database in the normal course of operation, like in a database-per-user situation, please note that the default password hash has been changed to a more robust variant. If for any reason you have to roll back to an earlier release, users created after the initial upgrade to 3.4.1 won’t be able to log in on the previous version. If you want to keep the possibility of a rollback, you can set the hashing algorithm to the previous default after upgrading by setting [chttpd_auth] hash_algoritm = sha. We strongly recommend setting this to sha256 (and enabling the auto-upgrade feature) after you have determined you don’t need to roll back any more.

New Features#

Nouveau: A New Full-Text Search Plugin#

This new feature deserves its own separate post.

QuickJS: An Alternative JavaScript Engine#

This new feature also deserves its own separate post.

xxHash: Faster Database Integrity Checks#

CouchDB ensures the integrity of any data it stores by attaching a checksum of the data when writing data to disk and validating it when reading data from disk. That way CouchDB knows the data has not been tampered with by other programs or SSD or hard drive faults.

Up until CouchDB 3.4.1 the default checksum algorithm has been md5. While md5 is deprecated for security relevant operations, the way CouchDB uses it remains a safe use of md5. But: md5 is rather old and there are faster alternatives available.

Enter xxHash. In CouchDB 3.4.1 you can change the default md5 algorithm to xxHash. Upsides include: it is ~30% faster for read and write operations for larger documents (128k and up) and no slower for smaller documents.

Why isn’t this enabled by default? — Excellent question! The answer lies in a mechanic that database operators prefer to have available when upgrading to a new version: rolling back to the old version in case something goes wrong with the new version.

For that to work, the old version needs to be able to understand the same data format as the new version. Now imagine you upgrade to the new version, it starts using xxHash, you try out a few things and realise you have to roll back. Now you have xxHash checksums in your database file and when you roll back, the old version does not understand them. Data integrity validation will fail and your only option is going to be rolling forward again (or better yet not rolling back in the first place).

The CouchDB project might make xxHash the default in the future when it can be sure, most users have upgraded to a CouchDB version that understands xxHash so it can be rolled back to.

We recommend not to enable xxHash until you have verified you have no need to roll back any deployments or you are okay with rolling back manually by setting up your old version of CouchDB and replicating data to it from CouchDB 3.4.1.

PBKDF2-SHA256: More Secure Password Hashing#

CouchDB stores your admin and user passwords as a PBKDF2 hash to make sure they cannot be exfiltrated as plain-text and to prevent offline password attacks. Up to CouchDB 3.3.3, the precise description of the algorithm used is PBKDF2-SHA1. While SHA1 has also been deprecated for use in security algorithms, this also remains a safe use of SHA1, but again, we can do better: PBKDF2-SHA256 is the state-of-the-art password hashing algorithm and CouchDB 3.4.1 uses it by default.

In addition, CouchDB can automatically upgrade password hashes in the SHA1 format to SHA256 when users authenticate. This saves you a manual upgrade task.

Higher Iteration Count#

To further shore up password security, CouchDB 3.4.1 raises the PBKDF2 iteration count to the industry standard 600000. The goal of the iteration count is to configure how long it takes to produce a hashed version of a plaintext password during authentication.

This has to be increased as hardware advances to ensure brute-force attacks against your passwords are not successful.

This also means, if you send CouchDB a plaintext password for authentication, it will take longer than it did with CouchDB 3.3.3. If you are authenticating to CouchDB with a plaintext password for every request, as with HTTP Basic Auth, CouchDB has to validate this password by hashing it with PBKDF2-SHA256 and the 600000 iterations.

If CouchDB didn’t help you in two important ways, this would mean that you would only be able to send a couple of requests per second to CouchDB and totally overwhelm your server.

So how does CouchDB 3.4.1 help with this:

1. The newly created password cache keeps an in-memory cache of low-iteration hashed passwords to compare plaintext passwords against after a successful authentication. The cache is only ever held in memory and entries last for at most 10 minutes (configurable). In addition, the entire cache can be disabled, if you are not using plaintext-authentication at all.
2. The newly created automatic account lockout feature will temporarily prevent a user from authenticating after a number of failed attempts. This ensures that malicious actors can not issue a denial of service attack against your server. Accounts are automatically locked after five failed attempts for a duration of five minutes beginning with the first failed attempt. After five minutes, the account is automatically unlocked. All thresholds here are configurable in the [chttpd_auth_lockout] config section.

What should I do?#

Neighbourhoodie recommends upgrading to CouchDB 3.4.1 at your earliest convenience.

Do consult the full change log for additional information.

Finally, if you have any questions, do get in touch.