neighbourhoodie-nnh-logo

Partial Data Fetching on Initial Load with PouchDB and CouchDB posted Wednesday, July 16, 2025 by The Neighbourhoodie Team pouchdbCouchDBsynctip

If you’re building an offline-first or local-first app, you’ll eventually run into a problem concerning the initial loading of data onto the device: it gets a bit slow, and users end up having to wait for the initial sync to complete before they can do anything. This is such a prevalent issue that there was even a session on it at Local-First Conf recently. We’d like to share a strategy for dealing with “Partial Data Fetching on Initial Load” from experience with PouchDB and CouchDB. Get fast time-to-idle and synced data at the same time!

One of the frequent UX issues with offline-first and local-first apps is an increasingly sluggish Initial Sync: apps like this are often designed to rely exclusively on replication (aka syncing) between a remote database and a local, in-browser database. We like building offline-capable apps this way because the CouchDB/PouchDB replication protocol automatically handles disconnects, retries and all the other networking issues one could have. The reasoning is that if none of our app code talks to the remote DB directly, none of it can fail when our user (or our server, for that matter) is offline.

The problem with this approach is that the local database is empty on first visit, or may be quite far behind the server database on a later visit, necessitating the replication of a lot of data from the server to the client before the user can actually do anything. This leads to loading spinners that leave users waiting, sometimes for minutes, while the app is replicating data the user isn’t immediately interested in.

To make matters worse, this issue tends to compound over time as your databases grow, which means that it also tends to sneak up on you as your app gets older and more successful: while the UX was snappy in the early days, initial sync for new users now takes ages, and existing users that return after a bit of a hiatus are also stuck looking at loading screens.

A strategy we like to employ here is switching dynamically between the remote and the local database, using the remote to quickly fetch all data needed to render the current route, and then switching to continuous replication in the background. A classic use case is a field work or survey app, where users can add data while offline, but they also need to stock up on data before they set off: they might need up-to-date measurement points, task lists, map layers and the like in order to do their job. So fetching new data periodically is often a requirement. As more and more data is collected, there’s a danger that this “stocking up” gets slower and slower. There are a number of angles of attack here, but for now, let’s assume that our users are not ok with staring at spinners for minutes, and focus on the data fetching.

The Strategy

Step by step:

  1. Identify the data we need to render the current view.
  2. Fetch only that data directly from the server.
  3. Render the view.
  4. Start a replication process for the remaining data in the background, once the page is idle and the user has something to look at.

Optional

  • Write the data to the local database so you don’t fetch it again during the background sync.
  • Keep fetching data from remote until the background sync is done, or:
  • If navigation happens, cancel the replication process, then do steps 1 to 4 again in the new view.

If your users are severely bandwidth constrained, you might want to avoid doing the remote requests and the background sync at the same time.

Prerequisites:

  • We’re actually able to connect to the remote DB. Just looking at navigator.onLine isn’t enough, because the remote might be offline. If you’re going to validate the session or auth the user in these scenarios anyway, that check is implicitly happening already.

Partial Data Fetching on Initial Load with PouchDB and CouchDB

PouchDB has a little trick up its sleeve: it can store data in the browser and it can act as a library for accessing a remote CouchDB, but best of all: it can do these things interchangeably: the API is the same for local and remote databases. So we can swap them out whenever we like.

Let’s have a quick refresher on PouchDB in the browser:

import PouchDB from 'pouchdb-browser'

let localDB: new PouchDB('local-db')
// Assuming you’ve got the user’s credentials
let remoteDB: new PouchDB('https://database.club/', {auth: {username, password}})

const someDataFromLocal = await localDB.get('my-first-document')
const someDataFromRemote = await remoteDB.get('my-first-document')

Both of these databases are instances of a PouchDB database and expose the same methods, which means you can point the same app code at a local or a remote database, and you can switch at any time:

import PouchDB from 'pouchdb-browser'

let localDB: new PouchDB('local-db')
let remoteDB: new PouchDB('https://database.club/', {auth: {username, password}})

let currentDB = localDB // or remoteDB!
const someData = await currentDB.get('my-first-document')

// Now consume `someData` to render your view/component

Let’s say you’re rendering a dashboard view that fetches a bunch of data:

  • the user’s profile
  • the user’s organisation info
  • a summary of some data for some fancy gauges
  • a list of the user’s tasks

…whatever it may be: the current view or component should know which data it needs to render itself, and also be in charge of fetching that data. Here’s the cool bit: you can assign any local or remote database to currentDB, as long as the expected documents are in there, the view will just work.

When do we Actually do This?

One question that depends entirely on your use case is when to talk to the remote in the first place. We’re assuming this is an app that will be used offline a lot, so kicking off remote requests whenever we load a view and having to wait for them to fail might not work for you. There are several options:

  • Always do the remote request first, but have a low request timeout before falling back to the local DB.
  • Tie the remote update to a different, preceding remote request, such as session checking or authentication.
  • Remember when the last sync occurred, and decide on remote or local depending on how much time has passed since then.

Again, this depends a lot on what your app does and what your users expect.

Waste not, want not

We’ve already loaded a bunch of data into for our view from the remote database, and it would be a bit of a waste to not stash it in the local database, so it doesn’t have to get synced again. Usefully, PouchDB actually lets you add documents to a database in the same way the replication protocol does, by writing them to the database without giving them a new revision. To do this, pass the new_edits: false option to bulkDocs():

try {
  // Try to get the data we need for this view from the remote
  const dataWeNeedForTheView = remoteDB.bulkGet({docs: aBunchOfDocumentIDs})
  // Store it in the local database as-is so it doesn’t need to be synced again
  await localDB.bulkDocs(dataWeNeedForTheView, {new_edits: false})
} catch (error) {
  console.log('Couldn’t fetch new data from remote')
}

// Rest of your view code that always reads from localDB

This is a rather naïve example: in the real world, the data might be split up across several local and remote databases (and these might not correspond 1:1 with each other), and you might not know the bunchOfDocumentIDs in advance, so some sort of query will be necessary here. If you actually do know the exact _ids of the documents you want, you can also do a filtered replication with an array of _ids. Depending on how comfortable you are with JS views in design documents, you might even be able to solve all your partial data fetching needs with filtered replication.

Several Options for Partial Data Fetching

It turns out we actually have a bunch options for this approach:

  1. Replace the local db with the remote db and then let our view code pull data from that instance, indifferent to which database is actually behind it (that was let currentDB = localDB // or remoteDB!).
  2. Don’t switch databases, but instead try to load all necessary data from the remote, write it to the local database with {new_edits: false}, and then have the view or component only ever read from the local instance.
  3. Use filtered replication to just get the subsets of data you need for a specific route or view, and then switch to your regular replication method once that’s done.

Depending on your bandwidth constraints, you also have the option of doing the remote requests (options 1 and 2) and the background sync concurrently or sequentially.

Conclusion

PouchDB and CouchDB give you several options for dealing with the “Partial Data Fetching on Initial Load”, from seamlessly switching to a remote DB to using filtered replication or even a more custom approach that sits somewhere in the middle.

There are also design considerations that might help you sync the more important pieces of data faster, eg. splitting up your data into many databases (by type, date, importance etc.) and managing the replication of these independently and in a more bespoke fashion. This gives you much more fine-grained control over your data than simply kicking off a continuous replication for a single database.

For more on document and database design with CouchDB and PouchDB, check back for future blog posts or subscribe to our RSS feed or newsletter.

« Back to the blog post overview
og-image-preview