From Dig Sites to Data Sync: iDAI.field and Offline-First Systems for Archaeology posted Thursday, March 20, 2025 by The Neighbourhoodie Team

Reliable data collection during archaeological field work in remote locations comes with a multitude of challenges. Even in the most isolated dig sites, capturing, storing and synching need to happen seamlessly and with zero data loss. Offline-First can solve half the problem. What about the half that needs to bring together teams and findings from across the globe?

We interviewed Lisa Steinmann at the German Archaeological Institute to learn about the app her team developed to help archaeology research groups record — and compare — their discoveries when conditions for connected tech are rarely guaranteed.

We talk about flexible data design for diverse methodologies, what really boosts collaboration and cooperation and how using CouchDB as a server simplifies things for non-technical contributors.

Could you describe what the German Archaeological Institute needs from a database technology and what your biggest IT challenges are?#

The German Archaeological Institute is an internationally active research institution subordinate to Germany’s Federal Foreign Office. With branches in Germany (Berlin, Bonn, Frankfurt am Main, Munich), as well as abroad (Athens, Istanbul, Cairo, Madrid, and Rome) the DAI is active in many different regions and countries. The departments and commissions oversee further field offices in Tehran, Sana’a, Baghdad, Damascus, and Beijing, as well as research centres in Lisbon, Ulaanbaatar, and Budapest which all contribute to a closely knitted cooperation with colleagues and institutions in the host countries. Individual local offices and local excavation houses complement the institute’s global research infrastructure.

The global research infrastructure is a crucial point: Located at the head office in Berlin, the central scientific services (Zentrale Wissenschaftliche Dienste/ZWD) support many online services unified under the name “iDAI.world”. This includes norm data, object databases, publications and library catalogues. We aim to support individual research projects of the DAI in using these systems, managing and storing their data, and finding suitable and consistent digital workflows. The department of software engineering/archaeoinformatics is a smaller unit within the ZWD, that takes care of developing the systems (including iDAI.field) according to the requirements and feedback of our researchers at the DAI.

In the end, we have many different requirements for many different database technologies employed in our various systems and services! Due to the international nature of our work, support for multiple languages and translations is always a key point. If we only think about a database that is to be used during archaeological field work, databases that can be used offline as well as online without any loss of information are crucial. Projects also need enough flexibility in the database to accommodate their different research methods and thus practices of data recording. Not to mention: It also has to be intuitive and easy to use, since projects often have fluctuation in their teams due to taking in students for seasonal field work and staff regularly changing after one project ends and another begins. This already says a lot about the challenges we are facing as developers.

What is iDAI.field and what does a fieldwork team’s workflow look like with versus without it?#

iDAI.field — which we now gradually start to simply call Field to be more inclusive of projects that are not associated with the German Archaeological Institute — is a documentation software for archaeological excavations. Using Field, archaeologists can document all their findings on an excavation, be it out in the field where they excavate new sites, or inside the excavation archives where they record the objects found on previous excavations.

A typical workflow on an archaeological dig starts at its most basic step with removing some soil. Soil is usually classified into different “features” or contexts by defining discolourations or shapes one can see in the earth. A feature is given a number and all observations made by the excavators are recorded. Without software like Field, archaeologists would keep a list of feature numbers on the dig where every one had to grab a number and record a name or basic data. If a group uses Field in a local network on their dig, they can see which numbers are taken and which are free, and immediately start recording without having to scribble hardly legible notes on a dirty piece of paper all of them share. Then a supervisor would describe their observations on a different piece of paper. Some excavations use structured forms with some lists to cross off and some areas for descriptions, others rely on freely recorded notes in a diary-style notebook.

With Field, they get pre-structured forms and can type and share all observations immediately with their colleagues. They can even search the database for similar features, or look up the dating and finds of a previous excavation without having to sift through thick folders of dusty paper. Together with the descriptions and observations, the locations of these features may be recorded in drawings made by hand in the field, or with measuring equipment such as advanced GPS devices or so-called total stations, which can determine precise coordinates. Field provides a map feature where such coordinates can be imported, though this typically does not happen directly on the dig. Of course, photographs also play a huge part in archaeological documentation. They can be attached to all records in Field. The biggest impact is probably the ability to share and create interoperable data across all collaborating supervisors, researchers and other personnel. That really gives a boost to collaboration and cooperation!

Using Field, the whole workflow can be achieved on a basic notebook ([the computer kind] not the pen & paper kind) and immediately shared with colleagues. However, we see varying degrees of digitisation in the workflow on different projects. Some projects still prefer pen & paper recording, and transfer their data into Field later on, but others go fully digital directly on the dig. That also depends on the kind of environment they are in. If you are out in the field without electricity, it is sometimes next to impossible to allow for a really digital working process. Weather conditions such as rain or extreme heat can be really hard on devices.

What strategies have you employed to maintain data compatibility when collaborating with partners and other institutions?#

We do not manage the data that research projects produce, and we do not want to ‘prescribe’ a certain set of definitions to them. All research projects have their own goals and methodologies, and we respect their various choices on how to structure their data. It is a reality of archaeological research that no two excavations or projects can truly be the same, as archaeology constantly wanders along the narrow ridge between sciences and humanities.

That being said, while Field allows a large degree of freedom it also has its own assumptions on hierarchy of information, processes and objects that are recorded on any excavation. All projects that use Field do have to conform to those assumptions and are thereby comparable to a certain degree. As the most basic example, the data model provides a category “Feature” and a category “Find”. A “Feature” is any unit of soil or even structure that has been excavated. It corresponds to a part of earth being removed during the excavation that will contain inclusions, maybe other features, or human-made artifacts. These artifacts are considered “Finds”, and a find can be a pottery vessel, a stone axe, a small bead and similar things. Users can add sub-categories to the “Feature” or “Find” definitions, and thus add their own descriptions and types of data that they want to record, but they are always denoted as a “Feature” or a “Find” and Field restricts which categories and their corresponding sub-categories can be placed at which level in the hierarchy. Thus, at least on the very basic level all projects that use Field are highly comparable.

As developers of this application, we collaborate with internal and external projects to define some default forms and fields that can be recorded. Projects using Field are free to use these default forms, and we have seen a lot of projects re-using and accepting many of the fields configured by default. Sometimes they alter them slightly, but this still reveals a lot of common ground between even very different projects. Especially for rather universal fields like measurement of width or height of an object, or the absolute dating to record the age of something, we see that projects gladly employ these defaults! While the configuration editor allows projects to define their own forms, it guides each entry into a comparable format by providing pre-defined value lists that can be extended.

In the end, we suggest as much consistency and re-use as possible without infringing on each project’s individual recording practices. We are currently working on a renewed publication platform that will simply demonstrate the value in reusing data models and structures, which will be another big step in the direction of more consistency and collaboration.

What has been an achievement of iDAI.field you’re most excited about? Is it something you expected or did it surprise you?#

It is always a happy surprise to see so many people are interested in and using the application, which was originally meant for internal use by the DAI but has now grown a small community of projects worldwide that are not necessarily associated with us at all. It was also great to gain the GBV, the common library network of seven German federal states, as a partner for cooperation on the development of Field! In the context of the archaeoDox system operated by the GBV for the Archaeological Museum Hamburg, our software has been used in the field of monument preservation since 2023. Field is also being used as part of the IT infrastructure in the KulturGutRetter project and evaluated in the context of NFDI4objects, which is great to see and will eventually lead to an even larger community.

By far the biggest achievement for us at the DAI is the fact that we have so many of our projects use the same software to document their findings! While data management was a bit harder to do in the times before Field, we can now provide centralized, regular backups of all the research data collected with Field and provide a way of keeping it safe and archiving it for our projects. Using Field Hub, the online synchronization server, all the projects can collaborate with their various colleagues on their data recording — even if they are scattered across the globe and only meet up for the excavation campaigns in the summer.

What lessons have you learned about managing data in challenging field conditions that other professionals in archaeology — or even other fields — might find useful?#

We have noticed how crucial the possibility of immediate cooperation and collaboration on data and the potential to “share” it with other colleagues is while everything is still a work in progress. One especially valuable thing Field is able to facilitate is the offline synchronization. It is very easy to set up even without technical knowledge, one just needs to run the desktop clients in a common network and enter an IP address of one of the clients. Every client can be a makeshift server — and the fact that this works in local networks is essential for excavations, since they do not always have internet access. And even if one person has to work outside of the network for a while: synchronization can happen immediately as soon as the device is connected to the same network as the others again. This way, everyone has access to all the data at all times. “Offline first!” has always been very important to us.

Thank you Lisa!#

The Neighbourhoodie team is very proud of this collaboration and its result. When the DAI team approached us, they wanted to upgrade the mobile Field application, perform a feature analysis and identify outstanding requirements. They were already working with CouchDB and PouchDB, which made us a great fit for the project.

The iDAI Field mobile app, which is a kind of light version of the more feature-complete desktop version, was built with React Native and Expo CLI and is compatible with both Android and iOS. The app also leverages Three.js, a 2D/3D JavaScript library that was used to render shapes, map layers and components without using native map SDKs from Google and Apple. Our team’s first task was to give the existing codebase some attention and perform some upgrades.

When we got our hands on it, we discovered that the mobile app was broken and dependent on shudderingly old packages. Our immediate goal became refactoring code, updating packages and getting the app up and running again. This activity didn’t come without its challenges though.

The expo packages and react versions were so old, we essentially created a new app in order to upgrade to the latest package versions. During refactoring, we also employed a different routing paradigm called Expo Router.

We successfully ensured that the app is able to render key screens without crashing. Hooray! Because our work didn’t focus on furnishing the tool with new features at this stage, we’ve been able to share a concept paper with the iDAI Field team outlining recommended development going forward. Accessibility and user experience are always a high priority for our team, so in addition to data management and multi-language input support, we also proposed:

• Image synchronisation, which also supports thumbnails for linked resources, project views and introduces file size limitation.
• Category-specific identifier prefixes, which automatically adds a prefix to resources in a category when selected.

Thank you to Lisa Steinmann for talking to us about what it takes to make a successful app for remote field work. We can’t wait to see what you do next — and hope we can help with the project!

CouchDB and Offline-First are technologies we love working with because of the impact they can have by making data safer and more reliably accessible. If you or a research team you work with want to give your apps offline powers, reach out! Our team is always curious.

« Back to the blog post overview