Tools & Data

Everything we build, we release

A method described in a paper helps the people who can reimplement it. A method released as working code, trained weights and documented training data helps everybody else — which, in historical research, is most people. All three of the systems below are open, and we would rather they were used than admired.

OccCANINE

What it does	_{Converts free-text occupational descriptions into HISCO codes automatically}
Coverage	_{13 languages, trained on 14 million description–code pairs from 22 sources}
Accuracy	_{96% accuracy, precision and recall}
Also outputs	_{OCC1950, OCCICEM and ISCO-68}
Authors	_{Christian Vedel, Christian Møller Dahl, Torben S. D. Johansen}
Get it	GitHub · Hugging Face · Paper

Occupational descriptions are the backbone of social and economic history — they are how a census tells you about class, skill, income and mobility. But they arrive as free text, in the language and spelling of the clerk who wrote them, and they have to be standardised into HISCO before anything can be compared across regions, countries or centuries. That work has traditionally been done by hand: it is slow, tedious, error-prone, and inconsistent between coders in ways that quietly undermine comparative research.

OccCANINE is a fine-tuned language model that does it automatically, turning work that took weeks into work that takes seconds. Because it is character-based rather than word-based, it copes with the misspellings, abbreviations and archaic forms that defeat dictionary-matching approaches.

DARE

What it does	_{Large-scale recognition of handwritten dates}
Published	_{International Journal on Document Analysis and Recognition}_{, 2026}
Authors	_{Christian Møller Dahl, Torben S. D. Johansen, Emil N. Sørensen, Christina E. Westermann, Simon F. Wittrock}
Get it	_DOI

Dates are the quiet workhorse of historical record linkage. A birth date, an admission date and a date of death are what let you decide whether two records describe the same person — and unlike names, they are unambiguous once read correctly. But they are written in dozens of formats, often partially, often with the year abbreviated or the month in words, and errors in them propagate directly into linkage failure. DARE is built specifically for that problem at scale.

HANA

What it is	_{A handwritten name database for offline handwritten text recognition}
Size	_{1.1 million images of personal names; 3.3 million names in total; 105,000+ unique names}
Source	_{Copenhagen police register sheets, 1890–1923 — all adults resident in the city}
Published	_{Explorations in Economic History}_{, 2023}
Authors	_{Christian Møller Dahl, Torben S. D. Johansen, Emil N. Sørensen, Simon F. Wittrock}
Get it	GitHub · DOI

Handwriting recognition research has long been held back by a shortage of large, labelled, real-world datasets — especially for proper names, which are precisely what historical linkage depends on and precisely what general-purpose language models cannot guess from context. HANA is a benchmark dataset built to close that gap, and it is openly available for anyone training or evaluating a recognition system.

OCR-Parser

What it does	_{A parameterised, end-to-end pipeline for turning scanned historical documents into structured records: multi-engine vision–language OCR, LLM-based field extraction, cross-engine alignment, and LLM judge adjudication with logged reasoning}
Built for	_{The HCNC corpus — 43,044 pages of Norwegian student yearbooks — and designed for reuse on other collections and languages}
Design	_{All OCR engines served behind standardised APIs; all LLM prompts stored as YAML templates, so the whole run is reproducible and adaptable to new document types}
Get it	GitHub · DOI

Where OccCANINE, DARE and HANA each solve one step, OCR-Parser is the harness that runs a whole corpus end to end — and, importantly, records how each value was arrived at. Every field in the output carries the number of engines that confirmed it, so a researcher can filter to the level of corroboration their question needs rather than trusting the pipeline wholesale.

That provenance layer came out of a finding worth reading before building any ensemble of your own: on this corpus, naive majority voting across three OCR engines made the data worse than the best single engine, because two of the three shared a model lineage and erred together. The HCNC page sets out what happened and what replaced it.

In development

CHAOS — Converting Historical Accounts into Occupational Scores

A replicable framework for generating occupational outcome estimates from historical occupational descriptions, extending the OccCANINE line of work from classification to measurement. With Matthew Curtis and Julius Koschnick.

Guidance on machine-learning bias in historical data

Not a tool but a protocol: how to detect and correct the systematic prediction error that automated transcription introduces into a historical dataset, and what it does to downstream estimates. See research theme 4.

Data resources built with partners

The Copenhagen Infant Health Nurse Records (CIHNR) cohort — an archive of handwritten health-visitor records turned into a documented, governed research cohort, with a published cohort profile so that other researchers can find it, cite it and apply to use it. International Journal of Epidemiology, 2023.

Datasets produced inside individual projects are described otheir own pages — see DANTRADE, Geolinking and Copenhagen Tax Books.

Help getting started
The tools are documented, but historical sources are idiosyncratic and the first run on a new archive rarely goes smoothly. If you are trying to use one of these on your own material and it is not working, write to cmd@sam.sdu.dk — we would genuinely like to know where it breaks.

Share on

Facebook X

Copy link

Search