<< List of all deliverables

D3.3.3 (a): Analysis tools for providing metadata for web data: language, country and register

Project: FIN-CLARIAH
Grant agreement: Research Council of Finland no. 367751
Start date: 01-01-2026
Duration: 24 months

WP 3.3: Report on analysis tools for providing metadata for web data: language, country and register
Date of reporting: 02-11-2026

Report authors: FTuomas Lundberg, Erik Henriksson, Tuukka Törö, Veronika Laippala (TurkuNLP, University of Turku)
Contributors: –
Deliverable location:

• Code repository: https://github.com/TurkuNLP/web-metadata-fin-clariah
• Demonstration: Google Colab notebook, https://colab.research.google.com/github/TurkuNLP/web-metadata-fin-clariah/blob/main/notebooks/webmeta_demo.ipynb
• Demo data: Finnish and Swedish HPLT 4.0 sample, https://huggingface.co/datasets/TurkuNLP/webmeta-demo-fi-sv

Keywords for the deliverable page: web data; metadata; language identification; multilingual documents; country; web registers; HPLT; SSH research

Description

Web-crawled corpora are among the largest text resources for research in the social sciences and humanities (SSH). For example, the newest HPLT version, HPLT 4.0, was built from over 10 petabytes of web archives, and its cleaned release holds 18.8 billion documents in 194 languages. When dealing with a corpus of these proportions, it’s hard to find the material a research question needs.

In this deliverable we publish webmeta, an open-source Python package that turns a large web corpus into research datasets. A researcher describes the subsample they need by its metadata: how many to draw, which documents qualify, and how to stratify. Webmeta then builds that subsample reproducibly, essentially reducing each study dataset to a set of settings. This enables a single corpus to serve many projects without writing new sampling and filtering code each time.

• Language. The main language of each document and the languages of its segments, so multilingual documents can be selected or excluded.
• Register. 25 web registers, such as news reports, recipes and opinion blogs, from a multilabel classifier with per-language thresholds. Documents that mix registers, as most web documents do (Henriksson et al. 2026b), can be selected as hybrids.
• Country. A country and UN region from the document’s web address. This is a proxy: where a domain such as .fi points to a country, generic domains such as .com have no such information, and furthermore, some country codes are sold and used worldwide.

HPLT 4.0 carries language identification and, for 92 languages, register labels from the TurkuNLP multilingual register classifier (Henriksson et al. 2026a). webmeta reads HPLT 4.0 samples drawn by TurkuNLP’s HPLT sampler, or a user’s own corpus, and exports each dataset along with a summary table and a record of the settings used in creating the dataset.

The first version of webmeta supports corpus-scale runs on the LUMI supercomputer, and additionally it can be used on one’s own computer as a plain Python package. In the companion Google Colab demo notebook, users can upload their own corpus or explore a Finnish and Swedish demo sample without writing their own code. The package is already used by TurkuNLP for building datasets for corpus linguistics. Licenses are Apache 2.0 for code, and CC0 for the demo data.

The deliverable acts as groundwork for the social media datasets of D3.3.3 (b) and the corpus analysis tools of D3.3.3 (c).

References

Henriksson, E., A. Myntti, S. Hellström, A. Eskelinen, S. Erten-Johansson, and V. Laippala. 2026a. “Automatic register identification for the open web using multilingual deep learning.” Natural Language Processing (accepted). arXiv:2406.19892.

Henriksson, E., A. Razzaghi, T. Lundberg, A. Kanner, and V. Laippala. 2026b. “Register Mixing Is the Norm on the Web.” Proceedings of NLP4DH 2026, 138–149. doi:10.18653/v1/2026.nlp4dh-1.14

HPLT 4.0: https://hplt-project.org/datasets/v4.0

HPLT sampler: https://github.com/TurkuNLP/hplt-sampler

FIN-CLARIAH project has received funding from the European Union – NextGenerationEU instrument and is funded by the Research Council of Finland under grant number 367751.

Last modified on 2026-11-06