Protection test

German: 5 documents

Five German documents, of companies in Germany under German law, run on the current rules: 147 of their 155 protected values were masked (94.8%), on macOS and on Windows alike, and 161 of the 316 symbols hid words that needed no protection. No round of fixes was made from these documents.

1. The test

• Test number. PT-261101-DE1, this test’s own code: the onboarding and the downloads name it.
• What it measures. How far rules written for English reach in German, on documents no fix was made from.
• First run. These documents ran once on round 5’s rules and again on round 6’s, with the same results on each system: round 6 was made from English documents only.
• What it meets. Papinian Guard’s word rules are written for English. Across the four languages, what stays visible is mostly amounts written in words, street lines, national telephone formats, company registry numbers and file references, and what is masked for nothing is mostly defined roles and legal words written with a capital, public codes and bodies, and honorifics. The tables list every case.
• The rules. Round 6 of the fixes, the newest code (1 November 2026).

A needless mask is counted among all the symbols that would leave; a document is refused when the outbound check finds something protected still in it.

2. Technical details

• Code. Papinian Guard at commit 47f4dd4747ce0ebedd85d86de55ed3c114591f16 of the open repository legal-machines/singularity (1e13e8289 in Legal Machines’ development tree).
• The measuring tool. papinian_masking_measure, from code/tests/papinian_masking_measure.cpp. For each Word file it reads the text with Singularity’s own extractor, cuts it into the passages a request carries, protects the whole document as one operation with Papinian Guard’s local model and its rules, and runs the outbound check that stops a request if anything protected would still leave. It writes each passage as read and as it would leave, and nothing else.
• The scorer. other/benchmarks/papinian-masking/measure.py writes the Word files with python-docx, runs the tool and scores what would leave. It finds each symbol by aligning a passage as read with the same passage as it would leave, so the mapping from symbols back to values is never written down.
• The model. Papinian NER 1.0.0, the package Singularity downloads (papinian-ner-1.0.0.zip, SHA-256 0f70a2a3eb1abac455e6a1cdba6e686e27d138a55eec05f0a13edcbded8ef91d); each of its nine files matched the hash Singularity pins for it. It is the open GLiNER2 privacy model fastino/gliner2-privacy-filter-PII-multi, exported to ONNX without further training, and runs on the processor through ONNX Runtime 1.23.2. A detection is used at a confidence of 0.5 or more.
• The setting. Standard protection.
• What is scored. Each document’s labels list, before its first run, the writings that must be masked: the names of people and organisations, identifiers, amounts and dates, as Papinian’s documented rule defines them. A protected value is each place such a writing stands. It is masked when every letter and figure of it is under symbols, and missed when any is left visible. A symbol over no labelled writing is a needless mask. Writings the rule leaves open, such as a public court or the law a contract chooses, are listed apart and not scored.
• The runs. The figures are from the run recorded on 1 November 2026 at 07:15 MSK on macOS and 07:16–07:29 MSK on Windows. The tables come from the same code run again the same day, at 10:46–10:47 MSK on macOS and 10:41–10:49 MSK on Windows, by measure.py at ab9527ac4, which also lists each masked value with the symbol that took its place. Both runs gave the same counts, misses and needless masks on each system.
• The first run. On round 5’s rules, at 06:57–06:58 MSK on macOS and 06:57–07:04 MSK on Windows, the documents gave the same results as on round 6.
• To run it again. With the open repository at the commit above, the tool built and the model package unpacked, run python3 other/benchmarks/papinian-masking/measure.py --tool TOOL --model MODEL --set SET --out SET.json for each set SET of de, TOOL being the built tool and MODEL the model’s folder. The download’s method/measure.py is the scorer at ab9527ac4.

3. Machines

Both systems ran every document of the test, one set after another, with no other heavy work on the computer. A wall time is the whole measure.py run: writing the Word files, starting the tool and loading the model included.

4. Dataset

• Five documents of Germany, under its law. A non-disclosure agreement between two companies, a law firm’s demand for payment, an employment contract, a commercial lease with bank details, and an email about a code-named deal.
• Written before they were run. Written on the PC on 1 November 2026 between 06:48 and 06:53 MSK, after round 5’s fixes and before any run on them, by writers who saw neither the guard’s code nor its tests, and sealed before the first run: sample_de.py, 31,099 bytes, SHA-256 50c8435309488b9ad1865dc98222ff6fb68542841cefd4014470ba97d62b79b5.
• Fictional. Every company, person, address and number is invented, and mail domains end in .example. Telephone numbers are in the Bundesnetzagentur’s ranges for fiction, its “Drama Numbers”. Identifiers carry valid check digits and are invented; in a dense national range one may coincide with a number already issued. No document names a real person or company.
• Counted. A protected value is each place a labelled writing stands; open writings are the labels’ list of what the rule leaves open.

• Download this test, 301 KB, SHA-256 6baccfa3886ccbd797a8909b00d7b6d79b4ae3fbf89e89006f7dd4896f1528d7: the Word files as the test protected them, their labels, both systems’ recorded runs and table runs of this test, each machine’s description and the scripts that wrote and scored the documents. Every test at once is on the index.

5. Masked

147 of the 155 protected values were masked, on macOS and on Windows alike. Each document’s table gives every value masked in it: Places counts where the value stands, and Masked as gives the symbol that would leave the computer in its place, with the words it replaced where they reach beyond the value. A row marked left open is one of the 6 places where a symbol hid a writing the rule leaves open: masking it or not is right, and it is not scored.

Kaltenbronn Geheimhaltungsvereinbarung

Lahnfalz Zahlungsaufforderung

Havelstrom Arbeitsvertrag Weissenfels

Vahlbruch Gewerbemietvertrag

Projekt Kormoran E-Mail

6. Missed

8 of the 155 protected values would have left the computer with a letter or a figure visible. Each passage quotes the words around the value as the scorer did, the value between << and >>; a slash marks a new paragraph.

Kaltenbronn Geheimhaltungsvereinbarung

Lahnfalz Zahlungsaufforderung

Havelstrom Arbeitsvertrag Weissenfels

Projekt Kormoran E-Mail

7. Masked needlessly

161 of the 316 symbols (50.9%) hid words that needed no protection. Each passage quotes the words around them as the scorer did.

Kaltenbronn Geheimhaltungsvereinbarung

Lahnfalz Zahlungsaufforderung

Havelstrom Arbeitsvertrag Weissenfels

Vahlbruch Gewerbemietvertrag

Projekt Kormoran E-Mail