Protection test

Italian: 5 documents

Five Italian documents, of companies in Italy under Italian law, run on the current rules: of their 152 protected values, 139 were masked on Windows (91.4%) and 138 on macOS (90.8%), and 117 of the 278 symbols on Windows and 117 of the 277 on macOS hid words that needed no protection. No round of fixes was made from these documents.

1. The test

• Test number. PT-261101-IT1, this test’s own code: the onboarding and the downloads name it.
• What it measures. How far rules written for English reach in Italian, on documents no fix was made from.
• First run. These documents ran once on round 5’s rules and again on round 6’s, with the same results on each system: round 6 was made from English documents only.
• What it meets. Papinian Guard’s word rules are written for English. Across the four languages, what stays visible is mostly amounts written in words, street lines, national telephone formats, company registry numbers and file references, and what is masked for nothing is mostly defined roles and legal words written with a capital, public codes and bodies, and honorifics. The tables list every case.
• One value differs between the systems. The street line “Via Bonazzi 18” scored 0.4988 on macOS, under the 0.5 threshold, and just above it on Windows: the same model, computed a hair apart on two processors. It is masked on Windows and missed on macOS; every other value, symbol and needless mask is the same on both.
• The rules. Round 6 of the fixes, the newest code (1 November 2026).

A needless mask is counted among all the symbols that would leave; a document is refused when the outbound check finds something protected still in it.

2. Technical details

• Code. Papinian Guard at commit 47f4dd4747ce0ebedd85d86de55ed3c114591f16 of the open repository legal-machines/singularity (1e13e8289 in Legal Machines’ development tree).
• The measuring tool. papinian_masking_measure, from code/tests/papinian_masking_measure.cpp. For each Word file it reads the text with Singularity’s own extractor, cuts it into the passages a request carries, protects the whole document as one operation with Papinian Guard’s local model and its rules, and runs the outbound check that stops a request if anything protected would still leave. It writes each passage as read and as it would leave, and nothing else.
• The scorer. other/benchmarks/papinian-masking/measure.py writes the Word files with python-docx, runs the tool and scores what would leave. It finds each symbol by aligning a passage as read with the same passage as it would leave, so the mapping from symbols back to values is never written down.
• The model. Papinian NER 1.0.0, the package Singularity downloads (papinian-ner-1.0.0.zip, SHA-256 0f70a2a3eb1abac455e6a1cdba6e686e27d138a55eec05f0a13edcbded8ef91d); each of its nine files matched the hash Singularity pins for it. It is the open GLiNER2 privacy model fastino/gliner2-privacy-filter-PII-multi, exported to ONNX without further training, and runs on the processor through ONNX Runtime 1.23.2. A detection is used at a confidence of 0.5 or more.
• The setting. Standard protection.
• What is scored. Each document’s labels list, before its first run, the writings that must be masked: the names of people and organisations, identifiers, amounts and dates, as Papinian’s documented rule defines them. A protected value is each place such a writing stands. It is masked when every letter and figure of it is under symbols, and missed when any is left visible. A symbol over no labelled writing is a needless mask. Writings the rule leaves open, such as a public court or the law a contract chooses, are listed apart and not scored.
• The runs. The figures are from the run recorded on 1 November 2026 at 07:15 MSK on macOS and 07:16–07:29 MSK on Windows. The tables come from the same code run again the same day, at 10:46–10:47 MSK on macOS and 10:41–10:49 MSK on Windows, by measure.py at ab9527ac4, which also lists each masked value with the symbol that took its place. Both runs gave the same counts, misses and needless masks on each system.
• The first run. On round 5’s rules, at 06:57–06:58 MSK on macOS and 06:57–07:04 MSK on Windows, the documents gave the same results as on round 6.
• To run it again. With the open repository at the commit above, the tool built and the model package unpacked, run python3 other/benchmarks/papinian-masking/measure.py --tool TOOL --model MODEL --set SET --out SET.json for each set SET of it, TOOL being the built tool and MODEL the model’s folder. The download’s method/measure.py is the scorer at ab9527ac4.

3. Machines

Both systems ran every document of the test, one set after another, with no other heavy work on the computer. A wall time is the whole measure.py run: writing the Word files, starting the tool and loading the model included.

4. Dataset

• Five documents of Italy, under its law. A non-disclosure agreement between two companies, a law firm’s demand for payment, an employment contract, a commercial lease with bank details, and an email about a code-named deal.
• Written before they were run. Written on the PC on 1 November 2026 between 06:48 and 06:53 MSK, after round 5’s fixes and before any run on them, by writers who saw neither the guard’s code nor its tests, and sealed before the first run: sample_it.py, 28,219 bytes, SHA-256 cd434ae850f27a1d8ff0cc6ecddeba99c3d4fee6d4a13a2e2adf4029c80d3282.
• Fictional. Every company, person, address and number is invented, and mail domains end in .example. Italy reserves no range for fiction, so the telephone numbers are invented in the national format and could match a real line. Identifiers carry valid check digits and are invented; in a dense national range one may coincide with a number already issued. No document names a real person or company.
• Counted. A protected value is each place a labelled writing stands; open writings are the labels’ list of what the rule leaves open.

• Download this test, 290 KB, SHA-256 261dc588ad7de76e985d351dee5874c6ad78d4010284e5deb59ba62bac283e3e: the Word files as the test protected them, their labels, both systems’ recorded runs and table runs of this test, each machine’s description and the scripts that wrote and scored the documents. Every test at once is on the index.

5. Masked

Of the 152 protected values, 139 on Windows and 138 on macOS were masked. Each document’s table gives every value masked in it: Places counts where the value stands, and Masked as gives the symbol that would leave the computer in its place, with the words it replaced where they reach beyond the value. A row marked left open is one of the 17 places where a symbol hid a writing the rule leaves open: masking it or not is right, and it is not scored.

Accordo di riservatezza Quaresmini Ravizza

Diffida ad adempiere Termoidraulica Bellandi

Contratto di lavoro Zappala

Contratto di locazione Pelletteria Anselmini

Email Progetto Tramontana

6. Missed

13 of the 152 protected values on Windows and 14 on macOS would have left the computer with a letter or a figure visible. Each passage quotes the words around the value as the scorer did, the value between << and >>; a slash marks a new paragraph.

Accordo di riservatezza Quaresmini Ravizza

Contratto di lavoro Zappala

Contratto di locazione Pelletteria Anselmini

Email Progetto Tramontana

7. Masked needlessly

117 of the 278 symbols on Windows (42.1%) and 117 of the 277 on macOS (42.2%) hid words that needed no protection. Each passage quotes the words around them as the scorer did.

Accordo di riservatezza Quaresmini Ravizza

Diffida ad adempiere Termoidraulica Bellandi

Contratto di lavoro Zappala

Contratto di locazione Pelletteria Anselmini

Email Progetto Tramontana