Protection test

English: 30 documents on the current rules

On the current rules, Papinian Guard masked all 542 protected values in the 30 English documents, on macOS and on Windows alike. One of its 550 symbols hid a word that needed no protection.

1. The test

• Test number. PT-261101-EN1, this test’s own code: the onboarding and the downloads name it.
• What it measures. Whether any protected value in the English documents would leave the computer with a letter or a figure still visible, and how many words that needed no protection were masked.
• What it shows. The rules were improved on these documents, round by round: each set after the first was written after a round of fixes and run once, and the next round was made from what it found. A set the fixes were made from shows their effect, not their reach. The reach is what the first runs measure: the sealed set’s and the 1 November set’s.
• The rules. Round 6 of the fixes, the newest code (1 November 2026).

A needless mask is counted among all the symbols that would leave; a document is refused when the outbound check finds something protected still in it.

2. Technical details

• Code. Papinian Guard at commit 47f4dd4747ce0ebedd85d86de55ed3c114591f16 of the open repository legal-machines/singularity (1e13e8289 in Legal Machines’ development tree).
• The measuring tool. papinian_masking_measure, from code/tests/papinian_masking_measure.cpp. For each Word file it reads the text with Singularity’s own extractor, cuts it into the passages a request carries, protects the whole document as one operation with Papinian Guard’s local model and its rules, and runs the outbound check that stops a request if anything protected would still leave. It writes each passage as read and as it would leave, and nothing else.
• The scorer. other/benchmarks/papinian-masking/measure.py writes the Word files with python-docx, runs the tool and scores what would leave. It finds each symbol by aligning a passage as read with the same passage as it would leave, so the mapping from symbols back to values is never written down.
• The model. Papinian NER 1.0.0, the package Singularity downloads (papinian-ner-1.0.0.zip, SHA-256 0f70a2a3eb1abac455e6a1cdba6e686e27d138a55eec05f0a13edcbded8ef91d); each of its nine files matched the hash Singularity pins for it. It is the open GLiNER2 privacy model fastino/gliner2-privacy-filter-PII-multi, exported to ONNX without further training, and runs on the processor through ONNX Runtime 1.23.2. A detection is used at a confidence of 0.5 or more.
• The setting. Standard protection.
• What is scored. Each document’s labels list, before its first run, the writings that must be masked: the names of people and organisations, identifiers, amounts and dates, as Papinian’s documented rule defines them. A protected value is each place such a writing stands. It is masked when every letter and figure of it is under symbols, and missed when any is left visible. A symbol over no labelled writing is a needless mask. Writings the rule leaves open, such as a public court or the law a contract chooses, are listed apart and not scored.
• The runs. The figures are from the run recorded on 1 November 2026 at 07:15 MSK on macOS and 07:16–07:29 MSK on Windows. The tables come from the same code run again the same day, at 10:46–10:47 MSK on macOS and 10:41–10:49 MSK on Windows, by measure.py at ab9527ac4, which also lists each masked value with the symbol that took its place. Both runs gave the same counts, misses and needless masks on each system.
• To run it again. With the open repository at the commit above, the tool built and the model package unpacked, run python3 other/benchmarks/papinian-masking/measure.py --tool TOOL --model MODEL --set SET --out SET.json for each set SET of main, holdout, holdout2, holdout3, holdout4, holdout5, TOOL being the built tool and MODEL the model’s folder. The download’s method/measure.py is the scorer at ab9527ac4.

3. Machines

Both systems ran every document of the test, one set after another, with no other heavy work on the computer. A wall time is the whole measure.py run: writing the Word files, starting the tool and loading the model included.

4. Dataset

• Six sets, each written before it was run. main holds the working day’s four NDAs and six documents written beside them; round 1 of the fixes was made from it. holdout, holdout2 and holdout3 were each written after a round of fixes, in October 2026, and run once before the next round was made from them. holdout4 was written and labelled on 1 November 2026 before any run on it, and round 5 was made from it. holdout5 was written and sealed on 1 November 2026 before round 5 began (holdout5.py, 10,187 bytes, SHA-256 922ac2c284ff87cef3089fba4a49554f5cd6a86212e0d0435ed7c61f05552afd), run once on round 5, and round 6 was made from it.
• Parties and law. US companies under US law, and one English company under English law, in the Saltmarsh Analytics NDA.
• Fictional. Every company, person, address and number is invented, and mail domains end in .example. Telephone numbers are in 555-01xx, the range the United States reserves for fiction. No document names a real person or company.
• Counted. A protected value is each place a labelled writing stands; open writings are the labels’ list of what the rule leaves open.

• Download this test, 1.1 MB, SHA-256 69344ca7d098120d662690958e0e698df9f0e755cedb8674d293f8ac2994addd: the Word files as the test protected them, their labels, both systems’ recorded runs and table runs of this test, each machine’s description and the scripts that wrote and scored the documents. Every test at once is on the index.

5. Masked

542 of the 542 protected values were masked, on macOS and on Windows alike. Each document’s table gives every value masked in it: Places counts where the value stands, and Masked as gives the symbol that would leave the computer in its place, with the words it replaced where they reach beyond the value. A row marked left open is one of the 8 places where a symbol hid a writing the rule leaves open: masking it or not is right, and it is not scored.

Ottermere Analytics NDA

Sablewood Distribution NDA

Fernhollow Engineering NDA

Kettleby Logistics NDA

Brightwater Bioinstruments NDA

Corvane Capital NDA

Ottermere NDA First Amendment

Westbrook Hale cover letter

Marsh consulting confidentiality

Ashgrove University research NDA

Riverbend Hospital NDA

Quarrystone Ventures NDA

Tallow Finch engagement letter

Employee confidentiality agreement

Brannock Precision supplier NDA

Okonkwo Lisle settlement letter

Lumenfield processing schedule

Saltmarsh Analytics NDA

Halvane board minutes

Kettleby open points memo

Wrenfield offer letter

Corvane demand letter

Larchmere term sheet email

Larkspur Commons office lease

Pellworth settlement agreement

Meridale master services agreement

Ostrander separation agreement

Ferncastle stock purchase agreement

Thornbury litigation hold notice

Ashgrove promissory note

6. Missed

None: no protected value would have left the computer with a letter or a figure visible.

7. Masked needlessly

1 of the 550 symbols (0.2%) hid words that needed no protection. Each passage quotes the words around them as the scorer did.

Thornbury litigation hold notice

The labels leave the place open as “Kent, Washington” but do not list “Kent” alone, and Papinian Guard masks a party’s places by its rule, so this mask stays as scored.