Protection test

English: the sealed set’s first run

Five English documents written and sealed before round 5 began, run once on round 5’s rules: 79 of their 81 protected values were masked (97.5%), on macOS and on Windows alike. Round 6 was made from this run, and on the current rules these documents mask all 81, as the first test shows.

1. The test

• Test number. PT-261101-EN2, this test’s own code: the onboarding and the downloads name it.
• What it measures. What a document the rules have not seen meets. When it ran, no fix had been made from these documents, and the work that made round 5 never saw them.
• Why it stays. Round 6 was made from this run, so these documents are now a set the fixes were made from. This first run stays the measure of what an unseen document met on round 5’s rules. The current rules have no such measure yet: it needs a new set, written before any fix is made from it.
• The rules. Round 5 of the fixes (1 November 2026).

A needless mask is counted among all the symbols that would leave; a document is refused when the outbound check finds something protected still in it.

2. Technical details

• Code. Papinian Guard at commit af667e7a3faaf5067cfc053f328fa7edf94564f9 of the open repository legal-machines/singularity (d81e6ca4 in Legal Machines’ development tree).
• The measuring tool. papinian_masking_measure, from code/tests/papinian_masking_measure.cpp. For each Word file it reads the text with Singularity’s own extractor, cuts it into the passages a request carries, protects the whole document as one operation with Papinian Guard’s local model and its rules, and runs the outbound check that stops a request if anything protected would still leave. It writes each passage as read and as it would leave, and nothing else.
• The scorer. other/benchmarks/papinian-masking/measure.py writes the Word files with python-docx, runs the tool and scores what would leave. It finds each symbol by aligning a passage as read with the same passage as it would leave, so the mapping from symbols back to values is never written down.
• The model. Papinian NER 1.0.0, the package Singularity downloads (papinian-ner-1.0.0.zip, SHA-256 0f70a2a3eb1abac455e6a1cdba6e686e27d138a55eec05f0a13edcbded8ef91d); each of its nine files matched the hash Singularity pins for it. It is the open GLiNER2 privacy model fastino/gliner2-privacy-filter-PII-multi, exported to ONNX without further training, and runs on the processor through ONNX Runtime 1.23.2. A detection is used at a confidence of 0.5 or more.
• The setting. Standard protection.
• What is scored. Each document’s labels list, before its first run, the writings that must be masked: the names of people and organisations, identifiers, amounts and dates, as Papinian’s documented rule defines them. A protected value is each place such a writing stands. It is masked when every letter and figure of it is under symbols, and missed when any is left visible. A symbol over no labelled writing is a needless mask. Writings the rule leaves open, such as a public court or the law a contract chooses, are listed apart and not scored.
• The runs. The figures are from the run recorded on 1 November 2026 at 06:57–06:58 MSK on macOS and 06:57–07:04 MSK on Windows. The tables come from the same code run again the same day, at 10:47 MSK on macOS and 10:55–10:56 MSK on Windows, by measure.py at ab9527ac4, which also lists each masked value with the symbol that took its place. Both runs gave the same counts, misses and needless masks on each system.
• To run it again. With the open repository at the commit above, the tool built and the model package unpacked, run python3 other/benchmarks/papinian-masking/measure.py --tool TOOL --model MODEL --set SET --out SET.json for each set SET of holdout5, TOOL being the built tool and MODEL the model’s folder. The download’s method/measure.py is the scorer at ab9527ac4.

3. Machines

Both systems ran every document of the test, one set after another, with no other heavy work on the computer. A wall time is the whole measure.py run: writing the Word files, starting the tool and loading the model included.

4. Dataset

• Written and sealed before round 5. Written on the PC on 1 November 2026 at 05:37 MSK, before round 5 began, and sealed outside the repository: holdout5.py, 10,187 bytes, SHA-256 922ac2c284ff87cef3089fba4a49554f5cd6a86212e0d0435ed7c61f05552afd. It entered the repository unchanged only after round 5’s fixes were pushed, and ran once on them.
• Parties and law. US companies under US law.
• Fictional. Every company, person, address and number is invented, and mail domains end in .example. Telephone numbers are in 555-01xx, the range the United States reserves for fiction. No document names a real person or company.
• Counted. A protected value is each place a labelled writing stands; open writings are the labels’ list of what the rule leaves open.

• Download this test, 232 KB, SHA-256 726c80320bcb16f9ceb6d19b5ab3d5b035e47cbd7c2dd3e219f9e35d730cff55: the Word files as the test protected them, their labels, both systems’ recorded runs and table runs of this test, each machine’s description and the scripts that wrote and scored the documents. Every test at once is on the index.

5. Masked

79 of the 81 protected values were masked, on macOS and on Windows alike. Each document’s table gives every value masked in it: Places counts where the value stands, and Masked as gives the symbol that would leave the computer in its place, with the words it replaced where they reach beyond the value. A row marked left open is one of the 3 places where a symbol hid a writing the rule leaves open: masking it or not is right, and it is not scored.

Meridale master services agreement

Ostrander separation agreement

Ferncastle stock purchase agreement

Thornbury litigation hold notice

Ashgrove promissory note

6. Missed

2 of the 81 protected values would have left the computer with a letter or a figure visible. Each passage quotes the words around the value as the scorer did, the value between << and >>; a slash marks a new paragraph.

Ferncastle stock purchase agreement

Round 6 fixed both: a company’s first word inside a quoted label, such as a wire reference, names that company, and a mailbox is an address.

7. Masked needlessly

3 of the 85 symbols (3.5%) hid words that needed no protection. Each passage quotes the words around them as the scorer did.

Meridale master services agreement

Ferncastle stock purchase agreement

Thornbury litigation hold notice

Round 6 fixed the first two: a finance department is a place inside a party, and a title names nobody, whatever the model took it for. The labels leave the place open as “Kent, Washington” but do not list “Kent” alone, and Papinian Guard masks a party’s places by its rule, so the third stays as scored.