Protection test

English: the 1 November set’s first run

Five English documents of kinds the earlier sets did not hold, written and labelled on 1 November 2026, run once on the rules before round 5: 93 of their 96 protected values were masked (96.9%), on macOS and on Windows alike. Round 5 was made from this run, and on the current rules these documents mask all 96, as the first test shows.

1. The test

• Test number. PT-261101-EN3, this test’s own code: the onboarding and the downloads name it.
• What it measures. What a document the rules had not seen met: when it ran, no fix had been made from these documents.
• Why it stays. Round 5 was made from this run, so these documents are now a set the fixes were made from. This first run stays the measure of what an unseen document met on the rules before round 5.
• The rules. The rules before round 5: four rounds of fixes made in October 2026.

A needless mask is counted among all the symbols that would leave; a document is refused when the outbound check finds something protected still in it.

2. Technical details

• Code. Papinian Guard at commit 8708dbbab96981cf9fef1ac4bd032e5bb72f2a01 of the open repository legal-machines/singularity (7fcd14e2 on Windows and 7647de63a on macOS, the same guard in Legal Machines’ development tree).
• The measuring tool. papinian_masking_measure, from code/tests/papinian_masking_measure.cpp. For each Word file it reads the text with Singularity’s own extractor, cuts it into the passages a request carries, protects the whole document as one operation with Papinian Guard’s local model and its rules, and runs the outbound check that stops a request if anything protected would still leave. It writes each passage as read and as it would leave, and nothing else.
• The scorer. other/benchmarks/papinian-masking/measure.py writes the Word files with python-docx, runs the tool and scores what would leave. It finds each symbol by aligning a passage as read with the same passage as it would leave, so the mapping from symbols back to values is never written down.
• The model. Papinian NER 1.0.0, the package Singularity downloads (papinian-ner-1.0.0.zip, SHA-256 0f70a2a3eb1abac455e6a1cdba6e686e27d138a55eec05f0a13edcbded8ef91d); each of its nine files matched the hash Singularity pins for it. It is the open GLiNER2 privacy model fastino/gliner2-privacy-filter-PII-multi, exported to ONNX without further training, and runs on the processor through ONNX Runtime 1.23.2. A detection is used at a confidence of 0.5 or more.
• The setting. Standard protection.
• What is scored. Each document’s labels list, before its first run, the writings that must be masked: the names of people and organisations, identifiers, amounts and dates, as Papinian’s documented rule defines them. A protected value is each place such a writing stands. It is masked when every letter and figure of it is under symbols, and missed when any is left visible. A symbol over no labelled writing is a needless mask. Writings the rule leaves open, such as a public court or the law a contract chooses, are listed apart and not scored.
• The runs. The figures are from the run recorded on 1 November 2026 at 06:04 MSK on macOS and 05:39–05:43 MSK on Windows. The tables come from the same code run again the same day, at 10:47 MSK on macOS and 10:56–10:57 MSK on Windows, by measure.py at ab9527ac4, which also lists each masked value with the symbol that took its place. Both runs gave the same counts, misses and needless masks on each system.
• To run it again. With the open repository at the commit above, the tool built and the model package unpacked, run python3 other/benchmarks/papinian-masking/measure.py --tool TOOL --model MODEL --set SET --out SET.json for each set SET of holdout4, TOOL being the built tool and MODEL the model’s folder. The download’s method/measure.py is the scorer at ab9527ac4.

3. Machines

Both systems ran every document of the test, one set after another, with no other heavy work on the computer. A wall time is the whole measure.py run: writing the Word files, starting the tool and loading the model included.

4. Dataset

• Written before any run. Five documents of kinds the earlier sets did not hold, written and labelled on 1 November 2026 before any run on them (commit b2bc2ce32 of the development tree).
• Parties and law. US companies under US law.
• Fictional. Every company, person, address and number is invented, and mail domains end in .example. Telephone numbers are in 555-01xx, the range the United States reserves for fiction. No document names a real person or company.
• Counted. A protected value is each place a labelled writing stands; open writings are the labels’ list of what the rule leaves open.

• Download this test, 230 KB, SHA-256 23a48df16c41889534ac728d3118ca94030859e6a0172bd15900d9cc34919c63: the Word files as the test protected them, their labels, both systems’ recorded runs and table runs of this test, each machine’s description and the scripts that wrote and scored the documents. Every test at once is on the index.

5. Masked

93 of the 96 protected values were masked, on macOS and on Windows alike. Each document’s table gives every value masked in it: Places counts where the value stands, and Masked as gives the symbol that would leave the computer in its place, with the words it replaced where they reach beyond the value. A row marked left open is one of the 4 places where a symbol hid a writing the rule leaves open: masking it or not is right, and it is not scored.

Wrenfield offer letter

Corvane demand letter

Larchmere term sheet email

Larkspur Commons office lease

Pellworth settlement agreement

6. Missed

3 of the 96 protected values would have left the computer with a letter or a figure visible. Each passage quotes the words around the value as the scorer did, the value between << and >>; a slash marks a new paragraph.

Corvane demand letter

Round 5 fixed all three: a short name a definition gives right after a party’s full name is that party’s name wherever it is written.

7. Masked needlessly

2 of the 99 symbols (2.0%) hid words that needed no protection. Each passage quotes the words around them as the scorer did.

Wrenfield offer letter

Larkspur Commons office lease

Round 5 fixed both: a department named by its function words alone is a place inside a party, and a manager’s title, by what the manager manages, names nobody.