Protection test
Before a request reaches an external model, Papinian Guard replaces the client’s names, identifiers, amounts and dates on the lawyer’s own computer, and restores them in the answer there. These tests measure how completely it does so, on fictional legal documents labelled before they were run. Each test has its own page: what it measures, how it ran and on which machines, its documents to download, and every value it masked, missed or masked needlessly.
1. The tests
| Test | Masked | Missed | Needless masks |
|---|---|---|---|
| PT-261101-EN1: English, current rules | 542 of 542 (100%) | 0 | 1 of 550 |
| PT-261101-EN2: English, sealed set’s first run | 79 of 81 (97.5%) | 2 | 3 of 85 |
| PT-261101-EN3: English, 1 November set’s first run | 93 of 96 (96.9%) | 3 | 2 of 99 |
| PT-261101-FR1: French | 128 of 151 (84.8%) | 23 | 86 of 234 |
| PT-261101-DE1: German | 147 of 155 (94.8%) | 8 | 161 of 316 |
| PT-261101-IT1: Italian | 139 of 152 (91.4%) | 13 | 117 of 278 |
| PT-261101-ES1: Spanish | 119 of 149 (79.9%) | 30 | 74 of 213 |
Every test ran on 1 November 2026 on a MacBook Pro with macOS and on a Windows PC, with the same results on both but for one Italian street line, masked on Windows and missed on macOS. The table gives Windows’ figures; on macOS, Italian masked 138 of 152 (90.8%), with 117 needless masks of 277.
2. Which figure to read
- English on the current rules: 100%. Every one of the 542 protected values in the 30 English documents was masked, on both systems. The rules were improved on these documents round by round, so this shows that every failure found in them is fixed, not how a new document will fare.
- A new document: the first runs. A set run once, before any fix is made from it, measures what a document the rules have not seen meets. The sealed set’s first run masked 79 of 81 (97.5%), and the 1 November set’s 93 of 96 (96.9%). The next round of fixes then masked every one of their values, as the first test shows. The current rules have no first run of their own yet: that needs a new set.
- French, German, Italian and Spanish. No round was made from these documents, so their figures measure the reach of rules written for English: they miss more, and mask many words that need no protection.
3. How every test runs
- The pipeline. Each document is written as a Word file, read by Singularity’s own extractor, cut into the passages a request carries, and protected by Papinian Guard, its local model and its rules, as one operation, as the app protects it. Nothing is sent anywhere.
- What must be masked. The names of people and organisations, identifiers, amounts and dates, as Papinian’s documented rule defines them, listed for each document before its first run. What the rule leaves open, such as a public court or the law a contract chooses, is not scored.
- The scores. A protected value is masked when no letter or figure of it is left visible, and missed otherwise. A needless mask is a symbol over no protected value. The scorer finds the symbols by aligning each passage as read with the passage as it would leave, so the mapping from symbols back to values is never written down.
- The model. Papinian NER 1.0.0, the package Singularity downloads, under Standard protection. Each test’s page gives the code, the tool, the machines and the times.
4. Download
- Every test at once, 2.1 MB, SHA-256
7e7a4d80f36fc31280a651642c9576369026f40ffdacd86a025c3e5ea27117c0: every document, label and run of these tests on both systems, each machine’s description, and the scripts that wrote and scored the documents. Each test’s page has its own download.
5. The first measurement, 22 October 2026
The first measurement, made on Legal Machines’ development tree before the open repository was created on 31 October 2026. It found how often the guard masked words that need no protection, and four rounds of fixes followed, each from the set before. Its runs kept counts only, not the values behind them.
- The first set, 10 documents: misses fell from 1 in 197 (0.5%) to none, and needless masks from 22 in 220 symbols (10.0%) to none.
- Written after round 1, 4 documents: 3 misses in 78 (3.8%) and 3 needless masks in 78 symbols (3.8%) when first run.
- Written after round 2, 3 documents: 1 miss in 48 (2.1%) and 12 needless masks in 59 symbols (20.3%) when first run.
- Written after round 3, 3 documents: no miss in 42 and 3 needless masks in 44 symbols (6.8%) when first run.
- After round 4: no miss and no needless mask in any of the 20 documents.
6. The limits of these tests
- These documents, not every document. The figures describe the documents published here. A new kind of document can show a new kind of failure, which is why every set is published with its first run, before any fix made from it.
- A set the fixes were made from shows their effect, not their reach. The first runs, and the four other languages, measure the reach.
- Written by Legal Machines. The documents are fictional and were written to resemble real ones. They are not client documents.
- One model package and one setting. Every run used Papinian NER 1.0.0 under Standard protection.
- Times are a measuring tool’s. They include writing each Word file and starting the tool, and show how the systems compare, not how fast the app protects a request.