项目的 README,来自 github.com/sherwinvishesh/SurrogateShield。合并为一个 markdown 文件:README.md。
◆ ──────────────────────────────────── ◆
S U R R O G A T E
███████╗██╗ ██╗██╗███████╗██╗ ██████╗
██╔════╝██║ ██║██║██╔════╝██║ ██╔══██╗
███████╗███████║██║█████╗ ██║ ██║ ██║
╚════██║██╔══██║██║██╔══╝ ██║ ██║ ██║
███████║██║ ██║██║███████╗███████╗██████╔╝
╚══════╝╚═╝ ╚═╝╚═╝╚══════╝╚══════╝╚═════╝
◆ ──────────────────────────────────── ◆
Privacy-preserving proxy for LLMs
Masks PII before it leaves your device
SurrogateShield replaces the personal data it detects with realistic, consistent surrogates before your message reaches an LLM provider, and puts the real values back in the reply. On the sealed test-3 benchmark of real chat messages it let 0.8% of 3,677 personal values through, against 9.1% for GLiNER-PII and 31.6% for Microsoft Presidio; the few places it trailed are listed in one line under Limitations. All cryptographic operations run locally. Measured results, each with its command, are in bench/results/README.md.
User message
│
▼
[ServiceQueryDetector]
├─ service query + street address → fuzz house number ±1, preserve city/state
├─ service query, no street addr → send unchanged (location not PII here)
└─ not a service query → full detection cascade
│
▼
SentinelLayer (PatternScan + canonicaliser → EntityTrace → PII tagger
[→ ContextGuard] → structural passes → relation gate → resolver)
│
▼
MimicGen → generate type-consistent surrogate values
│
▼
Apply substitutions → sanitised message
│
▼
ShadowMap.update({surrogate: original}) + save (AES-256-GCM encrypted)
│
▼
[Optional] RAG query → prepend anonymised context
│
▼
LLM API ← receives the masked text
│
▼
ResolvePass → restore original values in response
│
▼
Display to user
balanced, strict, fast and classicsurrogateshield (pip install ./python-library), a self-contained API with no dashboard dependency, for embedding directly into any Python application or serviceThe stages run in sequence: PatternScan, the canonicaliser (the same patterns on rewritten views such as "priya dot raman at gmail dot com"), EntityTrace, ContextGuard and the PII tagger, then the structural passes, a relation gate and a resolver. Each stage reads the text with what earlier stages found masked out, so no two stages fight over the same value. Which stages run is set by the detection preset: balanced (default), strict, fast or classic (see CONFIGURATION.md). The code lives in python-library/surrogateshield/core/; the top-level detection/, generation/, storage/ and reconstruction/ modules are aliases of it, or thin wrappers that apply config.py.
In balanced, the PII tagger (a DeBERTa-v3-small token classifier, sherwinvishesh/surrogateshield-pii-tagger on Hugging Face) reads names, organisations and places and adds to the patterns on phones, IDs, addresses, handles and credentials; spaCy reads places, and its names stand in only where no stage named anyone and the name is tied to a person; ContextGuard is off. Its weights (~565 MB) download once on first use, at a fixed commit, and are checked against their SHA-256 pin. On a machine that cannot fetch them, use the classic preset.
detection/pattern_scan.py)Regex-based structural detection. Runs first so structured PII is masked before any NER model sees it. Validators (Luhn, ABA checksum) reject false positives before any entity is emitted.
| Internal type | Examples (each is found when scanned on its own) | Validator |
|---|---|---|
address (the structured parser) |
99 Cathedral Close, Tempe, AZ 85281, PO Box 1234, Tempe, AZ 85281, 1500 Highway 50, Carson City, NV |
Plausibility rules (suffix tiers, "3 point turn" rejected) |
ssn |
512-44-7810 |
None |
email |
[email protected] |
None |
phone_us |
+1-480-555-1234, (480) 555-1234 |
None |
phone_uk |
+44 7911 123456 |
None |
phone_intl |
+49 8234 927461 |
None |
credit_card |
4111 1111 1111 1111 |
Luhn algorithm |
dob |
01/15/1990, March 14 1990, DOB 1990-03-14 |
None |
age |
34 years old, 7-year-old, age: 52 |
Shape guard (an age, not a count) |
ip_address |
192.168.1.100 |
None |
mac_address |
00:1A:2B:3C:4D:5E |
None |
url |
https://example.com/u/priya |
None |
handle |
username: priya_r |
Context-gated (label or @) |
api_key |
sk-ant-api03-abcdefghijklmnopqrstuvwxyz0123456789, ghp_abcdefghijklmnopqrstuvwxyz0123456789, AKIAIOSFODNN7EXAMPLE, AIzaSyA1234567890abcdefghijklmnopqrstuv, Bearer eyJhbGciOiJIUzI1NiJ9.e30.abc |
None |
credential |
password: Tr0ub4dor&3, PIN: 4821, one-time code 839201 |
Context-gated (keyword required) |
crypto |
1BvBMSEYstWetqTFn5Au4m4GFg7xJaNVN2, bc1qar0srrr7xfkvy5l643lydnw9re59gtzzwf5mdq, 0x52908400098527886E0F7030069857D2E4169EE7 |
None (Bitcoin P2PKH / P2SH / Bech32, Ethereum) |
iban |
GB29 NWBK 6016 1331 9268 19, DE89 3704 0044 0532 0130 00 |
IBAN mod-97 |
us_bank_number |
routing 021000021, routing number 122105155 |
ABA 9-digit checksum, context-gated |
us_driver_license |
DL B7654321, DL F123456789012, driver's license number B7654321 |
Context-gated (keyword required) |
passport |
passport C1234567, passport number 512345678 |
Context-gated |
license_plate |
plate 7ABC123, licence plate AB12 CDE |
Context-gated |
vin |
1HGCM82633A004352 |
VIN alphabet (no I, O, Q) |
id_number |
employee ID 48213977, patient MRN 00482913 |
Context-gated |
gender_indicator |
gender: female, she/her, I am a man |
None |
postcode_uk |
SW1A 1AA |
None |
zip_us |
zip 85281, 85281-1234 |
Context-gated for the 5-digit form |
The tests derive this table from pattern_scan._PATTERNS and scan every example (python-library/tests/test_detection_contract.py).
Pattern order matters; patterns claim character spans; later patterns cannot overlap earlier ones. In particular: crypto and us_bank_number run before zip_us so that 9-digit routing numbers and long hex strings are claimed before the ZIP pattern can fragment them.
ABA checksum (_aba_routing_valid): (3·d₀ + 7·d₁ + d₂ + 3·d₃ + 7·d₄ + d₅ + 3·d₆ + 7·d₇ + d₈) mod 10 = 0. Eliminates false positives; random 9-digit numbers almost always fail.
Driver's license is context-gated: the regex requires a keyword (driver's license, license number, DL, D.L.) within the same phrase. Only the license value itself (captured in group 1) is marked as an entity; the keyword prefix is left in the sanitised text.
detection/entity_trace.py)spaCy en_core_web_lg NER. Extracts PERSON, GPE, LOC, ORG, and FAC entities; in the default preset it is used for places, and for names only as the ensemble fallback (a name tied to a person, in a message where no stage named anyone). Returns two tiers:
Includes ORG→GPE reclassification when location prepositions appear before an organisation name (e.g. "lives in Google").
detection/context_guard.py)dslim/distilbert-NER (~250 MB, downloaded once from HuggingFace Hub on first use, no server required). Off in the balanced and fast presets, on in strict and classic. Verifies borderline entities from EntityTrace and independently detects anything missed. Applies word-piece artefact cleaning and a blocklist of short / title tokens that commonly cause false positives.
detection/logic.py)Four additional passes run on the combined entity set:
| Pass | What it does |
|---|---|
| A: Structural ORG | Regex for `[the/a/an] [corporation |
| B: Email-username reclassification | Corrects ORG→PERSON when the entity text is a prefix of a detected email username |
| C: PERSON component dedup | Removes standalone surnames that are sub-components of already-detected full names |
| D: Topical geo-entity filter | Drops a GPE/LOC only if it appears exclusively in knowledge-query sub-clauses |
detection/address_parser.py) — v2The canonical structured parser is the single source of truth for "what is an address". It captures the FULL span — house number, pre/post directionals, street name (including ordinals like "5th"), suffix, apartment/suite/unit, city, state (full names + uppercase 2-letter abbreviations), and ZIP/ZIP+4 — as ONE entity, so components are never surrogated independently. PO boxes, multi-line mailing blocks, comma-free formats, and highway forms ("1500 Highway 50") are all recognized, and a two-tier suffix system plus a plausibility validator rejects prose like "3 point turn" or "5 stars".
How the surrogate is built is controlled by ADDRESS_MODE in config.py:
| Mode | Behaviour |
|---|---|
"shift" |
Only the house number changes, by up to ±ADDRESS_SHIFT_RANGE (default ±1, ~one building). Street, city, state, ZIP, and formatting preserved byte-for-byte. |
"replace" |
Structure-preserving fake — every component replaced by a fake of the same shape, in the same slot. No real component reaches the LLM. |
"auto" (default) |
Context-aware: shift for service queries, replace otherwise. |
A shifted address never collides with another real address in the conversation, house number 1 always shifts up (never to 0), and leading zeros / alpha suffixes ("0123", "123A") are preserved.
detection/service_query.py)Identifies messages like "restaurants near 1126 E Apache Blvd, Tempe, AZ" and applies a lighter touch:
ADDRESS_MODE="auto"verify_addresses is now a no-op)detection/quasi_identifier.py)Based on Sweeney's k-anonymity research. Detects risky entity-type combinations and issues warnings:
| Combination | Risk |
|---|---|
| ZIP + DOB + Gender | High: 87% of US population uniquely identifiable (Sweeney 2000) |
| Postcode + DOB | High |
| Name + SSN | High |
| Name + DOB | High |
| Phone + Name | High |
| IP Address + Name | High |
| Name + Employer + City | Medium |
| Email + Location | Medium |
| Phone + Location | Medium |
| DOB + Location + Employer | Medium |
generation/logic.py)Generates type-consistent surrogates using Faker. Guarantees no collisions within a session via a used_surrogates set. Every surrogate is unique and realistic for its type:
| Entity type | Generated surrogate looks like |
|---|---|
PERSON |
Adya Nath |
email |
[email protected] |
ssn |
XXX-XX-XXXX (valid format) |
phone_us |
+1-###-###-#### |
phone_uk |
+44 7### ###### |
phone_intl |
+49 8234 927461 |
address (shift mode) |
790 Crescent Row, Springfield, IL for 789 Crescent Row, Springfield, IL — number ±1, everything else byte-identical |
address (replace mode) |
412 Nguyen Row, South Debra, PR 90830 — same structure, every component fake |
credit_card |
Valid Luhn-format number |
dob |
MM/DD/YYYY (age 18–80) |
ip_address |
10.x.x.x |
zip_us / postcode_uk |
Correct format |
api_key |
sk- + 32 random chars |
GPE / LOC / ORG / FAC |
Faker city/company names |
gender_indicator |
Grammatically valid gender expression |
crypto |
Bitcoin P2PKH format: 1 + Base58 chars, 26–35 chars |
us_bank_number |
Valid 9-digit ABA routing number (passes checksum) |
us_driver_license |
CA-format: letter + 7 digits (e.g. B4923817) |
storage/logic.py)An encrypted, per-conversation mapping of surrogate → original.
| Property | Detail |
|---|---|
| Encryption | AES-256-GCM with a fresh 12-byte nonce per save |
| Key derivation | HKDF-SHA256: device secret as IKM, conversation ID as salt, file kind (shadowmap-v1 / transcript-v1) as info |
| Binding | Kind and ID are AES-GCM associated data: a file only opens as what it was written for |
| Device secret | Generated once at ~/.surrogateshield/device.key ($SURROGATESHIELD_HOME overrides the directory); 0o600, directory 0o700, loose permissions are tightened, a key of the wrong length is refused |
| File location | ~/.surrogateshield/conversations/<conv_id>.shadowmap; IDs must match ^[A-Za-z0-9_-]{1,64}$ |
| Writes | Atomic (temp file, fsync, rename), files 0o600 |
| Missing file | Empty mapping |
| Corrupt file | Moved aside to <name>.corrupt-<ts> and an error is raised: never silently replaced by an empty map |
| Key unavailable | Error (ShadowMapStorageError); an ephemeral key is opt-in via config.ALLOW_EPHEMERAL_KEY |
reconstruction/logic.py)Four-pass restoration of original values in LLM responses. Every pass matches whole values only ("Lee" never matches inside "Leeds"), and a span one pass rewrote is protected from the later ones:
790 Crescent Row from a longer address), scoped to unresolved surrogates only to prevent corruption of adjacent textpartial_ratio with configurable threshold (default 85), for multi-word surrogates only, never numbersEvery failure is categorised as exact_miss, fuzzy_hit, or fuzzy_miss for research analysis.
chatbot/rag.py)Local Retrieval-Augmented Generation backed by ChromaDB and sentence-transformers (all-MiniLM-L6-v2).
~/.surrogateshield/rag (mode 0700; RAG_DIR in config.py)rag_global ShadowMap. In a RAG chat, a name that already appears in a document gets the document's surrogate, so retrieval matches it and quoted excerpts are restored in the answerpython main.py rag lists the indexed documents. python main.py rag --forget <file> removes a document's chunks and every mapping that no other document still usespython-library/surrogateshield)Every component above (the detection stages, MimicGen, ShadowMap and ResolvePass) lives in this self-contained pip-installable library. It imports nothing from the main application; the application is built on top of it. The public surface is a Session class plus the module functions config, scan, mask, mask_result, unmask, forget and flush, behind a single import line. See the Python Library section below for usage.
| Provider | Model | Env var required |
|---|---|---|
| Claude (default) | claude-sonnet-4-6 (SURROGATESHIELD_CLAUDE_MODEL) |
ANTHROPIC_API_KEY |
| Gemini | gemini-2.5-flash (SURROGATESHIELD_GEMINI_MODEL; SDK google-genai) |
GEMINI_API_KEY |
| ChatGPT | gpt-4o-mini (SURROGATESHIELD_OPENAI_MODEL) |
OPENAI_API_KEY |
| Local (Ollama) | llama3.2 (configurable) |
None: runs fully offline |
Switch providers from the Settings menu inside the dashboard (press S).
Thirteen public types; each groups the internal names the detectors report:
| Public type | Internal names |
|---|---|
PERSON |
PERSON, person |
ORG |
ORG |
LOCATION |
GPE, LOC, FAC, implicit_location |
ADDRESS |
address, zip_us, postcode_uk |
EMAIL |
email |
PHONE |
phone_us, phone_uk, phone_intl |
URL |
url, hostname |
HANDLE |
handle |
ID |
id_number, ssn, passport, us_driver_license, license_plate, vin, credit_card, iban, us_bank_number, crypto |
CREDENTIAL |
credential, api_key |
NETWORK |
ip_address, mac_address |
AGE |
age |
DATE_OF_BIRTH |
dob |
Two more labels exist: GENDER (gender_indicator) and OTHER (any label the map does not know, such as a plugin's own). Combination risk is scored separately: quasi-identifier sets per Sweeney k-anonymity.
You must activate a virtual environment before installing or running. Installing into the system or base conda Python is the most common cause of "package not found" errors at runtime.
# Clone
git clone https://github.com/sherwinvishesh/SurrogateShield.git
cd SurrogateShield
# Create and activate a virtual environment ← required
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
# Download the spaCy model
# Required by EntityTrace (stage 2 NER detection).
# Also required by the Presidio comparison panel if you enable it.
python -m spacy download en_core_web_lg
# The PII tagger (default preset) downloads once from Hugging Face
# (sherwinvishesh/surrogateshield-pii-tagger, ~565 MB) on first use;
# `surrogateshield download` fetches it now instead.
# ContextGuard (dslim/distilbert-NER, ~250 MB) is off by default; it
# downloads from HuggingFace Hub on first use only if you turn it on.
# Set your API key: create .env readable only by you, then add the key with
# an editor (not `echo`, which leaves it in your shell history)
touch .env && chmod 600 .env
# Launch
./run.sh
The first mask after start-up loads the models and takes a few seconds; later messages take tens of milliseconds.
Troubleshooting; Presidio shows "not installed": this almost always means the packages were installed into a different Python than the one running the app. Fix: activate your venv first, then re-run
pip install -r requirements.txtandpython -m spacy download en_core_web_lgfrom inside the venv.
The Presidio comparison panel is off by default. It adds a side-by-side Microsoft Presidio result below every PII Finder detection, which is useful for research comparisons but adds latency and screen noise during normal use.
To enable it:
pip install presidio-analyzer presidio-anonymizer
python -m spacy download en_core_web_lg # also used by EntityTrace
To disable it again, press S → C from the dashboard.
anthropic>=0.25.0 # Claude API client
python-dotenv>=1.0.0 # .env file loading
spacy>=3.7.0 # EntityTrace NER + Presidio NLP backend
faker>=24.0.0 # Surrogate generation
cryptography>=42.0.0 # AES-256-GCM, HKDF
rapidfuzz>=3.6.0 # Fuzzy reconstruction matching
typer>=0.12.0 # CLI framework
rich>=13.7.0 # Terminal UI
chromadb>=0.4.0 # RAG vector store
sentence-transformers>=2.7.0 # RAG embeddings
transformers>=5.0.0 # The PII tagger and ContextGuard (distilbert-NER)
torch>=2.0.0 # ContextGuard inference
bert-score>=0.3.13 # Utility preservation scoring (BERTScore comparison)
ollama>=0.1.8 # Local LLM (optional)
presidio-analyzer>=2.2.0 # Presidio comparison panel in PII Finder
presidio-anonymizer>=2.2.0 # Presidio anonymization (companion to analyzer)
spaCy model:
en_core_web_lgis required by EntityTrace (always) and by the optional Presidio comparison panel. One download covers both:python -m spacy download en_core_web_lg.PII tagger weights: downloaded once from Hugging Face,
sherwinvishesh/surrogateshield-pii-tagger(565 MB, MIT), at a fixed commit, then checked against their SHA-256 pin and read from the local cache with no network./.cache/surrogateshield/modelssurrogateshield downloadfetches them ahead of time. A folder of the same name under$SURROGATESHIELD_MODELS(default `) is used instead when there is one. If they cannot be fetched (orHF_HUB_OFFLINEis set and they are not cached) the default preset raisesDetectorUnavailable; theclassic` preset runs without them (it uses ContextGuard instead, which downloads the same way).ContextGuard model:
dslim/distilbert-NER(~250 MB) is off by default and is downloaded from HuggingFace Hub on first use only when it is turned on (thestrictandclassicpresets turn it on;surrogateshield download --allfetches it ahead of time).BERTScore:
bert-scoreand itsroberta-largemodel (~1.4 GB) are only needed if you enable BERTScore fields in JSON Test. The model is downloaded automatically on first use.
Create a .env file in the project root and make it private
(chmod 600 .env; run.sh warns when other users can read it). The keys are
read by python-dotenv; a variable already exported in your shell takes
precedence over the file.
ANTHROPIC_API_KEY=sk-ant-... # Required for Claude
ANTHROPIC_WORKSPACE_ID=wrkspc_... # Only for a key not scoped to a workspace
GEMINI_API_KEY=... # Required for Gemini
OPENAI_API_KEY=sk-... # Required for ChatGPT
These are changed interactively from inside the app and persist across sessions in ~/.surrogateshield/settings.json.
| Key | Default | What it controls |
|---|---|---|
llm_provider |
claude |
Active LLM backend: Claude / Gemini / ChatGPT / Local |
detailed_view |
false |
Show pipeline stage logs, per-entity PII table, and the API transparency panel in each chat turn |
presidio_comparison |
false |
Show the Presidio side-by-side panel below each PII Finder result. Off by default: requires presidio-analyzer, presidio-anonymizer, and python -m spacy download en_core_web_lg to be installed first (see Enabling the Presidio comparison panel above) |
config.py)Hard-coded thresholds and flags. Edit the file directly to change them; no restart required for PII Finder (restart required for chat sessions).
| Setting | Default | Description |
|---|---|---|
ENTITY_TRACE_HIGH_THRESHOLD |
0.90 |
spaCy score above which an entity is immediately confirmed |
ENTITY_TRACE_LOW_THRESHOLD |
0.70 |
spaCy score above which an entity is forwarded to ContextGuard |
CONTEXT_GUARD_ENABLED |
False |
Run ContextGuard; off in the default preset, where the PII tagger reads names |
CONTEXT_GUARD_CONFIDENCE_THRESHOLD |
0.70 |
distilbert score required to confirm a borderline entity |
ENTITY_TRACE_FALLBACK_THRESHOLD |
0.75 |
Score used when ContextGuard is disabled to promote borderline entities |
FUZZY_MATCH_THRESHOLD |
85 |
rapidfuzz threshold for ResolvePass reconstruction |
ADDRESS_MODE |
"auto" |
Address surrogation: "auto" (shift for service queries, replace otherwise), "shift" (house number ±range), or "replace" (structure-preserving fake) |
ADDRESS_SHIFT_RANGE |
1 |
Max house-number delta for shift mode (~one building at ±1) |
SERVICE_QUERY_DETECTION_ENABLED |
True |
Suppress standalone city/state replacement for location queries; drives ADDRESS_MODE="auto" |
CONTEXT_GUARD_DEVICE |
-1 |
ContextGuard inference device: -1 = CPU, 0+ = GPU id |
SHOW_API_TRANSPARENCY |
True |
Show the sent / received / restored transparency panel after each chat turn |
RAG_TOP_K |
3 |
Number of document chunks retrieved per RAG query |
RAG_CHUNK_SIZE |
512 |
Characters per chunk when splitting indexed documents |
CLAUDE_MODEL |
claude-sonnet-4-6 |
Claude model identifier |
SPACY_MODEL |
en_core_web_lg |
spaCy model used by EntityTrace and Presidio |
EMBEDDING_MODEL |
all-MiniLM-L6-v2 |
sentence-transformers model for RAG embeddings |
# Interactive dashboard (recommended)
python main.py
./run.sh
# Start a new conversation directly
python main.py chat
# Continue a saved conversation
python main.py chat --load <conversation-id>
# New conversation with RAG enabled
python main.py chat --rag
# List all saved conversations
python main.py list
# PII detection sandbox (no API calls, no credits)
python main.py pii-finder
# Index a document into the RAG store; list or remove indexed documents
python main.py add-doc path/to/document.txt
python main.py rag
python main.py rag --forget document.txt
| Key | Action |
|---|---|
N |
New conversation |
R |
New conversation with RAG mode |
P |
PII Finder: test detection without any API call |
1–9 |
Open saved conversation by number |
D1–D9 |
Delete conversation by number |
J |
JSON Test: batch-process a question file |
E |
Evaluation: score pipeline quality against ground-truth |
A |
Attacker Experiment: simulate adversarial PII recovery |
S |
Settings (provider, view mode) |
H |
Help |
Q |
Quit |
Place a question file in experiment/<name>.json:
[
{ "input": "My name is Priya Raman and my SSN is 512-44-7810. What are Wyoming's tax benefits?" },
{ "input": "My email is [email protected] and phone is 480-555-1234. Draft a resignation letter." }
]
Press J in the dashboard, enter the filename, select which fields to capture, and run. Output is saved to experiment/<name>_answers.json with progress flushed every 25 questions; safe to interrupt and resume.
| Field | What it captures |
|---|---|
question |
The original question text |
pattern_scan_pii |
Entities detected by PatternScan (regex stage) |
entity_trace_pii |
Entities detected by EntityTrace (spaCy NER) |
context_guard_pii |
Entities detected by ContextGuard (distilbert-NER) |
confirmed_pii |
Final combined confirmed entity list |
pii_detail |
Per-entity type, score, and source |
pii_spans |
Per-entity offsets, type and whether it was replaced (what the evaluator scores) |
quasi_id_risks |
Quasi-identifier combination risks detected |
surrogate_map |
Mapping of original → surrogate for every replaced entity |
sanitized_input |
The exact text sent to the LLM |
llm_response |
Raw LLM response (surrogates, before restoration) |
final_output |
The LLM response with the surrogates restored to the real values |
stage_timings_ms |
PatternScan / EntityTrace / ContextGuard / surrogate gen / LLM latency in ms |
presidio_sanitized_input |
Presidio [TYPE]-placeholder redaction (baseline for BERTScore comparison) |
presidio_found_piis |
Presidio raw detected entities: type, value, offsets, score |
recognized_not_replaced |
Entities detected as PII but intentionally skipped (e.g. topical geo in service queries) |
presidio_llm_response |
The LLM's answer to the Presidio-redacted text (one extra API call per question) |
clean_llm_response |
The LLM's answer to the original text (one extra API call per question; synthetic data only) |
bertscore_ss |
BERTScore (rescaled) of the SurrogateShield arm: input fidelity, and the answer against the clean answer when one was captured |
bertscore_presidio |
BERTScore (rescaled) of the same pairs for the Presidio arm |
By default, presidio_sanitized_input, presidio_found_piis, presidio_llm_response, clean_llm_response, bertscore_ss, and bertscore_presidio are off: enable them in the field selection screen to generate data for the Presidio comparison and BERTScore tables in Evaluation.
Every row also carries address_mode, the address setting the row was produced under. A question whose processing failed is written as question, error and error_type only, and is reprocessed when the run is resumed.
offline_eval.py scores the detection cascade directly against a key file — no
LLM calls, nothing leaves the machine. It is the fastest way to measure
detection accuracy while iterating:
# Overall + per-type precision / recall / F1 on the seeded dev split
# (experiment/make_dataset.py; tune on dev, never on synth_test_key.json)
python offline_eval.py --key experiment/synth_dev_key.json
# Protection check: gold values that reach the sent text, intended or not
python offline_eval.py --key experiment/synth_dev_key.json --protection --json out.json
# Focus on one type or cap the run
python offline_eval.py --key experiment/synth_dev_key.json --types address --limit 200
# Ablation: re-run the cascade with stages switched off and score each
python offline_eval.py --key experiment/synth_dev_key.json --ablation --json ablation.json
# Lint a key file for corrupt ground-truth values (values absent from the
# question, addresses that don't parse) — no models needed
python offline_eval.py --key experiment/experiment_key.json --lint-key
experiment/test_key.json (1,124 questions) is the legacy key. The detector
was tuned on it and the key was edited in the same commits, and
experiment/lint_key.py reports label errors in it (B1–B3), so a score on
it measures fit, not accuracy. The 0.9996 F1 in
experiment/offline_eval_v2.json came from that key and is not a claim.
The numbers to quote come from the sealed test-3 split of real chat messages,
scored with
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 .venv/bin/python -m bench.realdata.score --split test3 --out bench/results/realdata_test3.json;
see Measured results and bench/results/README.md.
Default matching is span-aware: because v2 detects addresses as full spans
(6720 Palm Dr, Phoenix, AZ) while the ground truth annotates components
separately, a key value counts as found when a detected span contains it.
Exact-boundary matches are counted alongside (the exact= figure in the
report).
Pair your question file with an answer-key file at experiment/<name>_key.json:
[
{
"Question": "My name is Priya Raman and my SSN is 512-44-7810...",
"Answer-Key": {
"name": "Priya Raman",
"ssn": "512-44-7810"
}
}
]
Press E in the dashboard to score a completed answers file against its key.
The evaluator maps flexible label names to internal types:
| Label(s) in key file | Internal type |
|---|---|
name, person, PERSON |
PERSON |
email |
email |
phone, phone_us, phone_uk, phone_intl |
phone |
ssn |
ssn |
address |
address |
dob, date_of_birth |
dob |
org, ORG, organization |
ORG |
gpe, GPE, location |
GPE |
loc, LOC |
LOC |
fac, FAC |
FAC |
credit_card |
credit_card |
api_key |
api_key |
ip_address, ip |
ip_address |
zip, zip_us, postcode, postcode_uk |
postal_code |
gender |
gender_indicator |
crypto, bitcoin, ethereum, wallet |
crypto |
bank_number, bank_account, us_bank_number, routing_number, routing |
us_bank_number |
driver_license, us_driver_license, drivers_license, dl, license |
us_driver_license |
url |
url |
handle, username |
handle |
credential, password |
credential |
age |
age |
id_number |
id_number |
passport |
passport |
iban |
iban |
vin |
vin |
mac_address |
mac_address |
license_plate |
license_plate |
| Metric | Description |
|---|---|
| Questions / Answered / Empty | Total count, non-empty responses, failures |
| Answer rate | Fraction of questions with non-empty LLM responses |
| Surrogate counts | Total found vs key total; averages per question |
| Precision / Recall / F1 / Accuracy | Overall surrogate detection quality |
| Error (miss rate) | Fraction of key PII values not detected |
| Stage timings | Average ms per pipeline stage |
| ResolvePass leak rate | Fraction of responses where a surrogate was not restored |
| Sanitisation quality | Fraction of questions where real PII reached the LLM |
| Per-entity-type breakdown | F1 / precision / recall per PII type |
| Presidio comparison (Table 1) | SS vs Presidio side-by-side per comparable type + overall |
| BERTScore comparison (Table 2) | Semantic utility preservation: SS vs Presidio vs no-anonymisation baseline |
| Ablation study (Table 4) | Not computed from answers files: the panel names python offline_eval.py --key <key file> --ablation |
The per-type comparison covers all types both systems can detect. Types SS detects that Presidio cannot are shown in a separate SS-Only table.
Comparable types (both SS and Presidio):
| Type | SS source | Presidio source |
|---|---|---|
| PERSON | EntityTrace / ContextGuard | Presidio NER |
| PatternScan | Presidio regex | |
| phone | PatternScan | Presidio regex |
| ssn | PatternScan | Presidio regex |
| credit_card | PatternScan (Luhn) | Presidio regex (Luhn) |
| ip_address | PatternScan | Presidio regex |
| dob (approximate) | PatternScan | Presidio DATE_TIME |
| GPE (approximate) | EntityTrace | Presidio LOCATION |
| crypto | PatternScan | Presidio CRYPTO |
| us_bank_number | PatternScan (ABA checksum) | Presidio US_BANK_NUMBER |
| us_driver_license | PatternScan (context-gated) | Presidio US_DRIVER_LICENSE |
SS-Only types (Presidio cannot detect these):
| Type | Notes |
|---|---|
api_key |
SK/Bearer/GHP/AKIA/AIzaSy prefixes |
address |
Structural street-address regex |
postal_code |
US ZIP + UK postcode |
gender_indicator |
Explicit gender declarations |
ORG |
Organisation names (NER) |
FAC |
Facility names (NER) |
BERTScore (roberta-large, rescaled with its baseline) is computed for two pairs:
clean_llm_response is collected (synthetic data only).Input fidelity favours realistic surrogates by construction, so only the utility pair says whether answers stay useful.
Measured on 298 real chat prompts from the sealed test-3 split (responder Claude Sonnet 4.6, judge Claude Opus 5.5):
| arm | utility: answer vs answer to the original (rescaled F1) |
|---|---|
| SurrogateShield | 0.442 |
| GLiNER-PII | 0.369 |
| Presidio + Faker | 0.365 |
| Presidio (placeholders) | 0.358 |
Paired difference in utility: +0.077 (95 % CI +0.053 to +0.103) against Presidio + Faker and +0.072 (+0.046 to +0.098) against GLiNER-PII. A pairwise judge preferred SurrogateShield's answer by +0.449 against Presidio + Faker and +0.051 against GLiNER-PII (132 wins to 117).
Command (makes provider calls): HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 .venv/bin/python -m bench.realdata.live utility --split test3 --out bench/results/utility_realdata_test3.json.
Enable the BERTScore SS and BERTScore Presidio fields in JSON Test to generate data for this table. The roberta-large model (~1.4 GB) is downloaded automatically on first use and can take 15–30 minutes to score on CPU.
The ablation measures how much each detection stage contributes. It is not
computed from an answers file: splitting the detected entities by the stage
that found them cannot measure how the stages interact, so the evaluation
screen's Ablation panel only names the command. offline_eval.py --ablation
re-runs the cascade on the key file once per configuration, with stages
switched off through the cascade options, and scores protection (replaced
spans only, span overlap):
| Configuration | Cascade options |
|---|---|
| PatternScan only | use_entity_trace=False, use_context_guard=False, use_tagger=False |
| PatternScan + EntityTrace (tagger and ContextGuard off) | use_context_guard=False, use_tagger=False |
| Every stage, post-passes A-C/E-H off | use_post_passes=False |
| Full cascade (shipped) | none |
For each configuration the report lists micro precision, recall and F1, the
F1 difference against the full cascade and a 95 % bootstrap interval of that
difference; --json also writes the macro scores and the options used.
--limit and --seed apply as for the main run.
python offline_eval.py --key experiment/synth_dev_key.json --ablation --json ablation.json
The Attacker Experiment (attacker.py) simulates an informed adversary who intercepts the sanitised text SurrogateShield sends to the LLM API and actively tries to recover the original PII values. It is the adversarial counterpart to the Evaluation suite and answers a critical research question: does replacing PII with realistic-looking surrogates (rather than [PLACEHOLDER] tokens) make it easier for an attacker to infer the originals?
Two variants are run on each question from an existing answers file:
| Variant | What the attacker sees | Goal |
|---|---|---|
| SurrogateShield | Sanitised text with realistic fake values (fake names, SSNs, emails, …) | Try to recover the originals from the surrogates |
| Presidio | Redacted text with [ENTITY_TYPE] placeholder tokens |
Try to recover the originals from the placeholders |
The attacker is given a carefully constructed adversarial prompt that discloses the PII types that were replaced and instructs the model to use every available inference technique; linguistic analysis, contextual reasoning, demographic inference, cross-field correlation, format patterns, and more.
The experiment counts how many values an attacker model recovers from each
arm, exactly or in part. A value that an arm left in plain text is counted
as leaked, not recovered. Results are listed in
bench/results/README.md.
Measured on the sealed test-3 split: 636 values in real chat messages, attacker Claude Opus 5.5. An attacker knows a value letter for letter when the arm left it in plain text or when the guess equals it exactly:
| arm | left in plain text | recovered exactly | known letter for letter |
|---|---|---|---|
| SurrogateShield | 7 | 4 of 620 | 11 of 627 (1.8 %) |
| GLiNER-PII | 73 | 6 of 556 | 79 of 629 (12.6 %) |
| Presidio + Faker | 201 | 8 of 428 | 209 of 629 (33.2 %) |
| Presidio | 201 | 14 of 428 | 215 of 629 (34.2 %) |
| LLM Guard | 364 | 19 of 264 | 383 of 628 (61.0 %) |
Commands: HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 .venv/bin/python -m bench.realdata.live attacker --split test3 --out bench/results/attacker_realdata_test3.json (makes provider calls), then HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 PYTHONPATH=.:python-library .venv/bin/python -m bench.realdata.attacker_exact --split test3 --out bench/results/attacker_exact_test3.json (no provider calls).
Press A in the dashboard. The four-screen flow will:
experiment/ (must contain surrogate_map, sanitized_input, and optionally presidio_sanitized_input / presidio_found_piis fields)Output files are written to experiment/:
| File | Contents |
|---|---|
<stem>_Attacker_Experiment.json |
Per-question recovery attempt details for both variants |
<stem>_Attacker_Experiment_Analysis.json |
Aggregated recovery rates, per-type breakdown, overall assessment |
The experiment supports resume: if interrupted, re-running with the same answers file picks up where it left off.
| Metric | Description |
|---|---|
| Questions available | Questions with SS / Presidio data to attack |
| Total targeted | Total PII values the attacker attempted to recover |
| Total recovered | Values where the attacker's guess matched the original (exact, lowercased) |
| Recovery rate | recovered / targeted: lower is better for privacy |
| Recovery rate (excl. address) | Rate excluding address-type entities (service queries fuzz addresses rather than fully replace them, so proximity recovery is theoretically possible; tracked separately) |
| By-type breakdown | Per-PII-type targeted/recovered counts and rate |
Street addresses in service queries receive house-number fuzzing (±address_shift_range, default 1) rather than full replacement. Exact address recovery is still impossible, but proximity-based recovery is theoretically possible. Address results are tracked separately via address_recovered_count / non_address_recovered_count and excluded from the primary recovery rate to keep the comparison fair.
Enable these fields in JSON Test before running so the experiment has both arms (the prompt itself carries no type metadata; an arm whose text is missing is recorded as unavailable):
| Field | Required for |
|---|---|
sanitized_input |
SS arm: the text to attack |
surrogate_map |
SS arm: which values were replaced, so a value still visible verbatim is scored as leak-through, not as an inference |
presidio_sanitized_input |
Presidio arm: the placeholder-redacted text to attack |
presidio_found_piis |
Presidio arm: which values Presidio redacted, for the same leak-through scoring |
tests/test_attacker.py)test_attacker.py covers attacker.py with a fake provider client; no real API key or network required:
| Test area | What is verified |
|---|---|
| Prompts | Neutral attacker prompts for both arms, placeholder-aware for Presidio, braces escaped |
| Models | The attacker model differs from the responder |
| Scoring | Match levels per value, leaked values and non-gold guesses kept separate, the same units for every arm |
| Failures | Failed or invalid attacker replies are counted as unavailable and excluded |
| Runner | Sampling, call budget, metadata and a deterministic plan |
| Screen | The attacker screen prints computed values only; old-format results files are refused |
# Fast, model-free suite (the default; no API key required)
.venv/bin/python -m pytest tests python-library/tests -q
# Tests that load spaCy / HuggingFace models, including the older
# check scripts in tests/, which run through tests/test_legacy_scripts.py
.venv/bin/python -m pytest tests python-library/tests -q -m heavy
Always run the tests through pytest. tests/ holds 141 pytest modules,
one per area (detection stages, surrogates, ShadowMap, ResolvePass,
settings and the CLI, RAG, the attacker experiment, the benchmark code,
the docs), plus the older check scripts that tests/test_legacy_scripts.py
runs and gates on. python-library/tests/test_public_api.py covers the
library's public API. No test needs an API key; provider clients are fakes.
SurrogateShield is also available as a standalone pip package (version 1.9.0); no dashboard, no CLI, just the core detection and masking pipeline as a clean Python API. The package is self-contained: it does not import from the main application, and the application is built on it.
Install it from the repository root, then the spaCy model:
pip install ./python-library
python -m spacy download en_core_web_lg
The PII tagger used by the default balanced preset is downloaded once from Hugging Face (sherwinvishesh/surrogateshield-pii-tagger, ~565 MB) the first time it is needed, or ahead of time with surrogateshield download; its weights are checked against their SHA-256 pin, and later runs read the local cache with no network. If they cannot be fetched, detection raises DetectorUnavailable instead of sending unmasked text. The classic preset works without them. The ContextGuard model (dslim/distilbert-NER) is off by default and downloads from HuggingFace Hub on first use only when context_guard_enabled=True or a preset that uses it is chosen.
import surrogateshield as shield
user_text = (
"Hi, I'm Priya Raman. My email is [email protected], "
"my SSN is 512-44-7810, and I was born on 04/12/1990."
)
sanitized = shield.mask(user_text)
# for example:
# "Hi, I'm Yashica Ahluwalia. My email is [email protected],
# my SSN is 856-14-3146, and I was born on 01/07/1992."
response = any_llm.chat(sanitized) # the detected values are sent as surrogates
restored = shield.unmask(response) # real values restored from the session map
shield.flush() # clear session before the next conversation
unmask() accepts native SDK response objects directly; no manual text extraction needed:
# Anthropic
import anthropic
import surrogateshield as shield
client = anthropic.Anthropic()
sanitized = shield.mask(user_text)
response = client.messages.create(model="claude-opus-4-8", max_tokens=1024,
messages=[{"role": "user", "content": sanitized}])
print(shield.unmask(response)) # passes Anthropic Message object directly
shield.flush()
# OpenAI
from openai import OpenAI
client = OpenAI()
sanitized = shield.mask(user_text)
response = client.chat.completions.create(model="gpt-4o",
messages=[{"role": "user", "content": sanitized}])
print(shield.unmask(response)) # passes ChatCompletion object directly
shield.flush()
# Gemini
from google import genai
client = genai.Client()
sanitized = shield.mask(user_text)
response = client.models.generate_content(model="gemini-2.5-flash", contents=sanitized)
print(shield.unmask(response)) # passes GenerateContentResponse directly
shield.flush()
# Any local model / plain string
raw_reply = ask_ollama(sanitized)
print(shield.unmask(raw_reply)) # plain strings work too
shield.flush()
| Function | Description |
|---|---|
shield.config(**kwargs) |
Set the detection preset, thresholds, storage mode, pii_off list, and display options |
shield.mask(text) |
Detect PII, replace it with surrogates, store the mapping |
shield.mask_result(text) |
Like mask, but also returns the detections (including pii_off ones sent as typed) and the replacements made |
shield.unmask(response) |
Restore original PII in an LLM response (any provider object or plain string) |
shield.scan(text) |
Detect PII and return a list of Detection (text, type, start, end, score, source, masked); as_dict=True returns {value: type}. No substitution, no shadow map update |
shield.pii_finder |
Alias for shield.scan |
shield.forget(original) |
Erase every mapping for one original value from the session |
shield.flush() |
End the current session; the next call starts a new one |
shield.Session(...) |
An explicit session with its own settings and mappings, for servers and concurrent use |
found = shield.scan(
"Contact Alice Nguyen at [email protected] or call +1-415-555-0198."
)
for d in found:
print(f"{d.type:15s} {d.text} ({d.source})")
# PERSON Alice Nguyen (pii_tagger)
# email [email protected] (pattern)
# phone_us +1-415-555-0198 (pattern)
shield.scan("...", as_dict=True)
# {"Alice Nguyen": "PERSON", "[email protected]": "email", "+1-415-555-0198": "phone_us"}
scan() returns every detected entity; one whose type is in pii_off has masked=False. It is safe to call in a read-only context; it never modifies the session.
shield.config(pii_off=["location", "org"])
sanitized = shield.mask(
"Emma Johnson works at Deloitte in New York, email [email protected]."
)
# "Emma Johnson" → replaced (PERSON not in pii_off)
# "[email protected]" → replaced (email not in pii_off)
# "Deloitte" → kept (org in pii_off)
# "New York" → kept (location in pii_off)
Available aliases: phone, name, location, org, email, ssn, dob, address, zip, postcode, postal_code, credit_card, ip_address, api_key, crypto, bank, license, gender_indicator. Raw type strings ("PERSON", "GPE", etc.) also accepted.
By default the session mapping is in-memory only. For web servers or multi-process deployments, point pii_mem at a directory and the map is encrypted to disk with AES-256-GCM:
import os, surrogateshield as shield
os.makedirs("/var/app/shadowmaps", exist_ok=True)
shield.config(pii_mem="/var/app/shadowmaps")
sanitized = shield.mask("My name is Clara Oswald, phone 555-123-4567.")
restored = shield.unmask(llm.chat(sanitized))
shield.flush() # deletes this session's .shadowmap file
| Parameter | Default | Effect |
|---|---|---|
detection |
"balanced" |
Detection preset: "balanced", "strict", "fast" or "classic", a DetectionConfig, or a dict of one (see CONFIGURATION.md) |
detailed_view |
False |
Print Rich tables after each call; they show the original values |
pii_mem |
"temp" |
"temp" for in-memory; a directory path for AES-256-GCM disk persistence |
pii_off |
[] |
PII types to detect but not replace |
service |
True |
Minimal address fuzzing for location queries instead of full replacement |
address_mode |
"auto" |
"auto" (shift for service queries, replace otherwise), "shift" or "replace" |
context_guard_enabled |
False |
Run the HuggingFace second-pass NER (dslim/distilbert-NER) |
fuzzy_threshold |
85 |
rapidfuzz score (0–100) used in the fuzzy reconstruction pass of unmask() |
spacy_model |
"en_core_web_lg" |
spaCy model for EntityTrace; swap for en_core_web_sm for faster but lower-accuracy NER |
The same detection pipeline as the full application, reporting the 13 public types listed under PII Types Detected: PERSON, ORG, LOCATION, ADDRESS, EMAIL, PHONE, URL, HANDLE, ID, CREDENTIAL, NETWORK, AGE and DATE_OF_BIRTH. PatternScan and the canonicaliser find the structured values, the PII tagger reads names, organisations and places, spaCy reads places, and ContextGuard can be switched on as an extra pass.
See python-library/README.md for the complete API reference, all config parameters, surrogate generation details, and troubleshooting.
SurrogateShield/
├── main.py # CLI entry point and interactive dashboard
├── pipeline.py # End-to-end message pipeline orchestration
├── config.py # All constants and thresholds (single source of truth)
├── util.py # Shared dataclasses (DetectedEntity, Conversation), logging helpers
├── settings_manager.py # Persistent user settings (~/.surrogateshield/settings.json)
├── evaluator.py # Precision/recall/F1 evaluation logic + Presidio/BERTScore/ablation study
├── json_tester.py # Batch JSON question processing
├── attacker.py # Adversarial PII recovery experiment (Attacker Experiment)
├── run.sh # Launcher: pre-flight checks, then the venv's python main.py
├── requirements.txt
│
├── detection/ # SentinelLayer — aliases or thin config.py wrappers of the library's detection modules
│ ├── logic.py # Cascade orchestration + post-processing passes A–D
│ ├── pattern_scan.py # PatternScan — regex-based structured PII detection
│ ├── entity_trace.py # EntityTrace — spaCy NER (en_core_web_lg)
│ ├── context_guard.py # ContextGuard — distilbert-NER (dslim/distilbert-NER)
│ ├── service_query.py # ServiceQueryDetector — address fuzzing for location queries
│ ├── quasi_identifier.py # Quasi-identifier combination risk scorer (k-anonymity)
│ └── geo_data.py # Geographic pass-through whitelist (US states, countries)
│
├── generation/
│ └── logic.py # MimicGen — type-consistent surrogate generation (Faker)
│
├── storage/
│ └── logic.py # ShadowMap — AES-256-GCM encrypted surrogate mapping store
│
├── reconstruction/
│ └── logic.py # ResolvePass — four-pass surrogate→original restoration
│
├── presidio/ # Presidio integration layer (optional comparison feature)
│ ├── engine.py # Lazy singleton AnalyzerEngine wrapper
│ ├── detect.py # detect(text) → list[PresidioEntity]
│ └── redact.py # redact(text, entities) → [TYPE]-placeholder string
│
├── chatbot/
│ ├── chat.py # Multi-provider LLM conversation handler (Claude/Gemini/OpenAI/Ollama)
│ └── rag.py # RAGStore — ChromaDB + sentence-transformers vector store
│
├── experiment/ # Batch test input/output files
│ ├── example.json # Sample question file
│ ├── example_key.json # Sample ground-truth answer key
│ ├── example_answers.json # Sample output from JSON Test
│ ├── *_Attacker_Experiment.json # Per-question attacker results (auto-generated)
│ └── *_Attacker_Experiment_Analysis.json # Aggregated recovery-rate analysis (auto-generated)
│
├── tests/ # Test suite, run with pytest (no API key required)
│ ├── conftest.py # Shared fixtures
│ ├── test_*.py # 141 pytest modules, one per area; model-loading ones are marked heavy
│ ├── test_legacy_scripts.py # Runs the older check scripts below and fails on any failed check
│ └── test1.py … test5.py # Older check scripts (run only through pytest)
│
├── python-library/ # Standalone pip package (surrogateshield)
│ ├── pyproject.toml # Package metadata and dependencies
│ ├── README.md # Full library API reference
│ ├── tests/
│ │ └── test_public_api.py # The library's public API
│ └── surrogateshield/ # Package source — self-contained, zero imports from the main app
│ ├── __init__.py # Public API: config, scan, mask, mask_result, unmask, forget, flush
│ ├── session.py # Session, Detection, MaskResult
│ ├── _state.py # Module-level cfg and session singletons
│ ├── _display.py # Optional Rich terminal output (graceful fallback if Rich absent)
│ ├── _response_parser.py # extract_text() — Anthropic / OpenAI / Gemini / plain string
│ └── core/
│ ├── entities.py # DetectedEntity, mask_spans, remove_span_overlap
│ ├── detection/
│ │ ├── pipeline.py # run_cascade(), deduplicate(), pii_off alias table
│ │ ├── pattern_scan.py # regex scanner (25 regex types plus the address parser) with Luhn / ABA / IBAN validators
│ │ ├── entity_trace.py # spaCy NER (model-name-keyed cache)
│ │ ├── context_guard.py # distilbert-NER second pass (optional)
│ │ ├── pii_tagger.py # PII tagger (DeBERTa-v3-small) for names, organisations, places
│ │ ├── canonical.py # Canonicaliser: rewritten views for PatternScan
│ │ ├── relation_gate.py # Decides whether a name is personal or public
│ │ ├── resolver.py # Keeps one candidate when several stages report a value
│ │ ├── config.py # DetectionConfig, presets, public types
│ │ ├── service_query.py # Service-query detection + address fuzzing
│ │ ├── quasi_identifier.py # Quasi-identifier combination risk scorer
│ │ └── geo_data.py # Geographic pass-through whitelist
│ ├── generation/
│ │ └── mimic.py # MimicGen — Faker-based surrogate generation
│ ├── storage/
│ │ └── shadow_map.py # ShadowMap — memory mode + AES-256-GCM disk mode
│ └── reconstruction/
│ └── resolve.py # ResolvePass — exact / component / fuzzy / given-name restoration
│
~/.surrogateshield/ # Runtime ($SURROGATESHIELD_HOME), 0700, created on first use
├── device.key # 32-byte device secret, 0600
├── settings.json
└── conversations/
├── <conv_id>.json # Encrypted transcript: display history (real values) + API history (surrogates)
└── <conv_id>.shadowmap # Encrypted surrogate→original mapping
| Component | Mechanism |
|---|---|
| Device secret | 32-byte random key at ~/.surrogateshield/device.key, 0o600 permissions, directory 0o700 |
| Per-conversation key | HKDF-SHA256: device secret as IKM, conversation ID as salt, file kind as info |
| ShadowMap encryption | AES-256-GCM with fresh 12-byte nonce per write; kind and ID as associated data |
| ShadowMap format | "SSv1" ‖ nonce (12 bytes) ‖ AES-GCM ciphertext: unreadable without device key |
| API transmission | Detected values are replaced by surrogates before the request is made |
| Conversation history | Stored locally in ~/.surrogateshield/conversations/, encrypted like the ShadowMap; it contains the real values you typed (display history) as well as the surrogate text sent to the API |
.gitignore |
*.shadowmap, conversations/, device.key, .env, and new experiment outputs (experiment/*_answers*.json, *_eval_results*.json, *_Attacker_Experiment*.json) excluded |
Measured on the sealed test-3 split: real chat messages from ShareGPT,
WildChat and oasst1 that were never used for tuning. Every number, the
commit it was run at and the per-dataset breakdown are in
bench/results/README.md. All commands below
start with HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 .venv/bin/python -m.
Personal values let through (3,677 values placed in real messages).
bench.realdata.score --split test3 --out bench/results/realdata_test3.json
| arm | values leaked | prompts sent clean | odd-format values leaked | harmless text edited |
|---|---|---|---|---|
| SurrogateShield | 0.8 % (30 of 3,677) | 97.4 % | 0.2 % (1 of 530) | 13.9 % |
| GLiNER-PII (threshold 0.3) | 7.3 % | 79.5 % | 7.5 % | 41.3 % |
| GLiNER-PII (threshold 0.5) | 9.1 % | 75.4 % | 8.3 % | 35.4 % |
| Presidio / Presidio + Faker | 31.6 % | 30.8 % | 43.6 % | 32.6 % |
| Presidio + RoBERTa | 54.0 % | 11.7 % | 66.0 % | 22.0 % |
| LLM Guard | 56.3 % | 9.1 % | 61.5 % | 16.4 % |
A second, outside dataset (Nemotron-PII, 1,000 records, 4,627 values).
bench.realdata.score --external nemotron_pii --out bench/results/external_nemotron_pii.json
Answer quality (298 prompts; makes provider calls).
bench.realdata.live utility --split test3 --out bench/results/utility_realdata_test3.json
Multi-turn conversations (60 conversations; makes provider calls).
bench.realdata.live multiturn --split test3 --out bench/results/multiturn_realdata_test3.json
Attacker (636 values; makes provider calls).
bench.realdata.live attacker --split test3 --out bench/results/attacker_realdata_test3.json, then
bench.realdata.attacker_exact --split test3 --out bench/results/attacker_exact_test3.json (with PYTHONPATH=.:python-library)
Speed and memory. bench.perf_arms --out bench/results/perf_arms_v4.json
| arm | median latency | peak memory |
|---|---|---|
| Presidio | 8.2 ms | 1,084 MB |
| Presidio + Faker | 8.4 ms | 1,010 MB |
| SurrogateShield | 43.5 ms | 1,348 MB |
| LLM Guard | 58.0 ms | 1,565 MB |
| GLiNER-PII (0.3 / 0.5) | 79.9 / 82.1 ms | 2,333 / 2,318 MB |
| Presidio + RoBERTa | 83.5 ms | 1,611 MB |
All ten checks in
PYTHONPATH=.:python-library .venv/bin/python -m bench.realdata.verdict --split test3 --out bench/results/verdict_test3.json
hold.
Limitations: it trailed on the 96 real values in natural text (29 leaked; GLiNER-PII 16–20, Presidio 27), on some single types (3 of 13 on test-3, 8 of 13 on Nemotron-PII, URL most of all) and on speed and memory (plain Presidio is about 5× faster and lighter); it came out ahead on every other test. Every number, with its command, is in bench/results/README.md.
Made with ❤️ by Sherwin Vishesh Jathanna