okasi/gliner2-privacy-filter-pii-multi-onnx

GLiNER2 Privacy Filter — ONNX (Q4 + Q8)

Memory-efficient ONNX Runtime packages for fastino/gliner2-privacy-filter-PII-multi, exported for CPU deployment with blockwise MatMulNBits weight quantization.

This repository ships two production variants:

Variant	Approx. size	Default	Best for
model_q8.onnx	510 MB	yes	Highest SPY accuracy and fastest CPU latency in our tests
model_q4.onnx	458 MB	no	Smallest footprint, strong accuracy

Both variants share the same tokenizer, label schema, and Python preprocessing/postprocessing contract.

Why this export exists

The upstream GLiNER2 model runs a schema-conditioned entity extractor backed by microsoft/mdeberta-v3-base. The original PyTorch forward() path uses a non-exportable preprocessing dataclass, so this package splits inference into:

1. Python preprocessing — GLiNER2 SchemaTransformer + ExtractorCollator
2. ONNX core — tensor-only graph producing count_logits and span_scores
3. Python postprocessing — span decoding back to character offsets

That split keeps the runtime small (no bundled PyTorch weights) while preserving the upstream label schema.

Entity labels

This package uses the SPY-aligned 7-label schema baked into entity_labels.json:

• name
• address
• email
• phone_num
• id_num
• url
• username

The schema token positions are fixed at export time (schema_positions in metadata). Do not change label order without re-exporting.

Evaluation — SPY 200 (char-exact, threshold 0.5)

Benchmark split: 200 English examples from the SPY PII benchmark (100 legal + 100 medical), character-exact span matching.

Benchmark protocol (matches the mks-logic/SPY loader and upstream Fastino evaluation):

• Source corpus: SPY placeholder templates with faker_random_seed=42 (official SPY.py recipe)
• Frozen slice: selection_seed=42, per_domain=100 (100 legal + 100 medical)
• Evaluation labels: the 7 SPY PII labels in entity_labels.json
• Character-exact span matching at threshold 0.5
• Examples are pre-materialized in benchmarks/spy_manifest.json / bundled result files; scoring reads the frozen slice only

Model	Avg F1	Legal F1	Medical F1	Mean latency (CPU)
openai/privacy-filter (PyTorch, measured)	0.368	0.357	0.379	—
fastino/gliner2-privacy-filter-PII-multi (PyTorch, measured)	0.490	0.482	0.497	—
This repo — ONNX Q8 (default)	0.534	0.521	0.546	704 ms
This repo — ONNX Q4	0.516	0.516	0.517	759 ms

Official full-SPY leaderboard numbers from the upstream model cards (full corpus, not this 200-doc slice):

Model	Avg F1	Legal F1	Medical F1
openai/privacy-filter	0.380	0.354	0.406
fastino/gliner2-privacy-filter-PII-multi	0.477	0.473	0.480

Comparison vs OpenAI Privacy Filter

On this English SPY-200 slice, both ONNX variants outperform openai/privacy-filter measured on the same data:

• Q8 (default) (0.534) is +45.2% vs OpenAI PyTorch (0.368)
• Q4 (0.516) is +40.5% vs OpenAI PyTorch (0.368)

Q8 also beats the measured upstream GLiNER2 PyTorch score on this slice (0.490) while using roughly 458–510 MB of ONNX weights instead of a full PyTorch checkpoint.

Detailed result files are included under benchmarks/.

Q8 per-label F1 (SPY overall)

Label	Precision	Recall	F1
address	0.727	0.846	0.782
email	0.360	0.946	0.522
id_num	0.610	0.734	0.667
name	0.224	0.897	0.358
phone_num	0.349	0.881	0.500
url	0.415	0.741	0.532
username	0.568	0.690	0.623

Q4 per-label F1 (SPY overall)

Label	Precision	Recall	F1
address	0.724	0.727	0.725
email	0.360	0.946	0.521
id_num	0.593	0.701	0.642
name	0.230	0.901	0.366
phone_num	0.348	0.820	0.488
url	0.397	0.656	0.495
username	0.572	0.636	0.603

Model files

.
├── model_q8.onnx              # Q8 graph (default)
├── model_q8.onnx.data
├── model_q8_metadata.json
├── model_q4.onnx              # Q4 graph
├── model_q4.onnx.data
├── model_q4_metadata.json
├── entity_labels.json         # fixed label schema
├── config.json
├── tokenizer.json
├── tokenizer_config.json
├── encoder_config/config.json
├── onnx_export_metadata.json
├── benchmarks/
│   ├── spy_q4.json
│   ├── spy_q8.json
│   ├── spy_fastino_pytorch.json
│   ├── spy_openai_pytorch.json
│   ├── spy_reference_scores.json
│   └── spy_manifest.json
└── examples/
    ├── onnx_runtime.py
    ├── predict.py
    ├── preprocess_feed.py
    └── transformers_js/
        ├── decode.mjs
        ├── index.html
        ├── infer.mjs
        ├── package.json
        └── smoke_inputs.json

Quantization uses ONNX Runtime MatMulNBits (symmetric, block size 128, MatMul weights only). External weight blobs use the .onnx.data sidecar format.

Quick start

Install

pip install onnxruntime gliner2 transformers

Download this repository:

hf download okasi/gliner2-privacy-filter-pii-multi-onnx --local-dir ./gliner2-onnx

Inference

from pathlib import Path
import sys

sys.path.insert(0, str(Path("./gliner2-onnx/examples")))
from onnx_runtime import Gliner2OnnxSession

session = Gliner2OnnxSession(Path("./gliner2-onnx"), onnx_model_file="model_q8.onnx")
text = "John Doe lives at 742 Evergreen Terrace, email john@example.com."
entities = session.predict_entities(text, threshold=0.5)
print(entities)

Or run the CLI helper:

python examples/predict.py --model-dir . --onnx-model-file model_q8.onnx

Transformers.js (browser + Node)

This repo includes a Transformers.js example that loads the tokenizer with @huggingface/transformers and runs the exported ONNX core with ONNX Runtime Web/Node.

Browser demo:

cd examples/transformers_js
npx serve .
# open index.html

Node demo:

cd examples/transformers_js
npm install --ignore-scripts
node infer.mjs

The browser demo uses examples/transformers_js/smoke_inputs.json for GLiNER2 schema preprocessing on the bundled smoke sentence. The Node demo uses the Python helper in examples/preprocess_feed.py for general text.

Choosing Q4 vs Q8

• model_q8.onnx — default; highest SPY F1 and best CPU latency in our tests
• model_q4.onnx — use when you need the smallest ONNX package (~50 MB less weights)

ONNX inputs and outputs

Name	Shape	dtype	Description
input_ids	[batch, seq]	int64	Token IDs
attention_mask	[batch, seq]	int64	Attention mask
text_word_indices	[batch, words]	int64	Word index map
text_word_counts	[batch]	int64	Words per example
count_logits	[batch, labels]	float	Span-count logits per label
span_scores	[batch, labels, words]	float	Word-level span scores

Export provenance

• Source model: fastino/gliner2-privacy-filter-PII-multi
• Export kind: gliner2_entity_onnx
• Opset: 17
• Generated: 2026-05-24T12:16:37.932073+00:00
• Publisher repo: okasi/gliner2-privacy-filter-pii-multi-onnx

Limitations

• Label schema is fixed at export time; dynamic label sets require re-export.
• Pre/post-processing currently depends on the gliner2 Python package.
• ONNX Runtime CPU MatMulNBits supports 2/4/8-bit weights; Q6 is not available. This repo intentionally ships Q4 and Q8 only.
• High-recall behavior on name and email can increase false positives versus precision-focused deployments; tune threshold for your use case.

License

This ONNX redistribution follows the upstream fastino/gliner2-privacy-filter-PII-multi terms (Apache 2.0). Attribution to Fastino for the base model is appreciated.

Citation

If you use the base GLiNER2 model, cite the upstream Fastino work and GLiNER2 paper linked from the base model card.