Perplexity Introduces PII-TRACE Benchmark and PII-Tracer On-Device Detector – Unite.AI
Perplexity on September 1, 2026, introduced PII-TRACE, a benchmark for evaluating detectors of personally identifiable information, and PII-Tracer, a compact 0.6B-parameter model built to flag PII on a user’s device before text is sent to cloud models. The announcement frames both releases as privacy infrastructure for the company’s hybrid compute architecture, in which cloud agents handle research, reasoning, and planning while a local model works with private files.
Under that architecture, a local privacy gate keeps sensitive content on the Mac, redacts detected private information, or requests approval before sending it to the cloud. Perplexity said the boundary only protects privacy if the device can recognize PII before text is sent to a remote model, and that detection becomes harder across long, multilingual conversations where the same identifier may appear several times across different turns. One missed mention can expose the information the system is meant to protect.
PII-Tracer supplies one local control signal for model routing by flagging spans predicted to contain PII. The application then enforces the routing policy: it keeps the relevant input local, redacts detected spans, or requests explicit approval before escalating to a cloud model.
The PII-TRACE Benchmark
PII-TRACE, short for Tracing Recurring PII Across Conversational Exchanges, contains 13,148 synthetic user-assistant conversations across 13 languages and 10 writing systems, with 37,431 identifier mentions labeled at the character level across nine PII types. A total of 41% of the conversations contain structured content. Among the 5,645 conversations with labeled PII, 63.8% include an identifier that appears more than once, and 28.7% include an identifier that appears across multiple turns.
Perplexity said it designed the benchmark around three behaviors that matter when a detector screens assistant conversations. The first is consistent coverage: when an identifier appears several times or crosses user and assistant turns, the detector needs to find every mention. The second is robustness to long context, with conversations ranging from fewer than 1,000 to more than 100,000 characters. The third is handling multiple languages and mixed-format content, since a conversation may switch languages and combine prose with code, tables, or structured records.
The benchmark measures performance at two levels. At the identifier level, a consistent detection score asks whether the detector covered every character in every mention of the same identifier, reported separately for identifiers with multiple mentions and for identifiers that repeat across turns. At the character level, precision measures how much of the text marked by a detector is labeled PII, recall measures how much labeled PII it finds, and F1 balances the two.
The dataset is synthetic but derived from production conversations. According to the company, multiple language models first mark nine types of PII in production user-assistant conversations, and a rule-based pass groups repeated identifiers of the same type under one entity ID. Each marked value is replaced by a typed placeholder, each turn is paraphrased with placeholders intact, and synthetic values matching each identifier’s type and format are inserted. Three automated gates check that replacements match the stored spans, that repeated mentions use the same value, and that marked source values are absent under a Presidio and regular-expression rescan. A second language model audits a sample, and humans review anything it flags as PII.
A Compact Detector for Local Use
PII-Tracer is a 0.6B bidirectional encoder adapted from a Qwen3 backbone. Perplexity said privacy screening differs from text generation because it requires finding relevant PII spans and returning their boundaries, so the model replaces Qwen3’s causal mask with padding-aware bidirectional attention, letting every token draw on both earlier and later turns within a 4,096-token window.
For each token, the encoder produces a 1,024-dimensional representation, and a linear tagging head scores 37 possible labels under the BIOES scheme for named entity recognition: one label for text outside a PII span, plus four span-position labels for each of the nine PII types. An auxiliary head predicts whether the conversation contains sensitive material, such as health or religious information. The company said it trained the model for three epochs on roughly 714,000 training samples combining multilingual assistant conversations with single-record examples. At inference time, a constrained Viterbi decoder searches for the highest-scoring valid label sequence and maps the result back to exact character spans for redaction or local routing.
Evaluation Results
Perplexity reported that PII-Tracer achieved the highest character F1 (0.629) among the 12 systems evaluated, along with the second-highest span-overlap F1 and span-containment F1. The company said frontier models, namely GPT-5.6-sol and Claude Sonnet 5, achieved comparable overall performance, with GPT-5.6-sol scoring higher on both span-level metrics but lower on character F1. It noted that those frontier models have hundreds of billions or even trillions of parameters and are closed-source models hosted in the cloud, making them unsuitable for screening text that must remain local, while other open-source PII detectors performed substantially worse than PII-Tracer.
On the consistency test, the evaluation set contains 899 identifiers that appear once and 959 that appear more than once, 790 of them spanning multiple turns. Perplexity reported that PII-Tracer finds every mention of 79.4% of recurring identifiers and 77.6% of cross-turn identifiers, while GPT-5.6-sol reaches 57.0% and 55.1% on the same two measures. PII-Tracer’s score moves from 0.917 for single-mention identifiers to 0.691 for identifiers appearing six to ten times.
Length remains a limitation. Single-window recall is 0.975 for conversations below 1,000 characters and 0.955 from 1,000 to 10,000, but drops to 0.687 at or above 10,000 characters. The company said decoding the same checkpoint with 50%-overlap sliding windows raises overall character recall from 0.830 to 0.965 and multi-mention consistent detection from 0.794 to 0.954, without retraining.
Across a six-language slice spanning Latin, Cyrillic, and Hangul scripts, PII-Tracer leads character F1 in German (0.735), French (0.633), Italian (0.676), and Russian (0.651), and sits within 0.016 and 0.036 of the best results in English and Korean, the company reported. On five external single-record benchmarks, Perplexity said PII-Tracer posted higher character F1 than the OpenAI Privacy Filter on every dataset, including 0.950 versus 0.907 on ai4privacy, 0.847 versus 0.709 on Nemotron-PII, and 0.594 versus 0.350 on TAB, the only benchmark in the group built from real, human-labeled text.
The research paper cautions that the conversations are synthetic reconstructions derived from the structure of production assistant traffic, so results should be read as measurements of conversational PII detection rather than estimates for any particular production workload. The paper also notes that tool calls, inter-agent messages, and multimodal inputs fall outside the benchmark’s scope, and that each baseline was evaluated under a single inference configuration. Perplexity said it plans to release both PII-TRACE and PII-Tracer soon; the paper states both will be released under the MIT license.
