PIIShield- Enterprise PII Redaction Pipeline

PIIShield — Enterprise PII Redaction Pipeline

PIIShield is an end-to-end document privacy pipeline designed to identify and redact Personally Identifiable Information (PII) from DOCX documents.

The system accepts a DOCX document, extracts its content through structured Bronze and Silver processing layers, detects multiple categories of sensitive information, resolves overlapping detection candidates, replaces detected values with deterministic placeholders, reconstructs a separate redacted DOCX, and records evidence in Gold-layer outputs.

The system supports nine PII categories:

PERSON EMAIL PHONE COMPANY ADDRESS SSN CREDIT_CARD DOB IP_ADDRESS

The project also includes a Streamlit-based web interface and a Render deployment for demonstrating the complete pipeline.

The Problem

Business documents frequently contain sensitive information such as names, email addresses, phone numbers, addresses, identification numbers, dates of birth, and financial information.

Manually removing this information is:

Time-consuming Error-prone Difficult to reproduce Difficult to audit Vulnerable to formatting damage

A practical privacy solution therefore needs to do more than simply find sensitive strings.

It must reliably detect PII, handle overlapping matches, preserve the original document structure, produce consistent replacements, and provide evidence showing what was detected and redacted.

The Solution

PIIShield implements a reproducible document-processing pipeline.

The workflow is:

DOCX Upload → Bronze Extraction → Silver Normalization → PII Detection → Candidate Resolution → Deterministic Replacement → DOCX Reconstruction → Gold Evidence

The application accepts a DOCX file through the Streamlit interface and produces a separate redacted document that can be downloaded.

The system deliberately keeps the public deployment safe by excluding the original prospectus and raw source PII from the public repository.

Architecture

1. Ingestion

The uploaded DOCX document is processed programmatically rather than manually.

The ingestion layer extracts relevant document content while retaining the information required for later reconstruction.

1. Bronze Layer

The Bronze layer represents the extracted document content in a form suitable for downstream processing.

This provides a structured representation of the original document before normalization and detection.

1. Silver Layer

The Silver layer prepares extracted content for reliable PII detection.

Normalization allows different representations of the same underlying content to be processed consistently.

1. PII Detection

The unified detector searches for the nine required PII categories.

Detection combines structured pattern-based techniques with category-specific validation rather than treating every numeric or textual value as PII.

1. Candidate Resolution

Multiple detectors can sometimes identify overlapping portions of the same text.

PIIShield resolves these candidates before replacement so that overlapping detections do not corrupt the resulting document.

1. Deterministic Replacement

Detected PII values are replaced using deterministic placeholders.

This makes the redaction process reproducible and prevents the output from containing the original sensitive values.

1. DOCX Reconstruction

A separate redacted DOCX is generated.

An important engineering goal was preserving document formatting instead of converting the document into plain text and rebuilding it from scratch.

1. Gold Evidence

The pipeline records evidence about the detected PII.

This provides an auditable representation of the detection and redaction process while keeping the original source documents out of the public repository.

Challenges Overlapping PII Matches

Different detection rules can identify overlapping text.

For example, a piece of text may simultaneously satisfy multiple detector patterns.

The pipeline therefore needs an explicit candidate-resolution stage before performing replacements.

Preserving DOCX Formatting

Simple text extraction and replacement can easily destroy formatting, runs, paragraphs, or other document structure.

PIIShield therefore treats document reconstruction as a separate engineering problem rather than simply generating a new plain-text document.

Deterministic Redaction

Random replacements would make testing and evaluation difficult.

Deterministic replacement ensures that the same input produces predictable redaction behavior.

Deployment Constraints

The public deployment must demonstrate the system without exposing the original source documents or raw PII.

The repository therefore contains the deployable project surface while intentionally excluding sensitive source data.

Memory Usage

DOCX processing can become memory-intensive.

The implementation was optimized to reduce rendering memory usage while preserving DOCX formatting.

Engineering Decisions Layered Data Processing

The Bronze → Silver → Gold structure separates extraction, normalized processing, and evidence generation.

This makes the pipeline easier to debug, test, and extend.

Unified Detector

Instead of scattering PII detection logic throughout the application, detection is centralized into a unified detector.

This makes the supported PII categories explicit and simplifies testing.

Candidate Resolution Before Replacement

Replacement is intentionally performed only after detection candidates have been resolved.

This prevents overlapping matches from producing corrupted output.

Deterministic Replacements

Deterministic replacement makes results reproducible and simplifies regression testing.

Separate Redacted Output

The original document is never modified in-place.

The system generates a separate redacted DOCX, reducing the risk of accidentally destroying the original document.

Public-Safe Repository

The GitHub repository intentionally excludes raw source documents and sensitive extraction layers.

This allows the application and engineering work to be demonstrated publicly without publishing source PII.

Automated Testing

The project includes regression tests covering detector and pipeline behavior.

The complete local package contains 69 automated tests.

Performance and Validation

The project was validated using automated tests and a labeled regression evaluation.

The complete local package currently reports:

69 automated tests passing

The evaluation layer includes labeled ground truth and metrics documentation.

The application was also deployed through Render, with the live application shell returning an HTTP 200 health response during verification.

The project focuses on both detection accuracy and practical document-processing reliability rather than treating PII detection as an isolated pattern-matching task.

Lessons Learned PII Detection Is More Than Regex

Detecting PII requires category-specific rules, normalization, validation, and conflict resolution.

A detector that simply searches for patterns can generate false positives and overlapping matches.

Document Preservation Matters

A privacy system is not useful if the redaction process destroys the structure and formatting of the business document.

The output document must remain usable.

Reproducibility Is Important

Deterministic processing makes the system easier to test, debug, evaluate, and trust.

Security Must Be Considered During Deployment

A privacy-focused application should not accidentally expose the very data it is designed to protect.

Separating deployable application code from source PII is therefore an important part of the architecture.

Evaluation Should Be Built Into the Pipeline

Maintaining labeled evaluation data and regression tests makes it possible to measure changes to the detector instead of relying only on manual inspection.

Future Work

Potential improvements include:

Adding more PII categories Improving contextual detection to reduce false positives Supporting additional document formats such as PDF and XLSX Adding configurable redaction policies Adding confidence scores for detections Introducing stronger entity-level NLP models Supporting batch document processing Adding role-based access control Adding encrypted temporary storage Expanding evaluation datasets Adding detailed monitoring and production observability Providing an API alongside the Streamlit interface Project Impact

PIIShield demonstrates how a privacy-oriented data pipeline can combine document processing, PII detection, deterministic transformation, automated testing, security considerations, and cloud deployment into a single reproducible system.

Rather than building only a PII detector, the project addresses the complete workflow from document ingestion to a usable redacted document and auditable detection evidence.

Core pipeline:

DOCX → Extract → Normalize → Detect → Resolve → Redact → Reconstruct → Audit