Document reformat at scale

The problem

A clinical operator needed to convert document corpora across PDF, TIFF, DOCX, structured XML, and clinical CDA. Single inputs reached tens of thousands of pages; naive in-memory conversion crashed on larger files. Structure preservation was inconsistent across formats; the audit trail was missing for transformations, so an analyst could not prove a value came from the original input.

The approach

We built a Python streaming pipeline with memory-bounded conversion. Each format adapter implements a streaming contract; very large documents process page-by-page without loading the whole corpus. Structure preservation (headings, tables, list semantics) survives format hops where the target format supports it. Every transformation step writes an audit record with input hash, output hash, and the adapter version, so reproducibility is verifiable by replay.

Stack and engineering choices

• Python streaming pipeline
• Memory-bounded format adapters
• PDF + TIFF + DOCX + CDA hops
• Structure preservation
• Per-step audit records
• Hash-based reproducibility
• Replay from audit trail

Outcome

Documents that previously crashed conversion now process predictably. Structure survives across format hops; analysts no longer reformat by hand on the other side. Reproducibility is verified by replay against the audit trail, so any output value can be traced back to its input.