A scalable pipeline turning unstructured medical notes into clean, structured datasets for analysis.
PythonPySparkFAISSLLMsPydanticETL
Overview
Medical encounter notes are written for people, not databases. This pipeline reads unstructured COVID-19 patient records and turns them into clean, validated, structured data that can be analyzed at scale, making free text medical notes queryable without losing the detail that makes them useful.
What it does
Structured extraction: a large language model reads each free text record and pulls out the fields that matter, with Pydantic schemas validating and standardizing every result.
Consistent medication matching: FAISS semantic search reconciles medication references that vary in wording across notes, so the same drug is recognized however it is written.
Population scale analysis: Apache Spark analyzes comorbidities, medication frequencies, and demographic trends across the dataset, with visualizations that support the findings.
Architecture and engineering
LLM extraction step: the initial pass turns unstructured notes into candidate structured records, handling the messiness and variability of real clinical text.
Validation as a gate: every extracted record passes through Pydantic validation, so downstream analysis always works from one consistent, well typed shape.
Semantic reconciliation: FAISS vector search matches medication mentions by meaning rather than exact string, keeping references consistent across the corpus.
Built for scale: Python drives the pipeline end to end, with PySpark handling analysis over large volumes and Parquet as the columnar storage format.
