From Chaos to Concurrency: Building a Scalable Medical Document Processor with AI

This is a technical deep-dive into how we tried (and failed, repeatedly) to turn a messy pile of unstructured medical documents — scanned PDFs, DOCX files, and images — into a clean, structured clinical database, and what we learned along the way. We began with a naive approach: a single Python script (document_processor.py) that extracted raw text using PyPDF2 and python-docx, then sent it all to Gemini with a massive prompt asking for 80 fields in JSON format. That broke quickly — the model returned incomplete or hallucinated data, with under 30% success, and most scanned PDFs were unreadable by PyPDF2. In Phase 2, we split the problem: a fallback pipeline used pdf2image and Tesseract OCR for robust text extraction, we introduced specialized prompts for each section (e.g., _extract_basic_data, _extract_medical_history), and added a lightweight AI classifier to detect form type (initial vs. follow-up) and select the correct taxonomy. This improved accuracy but killed performance, with each document requiring 5–6 sequential API calls and batch processing taking hours. So we moved to Phase 3: a parallel architecture using Docker. We introduced a SQLite task queue, split the system into two containers — medical-survey-processor (a Flask API that registers documents as PENDING) and worker.py (a Python service that processes jobs and updates their status) — and scaled using deploy: replicas: in docker-compose.yml to run multiple workers in parallel. The result is a resilient, scalable system combining deterministic logic with contextual AI, capable of processing hundreds of medical documents in parallel. The live demo will show a real file upload triggering the pipeline, key pieces of the code (OCR fallback, classification, prompts, worker orchestration), and no slides — just raw code, terminal logs, and real-world pain turned into working architecture.