Introducing Warp: Upload CCDA Patient Data With Horizontally Scalable Parallel Direct Database Imports
Hello OpenEMR Community! 
I’m excited to introduce Warp, a new high-performance data upload tool designed specifically for OpenEMR on EKS. Warp solves a critical challenge many healthcare organizations face: efficiently importing patient data from CCDA-compliant sources and other healthcare data formats.
What is Warp?
Warp is a direct database import tool that bypasses APIs and web interfaces entirely, writing patient data directly to OpenEMR’s MySQL database. This approach delivers much faster import speeds compared to traditional API-based methods.
Each Warp task can be configured to launch multiple workers and use configurable RAM and CPU resources and multiple Warp tasks can be launched simultaneously into the same cluster to import data in parallel at the same time. This allows for you to scale Warp horizontally and rapidly import even very large datasets. In testing I was able to achieve import rates of ~7.5-50 patients/second to our installation but in no way should this be considered an upper limit of the system.
Key Features:
-
Reliable: No API authentication or web session dependencies -
Kubernetes-Native: Designed to run as a resource-intensive pod on EKS -
Multiple Data Sources: Supports S3, local files, and other sources -
Auto-Discovery: Automatically finds credentials from Kubernetes/Terraform -
OMOP CDM Support: Loads OMOP Common Data Model data for direct database import -
Parallel Processing: Multi-worker architecture for maximum throughput -
Extensible Architecture: Designed with extensibility in mind, enabling future support for additional data formats and upload types beyond CCDA/OMOP
Why CCDA Data Upload Matters
CCDA (Consolidated Clinical Document Architecture) is a critical standard for healthcare interoperability. The ability to upload CCDA data to OpenEMR enables:
-
Interoperability: Seamlessly import patient records from other EHR systems that export CCDA-compliant documents
-
Data Migration: Efficiently migrate patient data when transitioning to OpenEMR from other systems
-
Research Integration: Import data from research datasets (like OMOP CDM) for clinical analysis
-
Compliance: Meet interoperability requirements for healthcare data exchange standards
-
Scalability: Handle large-scale data imports that would be impractical through web interfaces
Warp converts OMOP CDM (Common Data Model) format data—commonly used in healthcare research and analytics—into OpenEMR’s native database structure, enabling direct import of:
-
Patient demographics and basic information
-
Medical conditions and diagnoses
-
Medications and prescriptions
-
Clinical observations and lab results
Performance Benchmarks
In my testing with the synpuf-omop 1k dataset (1,000 patients with 160,322 conditions, 49,542 medications, and 13,481 observations):
-
Total Records Imported: 224,345 records
-
Processing Time: ~2.22 minutes (132.96 seconds)
-
Processing Rate: ~7.5 patients/second (single worker)
-
Success Rate: 100% (1,000 patients successfully uploaded, 0 failures)
Performance Comparison:
How It Works
Warp uses direct SQL INSERT statements that match OpenEMR’s internal newPatientData() function structure, ensuring:
-
Schema-Aware: Understands OpenEMR’s exact database schema
-
Transaction-Safe: Uses transactions for atomic operations
-
Parallel Processing: Multiple workers writing concurrently for maximum throughput
The tool automatically discovers database credentials from:
-
Kubernetes Secrets (highest priority)
-
Terraform outputs
-
Environment variables
This means you can run imports with minimal configuration—just point it at your data source!
Getting Started
Warp is currently in beta (version 0.1.0) and is actively being developed. You can find the project, documentation, and examples at:
GitHub Repository: openemr-on-eks/warp at main · openemr/openemr-on-eks · GitHub
The repository includes:
-
Complete documentation and usage examples
-
Kubernetes deployment manifests
-
Performance benchmarks and optimization guides
-
Developer documentation for contributing
Use Cases
Warp is particularly valuable for:
-
Healthcare Organizations migrating from other EHR systems
-
Research Institutions importing OMOP CDM datasets for analysis
-
Development Teams testing OpenEMR with realistic patient data
-
Production Deployments requiring bulk patient data imports
Future Extensibility: Warp’s modular architecture is designed to support additional data formats and upload types in the future. While currently focused on CCDA/OMOP data imports, the framework can be extended to accelerate other types of data uploads—such as lab results, imaging metadata, billing data, or custom data formats—all while maintaining the same high-performance direct database import approach.
Community & Feedback
I’m actively developing Warp and would love your feedback! If you:
-
Try Warp and have suggestions for improvements
-
Encounter issues or have questions
-
Want to contribute features or fixes
-
Have use cases we should support
-
Need support for additional data formats or upload types (Warp’s extensible architecture makes adding new data types straightforward)
Please feel free to:
-
Open issues on GitHub
-
Submit pull requests
-
Share your experiences in this thread
Important Notes
-
Warp is currently in beta and should be used with caution in non-production environments
-
Direct database access is required (this is by design for maximum performance)
-
Designed for OpenEMR on EKS
-
Production-ready status expected with version 1.0.0 release
Learn More
For detailed documentation, examples, and technical details, visit the Warp README on GitHub.
Questions? Comments? I’d love to hear from you! Feel free to reply to this thread or open an issue on GitHub.
Happy importing!
Warp is part of the OpenEMR on EKS project, providing production-ready OpenEMR deployments on Amazon EKS with fully managed infrastructure.