Lead Data Engineer

Cognitive

Posted 1mo ago

The employer’s full description could not be read from their board. This is a summary of the posting — follow the apply link for the original.

Own end-to-end PDE data pipeline ingestion, validation, editing, and storage
Apply and maintain CMS-defined business rule validations
Engineer and maintain vendor reference data edits from multiple sources
Build, maintain, and optimize Apache Airflow and PySpark transformations
Monitor PDE submissions for adjustment trends and informational edit rates
Provide analytical capability to link rejected PDEs with IDR records
Provide beneficiary and plan-level cost aggregations for year-end reconciliation
Support data transfer projects and new data share integrations with CMS
Maintain target unit test coverage for all new pipeline code
Support BDD and TDD practices
Track and surface real-time PDE processing metrics

Bachelor's degree in Computer Science, Data Engineering, or related field
8+ years in data engineering
4+ years building and operating high-volume batch ETL systems
Expert-level Python and PySpark experience
Production experience with Apache Airflow (AWS MWAA)
Production experience with AWS Redshift Serverless, Snowflake, and Amazon Athena
Hands-on experience with Databricks for large-scale data processing
Strong SQL skills for multi-source data transformation and editing logic
Experience with AWS S3, SQS, Lambda, and Event Bridge
Experience with Linux environments including RHEL, CentOS, and Amazon Linux 2
Experience with shell scripting in Bash
Experience with version control and CI/CD using GitHub, Jenkins, and JFrog Artifactory
Ability to pass CMS and internal background checks for public trust

Skills