Data Engineer (PySpark)

We are seeking an experienced Data Engineer (PySpark) to design, build, optimize, and maintain scalable data pipelines for production environments. The role requires strong hands-on experience in big data processing, pipeline optimization, and deployment using modern data engineering tools and frameworks.

Key Responsibilities

• Design, develop, and maintain robust, scalable data pipelines using Python and PySpark
• Perform data ingestion, transformation, cleansing, and validation across structured and unstructured datasets
• Conduct Exploratory Data Analysis (EDA) to identify data patterns, anomalies, and quality issues
• Apply data imputation techniques, data linking, and cleansing to ensure high data quality
• Implement feature engineering pipelines to support analytics and downstream use cases
• Optimize Spark jobs for performance, scalability, and cost efficiency
• Deploy and tune production-grade data pipelines, ensuring reliability and performance
• Automate workflows using Apache Airflow and/or Jenkins
• Collaborate with cross-functional teams to integrate data solutions into production systems
• Write and maintain unit tests to ensure code quality and reliability
• Manage source code, CI/CD, and deployments using Git, GitHub, and GitHub Actions

Requirements

To be considered for this role, you need to meet the following criteria:

Required Technical Skills

• Strong proficiency in Python
• Extensive hands-on experience with Apache Spark (PySpark)
• Experience working with Jupyter Notebooks
• Strong knowledge of SQL and NoSQL databases
• Proven experience with Git for version control and CI/CD
• Hands-on experience with Apache Airflow and/or Jenkins for scheduling and automation
• Solid understanding of data engineering best practices in production environments
• Demonstrated experience in Spark performance tuning and optimization
• Ability to write clean, testable, and maintainable Python code

Mandatory Requirement

• Previous production experience is a MUST, specifically in deploying, tuning, and maintaining data pipelines in production environments

Preferred Qualifications

• Experience working in high-volume or big data environments
• Strong problem-solving and analytical skills
• Ability to work independently in a fast-paced environment

Why Join?

• Competitive salary package
• Opportunity to work on production-scale data platforms
• Exposure to modern data engineering tools and practices
• Dubai-based role with a dynamic and collaborative work environment

To view other requirements we have, please visit our website - www.blackpearlconsult.com