Data Specialist - R01569712

Primary Skills

Databricks, Pyspark,sql

Specialization

• Databricks Engineering: Lead Data Engineer

Job requirements

Experience Range

With at least 5-6 years of experience in data engineering, including substantial hands-on expertise in Databricks and PySpark

Key Responsibilities

• Design and implement robust ETL pipelines using Databricks and PySpark to efficiently process large-scale datasets
• Develop, optimize, and maintain complex SQL queries, stored procedures, and scalable data models to support analytics and reporting
• Collaborate with data architects and business stakeholders to translate requirements into high-performance data solutions
• Integrate data from diverse sources into data warehouses and modern data platforms, ensuring data quality and consistency
• Troubleshoot and resolve issues related to data ingestion, transformation, and storage within Databricks environments
• Enforce best practices in data warehousing, data modeling, and Databricks platform fundamentals
• Monitor and tune performance of data processing jobs for optimal throughput and reliability
• Document data engineering processes and provide technical guidance to team members on Databricks and PySpark usage

Required Skills

• Advanced proficiency in Databricks platform
• Expertise in PySpark for distributed data processing and transformation
• Strong SQL skills for analytics and data manipulation
• Experience with data warehousing architectures
• Hands-on experience designing and maintaining stored procedures
• Proficiency in Apache Spark for large-scale data processing

Preferred Skills

• Experience with Azure Databricks or AWS Glue
• Familiarity with CI/CD pipelines for Databricks workflows
• Knowledge of Databricks performance optimization techniques
• Exposure to data governance and security best practices in cloud environments
• Experience with real-time data streaming frameworks such as Apache Kafka

Desired Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a closely related discipline
• Certification in Databricks or Apache Spark
• Certification in SQL or data warehousing technologies