Sr Data Engineer - Apache DolphinScheduler - Chennai

Key Responsibilities

• Develop, configure, and optimize Apache DolphinScheduler workflows for data processing and automation.
• Design and implement ETL pipelines, job scheduling, and workflow orchestration solutions.
• Troubleshoot and resolve performance, scalability, and reliability issues in DolphinScheduler.
• Integrate DolphinScheduler with big data tools such as Hadoop, Spark, Flink, Hive, and Kafka.
• Work closely with data engineers, DevOps, and software teams to enhance workflow automation.
• Develop custom plugins and extensions for DolphinScheduler as needed.
• Monitor and optimize job execution, resource allocation, and workflow efficiency.
• Maintain best practices for CI/CD, version control, and infrastructure automation.

Required Skills & Qualifications

• Strong experience with Apache DolphinScheduler in a production environment.
• Proficiency in Java, Python, or Scala for workflow scripting and automation.
• Experience with big data technologies like Hadoop, Spark, Hive, Flink, and Kafka.
• Understanding of workflow orchestration, DAG execution, and scheduling strategies.
• Strong problem-solving skills and ability to work in a fast-paced environment.