Lead Data Engineer

Workplace

White City, London - Hybrid 1-2 days a week on site

Contract Type

Permanent

The Role

As a Lead Data Engineer, you operate as a

technical orchestrator

and a

guardian of excellence

across our real-time operational data ecosystem. You are positioned at the critical intersection of high-level requirement analysis, distributed systems design, and hands-on delivery.

You will balance deep data architecture with infrastructure architecture, bridging the gap between data pipelines and cloud-native container platforms.

Your time will be balanced across three core pillars:

• 40% High-Level Design & Architecture: Reviewing incoming business requirements, driving technical discussions, and mapping out scalable, distributed data architectures.
• 40% Hands-On Engineering: Writing clean, testable code alongside the team, implementing core platform features, building reusable components and undertaking support requests.
• 20% Mentorship & Technical Governance: Cultivating the engineering squad's capabilities, managing technical debt, and defining quality frameworks.

Key Responsibilities

• Technical Orchestration: Translate complex business roadmaps into high-level designs, defining the structures, components, and interactions of event-driven streaming solutions.
• Data & Infrastructure Architecture: Architect high-throughput, real-time data ingestion pipelines while partnering with DevOps specialists to ensure alignment with cloud infrastructure frameworks.
• People Leadership & Mentoring: Oversee the technical growth and motivation of the engineering squad, running code reviews, sharing knowledge, and embedding engineering best practices.
• Delivery Flow: Partner with the Engineering Manager and Product Managers to refine requirements into technical tasks, ensuring iterative delivery and a lean backlog.
• Operational Excellence & SLA Management: Guard platform reliability against extreme traffic loads, establishing a proactive monitoring strategy via Grafana/Prometheus.
• Escalation & Support: Participate in the team’s weekly support rotation as a high-level escalation point, leading Root Cause Analysis when critical operational challenges arise.

What You’ll Bring

• Polyglot Engineering Literacy: Strong backend development capability with a willingness to design, challenge, and review logic across Python and Go, with Java or Scala as valuable secondaries.
• Cloud Native Expertise: Extensive experience with the GCP Stack (GKE, Pub/Sub, BigQuery, Dataflow, Cloud Functions) and event-driven architectures.
• Infrastructure as Code: Deep understanding of automated infrastructure deployment using Terraform and container orchestration via Kubernetes.
• Systems Thinking: Ability to map out complex, multi-project data flows and microservices environments, ensuring modularity and resilience.
• Strategic Communication: Ability to articulate abstract technical designs clearly to both technical engineers and non-technical stakeholders.
• Nice to Have: Familiarity with ESG and/or Sustainable Software Engineering principles to optimise cloud compute efficiency, or experience shifting data capabilities to "Edge" environments.

Minimum Criteria

• Demonstrable experience leading technical system designs and defining architecture in a high-scale, cloud-native operational data environment.
• Proven hands-on engineering experience handling high-throughput, real-time data ingestion and processing pipelines.
• Experience mentoring engineers, managing technical debt, and steering code-first DevOps cultures.