Databricks Platform Lead

KData

Posted 2mo ago

The employer’s full description could not be read from their board. This is a summary of the posting — follow the apply link for the original.

Provision Databricks workspaces, cloud storage integrations, and RBAC frameworks
Lead the implementation of Unity Catalog for centralized data lineage and access controls
Enforce compute policies, cluster sizes, and auto-scaling rules to minimize cloud spend
Diagnose bottlenecks and establish workspace best practices
Design and maintain scalable data pipelines using Spark (PySpark/Scala) and Delta Lake
Automate infrastructure and code deployment using Terraform, Databricks Asset Bundles (DABs), and Git/CI-CD pipelines
Leverage Delta Live Tables (DLT), Serverless compute, and Photon engine
Support data engineers, analysts, and data scientists by enabling workflows like MLflow, dbt, and Databricks SQL

7+ years of data engineering experience utilizing Apache Spark (PySpark/Scala) and SQL
Strong history of managing enterprise-scale Databricks infrastructure and implementing latest updates
Practical experience configuring Unity Catalog and Delta Lake storage optimization from scratch
Proficient with Terraform or Databricks Asset Bundles (DABs)
Experience securing Databricks environments (VPCs, Private Link, IAM/Active Directory) across AWS, Azure, or GCP
Self-driven and capable of working autonomously in a remote, part-time capacity

Skills