Azure Databricks Admin

Description

We are seeking a Senior Azure and Databricks Administrator to act as a Subject Matter Expert (SME) and lead the management, architecture, and optimization of our data platform infrastructure. With 6–8 years of experience, you will go beyond daily administration to design secure, enterprise-scale Databricks environments, implement comprehensive data governance strategies, and drive automation. You will partner closely with Data Architects, Data Engineers, and Cloud Security teams to ensure our Azure and Databricks ecosystems are highly resilient, cost-efficient, and aligned with industry best practices.

Key Responsibilities

Architecture & Enterprise Administration

Platform Architecture

Design and deploy enterprise-grade Azure Databricks topologies, including multi-workspace environments, complex VNet injection, and Hub-and-Spoke network architectures.

Unity Catalog & Data Governance Strategy

Lead the migration to and enterprise-wide implementation of Databricks Unity Catalog. Design complex data lineage, row/column-level security, and cross-workspace data sharing strategies (Delta Sharing).

Advanced Access Management

Architect scalable Identity and Access Management (IAM) frameworks using Azure Entra ID (Azure AD), SCIM provisioning, and complex RBAC/ABAC models.

Disaster Recovery & High Availability

Design and test failover strategies, backup routines, and Disaster Recovery (DR) plans for critical data platforms and metastores.

Infrastructure & Security Management

Advanced Azure Networking

Manage enterprise network security including Azure Firewall, Private Link / Private Endpoints, UDRs (User Defined Routes), and custom DNS configurations for secure data exfiltration prevention.

Storage & Secret Management

Oversee enterprise storage strategies (ADLS Gen2) and secure secret management pipelines using Azure Key Vault backed by strict compliance policies.

DevOps, Automation & Mentorship

Infrastructure as Code (IaC) Mastery

Build and maintain reusable Terraform modules or Bicep templates for zero-touch deployment of Databricks workspaces, clusters, and Azure resources.

Advanced CI/CD

Architect sophisticated MLOps and DataOps CI/CD pipelines using Azure DevOps or GitHub Actions, integrating automated testing and deployment of Databricks assets.

Technical Leadership

Mentor junior administrators and data engineers on platform best practices, Spark compute optimization, and secure coding standards. Act as the highest level of escalation for complex platform outages.

Required Skills & Qualifications

Experience

6–8 years of comprehensive experience in IT/Cloud Infrastructure, with at least 3–4+ years of dedicated, hands-on experience managing Azure Databricks in a large-scale production environment.

Cloud Architecture

Expert-level understanding of Azure ecosystem, specifically surrounding Big Data, networking (peering, Private Link, NSGs), and robust identity management.

Databricks Expertise

Deep, authoritative knowledge of the Databricks architecture (Data Plane/Control Plane), Unity Catalog, Delta Lake storage mechanics, and Databricks REST APIs/CLI.

Automation/DevOps

Strong scripting skills (Python, Bash, PowerShell). Proven expertise in writing and maintaining complex Terraform deployments for cloud infrastructure.

Performance Tuning

Strong ability to diagnose Spark UI metrics to assist data engineering teams in optimizing poorly performing clusters or jobs.