Databricks Architect

Job Description

Position: Databricks Platform Engineer

Location: Minneapolis ,MN

Databricks Platform Engineer to deploy, and manage our enterprise data platform. In this role, you will eliminate manual configuration by treating our entire data ecosystem as code. You will leverage Terraform as our primary Infrastructure as Code (IaC) tool to provision, secure, and scale multi-region Databricks workspaces across our Azure and AWS cloud stacks. Your work will directly enable our data engineering and analytics teams to deploy secure, compliant, and highly optimized pipelines seamlessly.

Key Responsibilities - Infrastructure Automation & Provisioning End-to-End Automation: maintain reusable Terraform modules to deploy multi-region Databricks workspaces from scratch.

Cloud Integration: Link Databricks workspaces with foundational cloud assets like AWS VPCs/IAM or Azure VNets/Private Endpoints.

Storage and Compute: Provision secure cloud storage (AWS S3, Azure ADLS Gen2) and define scalable cluster configurations via code.

State Management: Establish rigorous CI/CD workflows for Git-driven infrastructure deployments using GitHub Actions, GitLab CI, or Azure DevOps.

Governance, Security & Access Control Unity Catalog Implementation: Automate data governance architectures, managing catalogs, schemas, and external locations purely via the Databricks Terraform provider.

Identity Management: Configure SCIM provisioning, Service Principals, and Group memberships to enforce the principle of least privilege.

Secrets Management: Programmatically sync cloud secret managers (AWS Secrets Manager, Azure Key Vault) with Databricks secret scopes.

Performance & Cost Optimization Cluster Policies: Design and enforce cluster policies via IaC to control instance types, auto-termination rules, and spot/on-demand ratios.

Monitoring & Alerts: Set up automated workspace monitoring, log delivery (diagnostic logs), and budget alerts to keep infrastructure cost-efficient