SOLUTION ARCHITECT L1

Job Description

Position : Databricks Platform Engineer lead

Location : Minneapolis ,MN

seeking a Databricks Platform Engineer to deploy, and manage our enterprise data platform. In this role, you will eliminate manual configuration by treating our entire data ecosystem as code. You will leverage** Terraform** as our primary Infrastructure as Code (IaC) tool to provision, secure, and scale multi-region Databricks workspaces across our** Azure and AWS cloud stacks**. Your work will directly enable our data engineering and analytics teams to deploy secure, compliant, and highly optimized pipelines seamlessly.

Key Responsibilities - Infrastructure Automation & Provisioning

• End-to-End Automation: maintain reusable Terraform modules to deploy multi-region Databricks workspaces from scratch.

• Cloud Integration: Link Databricks workspaces with foundational cloud assets like AWS VPCs/IAM or Azure VNets/Private Endpoints.

• Storage and Compute: Provision secure cloud storage (AWS S3, Azure ADLS Gen2) and define scalable cluster configurations via code.

• State Management: Establish rigorous CI/CD workflows for Git-driven infrastructure deployments using GitHub Actions, GitLab CI, or Azure DevOps.

Governance, Security & Access Control

• Unity Catalog Implementation: Automate data governance architectures, managing catalogs, schemas, and external locations purely via the Databricks Terraform Provider

• Identity Management: Configure SCIM provisioning, Service Principals, and Group memberships to enforce the principle of least privilege.

• Secrets Management: Programmatically sync cloud secret managers (AWS Secrets Manager, Azure Key Vault) with Databricks secret scopes.

Performance & Cost Optimization

• Cluster Policies: Design and enforce cluster policies via IaC to control instance types, auto-termination rules, and spot/on-demand ratios.

• Monitoring & Alerts: Set up automated workspace monitoring, log delivery (diagnostic logs), and budget alerts to keep infrastructure cost-efficient.