Technical Architect (DPO)

Job Summary

We are seeking an experienced CI/CD Engineer or Lead to design, automate, and govern delivery pipelines across the Azure ecosystem (Azure DevOps, Databricks, ADF/Synapse) and adjacent CI/CD platforms. While the final tool choice may vary, the candidate must have significant hands-on experience with Jenkins and strong DevOps fundamentals. The role includes in-sprint testing enablement, platform governance (monitoring, administration, configuration), and cost-effective technology selection. Excellent communication and documentation skills are essential.

Key Responsibilities

• Pipeline Architecture & Delivery: Design and maintain end-to-end CI/CD pipelines using Azure DevOps, Jenkins, GitHub Actions, or GitLab CI. Implement linting and code review checks, integrate static analysis tools (Sonar Qube/CodeQL), architect multi-stage workflows, build reusable templates for Databricks, and manage artifact repositories.

• Azure Platform Governance & Operations: Monitor, administer, and configure Azure services (Key Vault, Storage/ADLS, ADF/Synapse, Databricks, AKS). Apply RBAC, secrets management, observability (Azure Monitor, Log Analytics), and cost governance.

• Source Control & ALM: Define branching strategy (Git Flow/Trunk), enforce PR policies in Bitbucket, and integrate Jira workflows with CI/CD.

• Automated & In-Sprint Testing: Embed unit, integration, SIT, and E2E/regression tests into pipelines for in-sprint execution. Collaborate with QA to align automated test coverage with sprint goals.

• Dev Sec Ops & Policy as Code: Integrate SAST/DAST, dependency scanning, SBOM generation, and enforce quality/security gates. Apply policy-as-code (OPA/Conftest, Azure Policy).

• Infrastructure as Code (IaC): Provision and promote infrastructure using Terraform (preferred) and/or Bicep; manage remote state and modules. Standardize container builds and AKS deployments.

• Data Storage Layer Management & Governance: Govern data storage across Unity Catalog (Databricks), Azure SQL / SQL Server, and ADLS. Implement data lineage, optimize storage tiers, and coordinate schema migrations.

• Communication & Collaboration: Communicate effectively with cross-functional teams including QA, Development, Product Management, and Architecture. Produce clear documentation, runbooks, and present status updates to stakeholders.

• Leadership (for Lead level): Define CI/CD standards, mentor engineers, lead design reviews, and drive continuous improvement.

Skill Requirements

Required Qualifications:

• Significant hands-on experience with Jenkins (declarative pipelines, shared libraries).

• Hands-on with Azure services (Key Vault, Storage, ADF, Databricks, AKS).

• Expertise in CI/CD best practices across multiple tools (Azure DevOps, GitHub Actions, GitLab CI).

• Solid understanding of in-sprint testing and test automation frameworks.

• Experience with linting, static analysis, and enforcing code quality gates.

• Scripting: Python, Bash, YAML proficiency.

• Familiarity with IaC using Terraform (preferred) and containerization (Docker/Kubernetes).

• Strong communication and documentation skills.

Preferred Qualifications:

• Databricks asset bundling/dbx, MLflow model promotion; Unity Catalog administration.

• Data quality frameworks (e.g., Great Expectations).

• Git Ops (Argo CD/Flux), Helm, Kustomize.

• SRE practices; DORA metrics instrumentation.

Cost-effective Tools & Technologies:

• CI/CD: GitHub Actions, GitLab CI, Jenkins (self-hosted), CircleCI.

• Code Quality: Sonar Qube/CodeQL, pre-commit, Black/Flake8, ESLint.

• Observability: Azure Monitor, Log Analytics; Cost Management tools like Azure Advisor.

• IaC: Terraform (preferred), Bicep; Secrets via Azure Key Vault.

• Data Storage: Unity Catalog, Azure SQL / SQL Server, ADLS Gen2; Catalog alternatives like Data Hub or Azure Purview.

KPIs / Success Metrics

• Lead time for changes, deployment frequency, change failure rate, MTTR (DORA).

• % of pipelines with linting, static analysis, test/security gates.

• % infra under Terraform; environment drift incidents; secrets coverage.

• Platform cost adherence: budget vs. actual; ingestion/storage optimization achieved.

• Data governance adoption: catalog completeness, Unity Catalog policies applied.

Other Requirements