Senior Technical Specialist

Job Summary

We are looking for a Databricks expert with strong experience in platform setup, architecture, and AI/ML capabilities across** Azure and/or GCP**. The ideal candidate should bring hands-on expertise in building Databricks-based AI solutions, not just traditional data pipelines.

Key Responsibilities

• Design and implement Databricks architecture on Azure and/or GCP (workspace, clusters, networking, security).

• Lead platform setup, governance, and optimization (Unity Catalog, cost control, access management).

• Build and deploy AI/ML solutions using Databricks (MLflow, Feature Store, Model Serving).

• Develop GenAI / LLM solutions (RAG pipelines, embeddings, vector search).

• Integrate LLMs (Azure OpenAI / Vertex AI / other APIs) within Databricks.

• Deliver scalable Lakehouse architecture solutions.

• Collaborate with engineering and business teams for end-to-end AI use cases.

• Ensure performance tuning, reliability, and security of workloads

Skill Requirements

• Strong experience in Databricks setup & architecture (must-have).

• Hands-on with Databricks AI stack:

MLflow, AutoML

• Feature Store

• Model Serving

• RAG / vector search / embeddings

• Experience with Py Spark, Spark optimization, Delta Lake.

• Strong exposure to cloud platforms:

Azure Databricks + Azure services

• GCP Databricks + GCP ecosystem (Big Query, GCS, Vertex AI)

• Experience integrating LLMs / GenAI solutions.

• Knowledge of Unity Catalog, governance, and security.

Other Requirements

• Experience in GenAI applications on Databricks

• Familiarity with Lang Chain / vector databases

• Exposure to real-time/streaming pipelines

• CI/CD and DevOps practices

--- Ideal Candidate

• Strong in Databricks platform + AI (not just data engineering)

• Hands-on in cloud setup (Azure/GCP) and AI delivery

• Able to design end-to-end AI solutions