Senior Technical Architect

Job Summary

Data Ingestion and Transformation: Design, develop, and maintain scalable data ingestion pipelines to onboard structured, semi-structured, and unstructured data from various sources into the Azure/Databricks

Data Deduplication and Tagging: Implement de-duplication strategies across large-scale datasets using deterministic and probabilistic matching techniques to ensure data integrity and reduce redundancy within the Data Lake.

Data Governance: Develop and enforce data tagging frameworks to classify, label, and annotate datasets with appropriate metadata to support data governance, discoverability, and compliance requirements.

Operationalization: Assist with the operationalization of deployments and support of Cloud services for ETL operations, including standardizing and automating processes and workflows.

Resource should have hands on work experience with Azure Databricks, Datalake.
Also should have knowledge of related AI tools.

Key Responsibilities

1. To design and architect large-scale solutions, ensuring scalability, performance, and security.

2. To train and develop team so as to ensure that there is an adequate supply of trained manpower in the said technology and delivery risks are mitigated.

3. To continuously upskill with cutting-edge tech to deliver high-quality, future-proof solutions meeting client expectations and industry standards.

4. To leverage domain/tech expertise to gather client needs, deliver solutions, and craft a technology strategy aligned with business goals.

5. Develop, manage, and optimize ADF pipelines. Design and implement Databricks notebooks for ETL processes. Write and optimize SQL scripts for large-scale datasets. Collaborate with BI teams to support dashboard and reporting solutions

Skill Requirements

Azure Databricks, DataLake

AI / ML tools.

Other Requirements

null