Technical Lead- AWS, Databricks

Job Description:

Experience Range

10+ years

Primary (Must have skills)* - To be Screened by TA Team

10+ years total experience in Data Engineering, Cloud Data Platforms

5+ years hands-on experience with Databricks (PySpark/Scala, Delta Lake, Lakehouse architecture)

4+ years experience in AWS Glue (Glue ETL jobs, Workflows, Crawlers, Glue Catalog).

3+ years experience designing data ingestion and transformation pipelines using Spark-based frameworks.

Job Description of Role

Technical Leadership

Drive end-to-end technical solutions, ensuring performance, scalability, and reliability.

Establish coding standards, best practices, and development guidelines.

Conduct code reviews, provide technical mentorship, and resolve complex technical issues.

Data Platform Architecture

Architect and implement large-scale data ingestion, transformation, and storage layers.

Design Lakehouse architecture using Delta Lake, Unity Catalog, Medallion model.

Ensure data quality, lineage, and governance across data assets.

Collaborate with cloud architects to integrate platform services across Azure and AWS.

Solution Development

Develop optimized Spark jobs (PySpark/Scala) with strong performance tuning practices.

Implement ETL/ELT pipelines using Databricks notebooks, Workflows, and Glue Jobs.

Integrate data from multiple sources into ADLS / S3 with appropriate security and metadata management.

Build CI/CD pipelines using Azure DevOps / GitHub Actions.

Cloud Infrastructure & Security

Define IAM roles, managed identities, ACLs, and storage permissions for secure operations.

Work with Terraform/CloudFormation for infrastructure automation (preferred).

Monitor cloud resource utilization and optimize cost.

Cross-Functional Collaboration

Interact with stakeholders, analysts, architects, and business teams to translate requirements into technical designs.

Present architectural recommendations and solution proposals to leadership.

Coordinate with DevOps and Cloud Engineering for deployment and operational readiness.