Databricks, Pyspark

Technology->Analytics - Solutions->SQL Server - Analytics Technology->Big Data - Data Processing->PySpark Technology->Data Engineering->Databricks

Key Responsibilities:

• Lead the development of end-to-end data pipelines on Databricks using PySpark for batch and incremental processing.
• Design scalable data models and curated datasets to support analytics and downstream consumption.
• Write and optimize advanced SQL for transformations, validations, and performance-critical queries.
• Implement robust data quality checks, reconciliation logic, and monitoring to ensure trusted datasets.
• Tune Spark jobs for performance and cost efficiency (partitioning, caching, file formats, cluster sizing).
• Establish coding standards, reusable frameworks, and review practices to improve maintainability.
• Collaborate with stakeholders to translate requirements into technical designs and delivery plans.
• Troubleshoot production issues, perform root-cause analysis, and drive preventive improvements.
• Mentor team members and provide technical guidance across design, implementation, and optimization. Minimum Qualifications:
• BTECH, MTECH, MCA, or MSC in Computer Science, Information Technology, or a related field.
• 6–8 years of overall experience in data engineering or large-scale data processing roles.
• Strong hands-on experience with PySpark for distributed data processing and transformation logic.
• Strong hands-on experience with Databricks for building, running, and managing data workloads.
• Proficiency in Advanced SQL including complex joins, window functions, and query optimization.
• Experience building reliable pipelines with strong focus on data quality, performance, and stability Preferred Qualifications:
• Experience designing lakehouse-style architectures and organizing curated layers for analytics readiness.
• Strong experience with Spark optimization techniques and handling large-scale datasets efficiently.
• Ability to build reusable PySpark utilities/frameworks for ingestion, transformation, and validation patterns.
• Experience with orchestration and scheduling approaches for dependable pipeline execution and recovery.
• Proven track record of technical leadership: mentoring, conducting reviews, and driving engineering best practices.