Spark-Scala, Databricks

Primary skills:Technology->Big Data - Data Processing->Spark Technology->Data Engineering->Databricks Technology->Functional Programming->Scala

Key Responsibilities: Data Engineering & ETL Development

• Design, develop, and maintain ETL pipelines using Spark-Scala on Databricks for batch and incremental processing.
• Implement data transformations, joins, aggregations, and validations to ensure accurate and consistent outputs.
• Build reusable Spark components and follow best practices for modular, maintainable code. Performance, Reliability & Operations
• Tune Spark jobs (partitioning, caching, shuffle optimization) to improve performance and cost efficiency.
• Monitor job execution, troubleshoot failures, and provide timely production support with root-cause analysis.
• Implement logging, error handling, and data quality checks to improve pipeline reliability. Collaboration & Delivery
• Work with cross-functional teams to gather requirements and translate them into technical solutions.
• Participate in code reviews, documentation, and knowledge sharing to uplift team standards.
• Support release cycles by validating outputs, ensuring backward compatibility, and maintaining deployment readiness. Minimum Qualifications:
• Bachelor's/Master's degree (BE/BTech/MSc/MCA/MTech or equivalent).
• 3–5 years of experience in data engineering or ETL development roles.
• Strong hands-on experience with Spark using Scala and working on Databricks.
• Solid understanding of ETL concepts, data transformations, and pipeline troubleshooting.
• Ability to write clean, testable code and collaborate effectively with technical and non-technical stakeholders. Preferred Qualifications:
• Experience building end-to-end pipelines on Databricks including notebooks, jobs/workflows, and cluster configuration basics.
• Strong SQL skills and experience integrating Spark pipelines with structured data sources and curated datasets.
• Familiarity with data quality frameworks, reconciliation strategies, and automated validation checks.
• Exposure to CI/CD practices for data engineering (version control, automated testing, release management).
• Proven ability to optimize distributed workloads and deliver measurable improvements in runtime and stability. Good to have skills: SQL, Delta Lake, Apache Airflow, Azure Data Lake Storage (ADLS), Git