Spark-Scala, Databricks

Job Description:

• Join a collaborative engineering team where you ll turn raw complex data into trusted insights that power real business decisions
• In this role you ll work hands on with Spark Scala and Databricks to build scalable data pipelines optimize distributed processing and help teams deliver reliable datasets for analytics and downstream applications
• You ll partner closely with data engineers analysts and stakeholders to understand requirements translate them into robust ETL solutions and continuously improve performance and quality
• If you enjoy solving data challenges learning modern cloud data practices and taking ownership from development through production support this is a great opportunity to grow your impact while working in a supportive high accountability culture

Key Responsibilities:

• Key Responsibilities
• Data Engineering ETL Development
• Design develop and maintain ETL pipelines using Spark Scala on Databricks for batch and incremental processing
• Implement data transformations joins aggregations and validations to ensure accurate and consistent outputs
• Build reusable Spark components and follow best practices for modular maintainable code
• Performance Reliability Operations
• Tune Spark jobs partitioning caching shuffle optimization to improve performance and cost efficiency
• Monitor job execution troubleshoot failures and provide timely production support with root cause analysis
• Implement logging error handling and data quality checks to improve pipeline reliability
• Collaboration Delivery
• Work with cross functional teams to gather requirements and translate them into technical solutions
• Participate in code reviews documentation and knowledge sharing to uplift team standards
• Support release cycles by validating outputs ensuring backward compatibility and maintaining deployment readiness

Technical Requirements:

• Primary skills Technology Big Data Data Processing Spark
• Technology Data Engineering Databricks
• Technology Functional Programming Scala

Additional Responsibilities:

• Minimum Qualifications
• Bachelor s Master s degree BE BTech MSc MCA MTech or equivalent
• 3 5 years of experience in data engineering or ETL development roles
• Strong hands on experience with Spark using Scala and working on Databricks
• Solid understanding of ETL concepts data transformations and pipeline troubleshooting
• Ability to write clean testable code and collaborate effectively with technical and non technical stakeholders
• Preferred Qualifications
• Experience building end to end pipelines on Databricks including notebooks jobs workflows and cluster configuration basics
• Strong SQL skills and experience integrating Spark pipelines with structured data sources and curated datasets
• Familiarity with data quality frameworks reconciliation strategies and automated validation checks
• Exposure to CI CD practices for data engineering version control automated testing release management
• Proven ability to optimize distributed workloads and deliver measurable improvements in runtime and stability
• Good to have skills
• SQL Delta Lake Apache Airflow Azure Data Lake Storage ADLS Git

Preferred Skills:

Technology->Big Data - Data Processing->Spark,Technology->Functional Programming->Scala,Technology->Data Engineering->Databricks