Data Engineer with Bigdata+Scala+Pyspark - immediate joiner

preferred skills

· 4+ years of software engineering experience developing big data pipelines.

· 3+ years of hands-on experience programming in Scala and PySpark.

· 3+ years of experience building solutions on AWS, including AWS Glue.

· 3+ years of experience working with big data file formats, including Parquet.

· 1+ years of experience with data validation, data profiling, or building rule-based data engines.

· 1+ years of experience leveraging AI coding assistants or code-generation tools (e.g., GitHub Copilot, Codex) in daily software development tasks.

primary responsibilities for this role.

· Design, build, and optimize a highly scalable, rule-based data validation engine capable of validating data for any data pipeline.

· Develop robust, high-performance big data pipelines and platforms using Scala, PySpark, and AWS Glue.

· Write clean, well-tested code utilizing optimized formats like Parquet for highly efficient storage and retrieval.

· Perform detailed data validation and profiling to ensure quality and consistency across various data systems.

· Use enterprise-approved AI tools to streamline software development workflows, automate tasks, and drive continuous improvement.

· Collaborate with cross-functional teams to translate business requirements into efficient, scalable technical solutions.

· Evaluate emerging trends and AI technologies to inform solution design and strategic innovation in data validation and big data.

· Participate in code reviews, testing, debugging, and profiling to ensure code meets quality and performance standards.