Sr. Big Data Engineer

Key Responsibilities:

• Design and develop scalable Big Data applications using Spark (Scala-Spark or PySpark), Hadoop, Hive, Kafka, HBase, and MongoDB
• Create and optimize data pipelines for ingestion, transformation, and storage
• Collaborate with cross-functional teams to translate business requirements into scalable data solutions
• Debug, troubleshoot, and optimize data processes and code
• Leverage cloud platforms (public/private) for application deployment and data processing
• Implement unit testing, CI/CD, and monitoring to support DevOps and Agile delivery
• Understand and support Machine Learning models integration with Big Data platforms
• Ensure data quality, security, and compliance in all deliverables
• Provide employment proof at the time of submission as per compliance requirements

Must Have Requirements:

• Minimum 7+ years of IT / development experience
• At least 3 years of experience designing and developing data pipelines using Scala-Spark
• 3+ years of hands-on experience with Big Data frameworks like Parquet, AVRO, ORC, Resource Management, Distributed Processing, and RDBMS
• Strong hands-on experience in Spark (Scala-Spark or PySpark), Hadoop, Hive, Kafka, HBase, and MongoDB
• Solid understanding of Spark-Scala or PySpark libraries for solving complex data engineering problems
• Experience in Agile methodologies with proficiency in tools like version control systems, CI/CD pipelines, build tools, TDD, and change management
• Working experience with cloud environments (AWS or other cloud platforms)
• Minimum 6 years working on data-related technologies