Senior Staff Engineer, Big Data Engineer

REQUIREMENTS:

• Total experience 9+ years.
• Excellent knowledge and experience in Big data engineer.
• Strong working experience with architecture and development in Apache Spark, Spark, Python, Azure Databricks, Data Pipelines, Azure Devops, Kafka, SQL Server/NoSQL.
• Strong expertise in Python, Django Rest Framework, Databricks and PostgreSQL.
• Hands on experience in building data pipelines and building data frameworks for unit testing, data lineage tracking, and automation.
• Familiarity with streaming technologies (e.g., Kafka, Kinesis, Flink).
• Experience with building and maintaining a cloud system.
• Familiarity with data modeling, data warehousing, and building distributed systems.
• Expertise in Spanner for high-availability, scalable database solutions.
• Knowledge of data governance and security practices in cloud-based environments.
• Problem-solving mindset with the ability to tackle complex data engineering challenges.
• Strong communication and teamwork skills, with the ability to mentor and collaborate effectively.

RESPONSIBILITIES:

• Writing and reviewing great quality code
• Understanding the clients business use cases and technical requirements and be able to convert them in to technical design which elegantly meets the requirements
• Mapping decisions with requirements and be able to translate the same to developers
• Identifying different solutions and being able to narrow down the best option that meets the clients requirements
• Defining guidelines and benchmarks for NFR considerations during project implementation
• Writing and reviewing design document explaining overall architecture, framework, and high-level design of the application for the developers
• Reviewing architecture and design on various aspects like extensibility, scalability, security, design patterns, user experience, NFRs, etc., and ensure that all relevant best practices are followed
• Developing and designing the overall solution for defined functional and non-functional requirements; and defining technologies, patterns, and frameworks to materialize it
• Understanding and relating technology integration scenarios and applying these learnings in projects
• Resolving issues that are raised during code/review, through exhaustive systematic analysis of the root cause, and being able to justify the decision taken
• Carrying out POCs to make sure that suggested design/technologies meet the requirements