Excellent knowledge and experience in Big data engineer.
Strong working experience with architecture and development in Apache Spark, Spark, Python, Azure Databricks, Data Pipelines, Azure Devops, Kafka, SQL Server/NoSQL.
Strong expertise in Python, Django Rest Framework, Databricks and PostgreSQL.
Hands on experience in building data pipelines and building data frameworks for unit testing, data lineage tracking, and automation.
Familiarity with streaming technologies (e.g., Kafka, Kinesis, Flink).
Experience with building and maintaining a cloud system.
Familiarity with data modeling, data warehousing, and building distributed systems.
Expertise in Spanner for high-availability, scalable database solutions.
Knowledge of data governance and security practices in cloud-based environments.
Problem-solving mindset with the ability to tackle complex data engineering challenges.
Strong communication and teamwork skills, with the ability to mentor and collaborate effectively.
RESPONSIBILITIES:
Writing and reviewing great quality code
Understanding the clients business use cases and technical requirements and be able to convert them in to technical design which elegantly meets the requirements
Mapping decisions with requirements and be able to translate the same to developers
Identifying different solutions and being able to narrow down the best option that meets the clients requirements
Defining guidelines and benchmarks for NFR considerations during project implementation
Writing and reviewing design document explaining overall architecture, framework, and high-level design of the application for the developers
Reviewing architecture and design on various aspects like extensibility, scalability, security, design patterns, user experience, NFRs, etc., and ensure that all relevant best practices are followed
Developing and designing the overall solution for defined functional and non-functional requirements; and defining technologies, patterns, and frameworks to materialize it
Understanding and relating technology integration scenarios and applying these learnings in projects
Resolving issues that are raised during code/review, through exhaustive systematic analysis of the root cause, and being able to justify the decision taken
Carrying out POCs to make sure that suggested design/technologies meet the requirements
