Position: Senior Data Engineer
Location: Pune/Nagpur
Type of Employment: FTE
Purpose of the Position:
This position requires a sound knowledge on Big Data using Cloud technologies such as such as Databricks, EMR, Athena, PySpark, S3, AWS Lambda etc. Strong foundation on database concepts and SQL is also required.
Key Result Areas and Activities:
- Technology Assessment and Design
Study existing technology landscape and understand current data integration frameworks and do impact assessment for the requirements.
- Documentation and Stakeholder Communication
Project documentation adheres to quality guidelines & schedules.
Works hand in hand with Architect & PM for successful delivery of project and provide estimation, scoping, scheduling assistance.
- Process Improvement and Automation
Suggests automation to improve existing processes.
Assist junior Data Engineers by providing expert advice or troubleshooting steps whenever required.
- Training and Knowledge Sharing
Create technology-focused training plans whenever required.
Deliver technology-focused training sessions with team members whenever required.
Essential Skills:
- In-depth knowledge of the following AWS services: S3, EC2, EMR, Athena, AWS Glue, Lambda
- Experience with at least one MPP database: AWS Redshift, Snowflake, SingleStore
- Proficiency in Big Data technologies: Apache Spark, Databricks
- Must have strong programming skills in Python
- Responsible for building data pipelines in AWS And Databricks
- Experience with Big Data table formats, such as Delta Lake (open source)
- Must have very strong SQL skills
- Experience with orchestration tools like Apache Airflow
- Expertise in developing ETL workflows with complex transformations such as SCD, deduplications, aggregations, etc.
- Should be a quick and self-learner, ready to adapt to new AWS services or Big Data technologies as required
- Strong understanding of data warehousing concepts
Desirable Skills:
- Cloud Databases – Snowflake, AWS Aurora
- Big Data – Hadoop, Hive
- Cloud Databases– AWS Aurora
- Associate Level or Professional Level AWS Certification or Databricks Certification
Qualifications:
- Overall, 7+ years of IT Exp
- 5+ years of relevant experience in AWS related project
- Bachelor's degree in computer science, engineering, or related field (master's degree is a plus)
- Demonstrated continued learning through one or more technical certifications or related methods.
Qualities:
- Hold strong technical knowledge and experience
- Should have the capability to deep dive and research in various technical related fields
- Self-motivated and focused on delivering outcomes for a fast-growing team and firm
- Able to communicate persuasively through speaking, writing, and client presentations
- Able to consult, write, and present persuasively
- Able to work in a self-organized and cross-functional team
- Able to iterate based on new information, peer reviews, and feedback
- Prior experience of working in a large media company would be added advantage
