Key Result Areas and Activities:
- Study existing technology landscape and understand current data integration frameworks and do impact assessment for the requirements.
- Should be able to design complex Big Data use cases using AWS Services and Databricks, under guidance of Architect
- Should be able to assist the architect in reasoning about various architecture choices in terms of cost, performance and durability.
- Should be able to suggest optimizations in existing designs.
- Project documentation adheres to quality guidelines & schedules.
- Works hand in hand with Architect & PM for successful delivery of project and provide estimation, scoping, scheduling assistance.
- Articulate design decisions very clearly with stakeholders.
- Perform Proof-of-Concepts and document all observations before proposing a new solution.
- Conduct design review sessions with other teams and identify scope of improvement.
- Suggests automation to improve existing processes.
Work and Technical Experience:
Must-Have:
- In-depth knowledge of the following AWS services: S3, EC2, EMR, Athena, AWS Glue, Lambda
- Experience with at least one MPP database: AWS Redshift, Snowflake, SingleStore
- Proficiency in Big Data technologies: Apache Spark, Databricks
- Must have strong programming skills in Python
- Responsible for building data pipelines in AWS And Databricks
- Experience with Big Data table formats, such as Delta Lake (open source)
- Must have very strong SQL skills
- Experience with orchestration tools like Apache Airflow
- Expertise in developing ETL workflows with complex transformations such as SCD, deduplications, aggregations, etc.
- Should be a quick and self-learner, ready to adapt to new AWS services or Big Data technologies as required
- Strong understanding of data warehousing concepts
Good to have:
- Cloud Databases Snowflake, AWS Aurora
- Big Data Hadoop, Hive
- Cloud Databases AWS Aurora
- Associate Level or Professional Level AWS Certification or Databricks Certification
Qualifications:
- Overall, 7+ years of IT Exp
- 5+ years of relevant experience in AWS related project
- Bachelor's degree in computer science, engineering, or related field (master's degree is a plus)
- Demonstrated continued learning through one or more technical certifications or related methods.
Qualities:
- Hold strong technical knowledge and experience
- Should have the capability to deep dive and research in various technical related fields
- Self-motivated and focused on delivering outcomes for a fast-growing team and firm
- Able to communicate persuasively through speaking, writing, and client presentations
- Able to consult, write, and present persuasively
- Able to work in a self-organized and cross-functional team
- Able to iterate based on new information, peer reviews, and feedback
- Prior experience of working in a large media company would be added advantage
