Job Profile- Sr Data Engineer
Exp- 10+ yrs
Location- Bangalore
Mode of work- Hybrid (2 Day WFO)
Responsibilities:
Develop, maintain, and optimize data pipelines and workflows and Feature Store to ensure seamless data ingestion and transformation as a scalable data solution.
- Design, develop, implement, and architect Data Engineering pipelines, considering performance & scalability including data storage and processing.
- Implement advanced data transformations and quality checks to ensure data accuracy, completeness, security and consistency of data.
- Seamlessly integrate data from diverse sources, for data ingestion, transformation and storage, leveraging AWS S3 Storage and possibly Snowflake as a SQL Data Warehouse.
- Create and implement advanced data models and schemas and ensure data governance and data management best practices.
Qualification and Desired Experiences:
- 7+ years of data analysis and engineering experience
- Bachelor's degree in computer science, Statistics, Informatics, Information Systems or another quantitative field.
- Working knowledge of API or Stream-based data extraction processes like Salesforce API and Bulk API and have hands-on experience in web crawling.
- Advanced Web-crawling & scraping methods and tools
- Building end-end Data Engineering pipelines for Semi and unstructured data (Text, all kinds of simple/complex table structures, images, video and audio data)
- Python, Pyspark, SQL, RDBMS
- Data Transformation (ETL/ELT) activities
- SQL Data warehouse (e.g. Snowflake) working / preferably administration
Good to have skills-
- Databricks
- Familiarity with AWS services : S3, Glue, EMR, EC2, RDS, monitoring and IAM
- Kafka, Spark & Kafka Streaming
- Workflow automation (e.g. using Github actions)
- Performing RCA
