Sr. Data Engineer

Responsibilities:

• Develop, maintain, and optimize data pipelines, workflows, and Feature Store for scalable data ingestion and transformation.
• Design, develop, and architect Data Engineering pipelines, ensuring performance, scalability, and maintainability.
• Implement advanced data transformations and quality checks to ensure accuracy, completeness, security, and consistency.
• Integrate data from diverse structured and unstructured sources, leveraging AWS S3 and Snowflake (or similar SQL Data Warehouses).
• Create and implement advanced data models and schemas while adhering to data governance and management best practices.
• Troubleshoot, perform root cause analysis (RCA), and optimize data pipelines and workflows.

Education and Experience Required:

• Bachelor's degree in Computer Science, Statistics, Informatics, Information Systems, or another quantitative field.
• 7+ years of experience in data analysis and engineering.
• Hands-on experience with API or stream-based data extraction (e.g., Salesforce API/Bulk API) and web crawling.

Technical Knowledge and Skills:

Primary: Advanced web-crawling & scraping methods, Python, PySpark, SQL, ETL/ELT processes, building end-to-end data engineering pipelines for semi-structured and unstructured data, SQL Data Warehouses (Snowflake preferred).

Secondary: Databricks, AWS (S3, Glue, EMR, EC2, RDS, IAM, monitoring), Kafka & Kafka Streaming, workflow automation (e.g., GitHub Actions).

Personal Attributes:

• Strong analytical and problem-solving skills; ability to handle sensitive information with accuracy and ethical awareness.
• Self-motivated, innovative, and confident working independently while thriving in a collaborative team environment.
• Effective time management; ability to anticipate stakeholder needs and deliver solutions proactively.
• Comfortable navigating ambiguity and change in a fast-paced environment.