Data Engineer - Python/SQL

Skills : Data engineer

Location : Remote

Experience : 4+ years

Notice : Immediate only

Key Skills

• Data Engineering Expertise : Bring 3+ years of experience in building data pipelines and managing a secure, modern data stack. This includes CDC streaming ingestion using tools like Debezium into a Hudi data lake that supports AI/ML workloads and a curated Redshift data warehouse.
• AWS Cloud Proficiency : At least 3 years of experience working with AWS cloud infrastructure, including Kafka (MSK), Spark / AWS Glue, and infrastructure as code (IaC) using Terraform.
• Strong Coding Skills : Write and review high-quality, maintainable code that enhances the reliability and scalability of our data platform. We use Python, SQL, and dbt extensively, and you should be comfortable leveraging third-party frameworks to accelerate development.
• Data Lake Development : Prior experience building data lakes on S3 using Apache Hudi with Parquet, Avro, JSON, and CSV file formats.
• Workflow Automation : Build and manage multi-stage workflows using serverless Lambdas and AWS Step Functions to automate and orchestrate data processing pipelines.
• Data Governance Knowledge : Familiarity with data governance practices, including data quality, lineage, and privacy, as well as experience using cataloging tools to enhance discoverability and compliance.
• CI/CD Best Practices : Experience developing and deploying data pipeline solutions using CI/CD best practices to ensure reliability and scalability.
• Data Integration Tools : Working knowledge of tools such as Stitch and Segment CDP for integrating diverse data sources into a cohesive ecosystem.
• Analytical and ML Tools Expertise : Knowledge and practical experience with Athena, Redshift, or Sagemaker Feature Store to support analytical and machine learning workflows is a definite bonus!