Senior Data Engineer

We are seeking a Senior Data Engineer to develop and manage scalable data solutions crucial for the Consumer-Packaged Goods (CPG) sector. This role ensures the integration, accuracy, and performance of data systems, enabling insights into consumer behavior and supply chain efficiency. The ideal candidate will optimize data workflows and support data-driven strategies to drive product innovation and business growth.

What will you do:

• Build and support scalable data engineering pipelines (load, extract, clean, transform, ingest, etc.) for our SaaS AI applications and Predictions-as-a-Service implementation and products.
• Build and deploy data engineering pipelines at scale using Kubeflow/Airflow.
• Own multiple data pipelines.
• Review, audit, and improve data engineering pipelines.
• Document and communicate technical and functional designs (HLD, LLD), timelines, and plans.
• Collaborate closely with AI/ML, DevOps, MLE, and other engineering and product teams.
• Work with engineering and implementation teams to understand business problems and implement efficient software solutions.
• Follow configurable programming practices (config files, modular programming, parameterization, etc.).
• Support the maintenance and enhancement of deployed pipelines, including bug fixes and performance improvements.
• Participate in organization-building activities (cross-team training, hiring, etc.) to further the Asper mission and culture.

Experience Required:

• Proficient coding skills in Python, PySpark, and SQL.
• Proven track record of writing and maintaining Spark code in a production environment.
• Experience building and supporting scalable data engineering pipelines (ETLs), including data extraction, loading, transformation, real-time monitoring, and alerting.
• Familiarity with orchestration technologies such as Airflow and Kubeflow.
• Experience with public clouds (AWS/Azure).
• Demonstrated production experience in big data infrastructure and data modelling.
• Performance optimization experience for data loading and ingestion.
• Knowledge of essential developer tools, including GitHub, Docker, and VSCode.
• Ability to thrive in a fast-paced, deadline-driven environment.