Job Description

Job Description

About the Role We are looking for a hands-on Data Engineer (7+years) to build and optimize scalable data pipelines and analytical datasets on the Databricks platform. You will work closely with Analytics/BI, Product, and Business teams to enable data-driven decision-making. Retail / eCommerce domain exposure is a strong plus, along with the ability to translate business needs into reliable and performant data solutions.

Key Responsibilities:

Design, develop, and maintain robust ETL/ELT pipelines using Databricks (Spark) and Python (PySpark).
Develop and optimize complex transformations using SQL (joins, window functions, CTEs, query tuning).
Build curated datasets and data models to support reporting, dashboards, and advanced analytics use cases.
Implement pipeline reliability best practices: data quality checks, monitoring, alerting, and reconciliation.
Optimize Databricks workloads for performance and cost (cluster sizing, partitioning strategies, caching, file formats).
Work with structured and semi-structured data (JSON, CSV, Parquet/Delta) and handle schema evolution.
Collaborate with stakeholders to understand business KPIs and deliver data solutions aligned to retail/eCommerce metrics (sales, orders, returns, inventory, customer cohorts).
Follow engineering best practices for version control (Git), documentation, reusable code patterns, and testing.
Good to have: Support or migrate Alteryx workflows into Python/Databricks pipelines.

Must-Have Skills & Qualifications :

7-13 years of experience in Data Engineering / Data Warehousing / Big Data.
Strong hands-on experience with Databricks (Jobs/Workflows, notebooks, cluster concepts, Spark tuning fundamentals).
Strong programming skills in Python (PySpark preferred).
Excellent SQL skills, including performance tuning and writing complex analytical queries.
Experience building scalable pipelines and working with large datasets in distributed environments.
Strong understanding of data engineering concepts: ETL/ELT, orchestration, data validation, and observability.
Familiarity with modern data storage formats and practices (Delta/Parquet, partitioning, incremental loads).

Good-to-Have Skills:

Retail / eCommerce domain knowledge (customer behavior, funnel metrics, pricing/promotions, inventory, catalog, order lifecycle).
Alteryx (workflow development, optimization, scheduling, or migration to Databricks).
Experience with Lakehouse patterns and Delta Lake features (e.g., MERGE, OPTIMIZE, Z ORDER).
Experience with orchestration tools (e.g., Airflow, ADF, Databricks Workflows).
Cloud experience: AWS / Azure / GCP (S3/ADLS/GCS, IAM basics, security controls).
CI/CD exposure for data pipelines, code reviews, and automated deployments. Preferred Traits
Strong problem-solving skills and a mindset for root-cause analysis.
Ownership and accountability for production-grade pipelines.
Ability to communicate with both technical and non-technical stakeholders.
Comfort working in fast-paced environments with evolving requirements.