Senior Data Engineer

Responsibilities:

Build Data Pipelines:

Utilize PySpark and Python to construct efficient and scalable data pipelines.

Integrate data from multiple source systems into a unified target system.

Orchestrate Pipelines with Airflow:

Use Apache Airflow to orchestrate and schedule data pipelines, ensuring timely and reliable execution.

Enhance Existing Pipelines:

Understand existing data pipelines and make enhancements based on evolving business requirements.

Implement improvements to optimize performance and maintainability.

Debugging and Root Cause Analysis:

Troubleshoot and resolve any failures in data pipelines promptly.

Conduct root cause analysis for pipeline failures and implement corrective measures.

Collaboration with Stakeholders:

Work closely with various stakeholders, both within and across teams.

Communicate effectively to understand and address business needs related to data processing.

Weekend and Shift Support:

Be available to work on weekends and in shifts if necessary to provide support for business operations.

Experience:

• 3-5 years of experience as a data engineer, demonstrating a solid understanding of data engineering principles.

Technical Skills:

• Proficient in SQL, Python, and PySpark for designing and implementing data solutions.
• Knowledge of data warehousing techniques and dimensional modeling.

Orchestration Tools:

• Experience with Apache Airflow for orchestrating complex data workflows.
• Familiarity with containerization using Docker and version control systems.

Data Modeling and Transformation:

• Exhibit strong proficiency in data modeling techniques, emphasizing expertise in designing and implementing effective data structures.
• Knowledge of DBT (Data Build Tool) for transforming and modeling data.

Cloud Platform:

• AWS knowledge is a plus, showcasing familiarity with cloud-based data services and infrastructure.