We are seeking a highly skilled Senior Data Engineer with 78 years of experience to design, develop, and optimize data pipelines and solutions. The ideal candidate will have expertise in PySpark, SQL, and Python, with solid knowledge of Azure Databricks. A strong understanding of agile methodologies will be an added advantage.
Key Responsibilities
- Design, build, and maintain scalable data pipelines and ETL processes.
- Work with large datasets to ensure high data quality, accuracy, and performance.
- Develop data solutions leveraging Azure Databricks, PySpark, and SQL.
- Collaborate with cross-functional teams including Data Scientists, Analysts, and Product teams to support data-driven initiatives.
- Troubleshoot and optimize data workflows to improve reliability and efficiency.
- Ensure compliance with best practices in data governance, security, and performance.
- Contribute to agile ceremonies and work in a collaborative, fast-paced environment.
Required Skills & Qualifications
- 78 years of experience in Data Engineering.
- Strong programming skills in Python and SQL.
- Hands-on experience with PySpark for large-scale data processing.
- Good working knowledge of Azure Databricks.
- Strong problem-solving and debugging skills.
- Experience in designing and optimizing ETL pipelines.
- Knowledge of Agile methodologies (Scrum/Kanban) is a plus.
Nice To Have
- Familiarity with other Azure services such as Data Lake, Synapse, or Data Factory.
- Exposure to CI/CD practices and version control systems (e.g., Git).
- Experience working in a cloud-first data ecosystem.
