Lead Data Engineer

• Highly skilled and experienced Senior Data Engineer to lead our data engineering & analytics team.
• The successful candidate will be responsible for designing, implementing, and managing our data infrastructure
• and analytical systems.
• This role requires exceptional technical expertise in data modeling, ETL processes, and database management.
• As the team lead, you will provide guidance and mentorship to other data engineers, ensuring high-quality deliverables and driving the teams success.

Major Responsibilities

• Design, develop, and implement data models, ETL processes, and data pipelines to ensure efficient and accurate data flow using Databricks.
• Lead and manage a team of data engineers, providing technical guidance, mentoring, and support.
• Implementing and enforcing architectural standards and frameworks to ensure a flexible and scalable data environment.
• Collaborating with cross-functional teams to gather requirements, analyze data, and design effective solutions.
• Hands-on development of Python/ Spark-based scripts and applications to support data processing and transformation.
• Utilizing tools such as Databricks Notebooks, Azure Data Factory, and Change Data Capture (CDC) for orchestrating and managing data workflows.
• Optimize and enhance existing data infrastructure and systems for performance, scalability, and reliability.
• Playing a key role in DevOps activities, including deployment of jobs and required infrastructure setup.
• Staying updated with the latest industry trends and technologies in data engineering and analytics.

Knowledge and Education

Degree in computer science or a similar field.

Work Experience

• 7+ Years of experience in the field of Data Engineering & Analytics.
• Strong understanding of Data warehousing, engineering & ETL concepts.
• Experience in Databricks development using Python, PySpark & DLT Pipelines.
• Experience working with databases and various types of data storage.
• Excellent communication and interpersonal skills
• Time and project management skills, with the ability to prioritize and multitask as needed.
• Goal orientation and holistic thinking and action.
• Experience working with Azure / AWS.
• Certifications in relevant data technologies will be an added advantage.

Skills and Competencies

• Team leading skills
• Troubleshooting and analytical skills.
• Good communication and collaboration skills.
• Client management skills.