We are seeking a highly skilled Azure Databricks Engineer to establish and maintain an efficient data ecosystem within our organization. You will play a critical role in designing and implementing scalable data pipelines, optimizing data workflows, and ensuring data quality using Azure Data Bricks, Python, and Spark. This position requires strong technical expertise and the ability to collaborate across various functional teams to drive data-driven decision-making.
Roles & Responsibilities:
- Design and implement scalable data pipelines using Azure Data Bricks.
- Develop ETL processes to efficiently extract, transform, and load data.
- Collaborate with data scientists and analysts to define and refine data requirements.
- Optimize Spark jobs for performance and efficiency.
- Monitor and troubleshoot production workflows and jobs.
- Implement data quality checks and validation processes.
- Create and maintain technical documentation related to data architecture.
- Conduct code reviews to ensure best practices are followed.
- Integrate data from various sources, including databases, APIs, and third-party services.
- Utilize SQL and Python for data manipulation and analysis.
- Collaborate with DevOps teams to deploy and maintain data solutions.
- Stay updated with the latest trends and updates in Azure Data Bricks and related technologies.
- Facilitate data visualization initiatives for better data-driven insights.
- Provide training and support to team members on data tools and practices.
- Participate in cross-functional projects to enhance data sharing and access.
Skills Required:
- Strong expertise in Azure Data Bricks and data lake concepts.
- Proficiency with SQL, Python, and Spark.
- Solid understanding of data warehousing concepts.
- Experience with ETL tools and frameworks.
- Familiarity with cloud platforms such as Azure, AWS, or Google Cloud.
- Excellent problem-solving and analytical skills.
- Ability to work collaboratively in a diverse team environment.
- Experience with data visualization tools such as Power BI or Tableau.
- Strong communication skills with the ability to convey technical concepts to non-technical stakeholders.
- Knowledge of data governance and data quality best practices.
- Hands-on experience with big data technologies and frameworks.
- A relevant certification in Azure is a plus.
- Ability to adapt to changing technologies and evolving business requirements.
QUALIFICATION:
- Bachelor's degree in Computer Science, Information Technology, or a related field.
