Design, develop, and optimize data pipelines using Azure Databricks. Collaborate with cross-functional teams to understand data requirements and deliver scalable solutions. Implement data transformation and processing tasks using PySpark. Ensure data quality and integrity across various data sources. Provide technical guidance and mentorship to junior team members.
Additional Responsibilities:
Minimum Qualifications Bachelor's degree in Computer Science, Information Technology, or a related field. Proven experience with Azure Databricks, with a minimum of 2 years in data engineering. Strong understanding of data processing frameworks and cloud technologies. Preferred Qualifications Experience with PySpark and its application in big data environments. Azure Databricks certification. Track record of successfully delivering data solutions in a consulting environment. Familiarity with data governance and security best practices. Excellent problem-solving skills and ability to work independently.
