Design, develop, and manage robust data pipelines using PySpark and SQL.
Work with AWS services to implement data solutions.
Utilize Databricks for data processing and analytics. Collaborate with data scientists, analysts, and other stakeholders to understand data requirements.
Ensure data quality and integrity throughout the data lifecycle. Optimize and maintain existing data architectures. Troubleshoot and resolve data-related issues.
Requirements :
Proven experience as a Data Engineer or similar role. Strong proficiency in PySpark, SQL, AWS, and Databricks. Experience in building and optimizing big data pipelines and architectures. Solid understanding of data warehousing concepts and ETL processes. Familiarity with data governance and data security best practices. Excellent problem-solving skills and attention to detail. Strong communication and collaboration skills.
