Big Data Lead Engineer

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Apache Spark, Databricks, Python, and PySpark.
• Build and optimize data processing workflows for structured, semi-structured, and unstructured data.
• Develop data solutions using Azure services including Azure Databricks and Azure Data Factory.
• Implement data lakehouse architecture using medallion design principles.
• Ensure data quality, reliability, security, and compliance across all pipelines and systems.
• Optimize Spark jobs using performance tuning techniques and debugging tools like Spark UI.
• Work with storage formats such as Delta Lake, Parquet, and Avro for efficient data handling.
• Develop automation solutions to reduce manual effort and improve delivery speed.
• Implement CI/CD practices using GitHub or GitLab for code versioning and deployment.
• Collaborate with stakeholders to understand requirements and translate them into technical solutions.
• Monitor production systems, troubleshoot incidents, and ensure system stability.
• Contribute to best practices, architecture standards, and knowledge sharing within teams.