Consultant | Data Engineer

Key Responsibilities...

• Migrate SQL Server Stored Procedures to Databricks Notebooks, leveraging PySpark and Spark SQL for complex transformations.
• Design, build, and maintain incremental data load pipelines to handle dynamic updates from various sources, ensuring scalability and efficiency.
• Develop robust data ingestion pipelines to load data into the Databricks Bronze layer from relational databases, APIs, and file systems.
• Implement incremental data transformation workflows to update silver and gold layer datasets in near real-time, adhering to Delta Lake best practices.
• Integrate Airflow with Databricks to orchestrate end-to-end workflows, including dependency management, error handling, and scheduling.
• Understand business and technical requirements, translating them into scalable Databricks solutions.
• Optimize Spark jobs and queries for performance, scalability, and cost-efficiency in a distributed environment.
• Implement robust data quality checks, monitoring solutions, and governance frameworks within Databricks.
• Collaborate with team members on Databricks best practices, reusable solutions, and incremental loading strategies.

All you need is...

• Bachelor's degree in computer science, Information Systems, or a related discipline.
• 4+ years of hands-on experience with Databricks, including expertise in Databricks SQL, PySpark, and Spark SQL. (Must)
• Proven experience in incremental data loading techniques into Databricks, leveraging Delta Lake's features (e.g., time travel, MERGE INTO).
• Strong understanding of data warehousing concepts, including data partitioning, and indexing for efficient querying.
• Proficiency in T-SQL and experience in migrating SQL Server Stored Procedures to Databricks.