Job Description
We are a team of data professionals who work with large datasets every day. With over 40 years of combined experience, we have a unique approach to data engineering.
Key Responsibilities
- Design and develop big data pipelines using Azure Databricks, PySpark, and Delta Lake.
- Work with Azure Data Lake, Azure Synapse, Azure SQL Database, and Azure Data Factory to implement robust data solutions.
- Develop and maintain ETL/ELT workflows for efficient data ingestion, transformation, and processing.
- Implement data governance, security, and compliance best practices in Azure environments.
- Optimize Databricks clusters, workflows, and cost efficiency in cloud environments.
- Collaborate with data scientists, analysts, and business stakeholders to ensure high-quality data solutions.
- Implement CI/CD pipelines for data engineering workflows using Azure DevOps.
Required Qualifications & Skills
- Databricks Certified Data Engineer Associate (Preferred)
- Databricks Certified Data Engineer Professional
- Azure Cloud Services : Azure Databricks, Azure Data Factory, Azure Data Lake, Azure Synapse, Azure Functions
- Big Data & ETL : PySpark, SQL, Delta Lake, Kafka (Preferred)
- Programming : Python, SQL, Scala (Optional)
- Orchestration & Automation : Airflow, Azure DevOps, GitHub Actions
- Data Governance & Security : Unity Catalog, RBAC, PII masking
- Performance Optimization : Spark tuning, Databricks cluster configuration
- Experience : years of experience in data engineering with a focus on Azure and Databricks.
- Strong understanding of data modeling, warehousing, and governance best practices
(ref:hirist.tech)
