Principal Consultant Data Engineer.

Inviting applications for the role of Principal Consultant - Data Engineer.

In this role, a Data Engineer will leverage cloud technologies to manage and analyze their data. This role demands expertise in Databricks, Azure Data Factory (ADF), Python, and PySpark and Unity Catalog to efficiently process and analyze large datasets. Data Engineer Is responsible for designing and implementing scalable data pipelines, optimizing data workflows, ensuring data quality, collaborating with cross-functional teams, and leveraging cloud technologies to enhance data processing and analytics capabilities.

Responsibilities

• Data Pipeline Development:

• Architect, build, and optimize data ingestion and transformation pipelines using Azure Data Factory and Azure Databricks.

• Implement data integration and transformation solutions using Azure Databricks.

• Data Quality and Governance:

• Ensure data quality frameworks, lineage, and monitoring are in place.

• Implement data quality checks, validation rules, and governance policies to ensure accuracy, reliability, and security of data assets.

• Data Management:

• Pull data from different sources, transform and stitch it for advanced analytics activities.

• Design, implement, and deploy data loaders to load data into the engineering sandbox.

• Develop and deploy data models and solutions using Azure services.

• Collaboration and Support:

• Collaborate with machine learning engineers and cloud engineers for the design and implementation of data management solutions.

• Work with data scientists and analysts to support their data requirements.

• Performance Optimization:

• Monitor and optimize data pipelines for performance and reliability.

• Troubleshoot and resolve data-related issues promptly.

• Security Measures:

• Implement data security and privacy measures to protect sensitive information.

• Implement Unity Catalog:

• Manage data governance and security using Unity Catalog to ensure compliance and protect sensitive information.

• Leadership and Mentorship:

• Mentor junior engineers and perform peer reviews.

• Participate in code reviews and provide feedback to improve data engineering processes

• Leverage data best practices and tools and assist ML engineer in pulling, filtering, tagging, joining, parsing, and normalizing data sets for use.

Qualifications We Seek in You!

Minimum Qualifications / Skills

• A bachelor's degree in computer science, Information Technology, Business, or a related field is required

• Experience in Databricks, Azure ADF, Python, Pyspark

• Experience in working on data ingestion or ETL from Workday or any other HR systems

• Experience in RBAC security models in Unity Catalog

• Exposure to CI/CD DevOps practices

• Expertise in Azure Databricks, including its features for big data processing and collaborative notebooks.

• Strong programming skills in Python for data manipulation and scripting.

• Extensive experience with PySpark and SQL for building scalable data transformation jobs, data querying, analysis, and data modelling

• Proficiency in data engineering tools like Databricks, Apache Spark, and Unity Catalogue

• Hands-on experience with Azure Data Factory for pipeline orchestration, scheduling, and monitoring.

Preferred Qualifications / Skills

• Mentoring data engineers and managing specific POD

• Certifications in Azure data engineering, databrickor related fields