Technology->Cloud Platform->Databricks AI Services,Technology->Data Engineering->Databricks
Key Responsibilities:
- Lead the design and development of scalable data pipelines and ETL/ELT workflows using Databricks on Azure.
- Architect and implement data lakehouse solutions, optimizing storage, compute, and query performance for large datasets.
- Develop, optimize, and maintain Databricks notebooks, jobs, and workflows for batch and near real-time data processing.
- Define and enforce coding standards, best practices, and reusable frameworks for Databricks-based data engineering.
- Collaborate with data architects, analysts, and business teams to understand requirements and translate them into technical designs.
- Implement robust data quality checks, validation rules, and monitoring to ensure reliability and trust in data products.
- Optimize Spark jobs for performance and cost efficiency, including cluster sizing, caching strategies, and partitioning.
- Ensure security, governance, and compliance of data solutions, including access control and data lineage within Databricks and Azure.
- Troubleshoot complex data issues, perform root cause analysis, and drive long-term fixes and improvements.
- Mentor junior engineers, provide technical guidance, conduct code reviews, and support knowledge sharing within the team.
- Contribute to roadmap planning, effort estimation, and delivery management for data engineering initiatives.
- Knowledge of more than one technology
- Basics of Architecture and Design fundamentals
- Knowledge of Testing tools
- Knowledge of agile methodologies
- Understanding of Project life cycle activities on development and maintenance projects
- Understanding of one or more Estimation methodologies, Knowledge of Quality processes
- Basics of business domain to understand the business requirements
- Analytical abilities, Strong Technical Skills, Good communication skills
- Good understanding of the technology and domain
- Ability to demonstrate a sound understanding of software quality assurance principles, SOLID design principles and modelling methods
- Awareness of latest technologies and trends
- Excellent problem solving, analytical and debugging skills
