Databricks Architecture

Responsibilities:

• Platform Setup and Configuration Lead the endtoend setup and configuration of a secure and highperforming Databricks platform ensuring alignment with best practices and security standards
• DevOps Implementation Design and implement a comprehensive DevOps framework for the Databricks environment including CICD pipelines for code deployment automated testing and infrastructure management
• Data Governance Framework Establish and enforce data governance policies and procedures within the Databricks platform ensuring data quality security compliance and proper data lineage
• AI Integration Strategy Define and execute a strategy for integrating AIML capabilities within the Databricks platform exploring potential use cases and leveraging relevant Databricks features like MLflow
• Use Case Identification and Development Work closely with the data engineering team and business stakeholders to understand customer business objectives identify specific use cases where Databricks can provide significant value and develop proofofconcept solutions to showcase platform capabilities
• Collaboration and Knowledge Sharing Partner effectively with the existing data engineering team providing guidance mentorship and knowledge transfer on Databricks best practices DevOps methodologies data governance principles and AI integration techniques
• Technical Leadership Serve as a technical leader and subject matter expert for Databricks providing guidance on architectural decisions performance optimization and troubleshooting
• Documentation Create and maintain comprehensive documentation for the Databricks platform setup DevOps processes data governance policies and developed use cases
• Continuous Improvement Stay abreast of the latest advancements in Databricks DevOps data governance and AI technologies and proactively identify opportunities for platform enhancement and process improvement

Qualifications

• Bachelors or Masters degree in Computer Science Engineering or a related field
• Minimum of 8 years of experience in data engineering with a strong focus on cloudbased data platforms
• Extensive handson experience with Databricks including platform setup Spark programming Python Scala SQL and Databricks Delta Lake
• Proven track record of designing and implementing DevOps pipelines for data platforms using tools like Jenkins GitLab CICD Azure DevOps or similar
• Deep understanding of data governance principles and experience in establishing data quality frameworks security measures and compliance protocols
• Experience with integrating AIML workflows within a data platform ideally using Databricks MLflow or similar MLOps tools
• Excellent problemsolving and analytical skills with the ability to translate business requirements into technical solutions
• Strong communication and collaboration skills with the ability to effectively work with both technical and nontechnical stakeholders including offshore and onshore teams
• Experience working in an offshore delivery model is highly preferred
• Ability to work independently and manage multiple priorities in a fastpaced environment
• Experience with other cloud platforms eg AWS Azure GCP and related data services is a plus

Preferred Qualifications

• Databricks certifications
• Experience in mention specific industry relevant to your company customers
• Familiarity with agile development methodologies