Roles & Responsibilities:
- Design, develop, and maintain complex ETL/ELT data pipelines in Databricks using PySpark, Scala, and SQL to process large-scale datasets
- Understand the biotech/pharma or related domains & build highly efficient data pipelines to migrate and deploy complex data across systems
- Design and Implement solutions to enable unified data access, governance, and interoperability across hybrid cloud environments
- Ingest and transform structured and unstructured data from databases (PostgreSQL, MySQL, SQL Server, MongoDB etc.), APIs, logs, event streams, images, pdf, and third-party platforms
- Ensuring data integrity, accuracy, and consistency through rigorous quality checks and monitoring
- Expert in data quality, data validation and verification frameworks
- Innovate, explore and implement new tools and technologies to enhance efficient data processing
- Proactively identify and implement opportunities to automate tasks and develop reusable frameworks
- Work in an Agile and Scaled Agile (SAFe) environment, collaborating with cross-functional teams, product owners, and Scrum Masters to deliver incremental value
- Use JIRA, Confluence, and Agile DevOps tools to manage sprints, backlogs, and user stories.
- Support continuous improvement, test automation, and DevOps practices in the data engineering lifecycle
- Collaborate and communicate effectively with the product teams, with cross-functional teams to understand business requirements and translate them into technical solutions
Must-Have Skills:
- Hands-on experience in data engineering technologies such as Databricks, PySpark, SparkSQL Apache Spark, AWS, Python, SQL, and Scaled Agile methodologies.
- Proficiency in workflow orchestration, performance tuning on big data processing.
- Strong understanding of AWS services
- Ability to quickly learn, adapt and apply new technologies
- Strong problem-solving and analytical skills
- Excellent communication and teamwork skills
- Experience with Scaled Agile Framework (SAFe), Agile delivery practices, and DevOps practices.
Good-to-Have Skills:
- Data Engineering experience in Biotechnology or pharma industry
- Experience in writing APIs to make the data available to the consumers
- Experienced with SQL/NOSQL database, vector database for large language models
- Experienced with data modeling and performance tuning for both OLAP and OLTP databases
- Experienced with software engineering best-practices, including but not limited to version control (Git, Subversion, etc.), CI/CD (Jenkins,Maven etc.), automated unit testing, and Dev Ops
Education and Professional Certifications
- Any Degree and 6-8 years of experience
- AWS Certified Data Engineer preferred
- Databricks Certificate preferred
- Scaled Agile SAFe certification preferred
