4 Technical Lead (Databricks Engineering)
Role Overview
Lead development of Databricks pipelines replacing SQL Server cleansing processes.
Key Responsibilities
- Lead development of API/FTP ingestion pipelines
- Oversee Spark-based cleansing & enrichment implementation
- Review code and enforce standards
- Optimize Delta tables & Spark jobs
- Guide migration from SQL logic to PySpark logic
- Implement monitoring & logging
- Troubleshoot production pipeline issues
Required Skills
- Strong Databricks hands-on experience
- Strong PySpark & Spark SQL
- Experience converting SQL procedures into Spark logic
- Experience with API ingestion
- Strong performance tuning expertise
- Healthcare data experience preferred
