Manager - Customer Service Operations

Experience

510 Years

Job Summary

We are seeking a Senior Databricks & Data Science Engineer with strong hands-on experience in building scalable data engineering, analytics, and machine learning solutions using Databricks on Azure and AWS. The role involves working with large-scale datasets, advanced analytics, and ML workflows while following Agile delivery practices and ITIL service management processes.

Databricks & Data Engineering Responsibilities

- Develop and maintain Databricks notebooks, workflows, and jobs
- Build ETL / ELT pipelines using Apache Spark, PySpark, Databricks SQL, and Delta Lake
- Ingest and process data from Azure Data Lake Gen2, AWS S3, relational databases, APIs, and streaming sources
- Optimize Spark workloads for performance, scalability, and cost
- Implement data validation, cleansing, and error-handling mechanisms

Data Science & Machine Learning Responsibilities

- Perform exploratory data analysis (EDA) using Databricks notebooks
- Perform feature engineering and feature selection
- Build, train, evaluate, and tune machine learning models
- Use Python libraries such as Pandas, NumPy, Scikit-learn, and Spark MLlib
- Track experiments and models using MLflow

Azure & AWS Integration

- Work with Azure Databricks and AWS Databricks environments
- Integrate Databricks with Azure Data Lake, Azure Synapse, and Azure Key Vault
- Integrate Databricks with AWS S3, IAM roles, and CloudWatch
- Ensure secure data access and cloud-native authentication
- Support cloud cost optimization and performance monitoring

Job Orchestration, Monitoring & Support

- Create and manage Databricks Jobs and schedules
- Monitor job execution, failures, retries, and SLA adherence
- Troubleshoot Spark errors, data quality issues, and pipeline failures
- Provide production support and ensure stability of data pipelines

Process Flow Agile & ITIL

- Work within Agile/Scrum teams, participating in sprint planning, stand-ups, reviews, and retrospectives
- Follow ITIL processes for Incident, Problem, Change, and Release Management
- Perform root cause analysis (RCA) for production incidents and drive preventive actions
- Ensure controlled releases and smooth promotion of data pipelines and ML models

Required Skills

- Databricks (Notebooks, Jobs, Workflows)
- Apache Spark, PySpark, Databricks SQL
- Delta Lake
- Python for data engineering and data science
- Machine learning fundamentals
- MLflow
- Azure and/or AWS cloud data services
- Git version control

Good to Have

- Delta Live Tables (DLT)
- Unity Catalog
- Spark Structured Streaming
- Advanced analytics and predictive modeling

Soft Skills

Strong analytical and problem-solving skills, ability to work with business stakeholders, good communication skills, and strong documentation practices.