Data Engineer

Role Title: Sr Data Engineer I

Responsibilities:

Pipeline Development & Operations:

• Design, maintain, and optimize Spark and Flink pipelines running in Databricks and Airflow.

• Ensure reliability and availability of data workflows in production.

• Debug and resolve issues in notebooks, jobs, and orchestration workflows.

Performance Optimization:

• Tune Spark jobs, clusters, and data pipelines for performance and cost efficiency.

• Analyze logs, queries, and workloads to identify bottlenecks and implement improvements.

Monitoring & Observability:

• Build and maintain dashboards, alerts, and metrics with Datadog, CloudWatch, or Prometheus.

• Proactively monitor pipeline health and escalate issues as needed.

Incident Response & RCA:

• Participate in incident response for pipeline issues, helping diagnose and resolve outages.

• Contribute to RCA and preventative actions to improve long-term stability.

Collaboration & Knowledge Sharing:

• Partner with data engineering, BI, and platform teams to support delivery and operations.

• Document best practices, contribute to runbooks, and mentor junior engineers.

Basic Qualifications:

• 6+ years of experience in data engineering or big data platform operations.

• Hands-on experience with Databricks, Spark, Flink, or Airflow (MWAA) in production.

• Strong SQL skills and experience with ETL/ELT pipeline development.

• Familiarity with AWS services (S3, Glue, Athena, EMR, EKS) for pipeline management.

• Working knowledge of observability tools (Datadog, CloudWatch, Prometheus).

• Proficiency in Python, Scala, or Java for pipeline development and automation.

• Familiarity with infrastructure-as-code (Terraform or similar).

• Strong collaboration and problem-solving skills, with experience in distributed teams.

Preferred Qualifications:

• Experience with incident response and RCA in production data environments.

• Familiarity with CI/CD pipelines (GitLab, Jenkins, or GitHub Actions).

• Exposure to data governance, access control, and cost optimization practices.

• Certifications such as Databricks Data Engineer Associate or AWS Solutions Architect – Associate.

Required Education:

• Bachelor's degree in Computer Science, Engineering, or related technical field (or equivalent experience).