Data Engineer

Role & responsibilities

8+ years experiences on relevant field below (Internship, prototype, and personal projects won't be counted)

Coding is required. (Ideally Python or Java)

Own end to end lifecycle (From development to deployment to production environment)

• Experience in building or deploying solution in the cloud.
• Either Cloud Native (Serverless): S3, Lambda, AWS Batch, ECS
• Or Cloud Agnostic: Kubernetes, Helm Chart, ArgoCD, Prometeus, Grafana.
• CICD experience: Github action or Jenkin.
• Infrastructure as code: e.g., Terraform

And experience in at least one of this focus area:

• Big Data: Building Big data pipeline or Platform to process petabytes of data: (PySpark, Hudi, Data Lineage, AWS Glue, AWS EMR, Kafka, Schema Registry)
• Or GraphDB: Ingesting and consuming data in Graph Database such as Neo4J, AWS Neptune, JanusGraph or DGraph

Preferred candidate profile

1. Specifically highlight Kafka expertise - include details like:

• Experience with Kafka cluster management and configuration
• Stream processing with Kafka Streams or KSQL
• Schema Registry implementation and management
• Kafka Connect for data integration

1. Put significant focus on PySpark skills:

• Experience building and optimizing PySpark jobs for batch processing
• Stream processing with Spark Structured Streaming
• Familiarity with Delta Lake, Hudi, or Iceberg for lakehouse implementation

1. Highlight data engineering skills that complement these technologies:

• Data pipeline design and implementation
• Experience with data quality, validation, and lineage tracking
• Performance optimization for large-scale data processing