Search by job, company or skills

Data Engineer Senior

Data Engineer Senior

Spark Eighteen Pvt. Ltd.
5-7 Years
Not Disclosed

This job is no longer accepting applications

Job Description

We're hiring a Data Engineer (5+ years) to independently design, deploy, and operate production-scale data platforms in fully air-gapped, secure customer environments — no cloud dependency, with manual deployment and troubleshooting expected. The ideal candidate has hands-on experience with Apache Airflow, Spark, PostgreSQL, Docker, Python, SQL, and S3-compatible storage (MinIO), along with strong knowledge of geospatial and time-series data, and is comfortable owning infrastructure end-to-end without managed cloud services — from data pipelines to deployment to troubleshooting in fully isolated environments.

Requirements

  • 5+ years of experience in Data Engineering and production-scale data platforms
  • Design and manage Apache Airflow pipelines for high-volume sensor, satellite, and third-party data ingestion within isolated environments
  • Build and optimize Apache Spark workloads for batch processing, geospatial analytics, and large-scale aggregations
  • Develop and maintain PostgreSQL, TimescaleDB, PostGIS, and DuckDB-based data solutions
  • Implement efficient data ingestion, transformation, and bulk-loading pipelines without external connectivity
  • Containerize and deploy applications using Docker and Docker Compose in air-gapped environments
  • Work with S3-compatible object storage (e.g., MinIO) for on-prem data storage
  • Proven experience deploying via Docker/Docker Compose in air-gapped or restricted-network environments
  • Comfortable working in Ubuntu-based environments with manual/local infrastructure management
  • Strong proficiency in Python, SQL, Airflow, Spark, and PostgreSQL
  • Experience with geospatial and time-series data
  • Collaborate with customer teams to translate business requirements into data products
  • Ensure data quality, lineage, observability, performance, and SLA compliance
  • Troubleshoot production issues on-site/remotely without cloud tooling support
  • Create and maintain technical documentation, data models, and operational run-books
  • Strong communication, stakeholder management, and problem-solving skills

Benefits

  • Comprehensive insurance coverage that gives you peace of mind, so you can focus on doing your best work
  • Flexible work arrangements designed to support sustained productivity, personal well-being, and work-life balance
  • Continuous learning and accelerated skill development through hands-on projects and mentorship from experienced industry leaders
  • Global client exposure across 20+ countries, offering real-world experience with diverse markets and business environments.
  • Opportunity to work on high-impact, large-scale projects that have collectively generated over $1B in measurable business value
  • Competitive, market-aligned compensation packages that recognize performance, expertise, and long-term contribution
  • Monthly demo days that celebrate innovation, showcase your work, and give you a real voice in what we build
  • Annual recognition programs and performance-driven awards in a truly meritocratic environment
  • Referral bonuses that reward you for helping grow a strong, like-minded team
  • A strong problem-solving culture with opportunities to tackle meaningful, real-world challenges
  • A positive, people-first workplace that supports happiness, balance, and long-term growth





More Info

Job Type:
Industry:
Employment Type:

Key Skills

Similar Jobs

3-5 yrs
Gurugram, Gurugram, India
Skills:
Ms Excel, Power Bi, SAS, Tableau, Sql
5-10 yrs
Noida, India
Skills:
S3, Github, Pyspark, Lambda, Ec2, Kinesis, Docker, Shell scripting, Python, AWS, Scala, Redshift, Sql, Git, Kubernetes, GenAI, Airflow, AI coding assistants, LLM tools, RAG pipelines, LangChain, prompt engineering, dbt, Glue, LlamaIndex
6-8 yrs
Gurugram, Gurugram, India
Skills:
Github, Pyspark, Pandas, Numpy, Gcp, Docker, Databricks, Azure, Python, AWS, scikit-learn, Great Expectations, Kedro
7-9 yrs
Gurugram, Gurugram, India
Skills:
Typescript, Pytorch, Docker, Terraform, Elasticsearch, Python, AWS, Sql, Gcp, Opencv, Kubernetes, Qdrant, HuggingFace, Ray, MLflow, LangSmith, LangGraph, GitHub Actions, OpenAI APIs, Weights Biases, FAISS, ArgoCD, Weaviate, BentoML
6-9 yrs
Gurugram, Gurugram, India
Skills:
Pyspark, Kafka, Sql, Azure Data Factory, Terraform, Spark, Databricks, Azure, Python, MLflow, Unity Catalog, GitHub Actions, Microsoft Purview, Event Hubs, Delta Lake, Structured Streaming, Bicep