Data Engineer
- Posted 13 hours ago
- Be among the first 10 applicants
Job Description
Responsibilities
- Design, build, and optimize ETL/ELT pipelines for batch and streaming data
- Develop and maintain data warehouses and data lakes (e.g., Snowflake, BigQuery, Redshift)
- Collaborate with data scientists, analysts, and product teams to understand data needs
- Ensure data quality, reliability, and observability across pipelines
- Build and manage orchestration workflows (e.g., Airflow, Prefect, Dagster)
- Implement best practices for data modeling, partitioning, and performance tuning
- Monitor and troubleshoot data infrastructure issues in production
- 5+ years of experience in data engineering or a related role
- Proficiency in SQL and Python (or Scala/Java)
- Experience with cloud platforms (AWS, GCP, or Azure)
- Hands-on with big data tools: Spark, Kafka, dbt, or similar
- Familiarity with data warehousing concepts and dimensional modeling
- Strong understanding of CI/CD and version control (Git)
- Experience with real-time/streaming pipelines (Kafka, Flink, Kinesis)
- Exposure to ML pipelines or MLOps tooling
- Knowledge of data governance and data catalog tools (e.g., Atlan, Alation)
More Info
Key Skills
About Company
Skills:
RAG (Retrieval-Augmented Generation), Azure Data Factory (ADF), Pyspark, Azure Sql, Sql, ELT, Devops, Azure Synapse Analytics, Azure Data Lake, Spark, Cosmos DB, Python, Etl, Fabric Lakehouse architectures, Infrastructure as Code, vector databases, CI/CD pipelines, Microsoft Fabric, Azure AI Search
Skills:
Aws Redshift, Data Modeling, Automated Testing, Scala, Apache Spark, ELT, Azure Synapse, Gcp, Databricks, Data Warehousing, Azure, Python, AWS, Etl, Lakehouse Architecture, Google BigQuery, Unity Catalog, Delta Lake, CI/CD, Infrastructure-as-Code, Observability, Databricks Workflows
Skills:
Pyspark, Azure Databricks, Data Modeling, Sql, ELT, Databricks, Etl, GenAI frameworks and platforms, Databricks Mosaic AI, Lakehouse Architecture, Medallion Architecture, Transformation Notebooks, Batch and real-time streaming data processing frameworks, Cloud-based data storage solutions
Skills:
Pyspark, Sql, Jenkins, Terraform, Docker, Spark, Data Modelling, Python, Kubernetes, Streaming data processing, Data pipeline development, Event-driven architectures
Skills:
Azure Functions, Sql, Python, Azure Data Factory, Azure Databricks, Pyspark, ADLS, Synapse, Databricks notebooks