Senior Data Engineer
- Posted 2 months ago
- Be among the first 20 applicants
Job Description
Experience Required: 4+ Years
Key Responsibility:
- Design, build, and maintain scalable and reliable data pipelines on Databricks, Snowflake, or equivalent cloud platforms.
- Ingest and process structured, semi-structured, and unstructured data from a variety of sources including APIs, RDBMS, and file systems.
- Perform data wrangling, cleansing, transformation, and enrichment using PySpark, Pandas, NumPy, or similar libraries.
- Optimize and manage large-scale data workflows for performance, scalability, and cost-efficiency.
- Write and optimize complex SQL queries for transformation, extraction, and reporting.
- Design and implement efficient data models and database schemas with appropriate partitioning and indexing strategies for Data Warehouse or Data Mart.
- Leverage cloud services (e.g., AWS S3, Glue, Kinesis, Lambda) for storage, processing, and orchestration.
- Build containerized solutions using Docker and manage deployment pipelines via CI/CD tools such as Azure DevOps, GitHub Actions, or Jenkins.
More Info
Key Skills
About Company
Skills:
snowflake , Apis, Csv, Pyspark, Kafka, Json, Sql, ELT, Kinesis, Spark, Data Lake, Python, Etl, Parquet, Databricks Delta Lake, dbt, Lakehouse, Glue, Data Mesh
Skills:
Data Factory, Devops, Spark Notebooks, Lakehouse architectures, Microsoft Fabric, Medallion Bronze Silver Gold data models, OneLake, Cloud data platforms, Infrastructure as Code, CI/CD, Azure Data Services
Skills:
Azure Data Factory, Apache Spark, Databricks, Sql, Python, Azure Data Lake Storage, Infrastructure as Code, Delta Lake, Unity Catalog
Skills:
Gcp, Apache Spark, Databricks, Tableau, Python, Sql, AWS, Airflow, Bitbucket Pipelines, dbt
Skills:
Data Engineer, Python, Sql, Pandas, BigQuery, Etl, Docker, Polar