Data Engineer
- Posted a month ago
- Over 50 applicants have applied
Job Description
Roles and Responsibilities
- Design, develop, and maintain large-scale data pipelines using Azure Data Factory (ADF) to extract, transform, and load data from various sources into Azure Databricks.
- Develop complex ETL processes using Python scripts and SQL queries to process large datasets stored in Azure Data Lake Storage.
- Collaborate with cross-functional teams to gather requirements for new data warehousing solutions and implement scalable architectures on Azure Data Warehouse.
- Troubleshoot issues related to ADF pipeline failures, Azure Synapse connections, and SQL queries.
More Info
Job Type:
Industry:
Function:
Employment Type:
About Company
Skills:
Common Table Expressions (CTEs), snowflake , BigQuery, PostgreSQL, Scala, Pyspark, SQL Server, Google Cloud, Apache Airflow, Azure Synapse, Databricks, Microsoft Azure, Advanced Sql, Python, AWS, Complex JOIN operations, dbt
Skills:
Java, Devops, Cloud Automation, Azure, Python, Google Cloud, Kubernetes, AWS
Skills:
Apache Airflow, Postgis, Docker, PostgreSQL, Apache Spark, Python, Sql, DuckDB, TimescaleDB
Skills:
Amazon Web Services (AWS), Java, Hadoop, Scala, Nodejs, Emr, Data Modeling, Sql, Hive, Spark, Python, Agentic AI tools, AI code generation tools, ETL pipelines
Skills:
Spark SQL, Distributed Computing, Pyspark, Apache Spark, Data Governance, Data Warehousing, Sql, ELT, Gcp, Agile, Databricks, Azure, Python, Etl, AWS, Databricks certification, Streaming frameworks, Auto Loader, Unity Catalog, Delta Lake, Lakehouse architecture, CI/CD for data pipelines