Azure Data Engineer / AWS Data Engineer

Key Responsibilities:

Data Pipeline Development

• Design and develop scalable data pipelines using Python, PySpark, and Pandas
• Process and transform large volumes of data on cloud platforms (AWS / Azure)
• Ingest both batch and streaming data from diverse sources

ETL and Analytical Applications

• Build and maintain ETL, OLAP-based, and analytical applications
• Write and optimize complex SQL queries across various RDBMS systems (Oracle, PostgreSQL, SQL Server)

Cloud Services Expertise

• Work with AWS services such as Lambda, Glue, Redshift, Athena, Kinesis, EMR, IAM, and S3
• Use Azure services including ADF, Databricks, Synapse Analytics, Azure SQL DB, ADLS, Azure Storage, and HDInsight
• Manage security and identity with Azure Key Vault and Azure Active Directory

Data Warehousing

• Experience with cloud data warehouses such as Snowflake or Amazon Redshift

DevOps and Automation

• Build and manage CI/CD pipelines and code repositories using Git and branching strategies
• Develop and maintain Airflow DAGs and work with infrastructure-as-code principles
• Automate deployments using modern DevOps tools

Integration and APIs

• Integrate systems and data using REST APIs
• Collaborate with cross-functional teams in Agile environments

Documentation and Communication

• Communicate clearly with technical and non-technical stakeholders
• Document processes, pipelines, and architecture effectively