Azure Data Engineer+Pyspark
JD:-
- PySpark and Spark SQL
- Working with Dataframes using different APIs
- Spark Functions
- Advanced Data transformations
- Good knowledge in SQL
- Applying UDFS
- Error handling, logging and monitoring
- Unix, Scheduling Tool
- Performance Tuning & Generic process development
- Banking Domain Knowledge
- Develop and maintain ETL pipelines using Azure Databricks (PySpark / Spark SQL)
- Process and transform large financial datasets (transactions, customer data)
- Implement data pipelines for AML, KYC, risk, and regulatory reporting
- Ensure data quality, governance, and lineage tracking
- Integrate data from banking systems (core banking, payments, trading)
- Optimize Spark jobs for performance and cost
- Collaborate with business and compliance teams
- Support audit and regulatory requirements
