Job Description
Design and develop scalable ETL/ELT pipelines using Databricks and Apache Spark.
- Build and maintain Data Lake and Lakehouse architectures.
- Develop data ingestion pipelines from multiple sources (RDBMS, APIs, Kafka, Files, Cloud Storage).
- Design dimensional models (Star Schema/Snowflake Schema) for reporting and analytics.
- Create Delta Lake tables and optimise them for performance.
- Develop SQL-based transformations and reporting datasets.
- Work with BI tools such as Power BI, Tableau, or Looker.
- Optimise Spark jobs for performance and scalability.
- Implement data quality checks and validation.
- Collaborate with Data Architects, Data Scientists, and Business Analysts.
- Manage production deployments and monitoring.
