As a Technical Specialist - Azure Databricks, you will play a crucial role in designing, building, and maintaining scalable data pipelines and infrastructure. You will work with a variety of technologies, including Scala, Python, Spark, AWS (Amazon Web Services) services, and SQL, to support our data processing and analytics needs.
Responsibilities: -
- Develop robust data platforms, and contribute to the organization's data-driven growth.
- Utilize expertise in Apache Airflow for orchestrating and automating complex data workflows.
- Help the other team members in developing solutions.
- Design solutions on Databricks including using delta lake/delta table, data warehouse, and more.
- Apply best practices while developing the solutions.
- Work towards creating reusable components.
- Ensure that the pipelines are designed to keep the operating cost low.
Educational Qualifications:
- Engineering Degree BE/ME/BTech/MTech/BSc/MSc.
- Technical certification in multiple technologies is desirable.
Skills:
Mandatory Technical Skills:-
- Hands-on experience in designing, developing, and optimizing scalable large volume pipelines to support the processing of structured and semi-structured data
- In-depth knowledge of SQL, PySpark, Databricks, delta lake, delta table, Azure data factory, Azure services Python, DevOps and Git, Relational Databases (Oracle, SQL Server, and more)
- Understanding of designing and implementing data solutions involving real-time streaming technologies, such as Apache Kafka, AWS MSK, Kinesis and Azure Event Hubs, ensuring seamless integration of streaming data into processing pipelines
- Exposure to the CI/CD pipelines for the Azure and AWS resources
- Possess excellent communication skills, with the ability to effectively convey complex analytical findings
- Understanding of FinOps/ observability patterns, data governance best practices
- AWS/GCP/Azure
