Job Summary
As a Sr. Developer you will play a pivotal role in designing developing and optimizing data solutions using cutting-edge technologies. Your expertise in Spark in Scala Delta Sharing and Databricks will drive innovation and efficiency in our data processes. Collaborate with cross-functional teams to enhance data management and analytics capabilities contributing to the companys growth and societal impact.
Responsibilities
- Develop and implement robust data solutions using Spark in Scala to optimize performance and scalability.
- Collaborate with teams to integrate Delta Sharing for seamless data exchange and interoperability.
- Administer Databricks Unity Catalog to ensure secure and organized data management.
- Utilize Databricks CLI to automate workflows and streamline data operations.
- Design and manage Delta Live Pipelines for real-time data processing and analytics.
- Implement Structured Streaming to handle continuous data streams efficiently.
- Apply risk management strategies to mitigate potential data-related risks and ensure compliance.
- Leverage Apache Airflow for orchestrating complex data workflows and processes.
- Integrate Amazon S3 for scalable and reliable data storage solutions.
- Utilize Amazon Redshift for efficient data warehousing and analytics.
- Develop scripts and applications using Python to enhance data processing capabilities.
- Optimize Databricks SQL queries for improved data retrieval and analysis.
- Manage Databricks Delta Lake to ensure data reliability and consistency.
- Oversee Databricks Workflows to automate and streamline data tasks.
- Implement PySpark for large-scale data processing and analytics.
Qualifications
- Possess extensive experience in Spark in Scala demonstrating proficiency in data processing and analytics.
- Have hands-on experience with Delta Sharing showcasing ability to facilitate data exchange across platforms.
- Demonstrate expertise in Databricks Unity Catalog Admin ensuring secure data governance.
- Show proficiency in Databricks CLI automating data workflows effectively.
- Experience with Delta Live Pipelines enabling real-time data processing.
- Skilled in Structured Streaming managing continuous data flows efficiently.
- Knowledgeable in risk management applying strategies to safeguard data integrity.
- Proficient in Apache Airflow orchestrating complex workflows seamlessly.
- Familiar with Amazon S3 and Amazon Redshift optimizing data storage and analytics.
- Adept in Python programming enhancing data processing capabilities.
- Experienced in Databricks SQL optimizing data queries for performance.
- Knowledgeable in Databricks Delta Lake ensuring data reliability.
- Skilled in Databricks Workflows automating data tasks efficiently.
- Proficient in PySpark handling large-scale data processing effectively.
