Position summary:
We are seeking a Senior Software Development Engineer – Data Engineering with 3-5 years of experience to design, develop, and optimize data pipelines and analytics workflows using Snowflake, Databricks, and Apache Spark. The ideal candidate will have a strong background in big data processing, cloud data platforms, and performance optimization to enable scalable data-driven solutions.
Key Responsibilities:
- Work with cloud-based data solutions (Azure, AWS, GCP).
- Implement data modeling and warehousing solutions.
- Developing and maintaining data pipelines for efficient data extraction, transformation, and loading (ETL) processes.
- Designing and optimizing data storage solutions, including data warehouses and data lakes.
- Ensuring data quality and integrity through data validation, cleansing, and error handling.
- Collaborating with data analysts, data architects, and software engineers to understand data requirements and deliver relevant data sets (e.g., for business intelligence).
- Implementing data security measures and access controls to protect sensitive information.
- Monitor and troubleshoot issues in data pipelines, notebooks, and SQL queries to ensure seamless data processing.
- Develop and maintain Power BI dashboards and reports.
- Work with DAX and Power Query to manipulate and transform data.
Basic Qualifications
- Bachelor's or master's degree in computer science or data science
- 3-5 years of experience in data engineering, big data processing, and cloud-based data platforms.
- Proficient in SQL, Python, or Scala for data manipulation and processing.
- Proficient in developing data pipelines using Azure Synapse, Azure Data Factory, Microsoft Fabric.
- Experience with Apache Spark, Databricks and Snowflake is highly beneficial for handling big data and cloud-based analytics solutions.
Preferred Qualifications
- Knowledge of streaming data processing (Apache Kafka, Flink, Kinesis, Pub/Sub).
- Experience in BI and analytics tools (Tableau, Power BI, Looker).
- Familiarity with data observability tools (Monte Carlo, Great Expectations).
- Contributions to open-source data engineering projects.
