Senior Data Engineer

About Cloudwick

Cloudwick, an AWS certified data lake and advanced analytics partner, established in 2010 and headquartered in California, is at the forefront of data-driven transformation. We specialize in empowering organizations to harness their data's potential for profound insights. Our in-house creation, Amorphic Data Cloud, drives our commitment to innovation. Visit our website to explore how we are revolutionizing the world of data.

Amorphic Data Platform

Amorphic data is a self-service data platform, simplifying data analytics by integrating with over 70+ AWS services. It offers a unified interface that empowers users to make data-driven decisions easily. With built-in data catalogue, robust governance, security measures and advanced ML and AI capabilities, it enables clients to manage and harness their data for advanced analytics.

About the role

We are looking for Senior Data Engineers to join our Data Engineering team and contribute to building scalable, reliable, and high-performance data solutions. The ideal candidate will have strong experience working with large-scale data processing using Apache Spark/PySpark, along with hands-on expertise in designing and developing robust data pipelines and ETL frameworks.

The role will focus on designing, building, and maintaining data platforms that enable the business to efficiently capture, process, transform, store, and consume large volumes of data for analytics and business applications.

Key Responsibilities

• Design, develop, and maintain ETL processes and data pipelines using AWS Glue with Pyspark.
• Collaborate with cross-functional teams to understand data requirements and deliver high-quality reliable data solutions.
• Optimize and tune data pipelines for performance and scalability.
• Ensure data quality and integrity through robust testing and validation processes.
• Implement data governance and security best practices.
• Monitor and troubleshoot data pipelines to ensure continuous data flow and address any issues promptly.
• Participate in code reviews, technical design, documentation, and adoption of engineering best practices.
• Design and implement scalable data lakes, data warehouses, and data processing architectures.
• Mentor junior engineers stay up-to-date with the latest trends and technologies in data engineering and cloud.

Required Qualification

• B.E/B.Tech in Computer Science Engineering or related field with relevant work experience.
• 5-8 years of experience in handling data and designing ETL pipelines with mandatory 4+ years experience in writing Spark.
• Experience with AWS services such as Glue, Athena, S3, and Redshift is a plus.
• Strong proficiency in Python, PySpark, and SQL, with a solid understanding of distributed computing and data processing.
• Hands-on experience designing and optimizing ETL/ELT pipelines, data ingestion, transformation, and integration workflows.
• Strong understanding of data modeling, database architecture, SQL optimization, and data warehousing concepts.
• Good understanding of data mapping, data processing patterns, and building applications for real-time and batch analytics
• Experience working with large datasets, data lakes, data warehouses, and formats such as Parquet, Avro, ORC, and JSON.
• Developing, constructing, testing, and maintaining architectures for data lakes, data pipelines, data warehouses, and large-scale data processing systems on Databricks.
• Strong problem-solving skills