Job Descriptions:
Job Title : Data Engineer / Senior Data Engineer
Job Type : Contract To Hire(3 Months)
Yrs of Experience : 3 to 5 yrs / 6 to 12 yrs
Notice Period : Immediate - 15 Days
Primary Skill Set : Data Engineer, GCP, DBT, Data flow, Composer, Python, SQL, Microservices
What are we required to do
- Design, build, and maintain scalable and robust data pipelines for processing large volumes of data.
- Develop ETL (Extract, Transform, Load) processes to ingest, clean, and transform data from various sources.
- Implement and optimize data storage solutions, including data lakes and warehouses, using GCP services like BigQuery, Cloud Storage, and Dataflow.
- Integrate data from multiple data sources, ensuring data consistency and quality.
- Develop and maintain automated data processing workflows, monitoring, and alerting systems.
- Manage and maintain metadata, data catalogs, and data lineage documentation.
- Work closely with data scientists, analysts, and other stakeholders to understand data requirements and provide appropriate solutions.
- Collaborate with cross-functional teams to ensure data availability and reliability for various use cases.
- Provide technical support and guidance to other team members on data engineering best practices.
- Optimize data processing pipelines for performance, scalability, and cost-efficiency.
- Monitor data pipeline performance, troubleshoot issues, and implement necessary improvements.
- Ensure data security and compliance with relevant regulations and best practices.
What you'll need
- experience in data engineering or related roles.
- Proven experience with Python for data processing, scripting, and automation.
- Hands-on experience with Google Cloud Platform (GCP) services, including BigQuery, Cloud Storage, Dataflow, and Pub/Sub.
- Strong understanding of data modeling, ETL processes, and data warehousing concepts.
- Strong problem-solving skills and attention to detail.
- Excellent communication skills, with the ability to explain complex technical concepts to non-technical stakeholders.
- Ability to work independently and as part of a team in a fast-paced environment.
