Data Engineer

Role Overview

We are looking for a Data Engineer to design, build, and maintain scalable data systems and infrastructure. In this role, you will collaborate with cross-functional teams to gather requirements, develop data pipelines, and implement best practices for data governance, security, and analytics. This position offers the opportunity to shape a core part of our data environment and directly influence how we leverage data for business insights and innovation.

Key Responsibilities

• Architect and Implement Data Solutions
• Design and build scalable data platforms using [AWS, Azure, or GCP] or on-premises technologies.
• Develop best practices for data storage, ingestion, and processing (batch and streaming).
• Data Pipeline Development
• Create and manage robust ETL/ELT workflows handling various data types (structured, semi-structured, unstructured).
• Optimize data pipelines for reliability, scalability, and performance.
• Data Governance and Security
• Define and enforce governance policies including data retention, cataloging, and lineage tracking.
• Ensure compliance with relevant data privacy regulations (GDPR, CCPA, HIPAA, etc.) through appropriate security controls and encryption.
• Metadata Management and Cataloging
• Implement and maintain data catalog solutions (e.g., AWS Glue, Apache Atlas, Collibra, or Alation).
• Automate the detection of new data sets, schema changes, and lineage updates.
• Data Quality and Monitoring
• Establish automated checks and alerts for data quality, completeness, and consistency.
• Troubleshoot and resolve data-related issues, providing root cause analysis and long-term fixes.
• Collaboration and Mentorship
• Work closely with cross-functional stakeholders to translate requirements into scalable technical solutions.
  
  

Qualifications

• Education & Experience
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field; or equivalent experience.
• 4+ years of hands-on experience in data engineering, data infrastructure, or related fields.
• Technical Skills
• Proven experience with cloud platforms (AWS, Azure, or GCP) and object storage (S3, ADLS, GCS).
• Proficiency in distributed data processing tools like Apache Spark or Databricks.
• Strong SQL skills and experience with one or more programming languages (Python, Scala, Java).
• Familiarity with real-time data streaming solutions (Kafka, Kinesis, etc.).
• Knowledge of data modeling and modern data architecture patterns.
• Data Governance & Security
• Track record of implementing data catalog and lineage solutions.
• Experience with data quality frameworks (e.g., Great Expectations, Deequ) is a plus.
• Understanding of RBAC, ABAC, encryption, and other security best practices.
• Soft Skills
• Excellent communication and stakeholder management skills.
• Ability to explain complex technical concepts to both technical and non-technical audiences.
• Experience in agile methodologies and cross-functional collaboration.
  
  

Nice-to-Have

• Certifications in AWS, Azure, or GCP (e.g., AWS Certified Data Analytics – Specialty, Azure Data Engineer).
• Experience with infrastructure-as-code (Terraform, CloudFormation).
• Familiarity with orchestration tools like Airflow, Luigi, or Dagster.
• Experience with containerization and CI/CD for data engineering pipelines.