We are looking for a skilledLeadData Engineerto join our growing team. You will play a pivotal role in designing,, building, and maintaining our data infrastructure and pipelines. Working closely with data analysts, data scientists, and various business stakeholders, you ll ensure data is reliable, efficient, and accessible in a scalable manner.
Key Responsibilities:
- Data Pipeline Orchestration
- Design, build, and maintain end-to-end data pipelines usingAirflow(including managed services likeAmazon MWAA) to orchestrate, schedule, and monitor batch/streaming workflows.
- Implement DAGs (Directed Acyclic Graphs) with retry logic, error handling, and alerting to ensure data quality and pipeline reliability.
- Data Ingestion & Transformation
- Integrate data from various sources usingAirbytefor ingestion anddbtfor transformations in a scalable and modular fashion.
- Collaborate with Data Analysts and Data Scientists to implement transformations and business logic, ensuring data is analytics-ready.
- Data Modeling & Warehousing
- Design and implement efficientdata modelsfor both structured and semi-structured data in AWS S3 (data lake) andSnowflake(data warehouse).
- Ensure data schemas and transformations support advanced analytics, BI reporting, and machine learning use cases.
- Data Governance & Security
- UtilizeAWS Lake Formation APIsand best practices to maintain data security, access controls, and compliance.
- Work closely with IT security to establish robust encryption standards, audit trails, and identity/role-based access.
- Performance Optimization
- OptimizeAWS Athenaqueries and configurations (e.g., data partitioning) for performance and cost efficiency.
- Monitor and tune Airflow DAGs, Snowflake queries, and data transformations to improve throughput and reliability.
- Collaboration & Stakeholder Management
- Partner with cross-functional teams, including DevOps, Platform Engineering, and Data Science, to ensure seamless integration of data workflows and systems.
- Communicate technical solutions effectively to non-technical stakeholders and leadership, translating requirements into actionable tasks.
- Continuous Improvement
- Participate in architecture reviews, code reviews, and troubleshooting sessions to ensure quality and alignment with best practices.
- Remain current with emerging trends in data engineering, orchestration tools (Airflow, MWAA), and cloud services (AWS, Snowflake).
Qualifications & Skills:
- Education & Experience
- Bachelor s or Master s degree in Computer Science, Engineering, or a related field.
- 10+ yearsof experience as a Data Engineer or in a similar role working with cloud-based data platform
Technical Skills:
Cloud & Orchestration
- Airflow(self-managed or managed services likeAmazon MWAA) for workflow orchestration, DAG development, and scheduling.
- Familiarity with best practices for Airflow DAG structure, dependency management, and error handling.
AWS Expertise
- Hands-on experience withAWS Lake Formation, S3, Athena, and related services (e.g., Lambda, Glue, IAM).
Snowflake
- Proficient in setting up data warehouses, configuring security, and optimizing queries on Snowflake.
Data Ingestion & Transformation
- Experience withAirbyteor similar tools for data ingestion.
- dbtor other SQL-based transformation frameworks for modular data processing.
Programming
- Proficiency inPythonand/orJava/Scalafor building data pipelines and custom integrations.
Query Languages
- Advanced knowledge ofSQLfor data manipulation and analysis.
- Soft Skills
- Strong problem-solving and analytical abilities.
- Excellent communication and collaboration skills, able to effectively work in cross-functional teams.
- Ability to operate in a fast-paced, agile environment and manage multiple priorities simultaneously
Preferred Certifications & Experience
- AWS certifications (e.g., AWS Certified Data Analytics Specialty, AWS Certified Solutions Architect) are a plus.
- Experience with CI/CD pipelines, containerization (Docker, Kubernetes), and infrastructure as code (Terraform, CloudFormation) is beneficial.
- Familiarity with DevOps best practices for managing Airflow environments (e.g., version control for DAGs, automated testing, monitoring).
- While familiarity with financial services data especially private equity and alternative investments is not necessary, it will be highly impactful for this role.
