Lead Data Engineer

At INDI, we're passionate about empowering individuals and businesses worldwide. Our cutting-edge recruiters connect leading companies with top talent, fostering a dynamic environment where innovation thrives. Join us in shaping the future of work.

Overview of the role:

The Lead Data Engineer position focuses on serving as core contributors to internal Data Platforms, responsible for building scalable, resilient pipelines that support Business Operations, Analytics, and cutting-edge AI/ML initiatives. This role involves designing, implementing, and optimizing data pipelines using a wide range of Big Data and cloud-native technologies while providing technical leadership, guiding personnel, and ensuring architectural consistency across projects.

Key responsibilities:

- Providing technical leadership and mentorship to other data engineers, helping them develop and succeed in their roles.

- Leading design reviews, enforcing coding standards, and guiding architectural decisions to ensure platform scalability and maintainability.

- Partnering with senior stakeholders and engineering leadership to prioritize and plan technical work that aligns with strategic business goals.

- Acting as technical points of contact for cross-team collaborations, participating in project planning, scoping, and retrospectives.

- Demonstrating curiosity, initiative, and capability to integrate AI-powered tools into day-to-day engineering workflows.

- Supporting AI/GenAI projects by building pipelines that feed LLM apps and recommendation systems.

- Driving adoption of data engineering best practices across organizations, including testing, CI/CD, and observability.

- Designing, building, and maintaining batch and streaming data pipelines using native data platforms.

- Ingesting data from various sources, including REST APIs, cloud services, and enterprise SaaS platforms.

- Implementing robust transformation logic using Python, PySpark, SQL, and Java across structured and semi-structured datasets.

- Ensuring data integrity, lineage, and performance across ingestion, transformation, and delivery layers.

Requirements:

- Data Engineering Experience: Substantial background (6+ years) as Data Engineer in enterprise-scale data infrastructures.

- Leadership Skills: Proven leadership in technical project management and personnel mentorship.

- Architectural Guidance: Track record of guiding architectural conversations in high-throughput data ecosystems.

- Programming Mastery: Advanced proficiency in SQL and Python programming, with expertise in PySpark or Java.

- Data Architecture: Comprehensive knowledge of data modeling, transformation methodologies, and data warehouse architectures.

- Hadoop Ecosystem: Proven expertise with Hadoop ecosystem technologies.

- Pipeline Development: Successful implementation of resilient, high-performance data pipelines meeting strict service level agreements.

- Integration Expertise: Deep understanding of API integration, performance optimization, and data schema adaptation.

- Language Proficiency: Advanced level of English.

Additional skills preferred:

- Workflow Tools: Practical implementation experience with Apache NiFi and Apache Airflow.

- Development Environment: Expertise in containerized environments, developer tooling, and API frameworks.

- AI/ML Platforms: Demonstrated capabilities in AI/ML platform support and generative AI pipeline development.

- Innovation Contributions: Proven contributions to internal technological innovations within data engineering workflows.

What to expect from us:

- Competitive Compensation: Excellent payment in USD or your preferred local currency.

- Paid Leave: Parental leave, vacation, and national holidays.

- Dynamic Work Culture: Innovative and multicultural environment.

- Elite Collaboration: Work with the global top 1% of talent.

- Growth and Support: Mentorship and career development opportunities.

If you are interested in being part of a team composed of the best professionals and working 100% goal-oriented in an innovative environment, do not hesitate to apply!