About the Role
DataWeave is looking for a Senior Data Engineer to join our core engineering team. You will be responsible for building and scaling a high-performance big data stack that delivers actionable competitive intelligence to some of the world's largest retailers and brands.
We operate with a flat hierarchy, where we value technical ownership, curiosity, and the ability to solve complex data challenges in a fast-paced environment.
What You'll Do (Job Functions)
- Architecture & Design: Design and develop robust, scalable data engineering solutions for both structured and unstructured data.
- Data Pipeline Engineering: Build and maintain high-performance data pipelines and big data stacks to aggregate and process billions of data points.
- Web Scraping & Crawling: Reverse-engineer complex web structures and write scalable Python-based crawler bots to collect public data at scale.
- API Development: Build and optimize robust APIs that power our delivery points, including dashboards, visualizations, and external integrations.
- System Optimization: Troubleshoot and debug complex data completeness and quality issues while continuously scaling overall system performance.
- Technical Leadership: Collaborate across teams to align data delivery with business insights and customer reporting requirements.
What We're Looking For
- Experience: 59 years of experience in Data Engineering or Backend Development.
- Core Skills: Expert-level proficiency in Python is mandatory.
- Data Stores: Hands-on experience with MySQL, Solr, ElasticSearch, and Cassandra.
- Big Data: Proven track record of building large-scale distributed systems and RDBMS.
- Web Mining: Experience in data mining, web crawling, and handling unstructured data.
- Mindset: A hacker mentalityyou enjoy reverse-engineering problems and building elegant, scalable solutions.
Why DataWeave
- Work on a platform that tracks billions of data points across the global web.
- Be part of a culture that encourages learning, experimenting, and taking total ownership of your work.
- Opportunity to work with a cutting-edge big data stack in a high-growth environment.
