Data Pipelines Engineering Manager – Python
Experience: 7 to 12 Years
Location: Remote (Work from Home) / Bangalore / India
Mode of Engagement: Full-time
No of Positions: 2
Educational Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry: IT / Data / AI / LegalTech / Enterprise Solutions
Notice Period: Immediate
What We Are Looking For:
- 7–12 years of experience in managing end-to-end data systems, including data collection, API-driven data delivery, and AI model integrations.
- Proven experience using large language models to automate and scale web data extraction, including building custom AI scraping pipelines for structured data collection at scale
- Proven expertise in Python-based data engineering, data scraping, and pipeline orchestration for large-scale analytics and automation projects.
- Strong background in building scalable APIs, microservices, and data pipelines using Python, Docker, and Kubernetes.
- Experience in customizing open-source language models (LLMs) and integrating them into enterprise solutions.
- Proficiency in AWS cloud services, PostgreSQL/MySQL, CI/CD, and DevOps tools for continuous delivery and monitoring.
- Ability to lead distributed engineering teams, manage delivery timelines, and drive innovation through automation and AI-driven workflows.
Responsibilities:
- Lead and mentor engineering teams responsible for designing, implementing, and maintaining data ingestion, transformation, and delivery pipelines.
- Architect data systems combine scraping, big data processing, and AI-based enrichment to support analytics and product intelligence.
- Oversee API architecture and system scalability, ensuring high performance and secure data flow across services.
- Manage public data scraping and data integration projects that involve large, unstructured datasets and complex relational mappings.
- Drive AI model customization using open-source frameworks (LangChain, LlamaIndex, or custom LLM integrations) to improve automation and search intelligence.
- Collaborate with product managers, AI teams, and data analysts to translate requirements into actionable data engineering tasks.
- Implement and monitor data quality, governance, and compliance standards throughout the data lifecycle.
- Optimize deployment processes using Docker, Kubernetes, AWS Lambda, and CI/CD pipelines.
- Conduct regular technical reviews, mentor developers, and establish best practices for Python, cloud deployment, and automation.
Qualifications:
- Bachelor's or master's degree in computer science, Engineering, or related field.
- Hands-on expertise in Python, Django, FastAPI, SQL, and API-driven system design.
- Strong understanding of data pipelines, scraping frameworks (BeautifulSoup, Scrapy), and distributed data systems.
- Experience with AI/LLM integration, automation frameworks, and data-driven architecture design.
- Proficiency in AWS cloud ecosystem, Docker, Kubernetes, and Git-based CI/CD workflows.
- Demonstrated leadership in managing multi-member engineering teams and delivering enterprise-grade products.
- Excellent analytical, communication, and problem-solving abilities.
- Familiarity with Agile methodologies, Jira/ClickUp, and project management best practices.
