AI Scraping & LLM Engineering Manager – Python
Experience: 9 to 15 Years
Location: Remote (Work from Home) / Bangalore / India
Mode of Engagement: Full-time
No of Positions: 2
Educational Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / IT
Industry: AI / Data Engineering / Enterprise Automation / SaaS / LegalTech
Notice Period: Immediate / 15 Days Preferred
What We Are Looking For:
- 9–15 years of experience in building large-scale web scraping, browser automation, and data extraction systems using Python.
- Proven expertise in Scrapy, Selenium, Playwright, Puppeteer, and modern browser automation frameworks for enterprise-grade scraping projects.
- Strong experience in building AI-powered scraping workflows using LLMs for intelligent extraction, data normalization, classification, and automation.
- Experience in designing scalable distributed scraping pipelines with proxy rotation, anti-bot handling, CAPTCHA solving, and high-concurrency crawling systems.
- Proven ability to lead engineering teams, manage delivery timelines, conduct code reviews, and drive architecture decisions for AI/data platforms.
Responsibilities:
- Lead the architecture and development of AI-driven scraping and browser automation platforms for structured and unstructured web data extraction.
- Build intelligent scraping pipelines leveraging LLMs, LangChain, LlamaIndex, OpenAI, Claude, or open-source AI models for automated extraction and enrichment.
- Design and manage scalable scraping infrastructure using Scrapy, Selenium, Playwright, Docker, Kubernetes, queues, and cloud-native deployment systems.
- Collaborate with AI, product, and engineering teams to create next-generation intelligent data acquisition and automation solutions.
- Mentor and manage scraping/data engineering teams while establishing coding standards, best practices, monitoring, and scalable engineering processes.
- Optimize system performance, reliability, scalability, and deployment workflows using AWS/GCP, CI/CD pipelines, databases, and distributed processing systems.
Qualifications:
- Bachelor's or Master's degree in Computer Science, Engineering, IT, or related field.
- Strong hands-on expertise in Python, Scrapy, Selenium, Playwright, FastAPI/Django, APIs, and scalable scraping architectures.
- Experience working with LLMs, AI automation frameworks, vector databases, semantic search, embeddings, and intelligent extraction systems.
- Proficiency in Docker, Kubernetes, AWS/GCP, PostgreSQL/MySQL/MongoDB, Redis, CI/CD, and cloud infrastructure management.
- Prior experience leading engineering teams and delivering enterprise-scale scraping, AI automation, or data intelligence products.
- Excellent analytical, communication, leadership, and problem-solving skills with experience working in Agile development environments.
