Python Developer (Web Crawling Exp) - 5+years - Kochi
Responsibilities:
- Develop and maintain efficient web crawlers with a focus on speed, performance, and scalability.
- Write clean, reusable, and optimized Python code for large-scale data extraction.
- Analyze and understand website structures, implement logic to handle dynamic and JS-rendered pages.
- Troubleshoot and debug crawl issues, optimize request handling, and improve crawler reliability.
- Collaborate with backend and data teams to ensure smooth data flow and integration.
- Monitor crawler performance, logs, and resolve runtime errors.
- Participate in code reviews, documentation, and continuous process improvements.
Skills Required:
- Strong coding skills in Python.
- Good understanding of how web pages, HTTP, and crawlers work.
- Hands-on experience with Docker, Postman, and PowerShell.
- Basic understanding of HTML, AngularJS, RabbitMQ, and Celery.
- Experience with Git and Bitbucket for version control and team collaboration.
- Familiarity with XPath, CSS selectors, and regular expressions for content extraction.
Preferred:
- Exposure to Scrapy, Puppeteer/Playwright, or other crawling frameworks.
- Understanding of Elasticsearch, HAProxy, or Curator for large-scale data handling.
