Python Developer Web Scraping

• Overview

Cloudhire is seeking a talented and motivated Python Developer specializing in web scraping to join our dynamic team. As a leading AI-based recruitment platform, we leverage cutting-edge technology to connect businesses with top talent. The ideal candidate will possess a strong background in Python programming and a passion for data extraction and manipulation. You will play a crucial role in enhancing our data acquisition capabilities, ensuring we have access to the most relevant information to drive our recruitment solutions.

Job Description

We are looking for a Python Developer with strong Web Scraping experience to build and maintain scalable scraping systems for collecting job listings and other data from websites, job boards, career pages, and online platforms.

• Responsibilities
• Build and maintain web scrapers and crawlers using Python.
• Scrape data from multiple websites and handle different website structures.
• Work with both static and JavaScript-based websites.
• Use tools such as Scrapy, Playwright, Selenium, BeautifulSoup, and Requests.
• Handle pagination, dynamic content, APIs, sessions, cookies, retries, and rate limiting.
• Build concurrent and scalable scraping workflows.
• Clean, normalize, validate, and deduplicate scraped data.
• Store and process large volumes of scraped data.
• Monitor scrapers and troubleshoot failures when websites change.
• Build supporting Python scripts, APIs, and automation as required.

Requirements

Strong hands-on experience with Python.

2+ years of experience in web scraping/crawling.

Strong experience with Scrapy and/or Playwright/Selenium.

Good understanding of HTTP, REST APIs, HTML, CSS, and JavaScript-rendered websites.

Experience with PostgreSQL/MySQL and Redis.

Understanding of concurrency, asynchronous programming, retries, and rate limiting.

Experience building production-grade scraping systems rather than basic one-off scripts.

Good debugging and problem-solving skills.

Good to Have

Experience with large-scale scraping systems.

Experience with AWS, Docker, Celery/RabbitMQ/SQS.

Data pipeline/ETL experience.

Experience scraping job boards, ATS platforms, and company career websites.

Experience with Elasticsearch/OpenSearch.

Location - Hyderabad (Remote)

Budget - 7 LPA