Web Scraper

Cloudhire is seeking a talented and motivated Python Developer specializing in web scraping to join our dynamic team. As a leading AI-based recruitment platform, we leverage cutting-edge technology to connect businesses with top talent. The ideal candidate will possess a strong background in Python programming and a passion for data extraction and manipulation. You will play a crucial role in enhancing our data acquisition capabilities, ensuring we have access to the most relevant information to drive our recruitment solutions.

Job Description

• We are looking for a Python Developer with strong Web Scraping experience to build and maintain scalable scraping systems for collecting job listings and other data from websites, job boards, career pages, and online platforms.

Details

Location - Hyderabad

Budget - 7 LPA

Year of experience - 3 years

Availability - Immediate joiners

• Responsibilities
• Build and maintain web scrapers and crawlers using Python.
• Scrape data from multiple websites and handle different website structures.
• Work with both static and JavaScript-based websites.
• Use tools such as Scrapy, Playwright, Selenium, BeautifulSoup, and Requests.
• Handle pagination, dynamic content, APIs, sessions, cookies, retries, and rate limiting.
• Build concurrent and scalable scraping workflows.
• Clean, normalize, validate, and deduplicate scraped data.
• Store and process large volumes of scraped data.
• Monitor scrapers and troubleshoot failures when websites change.
• Build supporting Python scripts, APIs, and automation as required.

Requirements

Strong hands-on experience with Python.

2+ years of experience in web scraping/crawling.

Strong experience with Scrapy and/or Playwright/Selenium.

Good understanding of HTTP, REST APIs, HTML, CSS, and JavaScript-rendered websites.

Experience with PostgreSQL/MySQL and Redis.

Understanding of concurrency, asynchronous programming, retries, and rate limiting.

Experience building production-grade scraping systems rather than basic one-off scripts.

Good debugging and problem-solving skills.

Good to Have

Experience with large-scale scraping systems.

Experience with AWS, Docker, Celery/RabbitMQ/SQS.

Data pipeline/ETL experience.

Experience scraping job boards, ATS platforms, and company career websites.

Experience with Elasticsearch/OpenSearch.