09
Oct

Scrapy Web Crawler Tuning

Posted By admin
web scraping

Project Title: Scrapy Web Crawler Tuning

Project Description:
I have an existing scrapy script that crawls redfin.com to extract all listings.

It takes a URL and starts crawling: https://www.redfin.com/city/12914/CA/Napa/filter/property-type=house+condo+townhouse+multifamily,max-sqft=2.5k-sqft,max-days-on-market=1wk,include=forsale+mlsfsbo+construction,status=active,viewport=38.42491:38.1441:-122.03737:-122.54274,no-outline

Recently, redfin started bot detection and blocked the crawl around 70 hits or so.

You need to know scrapy enough to fix that.

You can break down the geographical search area to reduce the number of hits per crawl.

Also, I would like to crawl listing of all the counties in California

Marin
Napa
Solano
Sonoma
Contra Costa
San Francisco
Alameda
San Mateo
Santa Clara
Yolo Sacramento
El Dorado
Los Angeles
Orange
San Bernardino
Riverside
San Diego

You can break down the counties into cities to keep the hits below threshold.

If some cities have too many lists (like Los Angeles), you’ll need to split the cities into zip codes.

I would like to be able to crawl the above areas at least twice weekly.

For similar work requirements feel free to email us on info@webscrapingexpert.com.

Comments
  • 2 weeks ago Maxted Lenz

    Scrape all Redfin.com listings for FOR-SALE and SOLD LAND listings in every county for the state of Georgia.

    Reply
  • 2 weeks ago Christopher S.

    We want you to scrape the property daily from this website realestate.com.au and return all the data to us in the excel sheet.

    Reply
  • 2 weeks ago Lydia Hartlaub

    Scrape Zillow.com for current sale listings and past rental data.

    Looking to pull data for all active sales in 3 major cities.

    Reply
  • 2 weeks ago Ethan M.

    The deliverable is scraping a list of URLs that I will provide and extracting the Company name, Owner name, Owner email and presenting the data into an excel sheet.

    Website link: remax.com

    Reply
  • 2 weeks ago Grace Loe

    We want to scrape all the property details listed on homes.com. Also, we require agents information. So, please provide your scraping quote accordingly.

    Reply
  • 2 weeks ago Brett M.

    Could you scrape property details for all cities of Australia from a website allhomes.com.au?

    Reply
  • 2 weeks ago Ashley E.

    What is the cost for the extraction of properties daily from the Remax.com? Thank you.

    Reply
  • 2 weeks ago Mary E.

    Can you please advise the price to scrape new property listings listed for sale daily on Zoopla.co.uk for 500 Zip-codes?

    Reply
  • 2 weeks ago Brady Brown

    Website: Remax.com

    How much does it cost to scrape property listings daily from above mentioned site?

    Reply
  • 2 weeks ago Shannon M.

    How much to scrape database for homes.com ? We require property as well as agents information. So provide us scraping quote accordingly.

    Reply

Add a comment