09
Oct

Scrapy Web Crawler Tuning

Posted By admin
web scraping

Project Title: Scrapy Web Crawler Tuning

Project Description:
I have an existing scrapy script that crawls redfin.com to extract all listings.

It takes a URL and starts crawling: https://www.redfin.com/city/12914/CA/Napa/filter/property-type=house+condo+townhouse+multifamily,max-sqft=2.5k-sqft,max-days-on-market=1wk,include=forsale+mlsfsbo+construction,status=active,viewport=38.42491:38.1441:-122.03737:-122.54274,no-outline

Recently, redfin started bot detection and blocked the crawl around 70 hits or so.

You need to know scrapy enough to fix that.

You can break down the geographical search area to reduce the number of hits per crawl.

Also, I would like to crawl listing of all the counties in California

Marin
Napa
Solano
Sonoma
Contra Costa
San Francisco
Alameda
San Mateo
Santa Clara
Yolo Sacramento
El Dorado
Los Angeles
Orange
San Bernardino
Riverside
San Diego

You can break down the counties into cities to keep the hits below threshold.

If some cities have too many lists (like Los Angeles), you’ll need to split the cities into zip codes.

I would like to be able to crawl the above areas at least twice weekly.

For similar work requirements feel free to email us on info@webscrapingexpert.com.

Comments
  • 2 years ago Maxted Lenz

    Scrape all Redfin.com listings for FOR-SALE and SOLD LAND listings in every county for the state of Georgia.

    Reply
  • 2 years ago Christopher S.

    We want you to scrape the property daily from this website realestate.com.au and return all the data to us in the excel sheet.

    Reply
  • 2 years ago Lydia Hartlaub

    Scrape Zillow.com for current sale listings and past rental data.

    Looking to pull data for all active sales in 3 major cities.

    Reply
  • 2 years ago Ethan M.

    The deliverable is scraping a list of URLs that I will provide and extracting the Company name, Owner name, Owner email and presenting the data into an excel sheet.

    Website link: remax.com

    Reply
  • 2 years ago Grace Loe

    We want to scrape all the property details listed on homes.com. Also, we require agents information. So, please provide your scraping quote accordingly.

    Reply
  • 2 years ago Brett M.

    Could you scrape property details for all cities of Australia from a website allhomes.com.au?

    Reply
  • 2 years ago Ashley E.

    What is the cost for the extraction of properties daily from the Remax.com? Thank you.

    Reply
  • 2 years ago Mary E.

    Can you please advise the price to scrape new property listings listed for sale daily on Zoopla.co.uk for 500 Zip-codes?

    Reply
  • 2 years ago Brady Brown

    Website: Remax.com

    How much does it cost to scrape property listings daily from above mentioned site?

    Reply
  • 2 years ago Shannon M.

    How much to scrape database for homes.com ? We require property as well as agents information. So provide us scraping quote accordingly.

    Reply

Leave a Reply to Brett M. Cancel reply