Cost for High Volume Website Scraping

Posted By admin

Project Title: Cost for High Volume Website Scraping

Project Description:
I am looking for a scraper that can crawl the top 25,000 or so U.S. websites (based on monthly users) which have “/ads.txt” and the end of the domain, and then compile which of those domains include specific ads.txt codes.

Domain: https://www.dmv.org/ads.txt (note that it ends in /ads.txt)

That url (which ends in /ads.txt) contains the following codes (among many others):

indexexchange.com, 186753, DIRECT, 50b1c356f2c5c8fc
appnexus.com, 9163, DIRECT
adtech.com, 10075, DIRECT

In this example, I need a crawler/scraper than can search all of the other 25k domains in the US ending in /ads.txt to find out which domains have any of those 3 codes.

I need the results in a simple excel doc.

I’m only interested in domains that end in /ads.txt

Can your product do this? Is there a cost?

For similar work requirements feel free to email us on info@webscrapingexpert.com.

  • 3 years ago Washington Mashingaidze

    Will be able to scrape 10 different eCommerce websites?

  • 3 years ago Micael Oliveira

    Need your help to extract data from walmart.com. The category I am looking for is: Toys & Video Games.

  • 3 years ago Sean Fallon

    We are having list of about 50 websites; whose store location data you will need to scrape. Can you assist us with this? Please guide.

  • 2 years ago Michael Brown

    need someone to extract all the products listed on the website ProductHunt https://www.producthunt.com/.

  • 2 years ago Zach Reiss

    We need help scraping data for ALL food products on https://www.kroger.com.

  • 2 years ago Brian Dell

    I need to scrape the website https://www.sba.gov/paycheckprotection/find.

  • 2 years ago Miguel Cabrer

    Would need to scrap data from a few different websites, sample data format is here:


  • 2 years ago Kathy MacKinnon


    I need an amazon review scraper. I can enter a URL and it collects all the reviews from that product.

  • 2 years ago Scott Booker

    Extract up to 20,000 data regarding Security agency companies (events securities within Yelp website) information within the USA.


Add a comment