Cost for High Volume Website Scraping

Posted By admin
Big Data text concept on blue background with world map and social icons.

Project Title: Cost for High Volume Website Scraping

Project Description:
I am looking for a scraper that can crawl the top 25,000 or so U.S. websites (based on monthly users) which have “/ads.txt” and the end of the domain, and then compile which of those domains include specific ads.txt codes.

Domain: https://www.dmv.org/ads.txt (note that it ends in /ads.txt)

That url (which ends in /ads.txt) contains the following codes (among many others):

indexexchange.com, 186753, DIRECT, 50b1c356f2c5c8fc
appnexus.com, 9163, DIRECT
adtech.com, 10075, DIRECT

In this example, I need a crawler/scraper than can search all of the other 25k domains in the US ending in /ads.txt to find out which domains have any of those 3 codes.

I need the results in a simple excel doc.

I’m only interested in domains that end in /ads.txt

Can your product do this? Is there a cost?

For similar work requirements feel free to email us on info@webscrapingexpert.com.

  • 2 months ago Washington Mashingaidze

    Will be able to scrape 10 different eCommerce websites?

  • 3 weeks ago Micael Oliveira

    Need your help to extract data from walmart.com. The category I am looking for is: Toys & Video Games.


Add a comment