Cost for High Volume Website Scraping
8 Comments
Project Title: Cost for High Volume Website Scraping
Project Description:
I am looking for a scraper that can crawl the top 25,000 or so U.S. websites (based on monthly users) which have “/ads.txt” and the end of the domain, and then compile which of those domains include specific ads.txt codes.
EXAMPLE:
Domain: https://www.dmv.org/ads.txt (note that it ends in /ads.txt)
That url (which ends in /ads.txt) contains the following codes (among many others):
indexexchange.com, 186753, DIRECT, 50b1c356f2c5c8fc
appnexus.com, 9163, DIRECT
adtech.com, 10075, DIRECT
In this example, I need a crawler/scraper than can search all of the other 25k domains in the US ending in /ads.txt to find out which domains have any of those 3 codes.
I need the results in a simple excel doc.
I’m only interested in domains that end in /ads.txt
Can your product do this? Is there a cost?
For similar work requirements feel free to email us on info@webscrapingexpert.com.
Will be able to scrape 10 different eCommerce websites?
Need your help to extract data from walmart.com. The category I am looking for is: Toys & Video Games.
We are having list of about 50 websites; whose store location data you will need to scrape. Can you assist us with this? Please guide.
need someone to extract all the products listed on the website ProductHunt https://www.producthunt.com/.
We need help scraping data for ALL food products on https://www.kroger.com.
I need to scrape the website https://www.sba.gov/paycheckprotection/find.
Would need to scrap data from a few different websites, sample data format is here:
https://www.herold.at/gelbe-seiten/ski–u-snowboardschulen/
Hi,
I need an amazon review scraper. I can enter a URL and it collects all the reviews from that product.