Cost for High Volume Website Scraping
Project Title: Cost for High Volume Website Scraping
I am looking for a scraper that can crawl the top 25,000 or so U.S. websites (based on monthly users) which have “/ads.txt” and the end of the domain, and then compile which of those domains include specific ads.txt codes.
Domain: https://www.dmv.org/ads.txt (note that it ends in /ads.txt)
That url (which ends in /ads.txt) contains the following codes (among many others):
indexexchange.com, 186753, DIRECT, 50b1c356f2c5c8fc
appnexus.com, 9163, DIRECT
adtech.com, 10075, DIRECT
In this example, I need a crawler/scraper than can search all of the other 25k domains in the US ending in /ads.txt to find out which domains have any of those 3 codes.
I need the results in a simple excel doc.
I’m only interested in domains that end in /ads.txt
Can your product do this? Is there a cost?
For similar work requirements feel free to email us on email@example.com.