Scraping USA Doctors Database

Project Title: Scraping USA Doctors Database

Project Description:
We are looking to scrape demographics data on several tens of thousands of physician profiles various doctor directory and verify with any other doctor’s directory. Directories have information on physicians embedded in their HTML code. Every physician has a publicly available unique “social security number” called an NPI number. We will provide the NPI numbers that will be the input to the search tool.

The tool locks after 20 or so searches with “429 Bad request….”

Ideally the tool would be web-based and there deliverable would be an excel file with the physician data.

Once logged in to director, go to URL for searching page.

We would like to provide a few second delay between each profile request. The ideal tool would auto login to the web site, enter the npi number, scrape the code, and export it into excel. Sample NPI number: 1013087600

“Inspect element” or however you evaluate the source code. Click on the link in the name “Patrick Muffley” with hyperlink to get demographics including email and NPI.

{“_id”:”529407″,”_score”:1.1360599,”sort”:null,”id”:529407,”full_name_with_credentials”:”Patrick Muffley, DO”,”first_name”:”Patrick”,”last_name”:”Muffley”,”credentials”:”DO”,”public_profile_photo_url”:”image link”,”public_url”:”source link

Desired variables to scrape:
– ID number
– NPI code
– URL
– First name
– Middle name
– Last name
– Name Suffix (Jr.)
– Full name with credentials
– Credentials (e.g. MD)
– Additional credentials
– Professional title (e.g. Assistant professor)
– Tenure
– Age
– Year of Birth
– Specialty name
– Subspecialty
– Street Address
– City
– State
– Zip code
– Country
– Phone number
– Fax number
– Email 1
– Email 2
– Email 3
– Email x
– Last year of fellowship
– Last year of residency
– Training type (e.g. residency)
– Training institution
– Training program
– Training start year
– Training end year
– Training state

For similar work requirement feel free to email us on info@webscrapingexpert.com.