save_htmls: Download multiple results pages as html files

In nealhaddaway/GSscraper: Scrapes Citation Information From Google Scholar Results

Download multiple results pages as html files

Description

Downloads one or more Google Scholar URLs as html files with a specific wait-time to avoid IP address blocking.

Usage

save_htmls(urls, pause = 0.5, backoff = FALSE)

Arguments

urls	One or more URLs corresponding to pages of Google Scholar search results.
pause	Integer specifying the number of seconds to wait between download attempts. The default value is 4 seconds.
backoff	A logical argument (TRUE or FALSE) specifying whether responsive backing-off should be used. If set to TRUE, the time between calls is varied depending on how long the server takes to respond to the original request. The responsive back-off time is set to multiple the response time by the 'pause' time: i.e. if the system takes 1.02 seconds to respond and 'pause' time is set to 4 seconds, a 4.10 second delay will be employed before the next call. The default for back-off is 'FALSE'.

Value

HTML files are downloaded as a string object Pause and success messages are printed to the console.

Examples

## Not run: 
urls <- c('https://www.google.com/search?q=site:www.sei.org+climate&start=0',
          'https://www.google.com/search?q=site:www.sei.org+climate&start=10')
htmls <- save_htmls(urls, pause = 5)

## End(Not run)

nealhaddaway/GSscraper documentation built on May 6, 2022, 10:52 a.m.