README.md
In cpsievert/rdom: Access the DOM of a webpage as HTML using phantomjs
rdom
Render and parse the DOM from R via phantomjs.
Installation
rdom depends on phantomjs, so make sure that is installed and visible to your PATH:
stopifnot(Sys.which("phantomjs") != "")
rdom currently isn't on CRAN, but you can install it with devtools
devtools::install_github("cpsievert/rdom")
Introduction
Web scraping packages such as XML, xml2 and rvest allow you to download and parse HTML files, but they lack a browsing engine to fully render the DOM.
To demonstrate the difference, suppose we want to extract the HTML table on this page:
XML::htmlParse() (and rvest::read_html()) returns the HTML page source, which is static, and doesn't contain the <table> element we desire (because JavaScript is modifying the state of the DOM):
XML::htmlParse("http://bl.ocks.org/cpsievert/raw/2a9fb8f504cd56e9e8e3/")
<!DOCTYPE html>
<html><body>
A Simple Table made with JavaScript
<p></p>
<script>
function tableCreate(){
var body = document.body,
tbl = document.createElement('table');
for(var i = 0; i < 3; i++){
var tr = tbl.insertRow();
for(var j = 0; j < 3; j++){
var td = tr.insertCell();
td.appendChild(document.createTextNode("Cell"));
}
}
body.appendChild(tbl);
}
tableCreate();
</script>
</body></html>
The main function in rdom, rdom(), uses phantomjs to render and return the DOM as an HTML string. Instead of passing the entire DOM as a string from phantomjs to R, you can give rdom() a CSS Selector to extract certain element(s).
tbl <- rdom::rdom("http://bl.ocks.org/cpsievert/raw/2a9fb8f504cd56e9e8e3/", css = "table")
tbl
<table>
<tbody>
<tr>
<td>Cell</td>
<td>Cell</td>
<td>Cell</td>
</tr>
<tr>
<td>Cell</td>
<td>Cell</td>
<td>Cell</td>
</tr>
<tr>
<td>Cell</td>
<td>Cell</td>
<td>Cell</td>
</tr>
</tbody>
</table>
In this case, we can use XML::readHTMLTable() (or rvest::html_table()) to convert the <table> node into a data frame.
XML::readHTMLTable(tbl)
V1 V2 V3
1 Cell Cell Cell
2 Cell Cell Cell
3 Cell Cell Cell
Render shiny apps
An interesting use case for rdom is for rendering (and testing) shiny apps.
library(shiny)
runExample("01_hello")
Listening on http://127.0.0.1:4870
Now, in another R session, pass this URL to rdom(). For this example, we'll just return the app's title.
header <- rdom::rdom("http://127.0.0.1:4870", "h2")
<h2>Hello Shiny!</h2>
This way it's easy to test your shiny apps!
library("testthat")
expect_identical(XML::xmlValue(header), "Hello Shiny!")
Using the CLI
rdom() is essentially a wrapper around phantomjs' command line interface. So, if you don't need R for your task, it might be more efficient to use the CLI.
$ git clone https://github.com/cpsievert/rdom.git
$ cd rdom
$ phantomjs inst/rdomjs/rdom.js inst/jsTable/jsTable.html table false 1 table.html
$ vi table.html
There are 4 arguments that the rdom.js script will respect:
url a URL of a web page (required).css a CSS selector.all This controls whether querySelector or querySelectorAll is used to extract elements from the page.timeout maximum time to wait for page to load and render, in seconds.filename Write HTML string to a file.
Everytime you call rdom(), it has to initiate a phantomjs process. Also, phantomjs has to open and fully render the site that you provide. If you need to render multiple sites,
Acknowledgements
Thanks to Winston Chang for webshot which inspired the design of this package.
cpsievert/rdom documentation built on July 31, 2019, 8:25 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
rdom
Render and parse the DOM from R via phantomjs.
Installation
rdom depends on phantomjs, so make sure that is installed and visible to your PATH:
stopifnot(Sys.which("phantomjs") != "")
rdom currently isn't on CRAN, but you can install it with devtools
devtools::install_github("cpsievert/rdom")
Introduction
Web scraping packages such as XML, xml2 and rvest allow you to download and parse HTML files, but they lack a browsing engine to fully render the DOM.
To demonstrate the difference, suppose we want to extract the HTML table on this page:
XML::htmlParse() (and rvest::read_html()) returns the HTML page source, which is static, and doesn't contain the <table> element we desire (because JavaScript is modifying the state of the DOM):
XML::htmlParse("http://bl.ocks.org/cpsievert/raw/2a9fb8f504cd56e9e8e3/")
<!DOCTYPE html>
<html><body>
A Simple Table made with JavaScript
<p></p>
<script>
function tableCreate(){
var body = document.body,
tbl = document.createElement('table');
for(var i = 0; i < 3; i++){
var tr = tbl.insertRow();
for(var j = 0; j < 3; j++){
var td = tr.insertCell();
td.appendChild(document.createTextNode("Cell"));
}
}
body.appendChild(tbl);
}
tableCreate();
</script>
</body></html>
The main function in rdom, rdom(), uses phantomjs to render and return the DOM as an HTML string. Instead of passing the entire DOM as a string from phantomjs to R, you can give rdom() a CSS Selector to extract certain element(s).
tbl <- rdom::rdom("http://bl.ocks.org/cpsievert/raw/2a9fb8f504cd56e9e8e3/", css = "table")
tbl
<table>
<tbody>
<tr>
<td>Cell</td>
<td>Cell</td>
<td>Cell</td>
</tr>
<tr>
<td>Cell</td>
<td>Cell</td>
<td>Cell</td>
</tr>
<tr>
<td>Cell</td>
<td>Cell</td>
<td>Cell</td>
</tr>
</tbody>
</table>
In this case, we can use XML::readHTMLTable() (or rvest::html_table()) to convert the <table> node into a data frame.
XML::readHTMLTable(tbl)
V1 V2 V3
1 Cell Cell Cell
2 Cell Cell Cell
3 Cell Cell Cell
Render shiny apps
An interesting use case for rdom is for rendering (and testing) shiny apps.
library(shiny)
runExample("01_hello")
Listening on http://127.0.0.1:4870
Now, in another R session, pass this URL to rdom(). For this example, we'll just return the app's title.
header <- rdom::rdom("http://127.0.0.1:4870", "h2")
<h2>Hello Shiny!</h2>
This way it's easy to test your shiny apps!
library("testthat")
expect_identical(XML::xmlValue(header), "Hello Shiny!")
Using the CLI
rdom() is essentially a wrapper around phantomjs' command line interface. So, if you don't need R for your task, it might be more efficient to use the CLI.
$ git clone https://github.com/cpsievert/rdom.git
$ cd rdom
$ phantomjs inst/rdomjs/rdom.js inst/jsTable/jsTable.html table false 1 table.html
$ vi table.html
There are 4 arguments that the rdom.js script will respect:
urla URL of a web page (required).cssa CSS selector.allThis controls whether querySelector or querySelectorAll is used to extract elements from the page.timeoutmaximum time to wait for page to load and render, in seconds.filenameWrite HTML string to a file.
Everytime you call rdom(), it has to initiate a phantomjs process. Also, phantomjs has to open and fully render the site that you provide. If you need to render multiple sites,
Acknowledgements
Thanks to Winston Chang for webshot which inspired the design of this package.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.