Nothing
Basics
In binman: A Binary Download Manager
The goal of this vignette is to describe the basic functionality of the
binman package.
Introduction
binman (Binary Manager) is an R package that allows the user to manage
the downloading of third party binaries. The downloading of binaries is
split into three parts: the pre-download, the download and the
post-download. Each part of the download is controlled by an appropriate
function: predlfunction, dlfunction and postdlfunction
respectively.
Pre-Download Function
The pre-download function should return up-to its final point a named list
of data.frames. The name of each data.frame corresponds to a "platform".
Each individual data.frame should be composed of three columns namely
"version", "url" and "file". This named list of data.frames can then be
passed to the assign_directory function.
mypredl_fun <- function(...) {
# code to return named list of data.frames
assign_directory(mynamedlist, "myappname")
}
How directories are assigned
Directories are assigned using the rappdirs package. See rappdirs::user_data_dir
for more OS specific information.
Download Function
The download function should take as its primary input the output given by
assign_directory that is a named list of data.frames with column names
"version", "url", "file", "dir" and "exists".
The download function should process the list downloading the "url" to the
given directory "dir" assigning it the file-name "file". Whether the
download is attempted may be dictated on the value of the "exists"
variable (For example an overwrite argument in the download function).
The download function should return a single data.frame with columns
"platform", "file" and "processed". The "platform" column indicates what
platform the file downloaded relates to. The "file" column gives the full
path to the downloaded file. The "processed" column indicates whether the
file was downloaded/processed.
mydl_fun <- function(dllist, ...) {
# dllist is output from a predlfunction with final output from assign_directory
# CODE here to perform download
data.frame(
platform = platform,
file = file,
processed = processed,
stringsAsFactors = FALSE
)
}
Post-Download Function
The post-download function takes as its primary input the output given by
a "download function" that is a data.frame with column names
"platform", "file" and "processed".
The purpose of the post-download function is to process the downloaded
files after the download. This could involve changing the file properties
such as the mode, unzip/untar a downloaded zip/tar file etc.
Application YAML file
The three functions mentioned previously are combined together to define a
process for downloading the binary file. The user stipulates the required
functions and arguments by way of a YAML format file. For example:
name: superduperapp
predlfunction:
"superduperapp::predl_superduper":
appname: "superduperapp"
dlfunction:
"superduperapp::download_superduper": []
postdlfunction:
"superduperapp::postdl_superduper": []
The defined YAML file is then processed by the process_yaml function.
The YAML file needs to define the three required functions with any
arguments and also define a name.
Browser Mob Proxy example
The following is an example of using binman to get the github assets from a project. The project is https://github.com/lightbody/browsermob-proxy/releases . When a new version is released a zipped binary is added as an "asset". A JSON representation of the project releases is available at https://api.github.com/repos/lightbody/browsermob-proxy/releases.
We shall breakdown the process into its three functional parts.
BMP Pre-Download
Firstly we note we have JSON data so we parse the JSON from the URL:
bmpURL <- "https://api.github.com/repos/lightbody/browsermob-proxy/releases"
bmpData <- jsonlite::fromJSON(bmpURL)
This gives us a list of 10 releases. A GitHub release should have a version number.
version <- bmpData[["tag_name"]]
version
[1] "browsermob-proxy-2.1.2" "browsermob-proxy-2.1.1"
[3] "browsermob-proxy-2.1.0" "browsermob-proxy-2.1.0-beta-6"
[5] "browsermob-proxy-2.1.0-beta-5" "browsermob-proxy-2.1.0-beta-4"
[7] "browsermob-proxy-2.1.0-beta-3" "browsermob-proxy-2.1.0-beta-2"
[9] "browsermob-proxy-2.1.0-beta-1" "browsermob-proxy-2.0.0"
This is our version data but we may want to tidy it up so:
versionregex <- c("browsermob-proxy-(.*)$", "\\1")
version <- gsub(versionregex[1], versionregex[2], version)
version
[1] "2.1.2" "2.1.1" "2.1.0" "2.1.0-beta-6" "2.1.0-beta-5"
[6] "2.1.0-beta-4" "2.1.0-beta-3" "2.1.0-beta-2" "2.1.0-beta-1" "2.0.0"
Each release has "assets" associated with it (a zip file).
assets <- bmpData[["assets"]]
length(assets)
[1] 10
The assets will contain the other information we need for our pre-download function.
From each assets item we would like the file-name:
bmpFiles <- vapply(assets, "[[", character(1), "name")
bmpFiles
[1] "browsermob-proxy-2.1.2-bin.zip" "browsermob-proxy-2.1.1-bin.zip"
[3] "browsermob-proxy-2.1.0-bin.zip" "browsermob-proxy-2.1.0-beta-6-bin.zip"
[5] "browsermob-proxy-2.1.0-beta-5-bin.zip" "browsermob-proxy-2.1.0-beta-4-bin.zip"
[7] "browsermob-proxy-2.1.0-beta-3-bin.zip" "browsermob-proxy-2.1.0-beta-2-bin.zip"
[9] "browsermob-proxy-2.1.0-beta-1-bin.zip" "browsermob-proxy-2.0.0-bin.zip"
and the URL for that file:
bmpURLs <- vapply(assets, "[[", character(1), "browser_download_url")
bmpURLs
[1] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.2/browsermob-proxy-2.1.2-bin.zip"
[2] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.1/browsermob-proxy-2.1.1-bin.zip"
[3] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0/browsermob-proxy-2.1.0-bin.zip"
[4] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-6/browsermob-proxy-2.1.0-beta-6-bin.zip"
[5] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-5/browsermob-proxy-2.1.0-beta-5-bin.zip"
[6] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-4/browsermob-proxy-2.1.0-beta-4-bin.zip"
[7] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-3/browsermob-proxy-2.1.0-beta-3-bin.zip"
[8] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-2/browsermob-proxy-2.1.0-beta-2-bin.zip"
[9] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-1/browsermob-proxy-2.1.0-beta-1-bin.zip"
[10] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.0.0/browsermob-proxy-2.0.0-bin.zip"
In the case of BMP we have a single platform which we could denote "generic".
So we could pass the following:
dllist <- list(
"generic" = data.frame(
version = version,
url = bmpURLs,
file = bmpFiles,
stringsAsFactors = FALSE
)
)
Then finally we pass this list to the assign_directory function:
predlOut <- assign_directory(dllist, "bmpApp")
str(predlOut, max.level = 2)
List of 1
$ generic:'data.frame': 10 obs. of 5 variables:
..$ version: chr [1:10] "2.1.2" "2.1.1" "2.1.0" "2.1.0-beta-6" ...
..$ url : chr [1:10] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.2/browsermob-proxy-2.1.2-bin.zip" "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.1/browsermob-proxy-2.1.1-bin.zip" "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0/browsermob-proxy-2.1.0-bin.zip" "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-6/browsermob-proxy-2.1.0-beta-6-bin"| __truncated__ ...
..$ file : chr [1:10] "browsermob-proxy-2.1.2-bin.zip" "browsermob-proxy-2.1.1-bin.zip" "browsermob-proxy-2.1.0-bin.zip" "browsermob-proxy-2.1.0-beta-6-bin.zip" ...
..$ dir :List of 10
..$ exists : logi [1:10] FALSE FALSE FALSE FALSE FALSE FALSE ...
In the binman package there is a function template for github asset type
downloads:
dllist <- predl_github_assets(
url = "https://api.github.com/repos/lightbody/browsermob-proxy/releases",
history = 3L,
platform = "generic",
appname = "bmproxy",
platformregex = "browsermob-proxy",
versionregex = c("browsermob-proxy-(.*)$", "\\1")
)
There are some extra arguments for generality but it basically performs
the operations we outlined above.
BMP Download
The download for BMP is straightforward:
dllist[["generic"]][1,]
version
1 2.1.2
url
1 https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.2/browsermob-proxy-2.1.2-bin.zip
file dir
1 browsermob-proxy-2.1.2-bin.zip /home/john/.local/share/binman_bmproxy/generic/2.1.2
exists
1 FALSE
For each platform (there is only one "generic" platform in this case) we
want to loop over the data.frame downloading the data at "url" to the "file"
in the "dir". If it already "exists" we may not perform the download.
In binman there is a simple template function for this:
dlfiles <- download_files(dllist)
So we simply need to pass the list from the pre download function to this
download function.
BMP Post Download
The files we receive from the Browser mob Proxy project are zipped. As
a post processing operation we would like to unzip these files. The result
of unzipping the files is a directory structure. In this case we do not
change the mode of the ultimate binary which is contained in this
directory structure.
Again binman has a simple template function to unzip/untar a downloaded
file so we pass our output from the download to this:
dlres <- unziptar_dlfiles(dlfiles)
BMP YAML
We can incorporate our three functional calls into a YAML file for the
Browser Mob Proxy application as follows:
name: bmproxy
predlfunction:
"binman::predl_github_assets":
url: "https://api.github.com/repos/lightbody/browsermob-proxy/releases"
platform:
- generic
history: 3
appname: "bmproxy"
platformregex:
- "browsermob-proxy"
versionregex:
- "browsermob-proxy-(.*)$"
- "\\1"
dlfunction:
"binman::download_files": []
postdlfunction:
"binman::unziptar_dlfiles": []
We have saved this file in the binman package:
ymlfile <- system.file("examples", "yaml", "bmproxy.yml", package = "binman")
We can now use the process-yaml function to download the BMP binaries:
process_yaml(ymlfile)
BEGIN: PREDOWNLOAD
BEGIN: DOWNLOAD
Creating directory: /home/john/.local/share/binman_bmproxy/generic/2.1.2
Downloading binary: https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.2/browsermob-proxy-2.1.2-...
Creating directory: /home/john/.local/share/binman_bmproxy/generic/2.1.1
Downloading binary: https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.1/browsermob-proxy-2.1.1-...
Creating directory: /home/john/.local/share/binman_bmproxy/generic/2.1.0
Downloading binary: https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0/browsermob-proxy-2.1.0-...
BEGIN: POSTDOWNLOAD
Looking at the resulting directory structure (this is for Linux the location
of directories will be OS dependent):
tree -d /home/john/.local/share/binman_bmproxy
/home/john/.local/share/binman_bmproxy
└── generic
├── 2.1.0
│ └── browsermob-proxy-2.1.0
│ ├── bin
│ │ └── conf
│ ├── lib
│ └── ssl-support
├── 2.1.1
│ └── browsermob-proxy-2.1.1
│ ├── bin
│ │ └── conf
│ ├── lib
│ └── ssl-support
└── 2.1.2
└── browsermob-proxy-2.1.2
├── bin
│ └── conf
├── lib
└── ssl-support
We see that binman has downloaded and unzipped the binaries to
appropriate directories.
Try the binman package in your browser
Any scripts or data that you put into this service are public.
binman documentation built on Sept. 1, 2022, 9:06 a.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
The goal of this vignette is to describe the basic functionality of the
binman package.
Introduction
binman (Binary Manager) is an R package that allows the user to manage
the downloading of third party binaries. The downloading of binaries is
split into three parts: the pre-download, the download and the
post-download. Each part of the download is controlled by an appropriate
function: predlfunction, dlfunction and postdlfunction
respectively.
Pre-Download Function
The pre-download function should return up-to its final point a named list
of data.frames. The name of each data.frame corresponds to a "platform".
Each individual data.frame should be composed of three columns namely
"version", "url" and "file". This named list of data.frames can then be
passed to the assign_directory function.
mypredl_fun <- function(...) { # code to return named list of data.frames assign_directory(mynamedlist, "myappname") }
How directories are assigned
Directories are assigned using the rappdirs package. See rappdirs::user_data_dir
for more OS specific information.
Download Function
The download function should take as its primary input the output given by
assign_directory that is a named list of data.frames with column names
"version", "url", "file", "dir" and "exists".
The download function should process the list downloading the "url" to the given directory "dir" assigning it the file-name "file". Whether the download is attempted may be dictated on the value of the "exists" variable (For example an overwrite argument in the download function).
The download function should return a single data.frame with columns "platform", "file" and "processed". The "platform" column indicates what platform the file downloaded relates to. The "file" column gives the full path to the downloaded file. The "processed" column indicates whether the file was downloaded/processed.
mydl_fun <- function(dllist, ...) { # dllist is output from a predlfunction with final output from assign_directory # CODE here to perform download data.frame( platform = platform, file = file, processed = processed, stringsAsFactors = FALSE ) }
Post-Download Function
The post-download function takes as its primary input the output given by a "download function" that is a data.frame with column names "platform", "file" and "processed".
The purpose of the post-download function is to process the downloaded files after the download. This could involve changing the file properties such as the mode, unzip/untar a downloaded zip/tar file etc.
Application YAML file
The three functions mentioned previously are combined together to define a process for downloading the binary file. The user stipulates the required functions and arguments by way of a YAML format file. For example:
name: superduperapp predlfunction: "superduperapp::predl_superduper": appname: "superduperapp" dlfunction: "superduperapp::download_superduper": [] postdlfunction: "superduperapp::postdl_superduper": []
The defined YAML file is then processed by the process_yaml function.
The YAML file needs to define the three required functions with any
arguments and also define a name.
Browser Mob Proxy example
The following is an example of using binman to get the github assets from a project. The project is https://github.com/lightbody/browsermob-proxy/releases . When a new version is released a zipped binary is added as an "asset". A JSON representation of the project releases is available at https://api.github.com/repos/lightbody/browsermob-proxy/releases. We shall breakdown the process into its three functional parts.
BMP Pre-Download
Firstly we note we have JSON data so we parse the JSON from the URL:
bmpURL <- "https://api.github.com/repos/lightbody/browsermob-proxy/releases" bmpData <- jsonlite::fromJSON(bmpURL)
This gives us a list of 10 releases. A GitHub release should have a version number.
version <- bmpData[["tag_name"]] version
[1] "browsermob-proxy-2.1.2" "browsermob-proxy-2.1.1" [3] "browsermob-proxy-2.1.0" "browsermob-proxy-2.1.0-beta-6" [5] "browsermob-proxy-2.1.0-beta-5" "browsermob-proxy-2.1.0-beta-4" [7] "browsermob-proxy-2.1.0-beta-3" "browsermob-proxy-2.1.0-beta-2" [9] "browsermob-proxy-2.1.0-beta-1" "browsermob-proxy-2.0.0"
This is our version data but we may want to tidy it up so:
versionregex <- c("browsermob-proxy-(.*)$", "\\1") version <- gsub(versionregex[1], versionregex[2], version) version
[1] "2.1.2" "2.1.1" "2.1.0" "2.1.0-beta-6" "2.1.0-beta-5" [6] "2.1.0-beta-4" "2.1.0-beta-3" "2.1.0-beta-2" "2.1.0-beta-1" "2.0.0"
Each release has "assets" associated with it (a zip file).
assets <- bmpData[["assets"]] length(assets)
[1] 10
The assets will contain the other information we need for our pre-download function.
From each assets item we would like the file-name:
bmpFiles <- vapply(assets, "[[", character(1), "name") bmpFiles
[1] "browsermob-proxy-2.1.2-bin.zip" "browsermob-proxy-2.1.1-bin.zip" [3] "browsermob-proxy-2.1.0-bin.zip" "browsermob-proxy-2.1.0-beta-6-bin.zip" [5] "browsermob-proxy-2.1.0-beta-5-bin.zip" "browsermob-proxy-2.1.0-beta-4-bin.zip" [7] "browsermob-proxy-2.1.0-beta-3-bin.zip" "browsermob-proxy-2.1.0-beta-2-bin.zip" [9] "browsermob-proxy-2.1.0-beta-1-bin.zip" "browsermob-proxy-2.0.0-bin.zip"
and the URL for that file:
bmpURLs <- vapply(assets, "[[", character(1), "browser_download_url") bmpURLs
[1] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.2/browsermob-proxy-2.1.2-bin.zip" [2] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.1/browsermob-proxy-2.1.1-bin.zip" [3] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0/browsermob-proxy-2.1.0-bin.zip" [4] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-6/browsermob-proxy-2.1.0-beta-6-bin.zip" [5] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-5/browsermob-proxy-2.1.0-beta-5-bin.zip" [6] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-4/browsermob-proxy-2.1.0-beta-4-bin.zip" [7] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-3/browsermob-proxy-2.1.0-beta-3-bin.zip" [8] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-2/browsermob-proxy-2.1.0-beta-2-bin.zip" [9] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-1/browsermob-proxy-2.1.0-beta-1-bin.zip" [10] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.0.0/browsermob-proxy-2.0.0-bin.zip"
In the case of BMP we have a single platform which we could denote "generic". So we could pass the following:
dllist <- list( "generic" = data.frame( version = version, url = bmpURLs, file = bmpFiles, stringsAsFactors = FALSE ) )
Then finally we pass this list to the assign_directory function:
predlOut <- assign_directory(dllist, "bmpApp") str(predlOut, max.level = 2)
List of 1 $ generic:'data.frame': 10 obs. of 5 variables: ..$ version: chr [1:10] "2.1.2" "2.1.1" "2.1.0" "2.1.0-beta-6" ... ..$ url : chr [1:10] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.2/browsermob-proxy-2.1.2-bin.zip" "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.1/browsermob-proxy-2.1.1-bin.zip" "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0/browsermob-proxy-2.1.0-bin.zip" "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-6/browsermob-proxy-2.1.0-beta-6-bin"| __truncated__ ... ..$ file : chr [1:10] "browsermob-proxy-2.1.2-bin.zip" "browsermob-proxy-2.1.1-bin.zip" "browsermob-proxy-2.1.0-bin.zip" "browsermob-proxy-2.1.0-beta-6-bin.zip" ... ..$ dir :List of 10 ..$ exists : logi [1:10] FALSE FALSE FALSE FALSE FALSE FALSE ...
In the binman package there is a function template for github asset type
downloads:
dllist <- predl_github_assets( url = "https://api.github.com/repos/lightbody/browsermob-proxy/releases", history = 3L, platform = "generic", appname = "bmproxy", platformregex = "browsermob-proxy", versionregex = c("browsermob-proxy-(.*)$", "\\1") )
There are some extra arguments for generality but it basically performs the operations we outlined above.
BMP Download
The download for BMP is straightforward:
dllist[["generic"]][1,]
version
1 2.1.2
url
1 https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.2/browsermob-proxy-2.1.2-bin.zip
file dir
1 browsermob-proxy-2.1.2-bin.zip /home/john/.local/share/binman_bmproxy/generic/2.1.2
exists
1 FALSE
For each platform (there is only one "generic" platform in this case) we want to loop over the data.frame downloading the data at "url" to the "file" in the "dir". If it already "exists" we may not perform the download.
In binman there is a simple template function for this:
dlfiles <- download_files(dllist)
So we simply need to pass the list from the pre download function to this download function.
BMP Post Download
The files we receive from the Browser mob Proxy project are zipped. As a post processing operation we would like to unzip these files. The result of unzipping the files is a directory structure. In this case we do not change the mode of the ultimate binary which is contained in this directory structure.
Again binman has a simple template function to unzip/untar a downloaded
file so we pass our output from the download to this:
dlres <- unziptar_dlfiles(dlfiles)
BMP YAML
We can incorporate our three functional calls into a YAML file for the Browser Mob Proxy application as follows:
name: bmproxy predlfunction: "binman::predl_github_assets": url: "https://api.github.com/repos/lightbody/browsermob-proxy/releases" platform: - generic history: 3 appname: "bmproxy" platformregex: - "browsermob-proxy" versionregex: - "browsermob-proxy-(.*)$" - "\\1" dlfunction: "binman::download_files": [] postdlfunction: "binman::unziptar_dlfiles": []
We have saved this file in the binman package:
ymlfile <- system.file("examples", "yaml", "bmproxy.yml", package = "binman")
We can now use the process-yaml function to download the BMP binaries:
process_yaml(ymlfile)
BEGIN: PREDOWNLOAD BEGIN: DOWNLOAD Creating directory: /home/john/.local/share/binman_bmproxy/generic/2.1.2 Downloading binary: https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.2/browsermob-proxy-2.1.2-... Creating directory: /home/john/.local/share/binman_bmproxy/generic/2.1.1 Downloading binary: https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.1/browsermob-proxy-2.1.1-... Creating directory: /home/john/.local/share/binman_bmproxy/generic/2.1.0 Downloading binary: https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0/browsermob-proxy-2.1.0-... BEGIN: POSTDOWNLOAD
Looking at the resulting directory structure (this is for Linux the location of directories will be OS dependent):
tree -d /home/john/.local/share/binman_bmproxy
/home/john/.local/share/binman_bmproxy
└── generic
├── 2.1.0
│ └── browsermob-proxy-2.1.0
│ ├── bin
│ │ └── conf
│ ├── lib
│ └── ssl-support
├── 2.1.1
│ └── browsermob-proxy-2.1.1
│ ├── bin
│ │ └── conf
│ ├── lib
│ └── ssl-support
└── 2.1.2
└── browsermob-proxy-2.1.2
├── bin
│ └── conf
├── lib
└── ssl-support
We see that binman has downloaded and unzipped the binaries to
appropriate directories.
Try the binman package in your browser
Any scripts or data that you put into this service are public.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.