In Enchufa2/bspm: Bridge to System Package Manager
Motivation
.pull-left-33[
]
.pull-right-66[.font120[
Comprehensive R Archive Network (CRAN)
- More than 17,700 packages (!)
- Many of them use compiled code (Fortran, C, C++...)
- Compilation offers performance, integration, use of libraries...
]]
--
.pull-right-66[.font120[
- Pick a package at random:
80% chance of requiring compilation
- Up to 50% of the 1% most popular require compilation
]]
--
.pull-right-66[.font120[
- Compiling requires an appropriate toolchain and libraries
- Compiling requires a lot of time
- Debugging build and compilation failures requires skills
]]
Motivation
--
Windows & macOS desktop users
--
Linux desktop users
--
Cloud deployments, containers, CI/CD systems...
We can do better
.left-column[
Entire tidyverse
in 36 seconds
on Fedora
]
.right-column[
]
Challenges
.pull-left[.font120[
OS support
- Many distributions
- Different package managers
- Different philosophies, conventions
- Different update cycles
Scalability
- Build and distribute thousands of packages
- Accomodate constant growth at CRAN
- Accomodate new platforms (hello, arm64)
]]
.pull-right[.font120[
Dependency management
- External dependencies
- Static vs. shared (system libraries)
Integration
- Built-in
install.packages
- Dependencies via system package manager
- Need for admin privileges
]]
Related work
.left-column[
Official repos
]
.right-column[.font120[
Most distributions maintain R and a number of R packages. In Fedora,
$ dnf repoquery --repo=fedora-source R-* | wc -l
## 384
]]
--
.right-column[.font120[
.pros[
- Dependency management and multi-tenancy
- Works out-of-the-box, no additional repos required
].cons[
- Does not scale: every R package requires a peer review (even for CRAN packages!)
- No integration: mixed used of distro-specific package manager (with admin privileges) and install.packages
]
]]
Related work
.left-column[
Official repos
cran2deb
cran2obs
]
.right-column[.font120[
- cran2deb started by Albrecht Gebhard, picked up by Dirk et al
- Significantly extented and polished by Charles Blundell
- Don Armstrong maintained a parallel setup called debian-r
- Michael Rutter maintains the c2d4u repos, based on cran2deb, in Launchpad
(~ 4.7k packages for several R versions) for Debian/Ubuntu
- Detlef Steuer maintains the autoCRAN repo, based on cran2obs, in OBS
(~16k packages) for openSUSE
]]
--
.right-column[.font120[
.pros[
- Dependency management and multi-tenancy
- Better scalability: scripts for automated conversion + PPA / OBS
].cons[
- No integration: mixed used of distro-specific package manager (with admin privileges) and install.packages
]
]]
Related work
.left-column[
Official repos
cran2deb
cran2obs
RSPM
]
.right-column[.font120[
- RStudio's Public Package Manager, announced on July 1, 2020
- Based on the proprietary product RSPM (RStudio Package Manager)
]]
--
.right-column[.font120[
.pros[
- Better OS support: builds for multiple distributions
- Better scalability: builds probably based on Docker containers
- Better integration: drop-in replacement for CRAN repo for install.packages
].cons[
- No dependencies:
- system package manager still required to install system dependencies
- version mismatches can happen
]
]]
class: inverse, center, middle
Meet cran2copr...
... and bspm
Meet cran2copr
.left-column[
A primer
]
.right-column[
$ sudo dnf install 'dnf-command(copr)'
$ sudo dnf copr enable iucar/cran
$ sudo dnf install R-CoprManager
]
--
.right-column[
install.packages("units")
## Install system packages as root...
## (1/3): R-CRAN-units-0.6.7-3.fc32.x86_64.rpm 5.2 MB/s | 787 kB 00:00
## (2/3): R-CRAN-Rcpp-1.0.5-2.fc32.x86_64.rpm 2.0 MB/s | 2.0 MB 00:00
## (3/3): udunits2-2.2.26-6.fc32.x86_64.rpm 420 kB/s | 617 kB 00:01
## Preparing : 1/1
## Installing : udunits2-2.2.26-6.fc32.x86_64 1/3
## Installing : R-CRAN-Rcpp-1.0.5-2.fc32.x86_64 2/3
## Installing : R-CRAN-units-0.6.7-3.fc32.x86_64 3/3
## Running scriptlet: R-CRAN-units-0.6.7-3.fc32.x86_64 3/3
## Verifying : R-CRAN-Rcpp-1.0.5-2.fc32.x86_64 1/3
## Verifying : R-CRAN-units-0.6.7-3.fc32.x86_64 2/3
## Verifying : udunits2-2.2.26-6.fc32.x86_64 3/3
]
Meet cran2copr
.left-column[
A primer
Under the hood
]
.right-column[.font120[
- cran2copr uses the Fedora Copr buildsystem to build packages and publish RPM contributed repos
- (Almost) the entire CRAN available (~17.3k packages and growing)
- Automated daily updates via GitHub Actions
- Full integration with install.packages via bspm (CoprManager)
]
knitr::kable(
data.frame(
c("OS support", "Dependencies", "Scalability", "Integration"),
c("●", "✔", "", ""),
c("●", "✔", "●", ""),
c("●", "✔", "✔", ""),
c("●", "✔", "✔","✔")
),
col.names = c("", "Official repos", "cran2deb", "cran2obs", "cran2copr"),
align = "rcccc",
escape = FALSE
)
]
Meet bspm
.left-column[
A primer
Under the hood
The magic
]
.right-column[.font120[
- bspm: Bridge to System Package Manager
- Extensible multi-distro integration (currently supports DNF and APT)
- cran2copr's CoprManager is just a branded version of bspm
]]
--
.right-column[
.font120[
- Available on CRAN, and as Debian/Ubuntu package (r-cran-bspm)
- Available in the rocker project (rocker/r-bspm)
- Provides integration to other projects
]
knitr::kable(
data.frame(
c("OS support", "Dependencies", "Scalability", "Integration"),
c("●", "✔", "", "<span class='badge'>✔</span>"),
c("●", "✔", "●", "<span class='badge'>✔</span>"),
c("●", "✔", "✔", "<span class='badge'>✔</span>"),
c("●", "✔", "✔","✔")
),
col.names = c("", "Official repos", "cran2deb", "cran2obs", "cran2copr"),
align = "rcccc",
escape = FALSE
)
]
Meet bspm
flowchart LR
subgraph Debian/Ubuntu
cran2deb[cran2deb
5k packages]
end
style Debian/Ubuntu fill:#CE0056,stroke:none,color:white
subgraph openSUSE
cran2obs[cran2obs
16k packages]
end
style openSUSE fill:#73BA25,stroke:none,color:white
subgraph Fedora
cran2copr[cran2copr
17k packages]
end
style Fedora fill:#3c6eb4,stroke:none,color:white
subgraph bspm[bspm package]
subgraph system service
backend --- apt
backend --- dnf
end
enable["enable()"]
is["install_sys(...)"]
is --> |desktop/server
via D-Bus|backend
is --> |containers
via root/sudo|backend
end
style bspm fill:none
subgraph R[R session]
ip["install.packages(...)"]
ip -.-o enable
ip --> is
is --> |not available|ip
end
apt --> cran2deb
dnf --> cran2obs
dnf --> cran2copr
Example: rspm vs. bspm
.left-column[
Missing dependencies
]
.right-column[
See another cool Ubuntu-based animation at https://dirk.eddelbuettel.com/blog/2020/08/26/
]
Example: geospatial stack via bspm
.left-column[
rocker/geospatial:
- 25 packages
Dependencies:
- 139 packages
- 69 system libraries
Size:
- ~300 MB download
- ~800 MB on disk
68 seconds
]
.right-column[
]
Thanks
.pull-left[
]
.pull-right[
Links:
- https://github.com/cran4linux/cran2copr
- https://github.com/cran4linux/bspm
- https://github.com/rocker-org/rocker/tree/master/r-bspm
]
class: inverse, center, middle
Appendix: installation instructions
.left-column[
Fedora:
Ubuntu:
openSUSE:
then...
]
.right-column[
$ sudo dnf install 'dnf-command(copr)'
$ sudo dnf copr enable iucar/cran
$ sudo dnf install R-CoprManager
$ sudo add-apt-repository ppa:marutter/rrutter4.0 # R v4.0 and higher
$ sudo add-apt-repository ppa:c2d4u.team/c2d4u4.0+ # R packages
$ sudo apt-get update
$ sudo apt-get install r-cran-bspm python3-{dbus,gi,apt}
$ echo "bspm::enable()" | sudo tee -a /etc/R/Rprofile.site
$ sudo zypper ar -r https://download.opensuse.org/repositories/devel:/languages:/R:/patched/openSUSE_Tumbleweed/devel:languages:R:patched.repo
$ sudo zypper ar -r https://download.opensuse.org/repositories/devel:/languages:/R:/autoCRAN/openSUSE_Tumbleweed/devel:languages:R:autoCRAN.repo
$ sudo zypper install R-patched python3-dnf python38-dbus-python python3-gobject
$ sudo ln -s /etc/zypp/repos.d /etc/yum.repos.d
$ sudo Rscript -e 'install.packages("bspm", repos="https://cran.r-project.org")'
$ echo "bspm::enable()" | sudo tee -a /usr/lib64/R/etc/Rprofile.site
install.packages(...) # enjoy!
]
Appendix: installation instructions
.left-column[
Other
]
.right-column[.font120[
General procedure
If you are trying bspm in another distro, or you are packaging it as a system package, please refer to the general procedure.
Containerized applications (e.g., a Docker images)
Refer to the rocker/r-bspm images.
]]
Enchufa2/bspm documentation built on Oct. 14, 2024, 10:36 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Motivation
.pull-left-33[
]
.pull-right-66[.font120[
Comprehensive R Archive Network (CRAN)
- More than 17,700 packages (!)
- Many of them use compiled code (Fortran, C, C++...)
- Compilation offers performance, integration, use of libraries... ]]
--
.pull-right-66[.font120[
- Pick a package at random:
80% chance of requiring compilation
- Up to 50% of the 1% most popular require compilation
]]
--
.pull-right-66[.font120[ - Compiling requires an appropriate toolchain and libraries - Compiling requires a lot of time - Debugging build and compilation failures requires skills ]]
Motivation
--
Windows & macOS desktop users
--
Linux desktop users
--
Cloud deployments, containers, CI/CD systems...
We can do better
.left-column[
Entire tidyverse
in 36 seconds
on Fedora
]
.right-column[
]
Challenges
.pull-left[.font120[
OS support
- Many distributions
- Different package managers
- Different philosophies, conventions
- Different update cycles
Scalability
- Build and distribute thousands of packages
- Accomodate constant growth at CRAN
- Accomodate new platforms (hello, arm64) ]]
.pull-right[.font120[
Dependency management
- External dependencies
- Static vs. shared (system libraries)
Integration
- Built-in
install.packages - Dependencies via system package manager
- Need for admin privileges ]]
Related work
.left-column[
Official repos
]
.right-column[.font120[ Most distributions maintain R and a number of R packages. In Fedora,
$ dnf repoquery --repo=fedora-source R-* | wc -l
## 384
]]
--
.right-column[.font120[
.pros[
- Dependency management and multi-tenancy
- Works out-of-the-box, no additional repos required
].cons[
- Does not scale: every R package requires a peer review (even for CRAN packages!)
- No integration: mixed used of distro-specific package manager (with admin privileges) and install.packages
]
]]
Related work
.left-column[
Official repos
cran2deb
cran2obs
]
.right-column[.font120[
- cran2deb started by Albrecht Gebhard, picked up by Dirk et al
- Significantly extented and polished by Charles Blundell
- Don Armstrong maintained a parallel setup called debian-r
- Michael Rutter maintains the c2d4u repos, based on cran2deb, in Launchpad
(~ 4.7k packages for several R versions) for Debian/Ubuntu
- Detlef Steuer maintains the autoCRAN repo, based on cran2obs, in OBS
(~16k packages) for openSUSE
]]
--
.right-column[.font120[
.pros[
- Dependency management and multi-tenancy
- Better scalability: scripts for automated conversion + PPA / OBS
].cons[
- No integration: mixed used of distro-specific package manager (with admin privileges) and install.packages
]
]]
Related work
.left-column[
Official repos
cran2deb
cran2obs
RSPM
]
.right-column[.font120[ - RStudio's Public Package Manager, announced on July 1, 2020 - Based on the proprietary product RSPM (RStudio Package Manager) ]]
--
.right-column[.font120[
.pros[
- Better OS support: builds for multiple distributions
- Better scalability: builds probably based on Docker containers
- Better integration: drop-in replacement for CRAN repo for install.packages
].cons[
- No dependencies:
- system package manager still required to install system dependencies
- version mismatches can happen
]
]]
class: inverse, center, middle
Meet cran2copr...
... and bspm
Meet cran2copr
.left-column[
A primer
]
.right-column[
$ sudo dnf install 'dnf-command(copr)' $ sudo dnf copr enable iucar/cran $ sudo dnf install R-CoprManager
]
--
.right-column[
install.packages("units")
## Install system packages as root... ## (1/3): R-CRAN-units-0.6.7-3.fc32.x86_64.rpm 5.2 MB/s | 787 kB 00:00 ## (2/3): R-CRAN-Rcpp-1.0.5-2.fc32.x86_64.rpm 2.0 MB/s | 2.0 MB 00:00 ## (3/3): udunits2-2.2.26-6.fc32.x86_64.rpm 420 kB/s | 617 kB 00:01 ## Preparing : 1/1 ## Installing : udunits2-2.2.26-6.fc32.x86_64 1/3 ## Installing : R-CRAN-Rcpp-1.0.5-2.fc32.x86_64 2/3 ## Installing : R-CRAN-units-0.6.7-3.fc32.x86_64 3/3 ## Running scriptlet: R-CRAN-units-0.6.7-3.fc32.x86_64 3/3 ## Verifying : R-CRAN-Rcpp-1.0.5-2.fc32.x86_64 1/3 ## Verifying : R-CRAN-units-0.6.7-3.fc32.x86_64 2/3 ## Verifying : udunits2-2.2.26-6.fc32.x86_64 3/3
]
Meet cran2copr
.left-column[
A primer
Under the hood
]
.right-column[.font120[
- cran2copr uses the Fedora Copr buildsystem to build packages and publish RPM contributed repos
- (Almost) the entire CRAN available (~17.3k packages and growing)
- Automated daily updates via GitHub Actions
- Full integration with install.packages via bspm (CoprManager)
]
knitr::kable( data.frame( c("OS support", "Dependencies", "Scalability", "Integration"), c("●", "✔", "", ""), c("●", "✔", "●", ""), c("●", "✔", "✔", ""), c("●", "✔", "✔","✔") ), col.names = c("", "Official repos", "cran2deb", "cran2obs", "cran2copr"), align = "rcccc", escape = FALSE )
]
Meet bspm
.left-column[
A primer
Under the hood
The magic
]
.right-column[.font120[
- bspm: Bridge to System Package Manager
- Extensible multi-distro integration (currently supports DNF and APT)
- cran2copr's CoprManager is just a branded version of bspm
]]
--
.right-column[
.font120[
- Available on CRAN, and as Debian/Ubuntu package (
]
r-cran-bspm)
- Available in the rocker project (rocker/r-bspm)
- Provides integration to other projects
]
knitr::kable( data.frame( c("OS support", "Dependencies", "Scalability", "Integration"), c("●", "✔", "", "<span class='badge'>✔</span>"), c("●", "✔", "●", "<span class='badge'>✔</span>"), c("●", "✔", "✔", "<span class='badge'>✔</span>"), c("●", "✔", "✔","✔") ), col.names = c("", "Official repos", "cran2deb", "cran2obs", "cran2copr"), align = "rcccc", escape = FALSE )
Meet bspm
flowchart LR
subgraph Debian/Ubuntu
cran2deb[cran2deb
5k packages] end style Debian/Ubuntu fill:#CE0056,stroke:none,color:white subgraph openSUSE cran2obs[cran2obs
16k packages] end style openSUSE fill:#73BA25,stroke:none,color:white subgraph Fedora cran2copr[cran2copr
17k packages] end style Fedora fill:#3c6eb4,stroke:none,color:white subgraph bspm[bspm package] subgraph system service backend --- apt backend --- dnf end enable["enable()"] is["install_sys(...)"] is --> |desktop/server
via D-Bus|backend is --> |containers
via root/sudo|backend end style bspm fill:none subgraph R[R session] ip["install.packages(...)"] ip -.-o enable ip --> is is --> |not available|ip end apt --> cran2deb dnf --> cran2obs dnf --> cran2copr
5k packages] end style Debian/Ubuntu fill:#CE0056,stroke:none,color:white subgraph openSUSE cran2obs[cran2obs
16k packages] end style openSUSE fill:#73BA25,stroke:none,color:white subgraph Fedora cran2copr[cran2copr
17k packages] end style Fedora fill:#3c6eb4,stroke:none,color:white subgraph bspm[bspm package] subgraph system service backend --- apt backend --- dnf end enable["enable()"] is["install_sys(...)"] is --> |desktop/server
via D-Bus|backend is --> |containers
via root/sudo|backend end style bspm fill:none subgraph R[R session] ip["install.packages(...)"] ip -.-o enable ip --> is is --> |not available|ip end apt --> cran2deb dnf --> cran2obs dnf --> cran2copr
Example: rspm vs. bspm
.left-column[
Missing dependencies
]
.right-column[ See another cool Ubuntu-based animation at https://dirk.eddelbuettel.com/blog/2020/08/26/ ]
Example: geospatial stack via bspm
.left-column[
rocker/geospatial:
- 25 packages
Dependencies: - 139 packages - 69 system libraries
Size: - ~300 MB download - ~800 MB on disk
68 seconds
]
.right-column[
]
Thanks
.pull-left[
]
.pull-right[
Links: - https://github.com/cran4linux/cran2copr - https://github.com/cran4linux/bspm - https://github.com/rocker-org/rocker/tree/master/r-bspm ]
class: inverse, center, middle
Appendix: installation instructions
.left-column[
Fedora:
Ubuntu:
openSUSE:
then...
].right-column[
$ sudo dnf install 'dnf-command(copr)' $ sudo dnf copr enable iucar/cran $ sudo dnf install R-CoprManager
$ sudo add-apt-repository ppa:marutter/rrutter4.0 # R v4.0 and higher $ sudo add-apt-repository ppa:c2d4u.team/c2d4u4.0+ # R packages $ sudo apt-get update $ sudo apt-get install r-cran-bspm python3-{dbus,gi,apt} $ echo "bspm::enable()" | sudo tee -a /etc/R/Rprofile.site
$ sudo zypper ar -r https://download.opensuse.org/repositories/devel:/languages:/R:/patched/openSUSE_Tumbleweed/devel:languages:R:patched.repo $ sudo zypper ar -r https://download.opensuse.org/repositories/devel:/languages:/R:/autoCRAN/openSUSE_Tumbleweed/devel:languages:R:autoCRAN.repo $ sudo zypper install R-patched python3-dnf python38-dbus-python python3-gobject $ sudo ln -s /etc/zypp/repos.d /etc/yum.repos.d $ sudo Rscript -e 'install.packages("bspm", repos="https://cran.r-project.org")' $ echo "bspm::enable()" | sudo tee -a /usr/lib64/R/etc/Rprofile.site
install.packages(...) # enjoy!
]
Appendix: installation instructions
.left-column[
Other
]
.right-column[.font120[
General procedure
If you are trying bspm in another distro, or you are packaging it as a system package, please refer to the general procedure.
Containerized applications (e.g., a Docker images)
Refer to the rocker/r-bspm images.
]]
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.