vignettes/Vignette.md

In SamA-DS/Rbduk: Assorted functions for BDUK analysts

Rbduk

code{white-space: pre-wrap;} span.smallcaps{font-variant: small-caps;} span.underline{text-decoration: underline;} div.column{display: inline-block; vertical-align: top; width: 50%;} div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;} ul.task-list{list-style: none;} code{white-space: pre;} a.sourceLine { display: inline-block; line-height: 1.25; } a.sourceLine { pointer-events: none; color: inherit; text-decoration: inherit; } a.sourceLine:empty { height: 1.2em; } .sourceCode { overflow: visible; } code.sourceCode { white-space: pre; position: relative; } div.sourceCode { margin: 1em 0; } pre.sourceCode { margin: 0; } @media screen { div.sourceCode { overflow: auto; } } @media print { code.sourceCode { white-space: pre-wrap; } a.sourceLine { text-indent: -1em; padding-left: 1em; } } pre.numberSource a.sourceLine { position: relative; left: -4em; } pre.numberSource a.sourceLine::before { content: attr(data-line-number); position: relative; left: -1em; text-align: right; vertical-align: baseline; border: none; pointer-events: all; display: inline-block; -webkit-touch-callout: none; -webkit-user-select: none; -khtml-user-select: none; -moz-user-select: none; -ms-user-select: none; user-select: none; padding: 0 4px; width: 4em; color: #aaaaaa; } pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; } div.sourceCode { } @media screen { a.sourceLine::before { text-decoration: underline; } } code span.al { color: #ff0000; font-weight: bold; } /* Alert */ code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */ code span.at { color: #7d9029; } /* Attribute */ code span.bn { color: #40a070; } /* BaseN */ code span.bu { } /* BuiltIn */ code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */ code span.ch { color: #4070a0; } /* Char */ code span.cn { color: #880000; } /* Constant */ code span.co { color: #60a0b0; font-style: italic; } /* Comment */ code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */ code span.do { color: #ba2121; font-style: italic; } /* Documentation */ code span.dt { color: #902000; } /* DataType */ code span.dv { color: #40a070; } /* DecVal */ code span.er { color: #ff0000; font-weight: bold; } /* Error */ code span.ex { } /* Extension */ code span.fl { color: #40a070; } /* Float */ code span.fu { color: #06287e; } /* Function */ code span.im { } /* Import */ code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */ code span.kw { color: #007020; font-weight: bold; } /* Keyword */ code span.op { color: #666666; } /* Operator */ code span.ot { color: #007020; } /* Other */ code span.pp { color: #bc7a00; } /* Preprocessor */ code span.sc { color: #4070a0; } /* SpecialChar */ code span.ss { color: #bb6688; } /* SpecialString */ code span.st { color: #4070a0; } /* String */ code span.va { color: #19177c; } /* Variable */ code span.vs { color: #4070a0; } /* VerbatimString */ code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */ // apply pandoc div.sourceCode style to pre.sourceCode instead (function() { var sheets = document.styleSheets; for (var i = 0; i < sheets.length; i++) { if (sheets[i].ownerNode.dataset["origin"] !== "pandoc") continue; try { var rules = sheets[i].cssRules; } catch (e) { continue; } for (var j = 0; j < rules.length; j++) { var rule = rules[j]; // check if there is a div.sourceCode rule if (rule.type !== rule.STYLE_RULE || rule.selectorText !== "div.sourceCode") continue; var style = rule.style.cssText; // check if color or background-color is set if (rule.style.color === '' && rule.style.backgroundColor === '') continue; // replace div.sourceCode by a pre.sourceCode rule sheets[i].deleteRule(j); sheets[i].insertRule('pre.sourceCode{' + style + '}', j); } } })(); body { background-color: #fff; margin: 1em auto; max-width: 700px; overflow: visible; padding-left: 2em; padding-right: 2em; font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; line-height: 1.35; } #TOC { clear: both; margin: 0 0 10px 10px; padding: 4px; width: 400px; border: 1px solid #CCCCCC; border-radius: 5px; background-color: #f6f6f6; font-size: 13px; line-height: 1.3; } #TOC .toctitle { font-weight: bold; font-size: 15px; margin-left: 5px; } #TOC ul { padding-left: 40px; margin-left: -1.5em; margin-top: 5px; margin-bottom: 5px; } #TOC ul ul { margin-left: -2em; } #TOC li { line-height: 16px; } table { margin: 1em auto; border-width: 1px; border-color: #DDDDDD; border-style: outset; border-collapse: collapse; } table th { border-width: 2px; padding: 5px; border-style: inset; } table td { border-width: 1px; border-style: inset; line-height: 18px; padding: 5px 5px; } table, table th, table td { border-left-style: none; border-right-style: none; } table thead, table tr.even { background-color: #f7f7f7; } p { margin: 0.5em 0; } blockquote { background-color: #f6f6f6; padding: 0.25em 0.75em; } hr { border-style: solid; border: none; border-top: 1px solid #777; margin: 28px 0; } dl { margin-left: 0; } dl dd { margin-bottom: 13px; margin-left: 13px; } dl dt { font-weight: bold; } ul { margin-top: 0; } ul li { list-style: circle outside; } ul ul { margin-bottom: 0; } pre, code { background-color: #f7f7f7; border-radius: 3px; color: #333; white-space: pre-wrap; } pre { border-radius: 3px; margin: 5px 0px 10px 0px; padding: 10px; } pre:not([class]) { background-color: #f7f7f7; } code { font-family: Consolas, Monaco, 'Courier New', monospace; font-size: 85%; } p > code, li > code { padding: 2px 0px; } div.figure { text-align: center; } img { background-color: #FFFFFF; padding: 2px; border: 1px solid #DDDDDD; border-radius: 3px; border: 1px solid #CCCCCC; margin: 0 5px; } h1 { margin-top: 0; font-size: 35px; line-height: 40px; } h2 { border-bottom: 4px solid #f7f7f7; padding-top: 10px; padding-bottom: 2px; font-size: 145%; } h3 { border-bottom: 2px solid #f7f7f7; padding-top: 10px; font-size: 120%; } h4 { border-bottom: 1px solid #f7f7f7; margin-left: 8px; font-size: 105%; } h5, h6 { border-bottom: 1px solid #ccc; font-size: 105%; } a { color: #0033dd; text-decoration: none; } a:hover { color: #6666ff; } a:visited { color: #800080; } a:visited:hover { color: #BB00BB; } a[href^="http:"] { text-decoration: underline; } a[href^="https:"] { text-decoration: underline; } code > span.kw { color: #555; font-weight: bold; } code > span.dt { color: #902000; } code > span.dv { color: #40a070; } code > span.bn { color: #d14; } code > span.fl { color: #d14; } code > span.ch { color: #d14; } code > span.st { color: #d14; } code > span.co { color: #888888; font-style: italic; } code > span.ot { color: #007020; } code > span.al { color: #ff0000; font-weight: bold; } code > span.fu { color: #900; font-weight: bold; } code > span.er { color: #a61717; background-color: #e3d2d2; }

Rbduk

General functions

is_integer64()

Within GCP, large numbers (eg UPRNs) are stored are 64-bit integers. R, on the whole, does not like 64-bit integer, and whilst it can cope, it’s easier to read in these columns are numeric, or convert them to numeric within R. To see if an object is currently a 64-bit integer, this function can be used.

is_integer64(100)
#> [1] FALSE

is_integer64("x")
#> [1] FALSE

pretty_postcode()

This function takes a postcode in any format and any case, and converts it into pretty format XX(X)(X)(Y)XXX, where X is the postcode, and Y is the specified seperator, which is a space by default. The case can also be specified, and is upper case by default.

pretty_postcode("SW1a2nP")
#> [1] "SW1A 2NP"

pretty_postcode("SW1a2nP", sep="")
#> [1] "SW1A2NP"

pretty_postcode("SW1a2nP", sep=".")
#> [1] "SW1A.2NP"

pretty_postcode("SW1a2nP", sep=".", uppercase=FALSE)
#> [1] "sw1a.2np"

is_postcode()

This function takes a string and returns TRUE or FALSE depedent on whether the string is in a valid UK postcode format or not. This may contain one space and still be valid. This does not indicate whether a postcode is an existing postcode, but that is has the format of one.

The following demonstrates valid postcodes and invalid postcodes:

is_postcode("SW1a2nP")
#> [1] TRUE

is_postcode("SW1a 2nP")
#> [1] TRUE

#Too short
is_postcode("S 2NP")
#> [1] FALSE

#Contains punctuation
is_postcode("Sw1a.2np")
#> [1] FALSE

#Too long
is_postcode("Sw1a2npX")
#> [1] FALSE

#Contains too many spaces
is_postcode("Sw1a  2np")
#> [1] FALSE

#Not valid number/letter combiantion
is_postcode("000000")
#> [1] FALSE

#Not valid number/letter combiantion
is_postcode("XXXXXX")
#> [1] FALSE

is_uprn()

This function takes a numeric or character vector and returns TRUE or FALSE depedent on whether it is in a valid UPRN format (all numeric, between 1 and 12 characters). It will also flag as a message any UPRNs that end 0000, as a common conversion error caused by scientific notation and reading/writing from excel can cause genuine UPRNs to end in a number of zeros, meaning they are no longer genuine. One or two of these messages is acceptable, but many indicates that this error has occured and that the UPRNs should be checked thoroughly.

The following demonstrates valid and invalid UPRNs:

is_uprn(1)
#> [1] TRUE

is_uprn(999999999999)
#> [1] TRUE

is_uprn("1")
#> [1] TRUE

is_uprn("999999999999")
#> [1] TRUE

#Too long
is_uprn(9999999999999)
#> [1] FALSE

#Too long
is_uprn("9999999999999")
#> [1] FALSE

#Empty
is_uprn("")
#> [1] FALSE

#Letters
is_uprn("ABC")
#> [1] FALSE

#Contains trailing characters
is_uprn("1,")
#> [1] FALSE

#Not an integer
is_uprn(111.999)
#> [1] FALSE

#Not an integer
is_uprn("111.999")
#> [1] FALSE

#Contains punctuation
is_uprn("111-999")
#> [1] FALSE

#Contains punctuation
is_uprn("111 999")
#> [1] FALSE

%notin% and %!in%

These functions serve the same purpose, and that is to give a more readable version of the opposite of %in%. They output the opposite logical response that %in% would provide.

2 %!in% c(1,3)
#> [1] TRUE

2 %notin% c(1,3)
#> [1] TRUE

"b" %!in% c("a","c")
#> [1] TRUE

"b" %notin% c("a","c")
#> [1] TRUE

2 %!in% c(1,2,3)
#> [1] FALSE

2 %notin% c(1,2,3)
#> [1] FALSE

"b" %!in% c("a","b","c")
#> [1] FALSE

"b" %notin% c("a","b","c")
#> [1] FALSE

all.equal(2 %!in% c(1,3), 2 %notin% c(1,3), !2 %in% c(1,3))
#> [1] TRUE

all.equal("b" %!in% c("a","c"), "b" %notin% c("a","c"), !"b" %in% c("a","c"))
#> [1] TRUE

GCP functions

bduk_bq()

THis function provides a shorthand for writing queries directly to bigquery within R. The input to this function must be a valid query, and the name of the project you are quering from within GCP. To run such a query a .json billing key must be stored on your machine. By default, this function will assume the key is the project name with "_bigquery.json" at the end. If this is not the case, the key name will need to be specified. By default, this function will assume the key is stored in the current working directory or project. If this is not the case, the key must be specified. This function also sets ‘bigint’ to integer64, allowing R to load in integer64 numbers (eg. UPRNs), but it is recommended that you use ‘CAST(VARIABLE as NUMERIC) as VARIABLE’ when running your queries, or converting these to numeric once they have been read in.

sql="SELECT pcds,Rurality FROM `dcms-datalake-staging.GEO_ONS.ONS_RURALITY` LIMIT 1 "

setwd("/home/dcms/keys")

bduk_bq(
      sql=sql,
      project="dcms-datalake-staging"
    )
#> Complete
#> Billed: 0 B
#> Downloading 1 rows in 1 pages.
#> 
Parsing [======================================================================] ETA:  0s
                                                                                         
#> # A tibble: 1 x 2
#>   pcds  Rurality
#>   <chr> <chr>   
#> 1 pcds  Rurality

bduk_bq(
      sql=sql,
      project="dcms-datalake-staging",
      key="dcms-datalake-staging_bigquery.json"
    )
#> Complete
#> Billed: 0 B
#> Downloading 1 rows in 1 pages.
#> 
Parsing [======================================================================] ETA:  0s
                                                                                         
#> # A tibble: 1 x 2
#>   pcds  Rurality
#>   <chr> <chr>   
#> 1 pcds  Rurality

bduk_bq(
      sql=sql,
      project="dcms-datalake-staging",
      keypath="/home/dcms/keys"
    )
#> Complete
#> Billed: 0 B
#> Downloading 1 rows in 1 pages.
#> 
Parsing [======================================================================] ETA:  0s
                                                                                         
#> # A tibble: 1 x 2
#>   pcds  Rurality
#>   <chr> <chr>   
#> 1 pcds  Rurality

sql="SELECT pcds,Rurality FROM `dcms-datalake-staging.GEO_ONS.ONS_RURALITY` LIMIT 1 "

#No key in this directory
setwd("/home/dcms/")

bduk_bq(
      sql=sql,
      project="dcms-datalake-staging"
    )
#> There is no json key saved in this directory. Please copy the file 'dcms-datalake-staging_bigquery.json' into the project or directory you are working within. The file can likely be found in ~/home/keys. If you have not made this folder, please contact the BDUK data and modelling team for additional support.

Spatial functions

geojson_to_sf()

Data in BigQuery is frequently stored in geojson format. This function provides a cimple shorthand to input a data frame or tibble that contains a geojson column (eg. a table read from BigQuery) as easily convert it to a simple feature. The inputs to this function are the data frame or tibble, and the column name of the geojson column as a string.

bq_table<-bduk_bq(
          sql="SELECT * FROM dcms-datalake-staging.GEO_ONS.shp_LA LIMIT 1" ,
          project="dcms-datalake-staging",
          keypath="/home/dcms/keys")
#> Complete
#> Billed: 0 B
#> Downloading 1 rows in 1 pages.
#> 
Parsing [======================================================================] ETA:  0s
                                                                                         

geojson_to_sf(
        data=bq_table,
        geojson_colname = "geom"
      )
#> Simple feature collection with 1 feature and 10 fields
#> geometry type:  POLYGON
#> dimension:      XY
#> bbox:           xmin: 440052.7 ymin: 525456.8 xmax: 454455 ymax: 537152
#> CRS:            4326
#> Warning in st_is_longlat(x): bounding box has potentially an invalid value range for
#> longlat data
#> # A tibble: 1 x 11
#>                        geom OBJECTID LAD20CD LAD20NM LAD20NMW  BNG_E  BNG_N  LONG   LAT
#>               <POLYGON [°]>  <int64> <chr>   <chr>   <chr>    <int6> <int6> <dbl> <dbl>
#> 1 ((448973.6 536745.3, 448…        1 E06000… Hartle… <NA>     447160 531474 -1.27  54.7
#> # … with 2 more variables: Shape__Are <dbl>, Shape__Len <dbl>

make_sf()

This function provides an easy shorthand to take any dataframe with coordinate columns, and convert it into a simple feature object. The inputs to this function are the data frame or tibble, the x and y column names, both as strings, and the coordinate system number. Coordinate reference system codes can be found at https://spatialreference.org/. This mostly serves as a wrapped to sf::st_as_sf(), but the defaults for this are set as latitude and longitude. Columns containing the string “lon” or “lat” in any case being used as the x_colname and y_colname respectively, and the coordinate reference system set to 4326. If multiple columns contain these strings, the function will return the data frame unchanged and produce an error message prompting you to specify the column names manually.

make_sf(
       data=data.frame(
          "Longitude"=c(1,2,3),
          "Latitude"=c(51,52,53)
          ),
       x_colname="Longitude",
       y_colname="Latitude",
       crs=4326
    )
#> Simple feature collection with 3 features and 0 fields
#> geometry type:  POINT
#> dimension:      XY
#> bbox:           xmin: 1 ymin: 51 xmax: 3 ymax: 53
#> CRS:            EPSG:4326
#>       geometry
#> 1 POINT (1 51)
#> 2 POINT (2 52)
#> 3 POINT (3 53)

make_sf(
       data=data.frame(
          "Longitude"=c(1,2,3),
          "Latitude"=c(51,52,53)
          )
    )
#> Simple feature collection with 3 features and 0 fields
#> geometry type:  POINT
#> dimension:      XY
#> bbox:           xmin: 1 ymin: 51 xmax: 3 ymax: 53
#> CRS:            EPSG:4326
#>       geometry
#> 1 POINT (1 51)
#> 2 POINT (2 52)
#> 3 POINT (3 53)

make_sf(
      data=data.frame(
        "Longitude_1"=c(1,2,3),
        "Longitude_2"=c(11,12,14),
        "Latitude"=c(51,52,53)
      )
    )
#> Error: Multiple columns contain 'lon', please specify the x_colname.
#>   Longitude_1 Longitude_2 Latitude
#> 1           1          11       51
#> 2           2          12       52
#> 3           3          14       53

Shiny functions

clearGroups()

A function to clear all groups from a leaflet object from within a shiny application. The prevents having to run clearGroup("groupname") for every group. The input to this is the name of the map, and then the groups wanting to be cleared as comma seperated strings.

library(shiny)
library(leaflet)
library(dplyr)

points_1<-data.frame("lng"=c(-2,-1,0),"lat"=c(51,52,53))
points_2<-data.frame("lng"=c(-2.1,-1.1,-0.1),"lat"=c(51,52,53))

ui<-fluidPage(
  fluidRow(
    actionButton("addgroups", "Add Groups"),
    actionButton("cleargroups", "Clear Groups"),
    leafletOutput("mymap")
  )
)

server<-function(input,output,session){

  output$mymap <- renderLeaflet({
    leaflet(options = leafletOptions(preferCanvas = TRUE)) %>%
      addTiles(options = tileOptions(opacity = 0.8), group = "Open Street Map") %>%
      setView(lng = -1, lat = 52, zoom = 7)
  })

  observeEvent(input$addgroups,{
    leafletProxy("mymap")%>%
      addCircles(data=points_1,lng=~lng,lat=~lat,color="Red",group="Group 1") %>%
      addCircles(data=points_2,lng=~lng,lat=~lat,color="Blue",group="Group 2")
  })

  observeEvent(input$cleargroups,{
    clearGroups(map="mymap","Group 1","Group 2")
  })
}

shinyApp(ui, server)

radioTooltip()

By default, shiny and shinyBS together allow us to add popup tooltips to shiny buttons and input fields. However, this tooltip will apply to all parts of the input fiels. For radio buttons and groups of checkboxes, this can be problematic. This function allows us to add a tooltip to specific parts of a radio button.

library(shiny)
library(leaflet)
library(dplyr)

ui<-fluidPage(
  fluidRow(
    radioButtons(
      "radio",
      "Radio Button",
      choices=c("Tooltip shows on mouseover here",
                "Tooltip shows on mouseover here as well",
                "Tooltip does not show on mouseover here")
                ),
    radioTooltip(
      id="radio",
      title="Tooltip message appears like this",
      choice=c(
        "Tooltip shows on mouseover here",
        "Tooltip shows on mouseover here as well"
      ),
      placement="right",
      trigger="hover"
    )
  )
)

server<-function(input,output,session){}

shinyApp(ui, server)

(function () { var script = document.createElement("script"); script.type = "text/javascript"; script.src = "https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; document.getElementsByTagName("head")[0].appendChild(script); })();

SamA-DS/Rbduk documentation built on May 1, 2021, 5:29 a.m.