In NOAA-OWP/hydrofabric: Easily Install and Load 'hydrofabric' tools

knitr::opts_chunk$set(echo = TRUE)
library(hydrofabric)
library(DBI)
library(RSQLite)
library(mapview)

We are now at the point where we have a common understanding of how hydrofabrics are sourced, manipulated for a given modeling task, and generated on a VPU basis.

This aim of this section is that each of us can build a subset network for a location of interest. Doing this requires an understanding of the following:

1. The CONUS network
2. Finding an Origin
3. The R based hfsubsetR::get
4. The CLI utility for micro service subsets.
5. RESTFUL API

The CONUS network file

A network file is distributed with each version/type of a hydrofabric. For more on data access patterns see the Data Vignette.

local   <- "/Users/mjohnson/hydrofabric"
s3      <- "s3://lynker-spatial/hydrofabric"
version <-  'v2.1.1'
type    <- "nextgen"
domain  <- "conus"


network_path = glue("{local}/{version}/{type}/conus_network")
net = open_dataset(network_path)

In the above schema you'll see that every relationship between the features in the current hydrofabric, the source hydrofabric, and the conus hydrolocations, have been exploded into a "many-to-many" table.

For example, we can look at the flowpath wb-1002 and find that it is defined by the aggregation of NHDPlusV2 COMID 1712220, 1712230, and 1712238.

glimpse(filter(net, id == "wb-1002"))

Or, that the terminal outflow of 'HUC12-010100100101' occurs at tnx-1000000569 which is fed by an aggregate flowpath made up of three source flowpaths (hf_id={816563, 816417, 816415})

glimpse(filter(net, hl_uri == 'HUC12-010100100101'))

Finding an Origin

By known COMID

findOrigin(network_path, comid = 101)

By known ID

findOrigin(network_path, id = 'wb-2430837')

By location (XY)

here = AOI::geocode("National Water Center, Alabama")

findOrigin(network_path, xy = c(here$x, here$y))

By Hydrolocation URI

# For a gage in Calfornia
findOrigin(network_path, hl_uri = "Gages-11123000")
# For the HUC12 of Atascadero Creek in Santa Barbara:
findOrigin(network_path, hl_uri = "HUC12-180600130201")
# For the dam on Horsetooth Reservoir
findOrigin(network_path, hl_uri = "NID-CO01659-1")

By NLDI Feature

# For a gage in Calfornia
findOrigin(network_path, 
           nldi_feature  = list(featureSource = "nwissite", 
                                featureID = "USGS-05428500"))

hfsubetR

The hfsubsetR library is core module in the NOAA-OWP/hydrofabric suite. It expedites the process of (1) finding an origin (2) traversing the network and (3) extracting the relevant features from the requested.

For example, lets get a basic subset of the network upstream of comid=101 complete with divides, nexus locations and flowlines:

subset101 = get_subset(comid = 101, 
                       lyrs = c("divides", "nexus", "flowlines"),
                       source   = "/Users/mjohnson/hydrofabric",
                       hf_version =  '2.1.1',
                       type    = "nextgen")

mapview::mapview(subset101)

The same request can be appended to request attribute information to enrich the network. For example, we can grab our precomputed divide attributes and the forcing weights needed for NGIAB.

subset101_enhanced = get_subset(comid = 101, 
           lyrs = c("divides", "nexus", "flowlines", "forcing-weights", "model-attributes"),
           source   = "/Users/mjohnson/hydrofabric",
           hf_version =  '2.1.1',
           type    = "nextgen",
           domain  = "conus")

subset101_enhanced$`model-attributes`

subset101_enhanced$`forcing-weights`

REST Service (BETA)

For workflows regardless of programming language, we offer an in-beta REST API. This API wraps the hfsubsetR library to provide the same subsetting capabilities across the web.

Currently, the API offers one endpoint /subset. To query this endpoint, we can use a tool like cURL:

API_URL="https://www.lynker-spatial.com/hydrofabric/hfsubset"
curl -o hydrofabric.gpkg "${API_URL}/subset?identifier=101&identifier_type=comid&version=2.1.1&subset_type=nextgen"

Running the above outputs a GeoPackage subset of the v2.1.1 NextGen hydrofabric, containing the following layers of upstream features from COMID 101:

  layer_name     geometry_type features fields             crs_name
1    divides           Polygon       20     10 NAD83 / Conus Albers
2  flowlines Multi Line String       20     11 NAD83 / Conus Albers
3      nexus             Point        8      6 NAD83 / Conus Albers
4    network                NA       86     18                 <NA>

Additionally, we can also subset forcing weights in two ways:

1. subset the pre-computed NextGen layer
2. compute the weight grid on-demand

For (1), we can accomplish this by explicitly setting the layer query parameter:

curl -o hydrofabric.gpkg "${API_URL}/subset?identifier=101&identifier_type=comid&version=2.1.1&subset_type=nextgen&layer=divides&layer=forcing-weights"

For (2), we can set the weights query parameter:

curl -o hydrofabric.gpkg "${API_URL}/subset?identifier=101&identifier_type=comid&version=2.1.1&subset_type=nextgen&layer=divides&weights=medium_range"

CLI Option

For those interested in using the NOAA NextGen fabric, without directly needing R, or within a non-interactive pipeline, we provide pre-built binaries for a Go-based CLI tool that creates and forwards requests to the REST API, preventing the need to construct URLs and use cURL as in the examples above.

The help output for this tool is as follows:

```{bash, eval = FALSE} hfsubset - Hydrofabric Subsetter

Usage: hfsubset [OPTIONS] identifiers... hfsubset (-h | --help)

Examples: hfsubset -o ./divides_nexus.gpkg \ -r "2.2" \ -t hl \ "Gages-06752260"

hfsubset -o ./poudre.gpkg -t hl "Gages-06752260"

# Using network-linked data index identifiers hfsubset -o ./poudre.gpkg -t nldi "nwissite:USGS-08279500"

# Specifying hydrofabric version and subset type hfsubset -o ./divides_nexus.gpkg -l divides,flowlines,nexus -r "2.1.1" -s "nextgen" -t hl "Gages-06752260"

# Finding data around a coordinate point hfsubset -o ./sacramento_flowlines.gpkg -l flowlines -t xy -121.494400,38.581573

Environment Variables: ${HFSUBSET_ENDPOINT} - Endpoint to use for subsetting, defaults to 'https://www.lynker-spatial.com/hydrofabric/hfsubset/'. Note: the endpoint must end with a trailing slash.

Details: * Finding POI identifiers can be done visually through https://www.lynker-spatial.com/hydrolocations.html

• When using identifier type 'xy', the coordinates are in OGC:CRS84 order, which is the same reference system as EPSG:4326 (WGS84), but uses longitude-latitude axis order rather than latitude-longitude.

• When using identifier type 'nldi', the identifiers follow the syntax

  :

  For example, USGS-08279500 is accessed with featureSource 'nwissite', so this gives the form 'nwissite:USGS-08279500'

Options: -debug Run in debug mode -dryrun Perform a dry run, only outputting the request that will be sent -l string Comma-delimited list of layers to subset. (default "divides,flowlines,network,nexus") -o string Output file name (default "hydrofabric.gpkg") -quiet Disable logging -s string Hydrofabric type, only "reference" is supported (default "reference") -t string One of: "hf", "comid", "hl", "poi", "nldi", or "xy" (default "hf") -v string Hydrofabric version (NOTE: omit the preceeding v) (default "2.2") -verify Verify that endpoint is available (default true) -w string Comma-delimited list of weights to generate over the subset. ```

NOAA-OWP/hydrofabric documentation built on Dec. 7, 2024, 11:24 a.m.