In dbca-wa/etlTurtleNesting: Data ETL of Turtle Nesting Census Data from ODK Central to WAStD

```{=html}

```r
knitr::opts_chunk$set(echo = TRUE)
library(etlTurtleNesting)
library(wastdr)
library(drake)

readd(odkc_ex_2019)
readd(odkc_tf_2019)
loadd(odkc_ex_2019)
loadd(odkc_tf_2019)

Currency

• This QA report ran on r Sys.time().
• The ODK Central data was downloaded on r odkc_ex_2019$downloaded_on.
• The WAStD Sites were downloaded together with the ODK Central data on r odkc_ex_2019$downloaded_on.

Export to GIS

WAStD Sites, as well as Site Visit Start and End records are exported here into GeoJSON, which can be viewed in GIS tools like Quantum GIS. SVS and SVE can inform how to adjust WAStD sites to capture and separate survey effort most efficiently.

odkc_ex_2019$svs %>% geojsonio::geojson_write(file = here::here("docs/svs.geojson"))
odkc_ex_2019$sve %>% geojsonio::geojson_write(file = here::here("docs/sve.geojson"))

What should happen

Data captured with ODK Collect is geo-referenced to point locations. This includes the Site Visit Start (SVS) and Site Visit End (SVE) points, turtle tracks (TRACKS), disturbances (DIST), and marine wildlife incidents (MWI).

WAStD stores a list of data capture sites with the polygonal extent of the respective beach and surrounding vegetation.

The data ETL process reconstructs Surveys from matching SVS and SVE points to the Site polygons, matching them by device_id (coming from the same tablet) and calendar date (captured on the same day). Where SVE were forgotten (or ended up across a site boundary and in the wrong site), surveys have no end time. Where SVS were forgotten, no Surveys are created and uploaded to WAStD.

WAStD reconstructs the Location and Site of Encounters like tracks / dist / mwi from matching their respective point locations to Site polygons.

WAStD's Surveys find and link Encounters by Encounter location and time. WASTD finally creates Surveys for orphaned Encounters (where SVS were forgotten), and ends Surveys with reconstructed end times 30 mins after their last Encounter record.

What could go wrong

WAStD Site polygons are inaccurate

Review your sites on the following map.

wastdr::map_sv_odkc(
  svs = odkc_ex_2019$svs,
  sve = odkc_ex_2019$sve,
  sites = odkc_ex_2019$sites
  )

SVS location at wrong Site

• ETL will create a Survey at wrong Site (but shouldn't).
• ETL will create no Survey at correct Site (but should).

If Encounters recorded at Site, a Survey will be reconstructed, but we will lose the SVS details (site photo, team, comments).

If no Encounters are recorded at Site, no Survey will be reconstructed, and confirmed absences / survey effort will not be recorded.

SVS location outside of any Site

• ETL will create no Survey.

If Encounters recorded at Site, a Survey will be reconstructed, but we will lose the SVS details (site photo, team, comments).

If no Encounters are recorded at Site, no Survey will be reconstructed, and confirmed absences / survey effort will not be recorded.

SVS forgotten

• ETL will create no Survey.

If Encounters recorded at Site, a Survey will be reconstructed, but we will lose the SVS details (site photo, team, comments).

If no Encounters are recorded at Site, no Survey will be reconstructed, and confirmed absences / survey effort will not be recorded.

Survey Ends missing

There are several explanations for Surveys without end point details.

SVE forgotten

• ETL will find no SVE for respective SVS.
• WAStD will reconstruct end time as start time plus 6 hours.
• If Encounters exist, WAStD will adjust end time to 30 mins after last Encounter.
• We will lose SVE details (photo, comments, precise time).

SVE in wrong Site

• ETL will discard SVE, as it mismatches other SVS's device_id.
• What if same device surveys "wrong" Site before or after surveying intented Site?

SVE outside of any Site

• ETL will find no SVE for respective SVS.
• WAStD will reconstruct end time as start time plus 6 hours.
• If Encounters exist, WAStD will adjust end time to 30 mins after last Encounter.
• We will lose SVE details (photo, comments, precise time).

Tablet switch between SVS and SVE

Reasons:

• Tablet out of battery, overheated, malfunctioning, or destroyed, use backup tablet
• Multi tablet survey

Causes:

• SVE and SVS within same site but different device_id between SVS and SVE.
• Since we match SVE to SVS by date, Site and device_id, we will lose the SVE of "switchers".

How can we tell (and fix)

SVS location at wrong Site

Map SVS with more than one count of site and calendar day. These are candidates for missing details in reconstructed Surveys.

WAStD would reconstruct a Survey for any Encounters at the intended Site.

Tabulate/map SVS and link to search WAStD for Surveys on that calendar day.

Identify reconstructed Survey in WAStD and backfill SVS details. (How should we do the photo?)

SVS or SVE location outside of any Site

Map SVS and SVE with no site_id.

If they are close to an existing site, it's possible the GPS error placed then just outside that site.

svs_na <- dplyr::filter(odkc_ex_2019$svs, is.na(site_id))
sve_na = dplyr::filter(odkc_ex_2019$sve, is.na(site_id))


if (nrow(svs_na) > 0 && nrow(sve_na) > 0){
wastdr::map_sv_odkc( 
  svs = dplyr::filter(odkc_ex_2019$svs, is.na(site_id)),
  sve = dplyr::filter(odkc_ex_2019$sve, is.na(site_id)), 
  sites = odkc_ex_2019$sites
)} else wastdr::wastdr_msg_noop("No SVS and SVE outside of known sites found.")

Find reconstructed WAStD Survey for that SVS and backfill details.

SVS forgotten

Not recoverable. Reconstructed in WAStD. Nothing to backfill.

SVE forgotten

Not recoverable. Reconstructed in WAStD. Nothing to backfill.

SVE at wrong Site

Map SVE with more than one count of site and calendar day. Identify Surveys with reconstructed end details and backfill details.

SVE outside of any Site

Map SVE with no site_id. Find reconstructed WAStD Survey ends for that SVS and backfill SVE details.

Tablet switch between SVS and SVE

Excluding matched SVE, match remaining SVE to SVS by only calendar date and Site. Find Survey (source_id) by SVS device_id in WAStD, backfill details.

Surveys without endpoints:

odkc_tf_2019$surveys %>% 
  dplyr::filter(is.na(end_source_id)) %>% 
  reactable::reactable(filterable = TRUE, sortable = TRUE, searchable = TRUE)

dbca-wa/etlTurtleNesting documentation built on Nov. 18, 2022, 8:03 a.m.