```{css, echo=FALSE} body, .main-container {max-width: 100%; margin-left: auto; margin-right: auto;}

```r
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
options(rmarkdown.html_vignette.check_title = FALSE)
library(wastdr)
library(sparkline)
drake::loadd("wastd_data")

x <- wastd_data %>% 
  wastdr::filter_wastd_turtledata(area_name = params$area_name)

area_id <- wastd_data$sites %>% 
  dplyr::filter(area_name==params$area_name) %>% 
  magrittr::extract2("area_id") %>% 
  magrittr::extract(1)

rt <- . %>% 
  reactable::reactable(
    ., 
    filterable = T, 
    sortable = T, 
    searchable = T,
    defaultColDef = reactable::colDef(html = TRUE)
  )

zipfn <- wastdr::urlize(params$area_name)
fn <- here::here("inst/reports", zipfn)
if (!fs::file_exists(fn)) {
  "Creating dir {fn}" %>% glue::glue() %>% wastdr::wastdr_msg_info()
  fs::dir_create(fn, recurse = TRUE)
}

Overview

This top level summary is exported both as just_the_data_thanks.csv and just_the_data_thanks.rds. The latter can be opened directly in R through readRDS("just_the_data_thanks.rds"). It shows totals of tracks, disturbance / predation, live sightings and strandings plus track tallies grouped and sorted per area, site, and calendar date (AWST). Individually encountered tracks are split up by nesting success. Track tallies, which are only collected in specific places like Cape Domett, are counted but the number of tracks therein is not further expanded.

A very detailed explanation of the columns in this table is hosted here.

jtd <- summarise_wastd_data_per_day_site(x) 
jtd %>% saveRDS(here::here(fn, "just_the_data_thanks.rds"))
jtd %>% readr::write_csv(here::here(fn, "just_the_data_thanks.csv"))
jtd %>% janitor::clean_names(case="sentence") %>% rt()

This report is a story about data. It explains, visualises, and rudimentarily summarises the data which is exported to CSV and various other formats. The audience of this report are data analysts and modellers who need to consume and understand the exported data. Suggestions and ideas for additional summaries or visualisations are welcome here.

x %>% wastdr::export_wastd_turtledata(outdir = fn, filename = zipfn, zip=FALSE)
# readRDS(fs::path(fn, "wastd_data.rds")) to restore x

Data sources

Data for this summary is supplied from WAStD, and is manipulated using helpers from the R package wastdr. Analysts are welcome to utilise wastdr or contribute ideas back as feature requests.

The raw data is exported into separate CSV files. The content of the individual CSV files is documented at wastdr::export_wastd_turtledata.

Additionally, an R object of class wastd_data containing all the data objects is saved to wastd_data.RData which can be read into R directly. All code in this workbook can be reproduced by running the code shown inline with the data from wastd_data.RData.

In this report, we filter the data to the location r params$area_name.

The data was downloaded from WAStD on r wastd_data$downloaded_on.

Training and test data is marked as species "Hatchback turtle (Corolla corolla)", surveys are "production=FALSE", disturbance causes are "training or test record". Where summaries hide the columns "species" and "production", respectively, training and test data are excluded.

Turtle nesting

The main caveat here is that some turtle monitoring programs monitor turtle nesting via the morning track counts, others conduct additional night time tagging.

As of now, tagging data is handled in another database, and an export of that database has to be manually edited into the track count data here.

The report here shows only data from track counts. This includes the missed tracks during turtle tagging.

This means:

This caveat will be resolved once the tagging database is fully synchronised with WAStD, and WAStD therefore contains all tagging records, which then are added to the descriptive statistics shown in this report.

Turtle Nesting Current Season

tr <- x$tracks %>% 
  wastdr::filter_wastd_season(params$season) %>% 
  wastdr::filter_realspecies() 

tr %>% wastdr::map_tracks(sites = x$sites)

For performance, we include the same map with tracks clustered by species. On slow machines with large datasets, this map will perform better than the above map with individual records.

tr %>% wastdr::map_tracks(sites = x$sites, cluster = TRUE)
tr %>% 
  dplyr::group_by(calendar_date_awst, species) %>% 
  dplyr::tally() %>% 
  dplyr::ungroup() %>% 
  tidyr::pivot_wider(names_from="species", values_from = "n", values_fill = 0) %>% 
  janitor::clean_names(case="sentence") %>% 
  gt::gt()

QA: tracks outside of known sites

tracks_nosite <- x$tracks %>% 
  wastdr::filter_realspecies() %>% 
  dplyr::filter(is.na(site_name)) %>% 
  dplyr::mutate(wastd=glue::glue("https://wastd.dbca.wa.gov.au{absolute_admin_url}"))

tracks_nosite %>% wastdr::map_tracks(sites = x$sites)
tracks_nosite %>% rt()

Missed tracks

This section is relevant for monitoring programs which tag turtles.

Missed tracks come from turtles which got away just before or during tagging. These records can be used to inform tagging start and end times:

Missed tracks recorded at the beginning of a tagging night means that tagging needs to start earlier in the day. Missed tracks recorded during tagging indicates that tagging beaches are understaffed.

Missed tracks are contained in the exported tracks.csv with nest_age "missed".

mt <- x$tracks %>% 
  wastdr::filter_wastd_season(params$season) %>% 
  wastdr::filter_realspecies() %>% 
  dplyr::filter(nest_age == "missed")

# mt %>% rt
# mt %>% gt::gt()
mt %>% wastdr::map_tracks(sites = x$sites)

Processed turtles

Encounters with living animals (training species excluded) in the current season include processed animals.

We will attach tag history shortly, but for now only show the raw encounters.

pt <- x$animals %>% 
  wastdr::filter_wastd_season(params$season) %>% 
  wastdr::filter_realspecies() %>% 
  wastdr::filter_alive()

pt %>% 
  dplyr::group_by(calendar_date_awst, species) %>% 
  dplyr::tally() %>% 
  dplyr::ungroup() %>% 
  tidyr::pivot_wider(names_from="species", values_from = "n",values_fill = 0) %>% 
  janitor::clean_names(case="sentence") %>% 
  gt::gt()
pt %>% map_mwi(sites = x$sites)

Tags

Tags (flipper/PIT) and biopsies encountered together with processed animals (re-sighted, applied new, etc) are listed here for the current season.

tags <- x$turtle_tags %>% 
  wastdr::filter_wastd_season(params$season) %>% 
  wastdr::filter_realspecies() %>% 
  dplyr::mutate(wastd=glue::glue("<a href=\"https://wastd.dbca.wa.gov.au{encounter_absolute_admin_url}\">{encounter_name}</a>"))

tags %>%
  dplyr::select(
    calendar_date_awst,
    tag_type,
    tag_location,
    name,
    status,
    encounter_reporter_name,
    wastd
  ) %>%
  janitor::clean_names(case="sentence") %>% 
  rt()

Disturbance, predation: nests and general

General disturbance and signs of predator presence is captured through the ODK form "Predator or Disturbance". Disturbance or predation of nests are captured through the ODK form "Turtle Track or Nest". As the data specific to the disturbance or predation is identical, both are presented here as one table.

In the exported nest_dist.csv, the encounter_type "nest" refers to disturbances or predation of turtle nests, whereas "other" refers to sightings of general disturbance or presence of predators.

x$nest_dist %>% 
  wastdr::filter_wastd_season(params$season) %>%
  wastdr::map_dist()

Disturbance causes

The following interactive table tallies both nest and general ("other") disturbance by season. Filter this table to inspect disturbance causes.

You can also select any combination of disturbance causes in the map above to inspect their spatial distribution across the studied area.

The numbers shown are absolute numbers and not normalised for survey effort or extent of surveyed area.

x$nest_dist %>% 
  wastdr::disturbance_by_season() %>% 
  dplyr::arrange(-season, -n) %>% 
  # dplyr::filter(n>1) %>% 
  rt()

Disturbance near nests

The following map show general disturbance or predator signs within 5m of nests and within 60 days of the nesting event. This translates roughly to disturbances that could have impacted nesting success and excludes disturbances seen before the nest was laid or after the end of the incubation period.

require(sf)
# GDA2020 MGA Zone SRIDS: https://epsg.io/?q=gda2020+zone
# Zone 49: 7849 # NIN
# Zone 50: 7850 # THV, KAR, DEL
# Zone 51: 7851 # PTH, BRO
# Zone 52: 7852 # CDO
wastd_crs <- 4326
local_crs <- 7850 # TODO select by area coordinates

tracks_gda2020 <- x$tracks %>%
  dplyr::filter(
    nest_type %in% c("successful-crawl", "nest", "track-unsure")) %>%
  dplyr::mutate(
    nest_laid = lubridate::ymd(calendar_date_awst),
    expected_hatching = nest_laid + lubridate::days(60)
  ) %>%
  sf::st_as_sf(coords = c("longitude", "latitude"), crs = wastd_crs) %>%
  sf::st_transform(crs = local_crs)
  # sf::st_buffer(dist = 5)

# sf::st_crs(tracks_gda2020) # Tracks are in GDA2020 MGA Zone 51 now
# mapview::mapview(tracks_gda2020)

dist_gda2020 <- x$nest_dist %>%
  dplyr::filter(encounter_encounter_type == "other") %>%
  dplyr::mutate(disturbance_date = lubridate::ymd(calendar_date_awst)) %>%
  sf::st_as_sf(coords = c("encounter_longitude", "encounter_latitude"),
               crs = wastd_crs) %>%
  sf::st_transform(crs = local_crs)

dist_near_nests <- dist_gda2020 %>%
  sf::st_join(tracks_gda2020, join = st_is_within_distance, dist = 5) %>%
  dplyr::filter(disturbance_date > nest_laid, 
                disturbance_date < expected_hatching) %>% 
  dplyr::select(
    encounter_id,
    encounter_reporter_name,
    encounter_leaflet_title,
    encounter_absolute_admin_url,
    encounter_name,
    disturbance_date,
    disturbance_cause,
    disturbance_severity,
    comments.x,
    comments.y,
    nest_type.y,
    species.y,
    nest_type_text.y,
    disturbance_date,
    geometry,
    leaflet_title,
    species_colours.y,
    nest_type_text.y,
    nest_laid,
    expected_hatching,
    season.x
  )

if (nrow(dist_near_nests) > 0) {
    mapview::mapview(dist_near_nests, zcol="disturbance_cause")
    # dplyr::filter(disturbance_cause == "vehicle", season.x==2020) %>% 
}

Turtle tracks and nests season totals

Tracks and nests are tallied by type and grouped by

To see only missed tracks, type "mis" into the filter box of column "(details_)nest_age".

This summary is exported as nesting_type_by_season_age_species.csv.

wastd_nesting_type_by_season_age_species <-
  x$tracks %>% 
  wastdr::filter_realspecies() %>% 
  wastdr::nesting_type_by_area_season_age_species() %>% 
  dplyr::arrange(-season)

wastd_nesting_type_by_season_age_species %>%
  readr::write_csv(here::here(fn, "nesting_type_by_season_age_species.csv"))

wastd_nesting_type_by_season_age_species %>% 
  # janitor::clean_names(case="sentence") 
  dplyr::select(
    area_name,
    season,
    species,
    nest_age,
    dplyr::contains("-"),
    dplyr::contains("nest")
  ) %>% 
  gt::gt(
    groupname_col=NULL,
    caption = "Nesting season totals"
  ) %>% 
  gt::tab_header(
    title="Seasonal nesting totals",
    subtitle="Nesting by rookery, species, nest age tallied by nesting success"
  ) %>%
  gt::cols_label(
    area_name = "Rookery",
    season = "Season",
    species = "Species",
    nest_age = "Nest age"
  )

Turtle tracks and nests daily totals

The same data, but grouped by day instead of season. This summary is exported as nesting_type_by_day_age_species.csv.

The data are shown as an interactive table with search, filtering, and sorting enabled. The table can be used to inspect nesting activity on individual days.

wastd_nesting_type_by_day_age_species <-
  x$tracks %>% 
  wastdr::filter_realspecies() %>% 
  wastdr::nesting_type_by_season_calendarday_age_species() %>% 
  dplyr::arrange(-season)

wastd_nesting_type_by_day_age_species %>%
  readr::write_csv(here::here(fn, "nesting_type_by_day_age_species.csv"))

wastd_nesting_type_by_day_age_species %>% rt

Night recaptures by species year and total

This section will be completed once the tagging database has been merged into WAStD.

Annual new turtles by species year and total

This section will be completed once the tagging database has been merged into WAStD.

Annual recaptures by species year and total

This section will be completed once the tagging database has been merged into WAStD.

Annual total turtles (individuals) by species year and total

This section will be completed once the tagging database has been merged into WAStD.

Annual total turtle emergences

Fresh tracks and nests are combined and tallied, grouped by season and species to form the total number of emergences in one season.

Older signs of nesting, such as incubating and hatched nests, as well as body pits are omitted from the tally.

Caveats:

x$tracks %>% 
  wastdr::exclude_training_species() %>% 
  dplyr::filter(
    nest_type %in% c(
      "successful-crawl",
      "false-crawl",
      "track-not-assessed",
      "track-unsure"
    )
  ) %>% 
  dplyr::group_by(season, species) %>%
  dplyr::tally() %>%
  dplyr::ungroup() %>%
  dplyr::arrange(-season, -n) %>%
  janitor::clean_names(case="sentence") %>% 
  gt::gt(
    groupname_col=NULL,
    caption = "Total turtle emergences"
  ) %>% 
  gt::tab_header(
    title="Seasonal emergence totals",
    subtitle="Emergences tallied by rookery, season, and species"
  ) %>%
  gt::cols_label(
    N = "Emergences"
  ) 

Number of nests tagged

Tagged nests are exported as individual records of encounters with tagged nests (including initial tagging and re-sighting).

We show the season tallies, and export the individual records as nest_tags.csv.

wastd_tagged_nests <- x$nest_tags %>% 
  wastdr::exclude_training_species()
wastd_tagged_nests %>% 
  dplyr::group_by(season, species) %>% 
  dplyr::tally() %>% 
  dplyr::ungroup() %>% 
  dplyr::arrange(-season) %>%
  janitor::clean_names(case="sentence") %>% 
  gt::gt(
    groupname_col=NULL,
    caption = "Total nest tags"
  ) %>% 
  gt::tab_header(
    title="Total nest tags",
    subtitle="Tally includes any encounter of a nest tag: deployment, re-sighting, and recovery"
  ) %>%
  gt::cols_label(
    N = "Tag encounters"
  ) 

Number of nests found

The data here is the same as above, nest type "successful-crawl" (confirmed fresh nest) and "hatched nest".

This summary is exported as fresh_and_hatched_nests.csv.

wastd_nests <- x$tracks %>% 
  wastdr::exclude_training_species() %>% 
  dplyr::filter(nest_type %in% c("successful-crawl", "hatched-nest")) %>% 
  dplyr::group_by(season, species, nest_type) %>% 
  dplyr::tally() %>% 
  dplyr::ungroup() %>% 
  dplyr::arrange(-season)

wastd_nests %>%
  readr::write_csv(here::here(fn, "fresh_and_hatched_nests.csv"))

wastd_nests %>% 
  dplyr::arrange(-season, -n) %>% 
  # dplyr::mutate(species = wastdr::humanize(species)) %>%
  janitor::clean_names(case="sentence") %>% 
  gt::gt(
    groupname_col=NULL,
    caption = "Total nests found"
  ) %>% 
  gt::tab_header(
    title="Total nests found",
    subtitle="Nest encounters grouped by season, species, and nest type"
  ) %>%
  gt::cols_label(
    N = "Nest encounters"
  ) 

Nest loggers

Nest loggers include both initial deployment and any recorded re-sighting. Multiple loggers can be associated with one nest.

For want of a better overview, we show an interactive table of all loggers here. The table supports search, filtering, and ordering.

The nest logger records are also exported as nest_loggers.csv.

x$nest_loggers %>% rt

Hatching and emergence success

Nest excavations are shown here as statistics grouped by season and species and exported as individual nest excavation records.

This summary is exported as nest_excavations_hatching_emergence_success.csv.

nest_excav <- x$nest_excavations %>%
  wastdr::hatching_emergence_success() %>% 
  dplyr::arrange(-season)

nest_excav %>% 
  readr::write_csv(
    here::here(fn, "nest_excavations_hatching_emergence_success.csv"))

nest_excav %>% 
  dplyr::arrange(-season, species) %>% 
  dplyr::mutate(species = janitor::make_clean_names(species, case="sentence")) %>% 
  janitor::clean_names(case="sentence") %>% 
  gt::gt(
    groupname_col=NULL,
    caption = "Hatching and emergence success"
  ) %>% 
  gt::tab_header(
    title="Hatching and emergence success",
    subtitle="Descriptive statistics from all excavated nests"
  )

Fan angle data

Fan angle data is exported as nest_fans.csv and previewed in the following table. More meaningful visualisations are in preparation.

x$nest_fans %>% 
  janitor::clean_names(case="sentence") %>% 
  rt

Tag return crossovers from or to by year

This section will be completed once the tagging database has been merged into WAStD.

Marine Wildlife Incidents

MWI include strandings, rescues, and live sightings of turtles and other marine megafauna.

Live sightings: rescues, in water, tagging

The ODK form "Turtle Sighting" captures turtles encountered during boat based surveys. The ODK form "Marine Wildlife Incident" also captures encounters with injured, entangled, rescued, but overall alive animals (including turtles). The ODK form "Turtle Tagging", as well as data import from the legacy turtle tagging database WAMTRAM, will show up as a live sighting with a nesting activity.

All of the above forms feed into WAStD AnimalEncounters, which are shown in the map below.

Current season data only.

x$animals %>% 
  wastdr::filter_wastd_season(params$season) %>% 
  wastdr::filter_alive() %>% 
  wastdr::map_mwi(sites = x$sites)

Dead sightings: strandings, mortalities

The ODK form "Marine Wildlife Incident" captures encounters with dead animals (including turtles), which then become WAStD AnimalEncounters.

Current season data only.

x$animals %>% 
  wastdr::filter_wastd_season(params$season) %>% 
  wastdr::filter_dead() %>% 
  wastdr::map_mwi(sites = x$sites)

Surveys

A summary of past seasons follows. Note the dates are "turtle dates" (calendar date of nesting night's evening).

This summary is exported as survey_season_stats.csv.

survey_season_stats <- x$surveys %>% 
  wastdr::survey_season_stats() 

survey_season_stats %>% 
  readr::write_csv(here::here(fn, "survey_season_stats.csv"))

survey_season_stats %>% 
  janitor::clean_names(case="sentence") %>% 
  gt::gt(
    groupname_col=NULL,
    caption = "Seasonal overview of survey effort"
  ) %>% 
  # gt::fmt_date(c("First day", "Last day")) %>% # Friday, October 20, 2017
  gt::tab_header(
    title="Seasonal overview of survey effort",
    subtitle="Survey stats grouped by rookery and season"
  )

The following summary splits up the above table by sites and is exported as survey_season_site_stats.csv.

The table is interactive and supports search, filtering and ordering.

survey_season_site_stats <- x$surveys %>% 
  wastdr::filter_realsurveys() %>% 
  wastdr::survey_season_site_stats()

survey_season_site_stats %>% 
  readr::write_csv(here::here(fn, "survey_season_site_stats.csv"))

survey_season_site_stats %>% 
  janitor::clean_names(case="sentence") %>% 
  rt

QA: Surveys with start points outside the nominated site

QA: Surveys with end points outside the nominated site

QA: Backwards Surveys

Incorrectly reconstructed Surveys can run backwards, i.e. with a start time past the end time.

Saving a "backwards" Survey in WAStD will automatically adjust the end time to 6h past the start time, then adopt Encounters within that time span. Saving the Survey again will set the end time to 30 min past the last adopted Encounter.

After QA, this list should be empty.

x$surveys %>%
  dplyr::filter(start_time > end_time) %>%
  dplyr::select(
    site_name,
    reporter,
    start_time,
    end_time,
    is_production,
    id
  ) %>%
  janitor::clean_names(case="sentence") %>% 
  dplyr::mutate(
    Edit = glue::glue(
        "<a href=\"https://wastd.dbca.wa.gov.au/",
        "admin/observations/survey/{Id}/change/\" target=\"_\">Edit</a>"
      ),
    View = glue::glue(
        "<a href=\"https://wastd.dbca.wa.gov.au/",
        "observations/surveys/{Id}\" target=\"_\">View</a>"
      )
  ) %>% 
  rt()

QA: Duplicate morning surveys

The following tables shows days and sites for which more than one production (= real, not training or duplicate) survey exists in WAStD. Typically, only one of them should be a "production" survey, and all other surveys could be marked and closed as duplicates.

Morning surveys are expected to start after 05:00 and before 15:00. More than one morning survey on the same site indicates a possible duplicate.

morning_survey_start_hour <- 5
night_survey_start_hour <- 15

duplicate_morning_surveys <- x$surveys %>% 
  dplyr::filter(lubridate::hour(start_time) > morning_survey_start_hour,
                lubridate::hour(start_time) < night_survey_start_hour) %>% 
  wastdr::duplicate_surveys()

duplicate_morning_surveys %>% 
  dplyr::arrange(-season, site_name) %>% 
  dplyr::select(-site_id) %>% 
  janitor::clean_names(case="sentence") %>% rt

QA: Duplicate night surveys

Night surveys are expected to start after 15:00 and before 5:00. More than one night survey on the same site and date indicates a possible duplicate.

duplicate_night_surveys <- x$surveys %>% 
  dplyr::filter(lubridate::hour(start_time) < morning_survey_start_hour,
                lubridate::hour(start_time) > night_survey_start_hour) %>% 
  wastdr::duplicate_surveys()

duplicate_night_surveys %>%
  dplyr::arrange(-season, site_name) %>% 
  dplyr::select(-site_id) %>% 
  janitor::clean_names(case="sentence") %>% rt

QA: Surveys with reconstructed duration of exactly 6 hours

Manually adjust end times as and if appropriate.

rec_surveys <- x$surveys %>% 
  dplyr::filter(
    lubridate::interval(
      start = start_time, 
      end = end_time) == lubridate::hours(6))

rec_surveys %>% 
  dplyr::select(
    season,
    site_name,
    reporter,
    calendar_date_awst,
    start_time,
    end_time,
    is_production,
    change_url
  ) %>% 
  janitor::clean_names(case="sentence") %>% rt

QA: Suspiciously long surveys

Surveys running for more than 12 hours could have been erroneously merged from two separate surveys, or they are truly long surveys.

Fixing these requires individual inspection and full fluency in WAStD's operation.

These can capture encounters from two separate surveys. Survey effort can be biased by showing only one survey where two have been conducted, and by showing the extra hours between actual fieldwork as survey duration.

Protocol:

long_surveys <- x$surveys %>% 
  dplyr::filter(
    lubridate::interval(
      start = start_time, 
      end = end_time) > lubridate::hours(12))

long_surveys %>%
  dplyr::select(
    season,
    site_name,
    reporter,
    calendar_date_awst,
    start_time,
    end_time,
    is_production,
    change_url
  ) %>% 
  janitor::clean_names(case="sentence") %>% 
  rt()

Personnel

The number of surveys lead by each volunteer is tallied here for each season.

If user mapping during data import has matched the wrong WAStD user, this table offers a link to the "Transfer data" form, where all records against one \ (incorrectly matched) WAStD user can be transferred within a rookery (pre-filled to r params$area_name) to another WAStD user (chosen by the QA operator).

x$surveys %>% 
  dplyr::group_by(season, reporter, reporter_id) %>% 
  dplyr::tally(name = "Surveys") %>% 
  dplyr::ungroup() %>% 
  dplyr::arrange(-season) %>%
  dplyr::mutate(
    "Transfer" = glue::glue(
        "<a href=\"https://wastd.dbca.wa.gov.au/users/",
        "{reporter_id}/transfer-at/{area_id}\" target=\"_\">Transfer data</a>"
      ),
  ) %>% 
  dplyr::select(-reporter_id) %>% 
  janitor::clean_names(case="sentence") %>% 
  rt


dbca-wa/etlTurtleNesting documentation built on Nov. 18, 2022, 8:03 a.m.