In weecology/NeonTreeEvaluation_package: Evaluate Individual Tree Segmentation Algorithms against Benchmark Data

TRAVIS <- !identical(tolower(Sys.getenv("TRAVIS")), "true")
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "www/README-",
  warning = FALSE,
  purl = TRAVIS,
  message = FALSE
)

A multi-sensor benchmark dataset for detecting individual trees in airborne RGB, Hyperspectral and LIDAR point clouds

Maintainer: Ben Weinstein - University of Florida.

This benchmark dataset contains a consistent annotation approach across a variety of ecosystems.

Installation

library(devtools)
install_github("weecology/NeonTreeEvaluation_package")

Download sensor data

To download evaluation data from the Zenodo archive (1GB), use the download() function to place the data in the correct package location. Download the much larger training data, set training=TRUE.

library(NeonTreeEvaluation)
download()

Getting Started

The package contains two vignettes. The 'Data' vignette describes each datatype and how to interact with it in R. The 'Evaluation' vignette shows how to submit predictions to the benchmark.

Submission Format

CSV bounding boxes

The format of the submission is as follows

• A csv file
• 5 columns: Plot Name, xmin, ymin, xmax, ymax

Each row contains information for one predicted bounding box.

The plot column should be named the same as the files in the dataset without extension (e.g. SJER_021 not SJER_021.tif) and not the full path to the file on disk. Not all evaluation data are available for all plots. Functions like evaluate_field_crowns and evaluate_image_crowns will look for matching plot name and ignore other plots.Depending on the speed of the algorithm, the simplest thing to do is predict all images in the RGB folder (see list_rgb()) and the package will handle matching images with the correct data to the correct evaluation procedure.

For a list of NEON site abbreviations: https://www.neonscience.org/field-sites/field-sites-map

Example

The package contains a sample submission file.

library(raster)
library(dplyr)
library(NeonTreeEvaluation)
head(submission)

Shp Polygons

Instead of bounding boxes, some methods may return polygons. To submit as polygons, create a single unprojected shapefile with polygons in image coordinates. Polygons must be complete with no holes. Here is an example of the above csv file in polygon format. Here the xmin, xmax, etc. columns are ignored since the information is stored in the geometry data.

head(submission_polygons)

Scores for an image-annotated crowns

The main data source are image-annotated crowns, in which a single observer annotated visible trees in 200 40m x 40m images from across the United States. This submission has bounding boxes in image coordinates. To get the benchmark score image-annotated ground truth data.

#Get a three sample plots to run quickly, ignore to run the entire dataset
df<-submission %>% filter(plot_name %in% c("SJER_052_2018","TEAK_061_2018","TEAK_057_2018"))

#Compute total recall and precision for the overlap data
results<-evaluate_image_crowns(predictions = df,project = T, show=F, summarize = T)
results

This returns the proportion of the recall and precision for boxes with intersection-over-union scores of greater than 0.4. It also returns the count error, defined as the difference between total number of trees in a plot and the predicted number of trees in a plot, divided by the total.

Scores for an field-annotated crowns

The second data source is a small number of field-annotated crowns from two geographic sites. These crowns were drawn on a tablet while physically standing in the field, thereby reducing the uncertainty in crown segmentation.

df <- submission %>% filter(plot_name=="MLBS_12_competition")
results<-evaluate_field_crowns(predictions = df,project = T)
results

The field-annotated crowns are spread out over the site. There are often only one or two in a single image. This is why algorithms are only scored on recall, the proportion of crowns correctly predicted, and not precision, since there are hundreds of predictions, most of them valid, that do not match any ground truth data.

Scores for an field-collected stems

The third data source is the NEON Woody Vegetation Structure Dataset. Each tree stem is represented by a single point. This data has been filtered to represent overstory trees visible in the remote sensing imagery.

df<-submission %>% filter(plot_name %in% c("JERC_049_2018"))
results<-evaluate_field_stems(predictions = df,project = T, show=T, summarize = T)
results

Predicted crowns in red are compared to field stems marking individual trees. Matching stems are in yellow, missing stems are in blue. Recall is the proportion of field collected stems correctly predicted. Not all trees in an image are field mapped.

If you would prefer not to clone this repo, a static version of the benchmark is here: https://zenodo.org/record/3723357#.XqT_HlNKjOQ

Sensor Data

RGB Camera

library(raster)
library(NeonTreeEvaluation)

#Read RGB image as projected raster
rgb_path<-get_data(plot_name = "SJER_021_2018",type="rgb")
rgb<-stack(rgb_path)

#Find path and parse
xmls<-get_data("SJER_021_2018",type="annotations")
annotations<-xml_parse(xmls)
#View one plot's annotations as polygons, project into UTM
#copy project utm zone (epsg), xml has no native projection metadata
xml_polygons <- boxes_to_spatial_polygons(annotations,rgb)

plotRGB(rgb)
plot(xml_polygons,add=T)

Lidar

To access the draped lidar hand annotations, use the "label" column. Each tree has a unique integer.

library(lidR)
path<-get_data("TEAK_052_2018",type="lidar")
r<-readLAS(path)
trees<-lasfilter(r,!label==0)
plot(trees,color="label")

We elected to keep all points, regardless of whether they correspond to tree annotation. Non-tree points have value 0. We highly recommend removing these points before predicting the point cloud. Since the annotations were made in the RGB and then draped on to the point cloud, there will naturally be some erroneous points at the borders of trees.

Hyperspectral

Hyperspectral surface reflectance (NEON ID: DP1.30006.001) is a 426 band raster covering visible and near infrared spectrum.

path<-get_data("MLBS_071_2018",type="hyperspectral")
g<-stack(path)
nlayers(g)
#Grab a three band combination to view as false color
f<-g[[c(52,88,117)]]
plotRGB(f,stretch="lin")

Submission

To submit to this benchmark, please see the evaluation vignette. Please submit a pull request, or contact the maintainer if you use these data in analysis and would like the results to be shown here.

weecology/NeonTreeEvaluation_package documentation built on Aug. 27, 2024, 10:53 a.m.