| reject_shots | R Documentation |
Flags the outlying shots (spectra) of each sample of a LIBS dataset, such as shots with a weak or missed plasma, or shots on an inclusion or a contaminated spot, before the shots are averaged.
reject_shots(
data,
sample,
method = c("intensity", "correlation"),
cutoff = 3.5,
wavelength = NULL,
drop = FALSE,
shot = NULL,
scale = c("floor", "sample", "pooled")
)
data |
A data frame with one shot per row: a sample column and spectral columns named by their wavelengths (other columns are kept but not used). |
sample |
The column identifying the sample of each shot, unquoted or as a string. |
method |
The criteria: one or more of |
cutoff |
The robust z-score above which a shot is rejected. Default is 3.5. |
wavelength |
An optional wavelength range (nm) in which the criteria are computed, for example to avoid saturated lines. |
drop |
A logical: return only the kept shots, without the added
columns ( |
shot |
An optional column (unquoted or as a string) giving the order of the shots in each sample, such as the shot number. Default is the order of the rows. |
scale |
The robust scale of the z-scores: |
Each shot is compared with the other shots of its sample, by robust z-scores of one or more criteria:
"intensity": the total intensity of the spectrum, on a log scale (its
deviation is relative to the median shot of the sample). Both weak and
unusually strong shots are flagged.
"correlation": the Pearson correlation r of the spectrum with
the median spectrum of the sample, which detects changes of the shape of
the spectrum (other lines, other line ratios) whatever its intensity.
The z-scores are those of Fisher's z = \tanh^{-1}(r): correlations
near 1 are bounded and skewed, so that small differences such as 0.998
and 0.9995 would otherwise look extreme. Only shots with a low
correlation are flagged.
"distance": the Euclidean distance of the spectrum to the median
spectrum of the sample, relative to the norm of the median spectrum, on
a log scale, which combines both. Only far shots are flagged.
Scale. A z-score is the deviation of a shot from the median of its
sample divided by a robust scale (the MAD). With the few shots of a sample
(often 5 to 10), the MAD of the sample is unstable: when its shots happen
to be very similar, a shot that differs by little gets a large z-score.
With scale = "floor" (default), the scale of a sample is its own MAD, but
not less than the pooled MAD of the deviations of all the samples (the
typical shot-to-shot variability of the data set); "sample" uses the MAD
of each sample alone, and "pooled" the pooled MAD for all samples, which
suits samples of similar variability. In simulations of 8 shots per sample
with noise varying between samples, "floor" with Fisher's z
rejected 0.1% to 0.5% of the regular shots (against 2.5% to 5% with the
MAD of each sample and the raw correlations), and still detected the shots
of a plasma weakened by a quarter or more, or with an extra line of 10% of
the strongest line. Lower cutoff to detect milder deviations.
A shot is rejected when any criterion exceeds cutoff (3.5 by default,
following Iglewicz and Hoaglin, 1993). Samples with fewer than 3 shots are
not checked. When a MAD is zero, the mean absolute deviation is used.
The result keeps the raw criteria (.intensity, the total intensity
relative to the median shot of the sample; .correlation; .distance),
easier to read than the z-scores, and the settings, which
plot_shots() uses to show the shots and the rejections. To average the
kept shots per sample, use average() on the result with drop = TRUE. In
a recipe, use step_reject_shots().
A tibble: data with the shot number in its sample .shot, a
logical column .rejected, the criteria that rejected each shot in
.reason (NA for kept shots), the raw criteria .intensity,
.correlation and .distance, and the robust z-score of each criterion
used (.intensity_z, .correlation_z, .distance_z). Its attribute
"reject_shots" holds the settings. With drop = TRUE, the kept rows of
data.
Christian L. Goueguel
Iglewicz, B., Hoaglin, D.C. (1993). How to Detect and Handle Outliers. ASQC Quality Press, Milwaukee.
Fisher, R.A. (1915). Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population. Biometrika, 10(4):507-521.
plot_shots(), step_reject_shots(), average(),
saturation_summary(), forageShots
data(forageShots)
# 8 shots of each of 20 forage samples
res <- reject_shots(forageShots, Measurement, shot = shot)
res[res$.rejected, c("Measurement", ".shot", ".reason", ".intensity", ".correlation")]
# the kept shots only
nrow(reject_shots(forageShots, Measurement, drop = TRUE))
# the former behavior, with the MAD of each sample alone
sum(reject_shots(forageShots, Measurement, scale = "sample")$.rejected)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.