This is an R package with utility functions to assist in harmonizing digital soil repositories. The purpose of the Soil Organic Carbon Data Rescue and Harmonization Repository (SOC-DRaHR) is to collect harmonization scripts for data sets related to soil carbon to enable scientific research. This project identifies soil carbon datasets that are publicly available, provides data harmonization scripts to integrate those data sets, and provides output scripts for a harmonized data product. Data can come from field surveys, field manipulation studies, and laboratory experiments.

This project is not a data repository or archive but instead a code repository. End users are responsible for complying with ALL data use policies of the orginal data providers, please check with the orginal archives and repositories to ensure you are complying with use policies.

Project links

Here are some important resources to get you started: 1) International Soil Carbon Network is our parent organization. 2) Our code of conduct is based on the The Contributor Covenant. 3) How to contribute 4) Our mailing list

Quick start

To quickly access ISCN3 data:

ISCN3 <- ISCN3(dataDir='YourDataDir')

Harmonization principles

1) Data is downloaded from an archived url link. 2) Meta data is encoded as a set of data table, and is ideally generated from the associated metadata on the archiving repository. 3) Data columns that share common variables are directly comparable with trivial unit conversions.

Data download

Data should be archived on a TRUST-ed repository (Lin et al, 2020) with an associated download url. SOC-DRaR2 scripts will then fetch the data from the identified url and may transform the data depending on the original structure. In general, R/read*.R scripts return a set of relational data tables that is wide or un-normalized where each row is a unique sample and each column an associated measurement or information about a measurement. The distinction between data and metadata is blurry and often varies between each contributing data set.

Meta-data as data tables

While we acknowledge that meta-data is often encoded as some XML tree structure, for this project we have opted to go with a data tables. The metadata has the following structure:

header | description ------------|------------- data_id | the name of the data product table_id | the name of the data table column_id| the name of the column header OR NA if the information is in the entry column variable | a valid ISCN measurement name that the column is associated with (see more on variables below) data_type | what kind of is is either in the column OR encoded in the data_entry entry | NA if this describes the values of a column OR the value of the data_type

For example, in ISC3:

data_id | table_id | column_id | variable | data_type | entry ---------|---------|-----------|----------|-----------|------ ISCN3 | NA | NA | NA | download_url | ftp:[...]xlsx ISCN3 | layer | 13c (‰) | 13c | value_number | NA ISCN3 | layer | NA | 13c | unit | permille

Note that this format is easy to derive from a wide table format

data_id | table_id | column_id | variable | data_type | value_string | unit ---------|----------|-----------|-----------|-----------|---------------|------------ ISCN3 | NA | NA | download_url | NA | ftp:[...]xlsx | NA ISCN3 | layer | 13c (‰) | 13c | value_number | NA | permille

Valid data_type entries include: value_number, value_string, unit, sigma, measurement_method, observation_note, and id.

The proposed vocabulary list from the ISCN 2016 data template is in data/vocabularyList.csv. ISCN 2016 has 295 terms grouped across 9 tables. This has been reduced to 123 terms across 5 tables for ISCN3 and many of the 2016 template terms were never used. In addition 58 new terms were added by ISCN3, mainly from the NRCS-Kellog Soil Science Lab. This is currently being document and cleaned up.

An example of the head of this vocabulary is:

data_table|data_column|variable|unit ----------|-----------|-------------------|---- site|site_name|Site Name|NA site|site_note|Site Notes|NA site|country|Country|NA site|province|Province|NA site|state|State|(us_states) site|county|County|NA site|lat|Latitude|dec. deg site|long|Longitude|dec. deg

Comparable data share variable

Data columns that are comparable with trivial conversions (generally unit conversion) share a variable. In some cases this requires unit conversions (eg grams to kilograms) but conversion factors should be coded as separate variables (eg loss on ignition times a conversion factor equals organic carbon fraction).

A current draft of the variable table is here data/vocabularyList.csv, it is under active development.

How to contribute

Please see the CONTRIBUTING document for more details on how to contribute, including how to identify datasets, contribute to the code, and everything else that is needed to run an open source community project. Contributors will be offered co-authorship for repository DOI but can not be guaranteed co-authorship on manuscripts, proposals or studies that utilize this repository. This repository is licensed here.


Repository versions will be assigned DOIs as needed. Any manuscripts which use these data harmonization scripts are asked to cite the appropreate repository version but are not required to list contributors as co-authors. The aggregation scripts here are licensed under BSD 2-clause. See LICENSE for details.


Lin, D., Crabtree, J., Dillo, I. et al. The TRUST Principles for digital repositories. Sci Data 7, 144 (2020).

Outside links of interest

International Soil Radiocarbon Database (ISRaD) a data curation project around soil radiocarbon and fractionation measurements.

Environmental Data Initiative has a GitRepo that has a similar approach to aggregating survey data.

Earth Sciences Information Partners

DataOne is a go-to resource for finding environmental data in general and searches many different established repositories. They have an R package for interacting with their search engine here

rOpenSci Project

Five ways consortia can catalyse open science, Nature, 2017

National Data Service

NCSS-USDA database description

ISCN/SOCDRaHR2 documentation built on May 26, 2023, 6:44 a.m.