In keltoskytoi/iSEGMound: iSEGMound - A Reproducible Workflow for Mound Detection in LiDAR-derived DTMs

Towards a Paradigm Shift in Archaeological Remote Sensing practice by means of Tool-driven Automated Analysis

In Chapter 2 the arguments of reproducibility and replicability have been addressed regarding the accessibility of data, code and workflow of the investigated studies. It was demonstrated that only 2 out of 31 studies have disclosed all three elements and are reproducible. It is important to discuss the exact definition of both expressions. In this Master’s thesis mainly the definitions of @marwickComputationalReproducibilityArchaeological2017b are followed. Reproducibility means that code and workflow of an analysis reproduces all results and figures in the respective publication from its original raw data. Replicability stands for the replication of the workflow (from data collection to data analysis) onto a new dataset. It was mentioned in the introduction that optimal reproducible open science would incorporate version controlled open code, clear workflows and open datasets or at least training datasets.

So then, how is the best way to implement and guarantee reproducibility? @marwickComputationalReproducibilityArchaeological2017b outlined four general principles of reproducible research: i) Data and code provenance, sharing and archiving (analogue to artifact provenience) ii) Scripted analyses (for reproducibility and ideally replicability) iii) Version control (transparency of decision points of any research and possibility of collaboration) iv) Self-contained computational environments (sharing the computational environment of the analysis for replicability).

Pushing forward along this road, @marwickPackagingDataAnalytical2018a propose R-packages as research compendia to organize transparent and reproducible research to approach the general principles laid out in 2017. Research compendia should organize files according to conventions: separate data, method and output (the latter being disposable because they can be reproduced) and define which exact computational environment was used for the analysis. Building an R package around a research project automatically forces one to keep the guidelines and conventions of package building which also provides quality control mechanisms. Research compendia come in many forms and complexities, as @marwickPackagingDataAnalytical2018a discusses. The author of this Master’s thesis experimented with a basic form of reproducible research compendium when analyzing the finds of an Iron Age cemetery in Hungary using multivariate statistical analysis (@schneiderMultivariateStatisticalAnalysis2019). The work was done basically in an R project with a fixed project structure which was uploaded in a github repository (https://github.com/keltoskytoi/Multivariate_Statistics_Szentloerinc). The repository contains an .Rproj file with the settings for the project, a folder called DATA for the raw data to be analyzed and not to be changed during the study, two .R files with the code used for the analysis in the study, a README.md file with basic information of the paper, thus the title and where to get it. An earlier study and research compendium from 2015, but already in a very mature form by Ben Marwick already shows the many of the elements of an optimal research compendium (listed in Figure 12), organized as an R package with the published compendium on figshare for the study @clarksonArchaeologyChronologyStratigraphy2015 which is also completely replicable: https://github.com/benmarwick/1989-excavation-report-Madjedbebe; https://figshare.com/articles/software/1989_excavation_report_Madjebebe/1297059 A recent use of a research compendium is Schmid 2019 with the compendium published at OSF. This study is also completely replicable: (https://github.com/nevrome/cultrans.bronzeageburials.article2019, https://osf.io/b6np2/) Based on the reproducibility of these studies and on the personal experience and motivation mentioned above, it was decided to create a research compendium for the Master’s thesis.

The optimal outline for a complex research compendium in an R environment is reproduced from @marwickPackagingDataAnalytical2018a, Figure 4:

knitr::include_graphics('C:/Users/kelto/Documents/iSEGMound/analysis/thesis/figures/Figure_17.png')

This optimal research compendium outline is implemented in the R package rrtools (‘reproducible research tools’, https://github.com/benmarwick/rrtools) by Ben Marwick, which helps to write reproducible research papers, setting up the structure propagated in @marwickPackagingDataAnalytical2018a. Although this Master’s thesis cannot give place to all of these desired steps and tools, it shall provide a reproducible workflow for burial mound detection using R.

R is an open-source, open-access statistical scripting language which can be used either through the command line or it’s GUI, R Studio. It has been used in several scientific domains, and lately it is one of the most commonly used scripting languages in Archaeological Sciences (@schmidtToolDrivenRevolutionsArchaeological2020b, 18). The advantages of scripting languages support the four general principles of reproducible research of @marwickComputationalReproducibilityArchaeological2017b (see above) which are embodied in an optimal research compendium. The use of scripting languages (can) facilitate the use of Automated Analysis methods in Archaeological Remote Sensing, by offering a clear and logical semantic syntax, delivering a (shared diachronic semantic) ontological consistency throughout the research project and for future use.

@schmidtToolDrivenRevolutionsArchaeological2020b – after discussing idea- and tool-driven revolutions in science – debate the role of tools in paradigm-shifts in Archaeology. Archaeology is in a paradoxical situation: for one it is mainly an idea-driven discipline, but it has seen multiple tool-driven paradigm shifts (e.g. with Clarke's Analytical Archaeology (@clarkeAnalyticalArchaeology1968) and the development of Computational Archaeology). Computational Archaeology, Archaeological Remote Sensing and Remote Sensing itself are disciplines where change is mainly tool-driven, as we can see based on Chapter 2, which analysed 31 papers. But tools also ignite ideas, so there is no clear distinction, more a self-induction.

Before going to the next step, the concept of “tool-driven” has to be investigated more thoroughly. @schmidtToolDrivenRevolutionsArchaeological2020b borrow the concept from @galisonImageLogicMaterial1997, who elaborated on a tool-driven change in particle physics in the twentieth century, meaning tools like digital devices (for instance computers in the case of Archaeology). They see R (or similar open source programming languages) as one projection of a tool-driven revolution, including sharing reproducible and replicable code (@schmidtToolDrivenRevolutionsArchaeological2020b, 19). In my opinion we have to go further - the concept of tool-driven should become more mundane and tangible: automated analysis in Archaeological Remote Sensing has to become tool-driven in order to present a robust scientific practice for Archaeology, which means that reproducible workflows have to be created to do specific tasks, such as feature extraction, segmentation, which can then be developed further and be learned from. Automated analysis in Archaeological Remote Sensing should embrace tool-driven reproducible research to the fullest to make it possible to learn from each other and to build upon the experience of each other to develop ideas further.

One important issue still needs to be addressed. Lately it has been stressed (@struplerRediscoveringArchaeologicalDiscoveries2021) that not all openly published data are comparable: only because the raw data, especially if they are older projects which do not comply yet with the Berlin Declaration on Open Access to Knowledge in the Social Sciences and Humanities, is openly published and the exact methods are known, it does not automatically grant reproducibility (@struplerRediscoveringArchaeologicalDiscoveries2021, 2.). This emphasizes, that not only new data, reproducible workflows and best practices need to be created and published according to guidelines, but also legacy data from on-going, long-term excavations should be revised and curated and updated, to fit the FAIR principles 'Findability', 'Accessibility', 'Interoperability', and 'Reuse' (@dehaasFAIRSurveyImproving2020). It is clearly a problematic issue in this case, because often the authors might not be retractable. Thus reproduction of previous studies is a good way to correct, curate and update these data sets and the code and data is perfect for learning and teaching workflows but also to learn how to deal with possible errors (@struplerRediscoveringArchaeologicalDiscoveries2021, 15). It has to be pointed out that it cannot be expected to have complete consistency between a data set processed in the original study using a GIS platform and then replicated using a scripting language. When using software other than a scripting language (where optimally all steps taken are documented), there are often steps one does not document. An error in a data set or analysis is a different matter and complicates reproducibility but should not have a negative connotation and should not be seen as a failure in the archaeological scientific community but treated with ‘full disclosure’, as a learning effect on how to make data sets and analysis better (@struplerRediscoveringArchaeologicalDiscoveries2021, 15). The valid point is raised, that when publishing data sets and analysis there should be a responsible person to turn to (@struplerRediscoveringArchaeologicalDiscoveries2021, 16-17). This is even true when e.g. in a project the data is collected by a different person than the one who will do the analysis. Often crucial information is already missing when doing the analysis in the first place and this also needs to be disclosed clearly in the analysis (metadata in any form suitable for the project) and is not to be left out. In the case of data sets made available in an online repository, the owner of the repository or provider (uploader) of the data set seems to be the logical person to make responsible if not disclosed otherwise. This can be of course problematic with legacy data sets.
Another kind of legacy data set is used in this Master's thesis: monographic publication of a certain archaeological object group in the landscape, treated from the archaeological point of view. Also in this case there is missing and perished information, which is on one hand due to the subjective point of view of observer(s): is there a burial mound or not and also to the diachronich nature of objects in the landscape: the change and perish with time. This will be discussed more in Chapter 5 and 6.

This notion comes well together with the development of best practices and publishing data sets with workflows as mentioned before and fits perfectly in the wider picture of the Reproducible Research Culture ([@nakoinzQuantitativeArchaologieUnd2021], 63). Reanalysis studies are the perfect means to test reproducibility and the quality of data and metadata (as seen above). Like Quantitative Archaeology, also Automated Archaeological Remote Sensing needs a paradigm shift from a closed and restrictive to a sustainable Open Science.

keltoskytoi/iSEGMound documentation built on Dec. 21, 2021, 5:24 a.m.