move_temporary_to_outputDB: Moves simulation output that was written to temporary text...

In Burke-Lauenroth-Lab/SoilWat_R_Wrapper: Simulation Framework for SOILWAT2

Description

Moves simulation output that was written to temporary text files to a SQL-database

Usage

move_temporary_to_outputDB(
  SFSW2_prj_meta,
  t_job_start,
  opt_parallel,
  opt_behave,
  opt_out_run,
  opt_verbosity,
  chunk_size = 1000L,
  dir_out_temp = NULL
)

move_temporary_to_outputDB_withChecks(
  SFSW2_prj_meta,
  t_job_start,
  opt_parallel,
  opt_behave,
  opt_out_run,
  opt_verbosity,
  chunk_size = 1000L,
  check_if_Pid_present = TRUE,
  dir_out_temp = NULL
)

Arguments

chunk_size

An integer value. The number of lines that are read at once from the temporary text files and processed in one SQL-transaction.

Details

move_temporary_to_outputDB: no checking of temporary text files is done. Any line that fails to be added to the database (for whatever reason including a record with identical combination of P_id and SoilLayerID is already present) is written to a new file ‘SQL_tmptxt_failed.txt’.

Initial tests suggest that performance degrades if chunk_size was small (e.g., 10); values around 1000 have been successful; values of 10,000 work about as fast as those of 1000, but memory usage is a bit larger – and the risk that an entire transaction fails increases with chunk_size.

move_temporary_to_outputDB_withChecks: temporary text files are checked for presence of table names and identification values (P_id and soil layer ID). If argument check_if_Pid_present is true and the record ID already exists in the database, then values are checked for agreement. The speed penalty for running the checks vs. move_temporary_to_outputDB was about 20

• Lines that have insufficient information or that fail to be added to the database are written to a new file ‘SQL_tmptxt_failed.txt’.

• Lines with record identified by Pid (and sl) that are already in database and data does agree (agreement information only available if check_if_Pid_present) are written to a new file ‘SQL_tmptxt_duplicates.txt’.

• Lines with record identified by Pid (and sl) is already in database, but data do not agree (agreement information only available if check_if_Pid_present) are written to a new file ‘SQL_tmptxt_repeats.txt’.

Burke-Lauenroth-Lab/SoilWat_R_Wrapper documentation built on Aug. 14, 2020, 5:17 p.m.