move_temporary_to_outputDB: Moves simulation output that was written to temporary text...
In DrylandEcology/rSFSW2: Simulation Framework for SOILWAT2

Moves simulation output that was written to temporary text files to a SQL-database

move_temporary_to_outputDB(
  SFSW2_prj_meta,
  t_job_start,
  opt_parallel,
  opt_behave,
  opt_out_run,
  opt_verbosity,
  chunk_size = 1000L,
  dir_out_temp = NULL
)

move_temporary_to_outputDB_withChecks(
  SFSW2_prj_meta,
  t_job_start,
  opt_parallel,
  opt_behave,
  opt_out_run,
  opt_verbosity,
  chunk_size = 1000L,
  check_if_Pid_present = TRUE,
  dir_out_temp = NULL
)

chunk_size

An integer value. The number of lines that are read at once from the temporary text files and processed in one SQL-transaction.

move_temporary_to_outputDB: no checking of temporary text files is done. Any line that fails to be added to the database (for whatever reason including a record with identical combination of P_id and SoilLayerID is already present) is written to a new file ‘SQL_tmptxt_failed.txt’.

Initial tests suggest that performance degrades if chunk_size was small (e.g., 10); values around 1000 have been successful; values of 10,000 work about as fast as those of 1000, but memory usage is a bit larger – and the risk that an entire transaction fails increases with chunk_size.

move_temporary_to_outputDB_withChecks: temporary text files are checked for presence of table names and identification values (P_id and soil layer ID). If argument check_if_Pid_present is true and the record ID already exists in the database, then values are checked for agreement. The speed penalty for running the checks vs. move_temporary_to_outputDB was about 20

Lines that have insufficient information or that fail to be added to the database are written to a new file ‘SQL_tmptxt_failed.txt’.
Lines with record identified by Pid (and sl) that are already in database and data does agree (agreement information only available if check_if_Pid_present) are written to a new file ‘SQL_tmptxt_duplicates.txt’.
Lines with record identified by Pid (and sl) is already in database, but data do not agree (agreement information only available if check_if_Pid_present) are written to a new file ‘SQL_tmptxt_repeats.txt’.