future_multihashes: Parallel Computation of SHA2 and SHA3 Hashes This function...

In SeanFobbe/databuilder: Create New Data Sets (esp. Large Text Corpora)

Parallel Computation of SHA2 and SHA3 Hashes This function parallelizes computation of both SHA2-256 and SHA3-512 hashes for an arbitrary number of files. It returns a data frame of file names, SHA2-256 hashes and SHA3-512 hashes. The function requires the existence of the openssl library (RPM) on the system.

Description

Please note that you must declare your own future evaluation strategy prior to using the function to enable parallelization. By default the function will be evaluated sequentially. On Windows, use future::plan(multisession, workers = n), on Linux/Mac, use future::plan(multicore, workers = n), where n stands for the number of CPU cores you wish to use. Due to the need to read/write to the disk the function may not work properly on high-performance clusters.

Usage

future_multihashes(x)

Arguments

x	A vector of filenames. Should be located in the working directory.

SeanFobbe/databuilder documentation built on July 20, 2022, 4:50 a.m.