storage: Virtual Distributed Storage Class

In DSL: Distributed Storage and List

Description

When using class DList the underlying ‘virtual’ storage plays an important role. It defines how to use the given storage (read/write methods, etc.), where the data is to be stored (i.e., the base directory on the file system), and how DMap as well as DReduce have to be applied.

Usage

DStorage(type = c("LFS", "HDFS"), base_dir, chunksize = 1024^2)
is.DStorage( ds )

Arguments

type	the type of the storage to be created. Currently only "LFS" and "HDFS" storage types are supported.
base_dir	specifies the base directory where data is to be stored.
chunksize	defines the size of each chunk written to the virtual storage.
ds	a virtual possibly distributed storage.

Value

An object which inherits from class DStorage, or, in case of is.DStorage() a logical indicating whether it inherits from "DStorage" or not.

Examples

## creating a new virtual storage using 50MB chunks
ds <- DStorage(type = "LFS", base_dir = tempdir(),
chunksize = 50 * 1024^2)
is.DStorage( ds )

Example output

[1] TRUE

DSL documentation built on Nov. 23, 2019, 3 a.m.