bdpar.Options: Object to handle the keys/attributes/options common to all...

In bdpar: Big Data Preprocessing Architecture

Object to handle the keys/attributes/options common to all pipeline flow

Description

This class provides the necessary methods to manage a list of keys or options used along the pipe flow, both those provided by the default library and those implemented by the user.

Usage

bdpar.Options

Details

By default, the application initializes the object named bdpar.Options of type BdparOptions which is in charge of initializing the options used in the defined pipes.

The default fields on bdpar.Options are initialized, if needed, as shown bellow:

[eml]

- bdpar.Options$set("extractorEML.mpaPartSelected", <<PartSelectedOnMPAlternative>>)

[resources]

- bdpar.Options$set("resources.abbreviations.path", <<abbreviation.path>>)

- bdpar.Options$set("resources.contractions.path", <<contractions.path>>)

- bdpar.Options$set("resources.interjections.path", <<interjections.path>>)

- bdpar.Options$set("resources.slangs.path", <<slangs.path>>)

- bdpar.Options$set("resources.stopwords.path", <<stopwords.path>>)

[twitter]

- bdpar.Options$set("twitter.consumer.key", <<consumer_key>>)

- bdpar.Options$set("twitter.consumer.secret", <<consumer_secret>>)

- bdpar.Options$set("twitter.access.token", <<access_token>>)

- bdpar.Options$set("twitter.access.token.secret", <<access_token_secret>>)

- bdpar.Options$set("cache.twitter.path", <<cache.path>>)

[teeCSVPipe]

- bdpar.Options$set("teeCSVPipe.output.path", <<outputh.path>>)

[youtube]

- bdpar.Options$set("youtube.app.id", <<app_id>>)

- bdpar.Options$set("youtube.app.password", <<app_password>>)

- bdpar.Options$set("cache.youtube.path", <<cache.path>>)

[cache]

- bdpar.Options$set("cache", <<status_cache>>)

- bdpar.Options$set("cache.folder", <<cache.path>>)

[parallel]

- bdpar.Options$set("numCores", <<num_cores>>)

[verbose]

- bdpar.Options$set("verbose", <<status_verbose>>)

Cache functionality

If the bdpar cache is configured through the "cache" and "cache.folder" options, the status of the instances will be stored after each pipe. This allows to avoid rejections of previously executed tasks, if the order and configuration of the pipe and pipeline is the same as what is stored in the cache.

If you want to remove the cache, the cleanCache method does this task.

Parallel functionality

The parallelization of instances is configured through the "numCores" option, which indicates the number of cores that will be used in the processing.

In the case of parallelisation, only the log by file will work to allow collecting all the information produced by the cores.

Log configuration

The bdpar log is configured through the configureLog function. This system manages both the place to display the messages and the priority level of each message showing only the messages with a higher level than indicated in the threshold variable.

If you want to deactivate the bdpar log, the disableLog method in bdpar.Options does this task.

Methods

• get: obtains a specific option.

  • Usage: get(key)

  • Value: the value of the specific option.

  • Arguments:

    • key: (character) the name of the option to obtain.

• add: adds a option to the list of options

  • Usage: add(key, value)

  • Arguments:

    • key: (character) the name of the new option.

    • propertyName: (Object) the value of the new option.

• set: modifies the value of the one option.

  • Usage: set(key, value)

  • Arguments:

    • key: (character) the name of the new option.

    • propertyName: (Object) the value of the new option.

• remove: removes a specific option.

  • Usage: remove(key)

  • Arguments:

    • key: (character) the name of the option to remove.

• getAll: gets the list of options.

  • Usage: getAll()

  • Value: Value of options.

• remove: resets the option list to the initial state.

  • Usage: reset()

• isSpecificOption: checks for the existence of an specific option.

  • Usage: isSpecificProperty(key)

  • Value: A boolean results according to the existence of the specific option in the list of options

  • Arguments:

    • key: (character) the key of the option to check.

• cleanCache: Cleans the cache of executed pipelines. Deletes all files and directories that are in the path defined in "cache.folder" option.

  • Usage: cleanCache()

• configureLog: Configures the bdpar log. In the case of parallelisation, only the log by file will work.

  • Usage: configureLog(console = TRUE, threshold = "INFO", file = NULL)

  • Arguments:

    • console: (boolean) Shows the log on console or not.

    • threshold: (character) The logging threshold level. Messages with a lower priority level will be discarded.

    • file: (character) The file to write messages to. If it is NULL, the log in file will not be enabled.

• disableLog: Deactivates the bdpar log.

  • Usage: disableLog()

• getLogConfiguration: Print the bdpar log configuration.

  • Usage: getLogConfiguration()

See Also

AbbreviationPipe, bdpar.log, Connections, ContractionPipe, ExtractorEml, ExtractorTwtid, ExtractorYtbid, GuessLanguagePipe, Instance, SlangPipe, StopWordPipe, TeeCSVPipe, %>|%

bdpar documentation built on Aug. 22, 2022, 5:08 p.m.