cleanDocument: cleanDocument

In j2scode/predictifyR: Word Prediction Language Model Evaluation

Description

cleanDocument cleans the HC Corpus document

Usage

cleanDocument(rawDocument)

Arguments

rawDocument

- the meta data and content for the file to be analyzed

Details

This function reads a corpus document and performs a series of conditioning, reshaping, normalization, and clean up tasks. The conditioning tasks include:

• ControlReplace UTF-8 control characters with spaces

• QuotationsConvert non-standard quotations to single quote

• EncodingConvert document from UTF-8 to ASCII enconding

The reshaping task uses the quanteda package https://cran.r-project.org/web/packages/quanteda/quanteda.pdf to reshape the corpus documents into sentences

The normalization tasks include the following:

• controlRemove ASCII control characters

• nonprintRemove ASCII non-printable characters

• emailsRemove email addresses

• urlsRemove urls

• hashtagsRemove twitter hashtags

• controlRemove ASCII control characters

• digitsRemove digits

• punctRemove punctuation except the apostrophe

• longWordsRemove words 40 characters or longer

• profanityRemove profanity

• correctCorrect contractions and common misspellings#'

Finally, the clean up tasks include:

• whiteSpaceRemove extra whitespace from documents

• punctRemove stray apostrophes and punctuation

• emptySentRemove empty sentences

Value

cleanDocument Cleaned text document in unlisted vector format.

Author(s)

John James, [e-mail]

See Also

Other text processing functions: analyzeCorpus, cleanCorpus, extractLines, getCorpus, getStats, summarizeAnalysis

j2scode/predictifyR documentation built on May 14, 2019, 10:34 a.m.