LargeDataSetForTextEmbeddings: Abstract class for large data sets containing text embeddings
In aifeducation: Artificial Intelligence for Education
LargeDataSetForTextEmbeddings R Documentation
Abstract class for large data sets containing text embeddings
Description
This object stores text embeddings which are usually produced by an object of class
TextEmbeddingModel. The data of this objects is not stored in memory directly. By using memory mapping these
objects allow to work with data sets which do not fit into memory/RAM.
LargeDataSetForTextEmbeddings are used for storing and managing the text embeddings created with objects of class
TextEmbeddingModel. Objects of class LargeDataSetForTextEmbeddings serve as input for objects of class
TEClassifierRegular, TEClassifierProtoNet, and TEFeatureExtractor. The main aim of this class is to provide a
structured link between embedding models and classifiers. Since objects of this class save information on the text
embedding model that created the text embedding it ensures that only embedding generated with same embedding model
are combined. Furthermore, the stored information allows objects to check if embeddings of the correct text
embedding model are used for training and predicting.
Value
Returns a new object of this class.
Super class
aifeducation::LargeDataSetBase -> LargeDataSetForTextEmbeddings
Methods
Public methods
-
-
-
LargeDataSetForTextEmbeddings$get_text_embedding_model_name()
-
-
-
-
-
-
-
-
-
-
-
LargeDataSetForTextEmbeddings$add_embeddings_from_EmbeddedText()
-
LargeDataSetForTextEmbeddings$add_embeddings_from_LargeDataSetForTextEmbeddings()
-
-
Inherited methods
aifeducation::LargeDataSetBase$get_all_fields()aifeducation::LargeDataSetBase$get_colnames()aifeducation::LargeDataSetBase$get_dataset()aifeducation::LargeDataSetBase$get_ids()aifeducation::LargeDataSetBase$load()aifeducation::LargeDataSetBase$n_cols()aifeducation::LargeDataSetBase$n_rows()aifeducation::LargeDataSetBase$reduce_to_unique_ids()aifeducation::LargeDataSetBase$save()aifeducation::LargeDataSetBase$select()
Method configure()
Creates a new object representing text embeddings.
Usage
LargeDataSetForTextEmbeddings$configure(
model_name = NA,
model_label = NA,
model_date = NA,
model_method = NA,
model_version = NA,
model_language = NA,
param_seq_length = NA,
param_chunks = NULL,
param_features = NULL,
param_overlap = NULL,
param_emb_layer_min = NULL,
param_emb_layer_max = NULL,
param_emb_pool_type = NULL,
param_aggregation = NULL
)
Arguments
model_namestring Name of the model that generates this embedding.
model_labelstring Label of the model that generates this embedding.
model_datestring Date when the embedding generating model was created.
model_methodstring Method of the underlying embedding model.
model_versionstring Version of the model that generated this embedding.
model_languagestring Language of the model that generated this embedding.
param_seq_lengthint Maximum number of tokens that processes the generating model for a chunk.
param_chunksint Maximum number of chunks which are supported by the generating model.
param_featuresint Number of dimensions of the text embeddings.
param_overlapint Number of tokens that were added at the beginning of the sequence for the next chunk
by this model.
param_emb_layer_minint or string determining the first layer to be included in the creation of
embeddings.
param_emb_layer_maxint or string determining the last layer to be included in the creation of
embeddings.
param_emb_pool_typestring determining the method for pooling the token embeddings within each layer.
param_aggregationstring Aggregation method of the hidden states. Deprecated. Only included for backward
compatibility.
Returns
The method returns a new object of this class.
Method is_configured()
Method for checking if the model was successfully configured. An object can only be used if this
value is TRUE.
Usage
LargeDataSetForTextEmbeddings$is_configured()
Returns
bool TRUE if the model is fully configured. FALSE if not.
Method get_text_embedding_model_name()
Method for requesting the name (unique id) of the underlying text embedding model.
Usage
LargeDataSetForTextEmbeddings$get_text_embedding_model_name()
Returns
Returns a string describing name of the text embedding model.
Method get_model_info()
Method for retrieving information about the model that generated this embedding.
Usage
LargeDataSetForTextEmbeddings$get_model_info()
Returns
list containing all saved information about the underlying text embedding model.
Method load_from_disk()
loads an object of class LargeDataSetForTextEmbeddings from disk and updates the object to the
current version of the package.
Usage
LargeDataSetForTextEmbeddings$load_from_disk(dir_path)
Arguments
dir_pathPath where the data set set is stored.
Returns
Method does not return anything. It loads an object from disk.
Method get_model_label()
Method for retrieving the label of the model that generated this embedding.
Usage
LargeDataSetForTextEmbeddings$get_model_label()
Returns
string Label of the corresponding text embedding model
Method add_feature_extractor_info()
Method setting information on the TEFeatureExtractor that was used to reduce the number of
dimensions of the text embeddings. This information should only be used if a TEFeatureExtractor was applied.
Usage
LargeDataSetForTextEmbeddings$add_feature_extractor_info(
model_name,
model_label = NA,
features = NA,
method = NA,
noise_factor = NA,
optimizer = NA
)
Arguments
model_namestring Name of the underlying TextEmbeddingModel.
model_labelstring Label of the underlying TextEmbeddingModel.
featuresint Number of dimension (features) for the compressed text embeddings.
methodstring Method that the TEFeatureExtractor applies for genereating the compressed text
embeddings.
noise_factordouble Noise factor of the TEFeatureExtractor.
optimizerstring Optimizer used during training the TEFeatureExtractor.
Returns
Method does nothing return. It sets information on a TEFeatureExtractor.
Method get_feature_extractor_info()
Method for receiving information on the TEFeatureExtractor that was used to reduce the number of
dimensions of the text embeddings.
Usage
LargeDataSetForTextEmbeddings$get_feature_extractor_info()
Returns
Returns a list with information on the TEFeatureExtractor. If no TEFeatureExtractor was used it
returns NULL.
Method is_compressed()
Checks if the text embedding were reduced by a TEFeatureExtractor.
Usage
LargeDataSetForTextEmbeddings$is_compressed()
Returns
Returns TRUE if the number of dimensions was reduced by a TEFeatureExtractor. If not return FALSE.
Method get_times()
Number of chunks/times of the text embeddings.
Usage
LargeDataSetForTextEmbeddings$get_times()
Returns
Returns an int describing the number of chunks/times of the text embeddings.
Method get_features()
Number of actual features/dimensions of the text embeddings.In the case a TEFeatureExtractor was
used the number of features is smaller as the original number of features. To receive the original number of
features (the number of features before applying a TEFeatureExtractor) you can use the method
get_original_features of this class.
Usage
LargeDataSetForTextEmbeddings$get_features()
Returns
Returns an int describing the number of features/dimensions of the text embeddings.
Method get_original_features()
Number of original features/dimensions of the text embeddings.
Usage
LargeDataSetForTextEmbeddings$get_original_features()
Returns
Returns an int describing the number of features/dimensions if no TEFeatureExtractor) is used or
before a TEFeatureExtractor) is applied.
Method add_embeddings_from_array()
Method for adding new data to the data set from an array. Please note that the method does not
check if cases already exist in the data set. To reduce the data set to unique cases call the method
reduce_to_unique_ids.
Usage
LargeDataSetForTextEmbeddings$add_embeddings_from_array(embedding_array)
Arguments
embedding_arrayarray containing the text embeddings.
Returns
The method does not return anything. It adds new data to the data set.
Method add_embeddings_from_EmbeddedText()
Method for adding new data to the data set from an EmbeddedText. Please note that the method does
not check if cases already exist in the data set. To reduce the data set to unique cases call the method
reduce_to_unique_ids.
Usage
LargeDataSetForTextEmbeddings$add_embeddings_from_EmbeddedText(EmbeddedText)
Arguments
EmbeddedTextObject of class EmbeddedText.
Returns
The method does not return anything. It adds new data to the data set.
Method add_embeddings_from_LargeDataSetForTextEmbeddings()
Method for adding new data to the data set from an LargeDataSetForTextEmbeddings. Please note that
the method does not check if cases already exist in the data set. To reduce the data set to unique cases call
the method reduce_to_unique_ids.
Usage
LargeDataSetForTextEmbeddings$add_embeddings_from_LargeDataSetForTextEmbeddings(
dataset
)
Arguments
datasetObject of class LargeDataSetForTextEmbeddings.
Returns
The method does not return anything. It adds new data to the data set.
Method convert_to_EmbeddedText()
Method for converting this object to an object of class EmbeddedText.
Attention This object uses memory mapping to allow the usage of data sets
that do not fit into memory. By calling this method the data set will be loaded and stored into memory/RAM.
This may lead to an out-of-memory error.
Usage
LargeDataSetForTextEmbeddings$convert_to_EmbeddedText()
Returns
LargeDataSetForTextEmbeddings an object of class EmbeddedText which is stored in the memory/RAM.
Method clone()
The objects of this class are cloneable with this method.
Usage
LargeDataSetForTextEmbeddings$clone(deep = FALSE)
Arguments
deepWhether to make a deep clone.
See Also
Other Data Management:
DataManagerClassifier,
EmbeddedText,
LargeDataSetForText
aifeducation documentation built on April 4, 2025, 2:01 a.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
| LargeDataSetForTextEmbeddings | R Documentation |
Abstract class for large data sets containing text embeddings
Description
This object stores text embeddings which are usually produced by an object of class TextEmbeddingModel. The data of this objects is not stored in memory directly. By using memory mapping these objects allow to work with data sets which do not fit into memory/RAM.
LargeDataSetForTextEmbeddings are used for storing and managing the text embeddings created with objects of class TextEmbeddingModel. Objects of class LargeDataSetForTextEmbeddings serve as input for objects of class TEClassifierRegular, TEClassifierProtoNet, and TEFeatureExtractor. The main aim of this class is to provide a structured link between embedding models and classifiers. Since objects of this class save information on the text embedding model that created the text embedding it ensures that only embedding generated with same embedding model are combined. Furthermore, the stored information allows objects to check if embeddings of the correct text embedding model are used for training and predicting.
Value
Returns a new object of this class.
Super class
aifeducation::LargeDataSetBase -> LargeDataSetForTextEmbeddings
Methods
Public methods
-
LargeDataSetForTextEmbeddings$get_text_embedding_model_name() -
LargeDataSetForTextEmbeddings$add_embeddings_from_EmbeddedText() -
LargeDataSetForTextEmbeddings$add_embeddings_from_LargeDataSetForTextEmbeddings()
Inherited methods
aifeducation::LargeDataSetBase$get_all_fields()aifeducation::LargeDataSetBase$get_colnames()aifeducation::LargeDataSetBase$get_dataset()aifeducation::LargeDataSetBase$get_ids()aifeducation::LargeDataSetBase$load()aifeducation::LargeDataSetBase$n_cols()aifeducation::LargeDataSetBase$n_rows()aifeducation::LargeDataSetBase$reduce_to_unique_ids()aifeducation::LargeDataSetBase$save()aifeducation::LargeDataSetBase$select()
Method configure()
Creates a new object representing text embeddings.
Usage
LargeDataSetForTextEmbeddings$configure( model_name = NA, model_label = NA, model_date = NA, model_method = NA, model_version = NA, model_language = NA, param_seq_length = NA, param_chunks = NULL, param_features = NULL, param_overlap = NULL, param_emb_layer_min = NULL, param_emb_layer_max = NULL, param_emb_pool_type = NULL, param_aggregation = NULL )
Arguments
model_namestringName of the model that generates this embedding.model_labelstringLabel of the model that generates this embedding.model_datestringDate when the embedding generating model was created.model_methodstringMethod of the underlying embedding model.model_versionstringVersion of the model that generated this embedding.model_languagestringLanguage of the model that generated this embedding.param_seq_lengthintMaximum number of tokens that processes the generating model for a chunk.param_chunksintMaximum number of chunks which are supported by the generating model.param_featuresintNumber of dimensions of the text embeddings.param_overlapintNumber of tokens that were added at the beginning of the sequence for the next chunk by this model.param_emb_layer_minintorstringdetermining the first layer to be included in the creation of embeddings.param_emb_layer_maxintorstringdetermining the last layer to be included in the creation of embeddings.param_emb_pool_typestringdetermining the method for pooling the token embeddings within each layer.param_aggregationstringAggregation method of the hidden states. Deprecated. Only included for backward compatibility.
Returns
The method returns a new object of this class.
Method is_configured()
Method for checking if the model was successfully configured. An object can only be used if this
value is TRUE.
Usage
LargeDataSetForTextEmbeddings$is_configured()
Returns
bool TRUE if the model is fully configured. FALSE if not.
Method get_text_embedding_model_name()
Method for requesting the name (unique id) of the underlying text embedding model.
Usage
LargeDataSetForTextEmbeddings$get_text_embedding_model_name()
Returns
Returns a string describing name of the text embedding model.
Method get_model_info()
Method for retrieving information about the model that generated this embedding.
Usage
LargeDataSetForTextEmbeddings$get_model_info()
Returns
list containing all saved information about the underlying text embedding model.
Method load_from_disk()
loads an object of class LargeDataSetForTextEmbeddings from disk and updates the object to the current version of the package.
Usage
LargeDataSetForTextEmbeddings$load_from_disk(dir_path)
Arguments
dir_pathPath where the data set set is stored.
Returns
Method does not return anything. It loads an object from disk.
Method get_model_label()
Method for retrieving the label of the model that generated this embedding.
Usage
LargeDataSetForTextEmbeddings$get_model_label()
Returns
string Label of the corresponding text embedding model
Method add_feature_extractor_info()
Method setting information on the TEFeatureExtractor that was used to reduce the number of dimensions of the text embeddings. This information should only be used if a TEFeatureExtractor was applied.
Usage
LargeDataSetForTextEmbeddings$add_feature_extractor_info( model_name, model_label = NA, features = NA, method = NA, noise_factor = NA, optimizer = NA )
Arguments
model_namestringName of the underlying TextEmbeddingModel.model_labelstringLabel of the underlying TextEmbeddingModel.featuresintNumber of dimension (features) for the compressed text embeddings.methodstringMethod that the TEFeatureExtractor applies for genereating the compressed text embeddings.noise_factordoubleNoise factor of the TEFeatureExtractor.optimizerstringOptimizer used during training the TEFeatureExtractor.
Returns
Method does nothing return. It sets information on a TEFeatureExtractor.
Method get_feature_extractor_info()
Method for receiving information on the TEFeatureExtractor that was used to reduce the number of dimensions of the text embeddings.
Usage
LargeDataSetForTextEmbeddings$get_feature_extractor_info()
Returns
Returns a list with information on the TEFeatureExtractor. If no TEFeatureExtractor was used it
returns NULL.
Method is_compressed()
Checks if the text embedding were reduced by a TEFeatureExtractor.
Usage
LargeDataSetForTextEmbeddings$is_compressed()
Returns
Returns TRUE if the number of dimensions was reduced by a TEFeatureExtractor. If not return FALSE.
Method get_times()
Number of chunks/times of the text embeddings.
Usage
LargeDataSetForTextEmbeddings$get_times()
Returns
Returns an int describing the number of chunks/times of the text embeddings.
Method get_features()
Number of actual features/dimensions of the text embeddings.In the case a TEFeatureExtractor was
used the number of features is smaller as the original number of features. To receive the original number of
features (the number of features before applying a TEFeatureExtractor) you can use the method
get_original_features of this class.
Usage
LargeDataSetForTextEmbeddings$get_features()
Returns
Returns an int describing the number of features/dimensions of the text embeddings.
Method get_original_features()
Number of original features/dimensions of the text embeddings.
Usage
LargeDataSetForTextEmbeddings$get_original_features()
Returns
Returns an int describing the number of features/dimensions if no TEFeatureExtractor) is used or
before a TEFeatureExtractor) is applied.
Method add_embeddings_from_array()
Method for adding new data to the data set from an array. Please note that the method does not
check if cases already exist in the data set. To reduce the data set to unique cases call the method
reduce_to_unique_ids.
Usage
LargeDataSetForTextEmbeddings$add_embeddings_from_array(embedding_array)
Arguments
embedding_arrayarraycontaining the text embeddings.
Returns
The method does not return anything. It adds new data to the data set.
Method add_embeddings_from_EmbeddedText()
Method for adding new data to the data set from an EmbeddedText. Please note that the method does
not check if cases already exist in the data set. To reduce the data set to unique cases call the method
reduce_to_unique_ids.
Usage
LargeDataSetForTextEmbeddings$add_embeddings_from_EmbeddedText(EmbeddedText)
Arguments
EmbeddedTextObject of class EmbeddedText.
Returns
The method does not return anything. It adds new data to the data set.
Method add_embeddings_from_LargeDataSetForTextEmbeddings()
Method for adding new data to the data set from an LargeDataSetForTextEmbeddings. Please note that
the method does not check if cases already exist in the data set. To reduce the data set to unique cases call
the method reduce_to_unique_ids.
Usage
LargeDataSetForTextEmbeddings$add_embeddings_from_LargeDataSetForTextEmbeddings( dataset )
Arguments
datasetObject of class LargeDataSetForTextEmbeddings.
Returns
The method does not return anything. It adds new data to the data set.
Method convert_to_EmbeddedText()
Method for converting this object to an object of class EmbeddedText.
Attention This object uses memory mapping to allow the usage of data sets that do not fit into memory. By calling this method the data set will be loaded and stored into memory/RAM. This may lead to an out-of-memory error.
Usage
LargeDataSetForTextEmbeddings$convert_to_EmbeddedText()
Returns
LargeDataSetForTextEmbeddings an object of class EmbeddedText which is stored in the memory/RAM.
Method clone()
The objects of this class are cloneable with this method.
Usage
LargeDataSetForTextEmbeddings$clone(deep = FALSE)
Arguments
deepWhether to make a deep clone.
See Also
Other Data Management:
DataManagerClassifier,
EmbeddedText,
LargeDataSetForText
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.