eval_clustering: Evaluate DSM on Clustering Task (wordspace)

Description Usage Arguments Details Value Author(s) See Also Examples

Description

Performs evaluation on a word clustering task by comparing a flat clustering solution based on semantic distances with a gold classification.

Usage

1
2
3
4
eval.clustering(task, M, dist.fnc = pair.distances, ...,
                details = FALSE, format = NA, taskname = NA,
                scale.entropy = FALSE, n.clusters = NA,
                word.name = "word", class.name = "class")

Arguments

task

a data frame listing words and their classes, usually in columns named word and class

M

a scored DSM matrix, passed to dist.fnc

dist.fnc

a callback function used to compute distances between word pairs. It will be invoked with character vectors containing the components of the word pairs as first and second argument, the DSM matrix M as third argument, plus any additional arguments (...) passed to eval.multiple.choice. The return value must be a numeric vector of appropriate length. If one of the words in a pair is not represented in the DSM, the corresponding distance value should be set to Inf.

...

any further arguments are passed to dist.fnc and can be used e.g. to select a distance measure

details

if TRUE, a detailed report with information on each task item is returned (see “Value” below for details)

format

if the task definition specifies POS-disambiguated lemmas in CWB/Penn format, they can automatically be transformed into some other notation conventions; see convert.lemma for details

taskname

optional row label for the short report (details=FALSE)

scale.entropy

whether to scale cluster entropy values to the range [0, 1]

n.clusters

number of clusters. The (very sensible) default is to generate as many clusters as their are classes in the gold standard.

word.name

the name of the column of task containing words

class.name

the name of the column of task containing gold standard classes

Details

The test words are clustered using the “partitioning around medoids” (PAM) algorithm (Kaufman \& Rousseeuw 1990, Ch. 2) based on their semantic distances. The PAM algorithm is used because it works with arbitrary distance measures (including neihbour rank), produces a stable solution (unlike most iterative algorithms) and has shown to be on par with state-of-the-art spherical k-means clustering (CLUTO) in evaluation studies.

Each cluster is automatically assigned a majority label, i.e. the gold standard class occurring most frequently in the cluster. This represents the best possible classification that can be derived from the clustering.

As evaluation metrics, clustering purity (accuracy of the majority classification) and entropy are computed. The latter is defined as a weighted average over the entropy of the class distribution within each cluster, expressed in bits. If scale.entropy=TRUE, the value is divided by the overall entropy of the class distribution in the gold standard, scaling it to the range [0, 1].

NB: The semantic distance measure selected with the extra arguments (...) should be symmetric. In particular, it is not very sensible to specify rank="fwd" or rank="bwd".

NB: Similarity measures are not supported by the current clustering algorithm. Make sure not to call dist.matrix (from dist.fnc) with convert=FALSE!

Value

The default short report (details=FALSE) is a data frame with a single row and the columns purity (clustering purity as a percentage), entropy (scaled or unscaled clustering entropy) and missing (number of words not found in the DSM).

The detailed report (details=TRUE) is a data frame with one row for each test word and the following columns:

word

the test word (character)

cluster

cluster to which the word has been assigned; all unknown words are collected in an additional cluster "n/a"

label

majority label of this cluster (factor with same levels as gold)

gold

gold standard class of the test word (factor)

correct

whether majority class assignment is correct (logical)

missing

whether word was not found in the DSM (logical)

Author(s)

Stefan Evert (http://purl.org/stefan.evert)

See Also

Suitable gold standard data sets in this package: ESSLLI08_Nouns

Support functions: pair.distances, convert.lemma

Examples

1

wordspace documentation built on Jan. 9, 2020, 1:08 a.m.