Evaluation of Word Alignment Quality

Share:

Description

It measures Precision, Recall, AER, and F_measurs metrics to evaluate word alignment quality.

Usage

1
2
3
4
5
6
7
8
9
Evaluation1(file_train1, file_train2, nrec = -1, 
            tst.set_sorc, tst.set_trgt, nlen = -1, 
            minlen1 = 5, maxlen1 = 40, minlen2 = 5, 
            maxlen2 = 40, ul_s = FALSE, ul_t = TRUE, 
            removePt = TRUE, all = FALSE, iter = 3, 
            method = c("fix", "Excel"), agn = c("an.agn", "my.agn"), 
            null.tokens = TRUE, excel1 = "gold.xlsx", excel2 = "align.xlsx",
            fixfile_gld = NULL, fixfile_agn = NULL, dtfile = NULL, 
            f1 = "fa", e1 = "en", alpha = 0.5)

Arguments

file_train1

the name of source language file in training set.

file_train2

the name of target language file in training set.

nrec

number of sentences in the training set to be read. If -1, it considers all sentences.

tst.set_sorc

the name of source language file in test set.

tst.set_trgt

the name of target language file in test set.

nlen

number of sentences in the test set to be read. If -1, it considers all sentences.

minlen1

a minimum length of sentences in training set.

maxlen1

a maximum length of sentences in training set.

minlen2

a minimum length of sentences in test set.

maxlen2

a maximum length of sentences in test set.

ul_s

logical. If TRUE, it will convert the first character of source language's sentences. When source language is an Arabic script, it can be FALSE.

ul_t

logical. If TRUE, it will convert the first character of target language's sentences. When target language is an arabic script, it can be FALSE.

removePt

logical. If TRUE, it removes all punctuation marks.

all

logical. If TRUE, it considers the third argument (lower = TRUE) in culf function.

iter

number of iterations for IBM Model 1.

method

character string including two values. If "fix", it uses fix.gold function to achieve gold standard. If "Excel", it uses consExcel function to achieve gold standard.

agn

character string including two values. If "my.agn", the user wants to evaluate one-to-many word alignment using the word_alignIBM1 function in this package. If "an.agn", the user applies another software or even another method to word alignment.

null.tokens

logical. TRUE means that the gold standard alignment is constructed based on "null tokens".

excel1

the name of the excel file for gold standarad.

excel2

the name of the excel file for alignment.

fixfile_gld

if NULL, it is the first time to build a gold standard using fix.gold function. In this case, the function will automatically save the created matrices of gold standard with an informative name which is a combination of the sentence number and 'RData'. As an example "1.RData", "2.RData" and ... . Note that, the aforementioned name must not be changed. For next times, it is sufficient to set fixfile_gld to any character, e.g. "a", "textfile" or "myproject".

fixfile_agn

it is similar to fixfile_gld, but when we use some other software or method to create alignment instead of creating gold standard.

dtfile

to run this function for the first time, it must be assigned to NULL. In this case, the function will automatically save required codes of word_alignIBM1 function with a name which is a combination of f1, e1, nrec and iter as "f1.e1.nrec.iter.RData". Note that, the abovementioned name must not been changed. For next times, it is sufficient to set dtfile to any character, e.g. "a", "textfile" or "myproject".

f1

it is an abbreviation of source language (default = 'fa').

e1

t is an abbreviation of target language (default = 'en')

alpha

is a parameter that sets the trade-off between Precision and Recall.

Details

To evaluate word alignment quality, we need to a "reference alignment" (a gold standard for the word alignment) of a test set. Two methods to enter this gold standard is considered. When method = "fix", means that fix.gold function is called and the user should press 'Enter' to continue and edit the matrix to enter Sure/Possible alignments (Sure=1,Possible=2). Furthermore, when the user applies another software or even another method to word alignment, he/she should set agn = "an.agn", and he/she based on another word alignment's results should press'Enter' to continue and edit the matrix to enter 3 for alignment. (Note that for each sentence pair, one matrix is created.)

If method = "Excel", means that the created excel file of consExcel has been used. In this method, the aforementioned excel file should be completed by an expert with codes 1 or 2 for Sure or Possible alignments first and then this excel file named excel1 (as a default: "gold.xlsx") is set as an input argument. Moreover, to evaluate word alignment quality using another software or even another method, the user can be used excel2 file (as a default: "align.xlsx") that had been completed by 3 for alignments.

Value

A list.

Recall

A decimal number.

Precision

A decimal number.

AER

A decimal number.

F_measure1

A decimal number.

F_measure2

A decimal number.

Note

Note that we have a memory restriction and just special computers with high cpu and big ram can allocate the vectors of this function. Of course, it depends on corpus size.

Author(s)

Neda Daneshgar and Majid Sarmad.

References

Fraser F., Marcu D. (2007), "MeasuringWord Alignment Quality for Statistical Machine Translation.", Computational Linguistics, 33(3), 293-303.

Koehn P. (2010), "Statistical Machine Translation.", Cambridge University, New York.

Och F., Ney H.(2003)."A Systematic Comparison Of Various Statistical Alignment Models.", 2003 Association for Computational Linguistics, J03-1002, 29(1).

Wang X. "Evaluation of Two Word Alignment Systems.", Final Thesis, Department of Computer and Information Science.

See Also

word_alignIBM1, fix.gold, consExcel