calcPredictionAccuracy: Calculate the Prediction Error for a Recommendation

In recommenderlab: Lab for Developing and Testing Recommender Algorithms

Calculate the Prediction Error for a Recommendation

Description

Calculate prediction accuracy. For predicted ratings, MAE (mean absolute error), MSE (mean squared error), and RMSE (root mean squared error) are calculated. For top-N lists, binary classification metrics such as precision, recall, TPR, and FPR are returned.

Usage

calcPredictionAccuracy(x, data, ...)

## S4 method for signature 'realRatingMatrix,realRatingMatrix'
calcPredictionAccuracy(x, data, byUser = FALSE, ...)

## S4 method for signature 'topNList,realRatingMatrix'
calcPredictionAccuracy(x, data, byUser = FALSE,
  given = NULL, goodRating = NA, ...)

## S4 method for signature 'topNList,binaryRatingMatrix'
calcPredictionAccuracy(x, data, byUser = FALSE,
  given = NULL, ...)

Arguments

x	Predicted items in a "topNList" or predicted ratings as a "realRatingMatrix"
data	Observed true ratings for the users as a "RatingMatrix". The users have to be in the same order as in x.
byUser	logical; Should the accuracy measures be reported for each user individually instead of being averaged over all users?
...	further arguments.
given	How many items were given to create the predictions. If the data comes from an all-but-x evaluation scheme (i.e., a negative value for given), supply a vector containing the number of items given for each prediction. This can be obtained from the evaluation scheme es using getData(es, "given").
goodRating	If x is a "topNList" and data is a "realRatingMatrix" then goodRating is used as the threshold for determining what rating in data is considered a good rating.

Details

The function calculates the accuracy of predictions compared to the observed true ratings (data) averaged over the users. Use byUser = TRUE to get the results for each user.

If the predictions are numeric ratings (i.e., a "realRatingMatrix"), the error measures RMSE, MSE, and MAE are calculated.

If the predictions are a "topNList", the confusion matrix entries (true positives TP, false positives FP, false negatives FN, and true negatives TN) and binary classification measures such as precision, recall, TPR, and FPR are calculated. If the data is a "realRatingMatrix", then goodRating must be specified to identify items that should be recommended (i.e., those with ratings at or above this threshold). Note that you need to specify the number of items given to the recommender to create predictions. The number of predictions by user (N) is the total number of items in the data minus the number of given items. The number of TP is limited by the size of the top-N list. Also, since the counts for TP, FP, FN and TN are averaged over the users (unless byUser = TRUE is used), they will not be whole numbers.

If the predictions are a "topNList" and the observed data is a "realRatingMatrix", goodRating determines which ratings in data count as good for the binary classification measures. An item in the top-N list counts as a true positive if its observed rating is at least goodRating.

Value

Returns a vector with the appropriate measures averaged over all users. For byUser=TRUE, a matrix with a row for each user is returned.

References

Asela Gunawardana and Guy Shani (2009). A Survey of Accuracy Evaluation Metrics of Recommendation Tasks, Journal of Machine Learning Research 10, 2935-2962.

See Also

topNList, binaryRatingMatrix, realRatingMatrix.

Other evaluation: error, evaluate(), evaluationResultList-class, evaluationResults-class, evaluationScheme(), evaluationScheme-class, plot()

Examples

### recommender for real-valued ratings
data(Jester5k)

## create 90/10 split (known/unknown) for the first 500 users in Jester5k
e <- evaluationScheme(Jester5k[1:500, ], method = "split", train = 0.9,
    k = 1, given = 15)
e

## create a user-based CF recommender using training data
r <- Recommender(getData(e, "train"), "UBCF")

## create predictions for the test data using known ratings (see given above)
p <- predict(r, getData(e, "known"), type = "ratings")
p

## compute error metrics averaged per user and then averaged over all
## recommendations
calcPredictionAccuracy(p, getData(e, "unknown"))
head(calcPredictionAccuracy(p, getData(e, "unknown"), byUser = TRUE))

## evaluate topNLists instead (you need to specify given and goodRating!)
p <- predict(r, getData(e, "known"), type = "topNList")
p
calcPredictionAccuracy(p, getData(e, "unknown"), given = 15, goodRating = 5)

## evaluate a binary recommender
data(MSWeb)
MSWeb10 <- sample(MSWeb[rowCounts(MSWeb) >10,], 50)

e <- evaluationScheme(MSWeb10, method="split", train = 0.9,
    k = 1, given = 3)
e

## create a user-based CF recommender using training data
r <- Recommender(getData(e, "train"), "UBCF")

## create predictions for the test data using known ratings (see given above)
p <- predict(r, getData(e, "known"), type="topNList", n = 10)
p

calcPredictionAccuracy(p, getData(e, "unknown"), given = 3)
calcPredictionAccuracy(p, getData(e, "unknown"), given = 3, byUser = TRUE)

recommenderlab documentation built on Oct. 5, 2026, 9:08 a.m.