Create OLAT Tests with R/exams

In c403: Exam Tools for Department of Statistics (c403), Uni Innsbruck

suppressPackageStartupMessages(library("exams"))

This vignette describes how to get started using the R exams package to create randomized OLAT tests (or quizzes). Several tutorials are available on r-exams.org which also help beginners with getting used to the features of the R exams package.

The following tutorials might be of interest:

• First Steps a first overview of the structure of R/exams.
• Stress Testing Dynamic R/exams Exercises automatic stress-test of randomized exercises.

Start to write R/exams questions

R/exams questions are written in either pure markdown (.md), R/markdown (.Rmd) or R/LaTeX using Sweave (.Rmd). For demonstration purposes this tutorial is only using R/Markdown files, more information can be found in the [First Steps][rexams-firsteps] tutorial on r-exams.org.

Write your first static question

To create a new question we write a new .Rmd file and edit the content with a text editor, e.g., a file called iso3-code.Rmd. The following output shows the content of this file (iso3-code.Rmd) which contains a simple 'static' single choice R/exams question:

cat(paste(readLines("iso3-code.Rmd"), collapse = "\n"))

Each question consists of three sections, namely "Question", "Solution", and "Meta-information".

• Question: contains the question itself, simply text which contains the task. The "Answerlist" subsection defines the possible answers (here 3).
• Solution: this is typically not shown to the participants, or only depending on the system settings (e.g., if the 'solutionswitch' in OpenOLAT is enabled). Can contain text with an extensive solution if needed, and contains a subsection "Answerlist" in the same order as the possible answers in the "Question" section.
• Meta-information: contains the name and type of the question, information about the correct solution, and additional settings. In this case it is a single choice question (extype: schoice), exsolution defines the correct answer. Three possible answers, the first one is correct (thus 100). exshuffle: True tells R/Exams to shuffle the possible questions (3 out of 3; random order).

Testing the question

Once we have saved our file (iso3-code.Rmd) we can test the question by calling:

# Renders the question as html
exams2html("iso3-code.Rmd")

This will open the html-rendered version of the question in your browser. As exshuffle is turned on you will get a different randomization every time the command above is called. It is suggested to do this multiple times to see if the randomization works as expected.

Testing the solution

The function exams_metainfo() allows to check/test if the solution defined in the Meta-information section is specified correctly.

set.seed(100)

# Test solutionstring
exams_metainfo(exams2html("iso3-code.Rmd"))

The output shows the correct answer is "2" which can be compared with the rendered html output (not shown here) to see if it is correct.

Stress-testing the question

Once one has checked that the randomization works, the texts are correct, and the answer is correct a stress-test for the question can be done by calling stresstest_exercise().

stresstest_exercise() renders the exercise multiple types (100 times by default) and returns some information about the randomization.

# Stresstesting 'iso3-code.Rmd' (100 times)
stress <- stresstest_exercise("iso3-code.Rmd")
plot(stress)

The output shows the runtime to generate the randomized questions and the position of the correct solution. As we use exshuffle: True with three possible answers this should be close to a uniform distribution between 1, 2, and 3.

Write our first dynamic question

R/exams questions can also be dynamic. The following example is the content of a second question stored in vector-subsetting.Rmd.

cat(paste(readLines("vector-subsetting.Rmd"), collapse = "\n"))

Testing the question/solution

Again, it is recommended to check the question multiple times by calling:

# Test html output and meta information (correct answer)
exams_metainfo(exams2html("vector-subsetting.Rmd"))

Stress-testing

In contrast to the static single choice exercise this exercise uses a randomized vector x, randomized vector indices (idx), and randomized correct values (res). Thus, in addition to the 'runtime', 'position of correct solution' and 'rank of correct solution' the function also returns the values of all scalars (here idx, res) of all randomizations during the stress test.

## Run the stresstest
#stress <- stresstest_exercise("vector-subsetting.Rmd")
## Plot overview
#plot(stress)

The additional information stored on stress$objects can be used to get more insights, e.g., how often each index idx $\in {1, ..., 5}$ has been chosen and what the correct answers were (res):

## Plot histograms of the two scalars idx/res from the
## 100 randomizations performed by the stresstest function
#par(mfrow = c(1, 2), mar = c(5, 5, 2, 1))
#hist(stress$objects$idx, col = "gray90",
#     breaks = seq(0.5, 5.5, by = 1),
#     main = "Histogram of idx")
#hist(stress$objects$res, col = "gray90",
#     main = "Histogram of res")

Generate OLAT Test for Upload

Once the questions have been tested and approved we can create the final test used to upload to OpenOLAT. This is done by calling the exams2openolat() function. The first input can be a single file (e.g., iso3-code.Rmd; results in a test with only one question) or a list of files. Each entry of the list contains the name of one .Rmd file and corresponds to one question in the quiz. If the list entry contains a character vector with multiple file names, R/exams will chose one of the files provided to generate the question (allows for additional randomization of questions).

Note: exams2openolat() has a wide range of additional input arguments to adjust/control the test settings (see R/exams). To generate the zip archive file for OpenOLAT we have to call the following:

# Using the c403 package
library("c403")
# Set seed (makes the randomization reproduceable
set.seed(321)
# Generate OpenOLAT test/exam resource using the c403 package
files <- list("iso3-code.Rmd", "vector-subsetting.Rmd")
c403::exams2openolat(file = files, quiet = TRUE,
                     n = 3L, name = "test-quiz",
                     maxattempts = 100L)

• file: the files with the questions to be converted.
• n: number of randomized tests to be generated. Default is n = 1L. If you create n = 10L tests for 50 participants each test might be re-used about 5 times (n does not limit the number of participants).
• name: name of the test (will generate <name>.zip as output file; default is \code{"olattest"}).
• maxattempts: number of attempts - number of times the participants are allowed to change the selected answer. Default is only 1 (maxattempts = 1L).

The function exams2openolat() from the c403 package automatically creates two files: the zip file (test-quiz.zip) to be uploaded to OpenOLAT and the file test-quiz.rds which contains the randomized questions/tests.

Alternatively one can use the exams2openolat() function from the exams package. Note: exams::exams2openolat() does not automatically save the randomized questions (.rds file), however, it is very important to store this information if one wants to evaluate the tests later on. It is also highly recommended to set a seed for pseudo-randomization. This allows to reproduce the generated tests if the .rds file gets lost. If one does not want to use the c403 package you can do it as follows using the exams package:

# -------------------------------------
# Or: load the exams package
# -------------------------------------
suppressPackageStartupMessages(library("exams"))
# Set seed (makes the randomization reproduceable
set.seed(321)
# Generate OpenOLAT test/exam resource
exams <- exams:exams2openolat(file = files,
                              n = 3L, name = "test-quiz-exams",
                              maxattempts = 100L)
# Save the generated tests for evaluation
saveRDS(exams, file = "test-quiz-exams.rds")

Note that c403::exams2openolat() and exams::exams2openolat() have different default arguments (e.g., the c403 package uses the German language by default).

Upload test to OpenOLAT

Once the test has been generated one can import the tests to OpenOlat. To do so, log in to OpenOLAT and go to the Authoring section. The zip file (test-exam.zip) can be uploaded via the Import button:

knitr::include_graphics("olat_test_import.jpg")

It is highly recommended to give the learning resource a clear and unique title such that one can easily find the resource later on. Once the test has successfully been imported a new course element has to be created. First, navigate to your course in OpenOLAT and enter the Administration area. Afterwards there are two options:

• Create a new course element of type Test,
• or copy an existing Test course element.

The latter option has the advantage that it also copies the settings of the course element! To 'attach' the uploaded learning resource select the course element, click on Test configuration and select or replace (in case you made a copy of an existing test course element) the 'file' (the uploaded test/zip file).

knitr::include_graphics("olat_create_test1.jpg")
knitr::include_graphics("olat_create_test2.jpg")
knitr::include_graphics("olat_create_test3.jpg")

Note: check and adjust all test settings from Title and description, accessibility, and course element options!

c403 documentation built on Oct. 20, 2023, 3:01 p.m.