In rstudio-education/grader: Automated Feedback for Student Exercises in 'learnr' Tutorials

library(learnr)
library(gradethis)
gradethis_setup()
knitr::opts_chunk$set(echo = FALSE)

Welcome

The gradethis package provides three ways to autocheck student code in learnr tutorials. You can use:

1. grade_result() to check that the student's code returns a correct result.
2. grade_result_strict() to check that the student's code returns a result that satisfies a complete set of conditions. This is akin to unit testing the result.
3. grade_code() to check that the student's code exactly matches the solution code.

You can also use gradethis to create your own customized checking code.

Syntax

In each case, you grade an exercise by passing a gradethis function to the -check code chunk associated with the exercise. This code chunk will share the same label root as the exercise code chunk, but the root will be suffixed with -check.

See the learnr documentation to learn more about code chunks in learnr tutorials.

Here is how an example exercise and -check chunk would appear in a learnr document.

`r ''````r
x + 1
```

`r ''````r
grade_result(
  pass_if(~ identical(.result, 2), "Good job!")
)
```

Goal

gradethis functions are designed to provide formative feedback to the student. When a student clicks "Submit Answer" in his or her learnr tutorial, the gradethis function immediately performs three tasks. It:

1. Displays whether or not the answer is correct
2. Returns an instructional message customized to the student's submission
3. Offers dynamically generated encouragement

gradethis does not itself calculate a final grade for the student.

Setup

To use gradethis inside a learnr tutorial, we suggest calling gradethis::gradethis_setup() inside the tutorial's setup chunk. This is how it will look in the learnr R Markdown file:

`r ''````r
library(gradethis)
gradethis_setup()
```

You can also see this in the first code chunk of the .Rmd file associated with this tutorial.

The remainder of this document looks at how to use individual gradethis functions.

grade_result()

grade_result() checks whether or not the student's code returns the correct result. It matches the result against one or more conditions and returns the message (and correct/incorrect status) associated with the first matched condition.

Here is an example of grade_result() in use.

• Calculate the average number of weeks in a month.

grade_result(
  pass_if(~ identical(.result, 365.25 / 12 / 7), "There are 4.348214 weeks on average in a month."),
  fail_if(~ identical(.result, 365 / 7 / 12), "Did you assume the average year has 365 days? Due to leap years, the average year actually has 365.25 days."),
  fail_if(~ identical(.result, 52 / 12), "Did you assume that the average year has 52 weeks? It is actually has a little more because 52 * 7 is only 364 days."),
  fail_if(~ identical(.result, 4), "Close, but four is the average number of whole weeks in a month."),
  fail_if(~ TRUE, "Not quite. Consider that there are 365.25 days in an average year, 12 months in a year, and 7 days in a week.")
)

And here is the code behind the example.

`r ''````r

```

`r ''````r
grade_result(
  pass_if(~ identical(.result, 365.25 / 12 / 7), "There are 4.348214 weeks on average in a month."),
  fail_if(~ identical(.result, 365 / 7 / 12), "Did you assume the average year has 365 days? Due to leap years, the average year actually has 365.25 days."),
  fail_if(~ identical(.result, 52 / 12), "Did you assume that the average year has 52 weeks? It is actually has a little more because 52 * 7 is only 364 days."),
  fail_if(~ identical(.result, 4), "Close, but four is the average number of whole weeks in a month."),
  fail_if(~ TRUE, "Not quite. Consider that there are 365.25 days in an average year, 12 months in a year, and 7 days in a week.")
)
```

Syntax

grade_result() should contain a sequence of fail_if() and pass_if() functions.

Each fail_if() and pass_if() function should contain:

1. A logical test prefixed by a ~
2. A character string to display if the logical test evaluates to true

Use .result to refer to the student's answer within the logical tests.

Execution

grade_result() will evaluate the _if functions in order, replacing .result with the student's result as it does.

grade_result() will stop and return the message of the first _if function whose condition evaluates to true. If that function is:

• pass_if(), the exercise will be marked correct
• fail_if(), the exercise will be marked wrong

Order matters! grade_result() will not continue to evaluate _if() functions after one returns a message.

Choose the best grading function for you

grade_result() will mark a result correct if it passes a single pass_if() statement (without first triggering a fail_if()).

If you would like to ensure that a result satisfies every pass_if statement use grade_result_strict().

grade_result() will not check the students code. Nor will grade_result() know if the student directly typed the correct result into the exercise box.

If you want to check the code the student used to get a result, use grade_code()

See ?grade_result for more information.

grade_result_strict()

grade_result_strict() is similar to grade_result(), but it requires a result to pass every pass_if() function contained in its function body. This method is analogous to creating unit tests that all need to pass.

Here is an example of grade_result_strict() in use:

Please make a function in the exercise space below, but do not assign the function to an object. The function should:

• Take a single x argument
• Add the integer 1 to the x value.

Then click Submit Answer.

function(x) {
  # solution is x + 1L
  x + 1
}

grade_result_strict(
  pass_if(~ .result(3) == 4),
  pass_if(~ identical(.result(0), 1)),
  pass_if(~ identical(sapply(1:10, .result), 2:11)),
  pass_if(~ sapply(1:10, .result) == 2:11),
  pass_if(~ all.equal(sapply(1:10, .result), 2:11)),
  pass_if(~ checkmate::test_function(.result, args = c("x")))
)

And here is the code behind the example.

`r ''````r
function(x) {
  # solution is x + 1L
  x + 1
}
```

`r ''````r
grade_result_strict(
  pass_if(~ .result(3) == 4),
  pass_if(~ identical(.result(0), 1)),
  pass_if(~ identical(sapply(1:10, .result), 2:11)),
  pass_if(~ sapply(1:10, .result) == 2:11),
  pass_if(~ all.equal(sapply(1:10, .result), 2:11)),
  pass_if(~ checkmate::test_function(.result, args = c("x")))
)
```

Syntax

grade_result_strict() takes a set of pass_if() functions. Each pass_if() function should contain:

1. a logical test prefixed by a ~

Use .result to refer to the student's answer within the logical tests.

Evaluation

grade_result_strict() will mark a result as correct only if passes every pass_if() statement. This is especially useful for grading function definitions.

See ?grade_result_strict for more information.

grade_code()

grade_code() to checks whether the student code matches the solution code. If the code does not match, grade_code() will tell the student exactly where their code begins to diverge from the solution and how to get back on track. Here's an example:

• Take the square root of the log of 2, then click Submit Answer.

sqrt(log(2))

grade_code("Don't worry, things will soon get harder.")

We can also provide a more specific feedback when we expect an assign statement, for example.

• The expected here is a <- sample(1:6, size = 1). Enter 123, then click Submit Answer.

a <- sample(1:6, size = 1)

grade_code("Don't worry, things will soon get harder.")

You should see a red box with "I expected you to assign something to something else with <- where you wrote 123."

Grading exercise errors

Since gradethis_setup() sets learnr::tutorial_options(exercise.error.check.code = "gradethis::grade_code()"), when an exercise submission produces an evaluation error, intelligent feedback is provided via grade_code() (when a solution is provided). Try it yourself by submitting mt (R error, incorrect answer), then cars (No error, incorrect answer), then mtcars (correct answer) below:

mtcars

grade_result(
  fail_if(~identical(.result, cars), "This is the cars (not mtcars) dataset."),
  pass_if(~identical(.result, mtcars))
)

And here is the code behind the example.

`r ''````r

```

`r ''````r
mtcars
```

`r ''````r
grade_result(
  fail_if(~identical(.result, cars), "This is the cars (not mtcars) dataset."),
  pass_if(~identical(.result, mtcars))
)
```

In the case that you wish to provide custom checking logic for errors, you can also provide a -error-check chunk. If the student submission generates an error, gradethis will run the checking code in the -error-check chunk instead of the checking code in the -check chunk. gradethis will only run the -error-check code if the student submission generates an error.

mtcars

grade_code(incorrect = "This code produces an error (press 'Run Code' to see it).", glue_incorrect = "{ .message } { .incorrect }")

grade_result(
  fail_if(~identical(.result, cars), "This is the cars (not mtcars) dataset."),
  pass_if(~identical(.result, mtcars))
)

And here is the code behind the example.

`r ''````r

```

`r ''````r
mtcars
```

`r ''````r
grade_code(incorrect = "This code produces an error (press 'Run Code' to see it).", glue_incorrect = "{ .message } { .incorrect }")
```

`r ''````r
grade_result(
  fail_if(~identical(.result, cars), "This is the cars (not mtcars) dataset."),
  pass_if(~identical(.result, mtcars))
)
```

Syntax

grade_code() requires a model solution to compare student code to. Supply this solution in a learnr -solution chunk, i.e. a chunk whose label is the label of the exercise it is associated with followed by -solution:

`r ''````r
sqrt(log(2))
```

grade_code() will compare the last expression in the student submission to the last expression in the solution chunk.

Teachers will usually pass grade_code() a character string to display to the student if their code successfully matches the solutiuon, e.g.

grade_code("Good job. Don't worry, things will soon get harder.")

Execution

grade_code() does not check the result of the student's code.

Instead, grade_code() parses the student code into a call tree. As it does, it standardizes argument names and accounts for the presence of pipes (%>%).

grade_code() then does the same for the solution code.

Finally, grade_code() recursively walks the two call trees.

grade_code() stops and returns a message if the student code:

1. contains malformed code that would cause an error if passed to match.call()
2. contains a different element than the solution
3. contains an extra element that is not in the solution
4. is missing an element that appears in the solution

If none of the above occurs, grade_code() marks the answer as correct.

Feedback

grade_code() attempts to supply helpful feedback that makes sense to the student without giving away the solution. To do this, grade_code() messages take the form of

"I expected X, where you wrote Y. Please try again."

Here Y may be an argument or function call that appears at the beginning, end, or middle of the student code.

grade_code() will catch mistakes one at a time in the order that they appear in the students code. This lets the student iteratively improve their code.

See ?grade_code for more information.

Custom Checking Code

gradethis can accept any checking method that returns a gradethis::graded object.

The example below returns a correct/incorrect answer with 50/50 probability.

`r ''````r
"Flip a coin"
```

`r ''````r
fifty_fifty_checker <- function(
  correct = "Correct!",
  incorrect = "May the odds be ever in your favor!",
  ...,
  user
) {
  is_correct <- (runif(1) < 0.5)
  gradethis::graded(
    correct = is_correct,
    message = ifelse(is_correct, correct, incorrect)
  )
}
fifty_fifty_checker()
```

Syntax

The custom checking function should return the output of gradethis::graded() which takes two arguments:

1. correct - TRUE is the answer is correct, FALSE otherwise
2. message - an optional character string to display to the student

Evaluation

To use the custom checking function, place it in an exercise -check chunk, as you would place grade_result(), grade_result_strict(), or grade_code().

Try it!

---

Are you feeling lucky?

If you are, click Submit Answer.

"Flip a coin"

fifty_fifty_checker <- function(
  correct = "Correct!",
  incorrect = "May the odds be ever in your favor!",
  ...,
  user
) {
  is_correct <- (runif(1) < 0.5)
  gradethis::graded(
    correct = is_correct,
    message = ifelse(is_correct, correct, incorrect)
  )
}
fifty_fifty_checker()

See ?graded for more information.

rstudio-education/grader documentation built on July 6, 2023, 8:48 a.m.