In dtkaplan/checkr: Facilities for checking R submissions in tutorials

library(learnr)
library(checkr)
knitr::opts_chunk$set(echo = FALSE)
tutorial_options(exercise.checker = checkr::check_for_learnr)

cruel_remarks <- c("Really?", "Think about it!", "Maybe you shouldn't major in this field.",
"Perhaps you should review the textbook.", "Look carefully at what you did.", "No!",
"{{V}} is not right.")
trig_2_check <- function(USER_CODE) {
  code <- for_checkr(USER_CODE)
  t1 <- if_empty_submission(code, message = "OK. Let's get you started. Pick a trig function such as sin(), cos(), tan(), giving the angle as an argument. Perhaps something like tan(40). This won't be right, but it will get you on track. ")
  t1 <- line_calling(t1, sin, cos, tan, message = "You should be using a trigonometric function.")
  t1 <- line_calling(t1, sin, message = "Make sure to pick the right trig function. cos() does horizontal lengths, sin() does vertical lengths.")
  if (failed(t1)) return(t1)
  a1 <- trig_radian_check(t1, 53*pi/180)
  if (failed(a1)) return(a1)
  t2 <- line_where(code, insist(F == "*"", "Remember to multiply by the length of the hypotenuse."))
  t3 <- arg_calling(t2, `*`)

  a1 <- arg_number(t3, 1, insist(V == 15, "How long is the hypothenuse? It says right on the diagram."))
  a2 <- arg_number(t3, 2, insist(V == 15, "How long is the hypothenuse? It says right on the diagram."))
  if (failed(a1 %or% a2)) return(a2)
  line_where(t2, insist(is.numeric(V)), 
             insist(abs(V - 11.98) < 0.01, "{{V}} is a wrong numerical result. It should be about 11.98."), 
             passif(TRUE, "Good!"))
}

cruel_trig <- function(USER_CODE) {
  code <- for_checkr(USER_CODE)
  line_where(code, 
             insist(abs(V - 11.97) < 0.01,  sample(cruel_remarks, 1)),
             passif(TRUE, "See, it wasn't so hard!"))
}

cruel_trig("11.98")

source(system.file("learnr_examples/internal-examples.R", 
                          package = "checkr"))

Scaffolding, hints, and solutions

A problem from trigonometry

Write an R statement to compute the numerical value of x.

..trig_function..( ..angle.. )

15 * sin(53 * pi / 180)

Is x a vertical or horizontal difference? Pick the appropriate trig function.

Sine is for vertical, cosine for horizontal.

The angle should be in radians.

The four questions of passing

• What's to keep students from paging through the hints? Are there consequences?
  • Nothing, unless you log each event.
• What's there to ensure that the final answer is correct?
  • Nothing, unless you log that event and check it later.
• What's in the hint for me?
  • A student who already knows the sin part of the answer still has to walk through that hint.
• What happens to the student who doesn't even know what to ask, and just presses "Run."

With a submit button

You can produce a "Submit" button by adding a -check chunk.

..trig_function..( ..angle.. )

```r
..trig_function..( ..angle.. )
```

```r
for_checkr(USER_CODE)
```

for_checkr(USER_CODE)

The positive message for running code is nice, but too generous!

Once more, with ~~feeling~~ feedback.

The trig problem

3

res <- pre_check(USER_CODE)
if (failed(res)) return(res)

# USER_CODE <- quote("15 * sin(53*pi/180) + 2")
# USER_CODE <- quote("sin(53*pi/180)")

trig_2_check(USER_CODE)

The man behind the curtain

Ordinarily, you would not show students the checkr statements implementing this behavior. But our purpose here is to introduce checkr, So here are the statements for the above exercise.

print_function_contents(
  trig_2_check)
#  from_file = system.file("learnr_examples/internal-examples.R", package = "checkr"))

Breaking this down, line by line:

• [1] accepts the user submission from learnr. The submission is always called USER_CODE. The function for_checkr() does some pre-processing of the user submission to turn it into evaluated code and format it for use in later checkr functions.
• [2] handles the case of an empty submission. Using this allows you to replace a scaffold with a starting suggestion. The nice thing is that the suggestion can be wrong, so student attention is focussed on a specific part of the problem.
• [3] looks for a line containing one of the trig functions. Not necessarily the right one. The student has to think about which one to use.
• [5] & [6] tell the student to keep working until she hits on the right trig function.
• [7] To reach here, the student must have used sin(). So we'll focus on the angle. You can write specialized checkr functions to handle specialized areas. Degrees vs radians will not be an issue in most tutorials.
• [10] makes sure that the multiply function is being used.

Depending on the submission, any of the checksmight fail. If a check fails, later checks that use the previous result will short circuit to a failed check. This allows checks to be chained, with the earliest failure determining the outcome.

Different pedagogies, different checks

Example: the cruel instructor

An instructor with a different pedagogical approach might prefer to structure the checking in an entirely different way. For instance, here are checkr statements that simply tell the user whether or not the submission did what was requested.

# Get started

cruel_trig(USER_CODE)

print_function_contents(cruel_trig)

Really?

Think about it!

Maybe you shouldn't major in this field.

Perhaps you should review the textbook.

Look carefully at what you did.

No!

{{V}} is not right.

Hints?

• I used feedback messages that are cruel.
• I could equally have used hints as feedback.

But to give hints without context seems strange, yet this is what the learnr -hint system does.

rlang, redpen, and checkr

The foundation

checkr is based on the rlang and redpen packages. Two basic technologies:

1. Associating an environment with every code excerpt, so that it can be evaluated in isolation.
2. Pattern binding that lets you associate any function call or argument with a name. Per (1), the name can then be checked or evaluated.

# .(label) refers to an expression
# ..(label) refers to a value
submission <- quote(15 * sin(53 * pi / 180))
redpen::node_match(submission,
                   15 * sin(.(ang)) ~ ang)
redpen::node_match(submission,
                   15 * sin(..(ang)) ~ ang)

checkr adds ...

• Dealing with multiple lines of code.
• A framework for dealing with assignment,
• Patterns written in sets of possibilities (e.g. sin, cos, tan)
• Logic linking success tests
• Integreation with learnr.

Careful!

You need to be careful about whether an expression can be evaluated.

For instance, here, hp and mpg only make sense in the context of mtcars. And it's select() that provides that context.

submission <- quote(mtcars %>% select(hp, mpg))
redpen::node_match(submission,
                   mtcars %>% select(.(v1), .(v2)) ~ c(v1, v2))
redpen::node_match(submission,
                   mtcars %>% select(..(v1), ..(v2)) ~ c(v1, v2))

Checking logic needs to be careful about names versus values when referring to variables.

Writing checking functions

A process

A good way to start when developing a checking function:

a. Write down examples of the correct and incorrect submissions you anticipate. - Present an exercise in class to gain experience. - Deploy a development problem with minimal feedback and harvest the submissions. b. Create the feedback message for each of these submissions. c. ... the magic happens here ... Figure out the checking logic to associate (a) and (b).
Some easy cases: - Did they use a distinctively wrong function? line_calling() %>% misconception() - Cast as a fill-in-the-blanks - Are there named arguments? named_arg()

Debugging

Checkr can be run independent of learnr, so regular test-case and debugging tools can be used.

Fill in the blanks

These problems have a predictable structure, making them easier to check.

Exercise 14: Fill in the blanks in the following code to create a ggplot2 command that will produce the following scatter plot with the mtcars data.
```r library(ggplot2) ggplot(mtcars, aes(x = mpg, y = hp, color = cyl)) + geom_point()

>
> There are four blanks. You'll have to replace all of them with the correct contents to generate the plot.

```r
library(ggplot2)
ggplot(mtcars, aes(x = ____ , y = ____, color = ____)) +
  ____()

check_exer_14(USER_CODE)

The checking code:

print_function_contents(
  check_exer_14,
  from_file = system.file("learnr_examples/internal-examples.R", 
                          package = "checkr"), 
  just_the_body = FALSE)

Pre-evaluation checking

In the ggplot example, pressing "Submit" on the scaffold produced a native R run-time error as the message.

• The learnr system evaluates the submission before handing it to the checker.
• Use a -code-check chunk to hand the submission to the checker before evaluating it.

library(ggplot2)
ggplot(mtcars, aes(x = .... , y = .... , color = .... )) +
  ....()

3 # chunk must have executable content

res <- pre_check(USER_CODE)
if (failed(res)) return(res)
check_exer_14(USER_CODE)

s1 <- "library(ggplot2); ggplot(mtcars, aes(x = ...., y = ...., color = ....)) + ....()"
pre_check(s1)
check_exer_14(s1)

Para-evaluation checking

We can only know if a name is valid by evaluating it in the context of earlier computations.

• The evaluate-then-check process doesn't support this.
• Error messages from pressing "Run" are native, but from "Submit" can be tailored.

Suggestions:

• Make pre-check the default for submissions.
• Add a -run-check option to let "Run" call a checking function, too.
• Let the checking program do all the evaluation.

Feedback on feedback

Providing student feedback is largely an empirical problem, not a logical one. We need to know what student misconceptions are in order to know how to give formative feedback.

• Use the learnr tutorial-event-recorder function: collect submissions.
  • The submittr package (that's another talk!) can handle this with user authentication, etc.
• Can look through these to identify common misconceptions, then test for these.

Present status

• Available via install_github("dtkaplan/checkr")
• Vignette with many examples.
• This document at https://dtkaplan.shinyapps.io/RStudioConf2018-slides

Among other things:

• needs better parse-time diagnostics (e.g. https://github.com/stephdesilva/helpPlease, https://github.com/noamross/zero-dependency-problems)
• better spell-check (e.g. https://github.com/wrathematics/Rdym)
• experience programming with checkr may lead to better high-level functions for simplifying checking, like check_blanks. We need to dig the Pit of Success

dtkaplan/checkr documentation built on May 15, 2019, 4:59 p.m.