In dtkaplan/checkr: Facilities for checking R submissions in tutorials

library(checkr)
library(ggplot2)
library(mosaic)
library(dplyr)
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

Computer tutorials are systems for guiding the interaction between the student and the computer. In a typical tutorial exercise, the student is presented with a task and asked to modify or construct computer commands to carry out the task. Several things are accomplished by tutorial systems such as the RStudio's learnr system, the swirl package, or DataCamp's browser-based GUI and testwhat package:

1. Integrating explanatory narrative with exercises.
2. Establishing a path through the narrative and exercises that can be covered at a pace set by the student.
3. Providing a sandbox in which the student can safely evaluate commands.
4. Examining student command submissions to determine whether the exercise task has been appropriately completed, and, if it has not, provide hints to guide the student.
5. Store a record of student progress through tutorials.

The checkr package supports a facility for (4) that can be integrated into tutorial systems such as learnr. It is hoped that checkr will enable instructors to write exercises that engage in a learning dialog with students, a dialog that can approximate the natural interaction between instructor and student that occurs when watching a student construct computer commands.

The checkr framework

An instructor uses checkr by writing a custom checking function that takes computer code as input, analyzes that code, and depending on what it finds generates a pass-or-fail output containing a message that ideally is relevant and useful to the student. Let's look at the components of the description in the previous sentence.

• "computer code as input" -- The computer code is in the form of text generated by the student.
• "analyzes that code" -- To analyze is to "take apart." The computer code consists of one or more commands (which I'll call "lines of code"). Functions in the checkr package can separate the code into components according to patterns specified by the instructor. A simple pattern, for instance, is whether the line invokes some specified function such as plot().
• "depending on what it finds" -- Once a relevant component of code has been isolated, tests can be performed on the elements of that component. Each such test can be associated with a message and a pass/fail/ok result. The custom function assembles the results from the various code components and integrates them into a single, overall result.
• "relevant and useful" -- The instructor, in writing the checking function, can check the submitted code into it's components at any level of detail and provide context-sensitive feedback regarding problems with specific components.

A rich example

Suppose the task set for the student is:

Exercise 1: Construct an appropriate linear regression model of the mpg of cars in the mtcars data set. Your model should have an R-squared of at least 0.30.

It's up to the instructor to decide how finely to look at the student's submission. Suppose you have decided to check:

a. Is the lm() function being used? b. Is mtcars the data argument to lm()? c. Is mpg on the left-hand side of the formula argument to lm()? d. Is the R-squared at least 0.30?

Consider several possible student submissions:

s1 <- "lm(mpg ~ hp, data = mtcars)" # Right!
s1wrong <- "lm(mpg ~ hp, data = head(mtcars))"
s2 <- "mod <- lm(mpg ~ hp, data = mtcars); summary(mod)" # Right!
s2wrong <- "mod <- lm(hp ~ mpg, data = mtcars); summary(mod)"
s3wrong <- "for_me <- mtcars; mosaic::rsquared(lm(data = for_me, mpg ~ 1))"

Some of these are right and some are wrong. (s2wrong will produce a model with a passing R-squared, but it's a model of hp rather than mpg. s3wrong has a model formula that cannot produce any R-squared but zero.) The commands have different structures. In s1, even though it's a satisfactory model, the R-squared is not calculated at all. (Perhaps the student "cleaned up" their code before submitting.)

One pedagogical choice would be to insist that the student submission follow a specific template. The pedagogical choice I'll make here is to focus on the model itself and accept variant forms of the submission so long as they make a model with the specified qualities.

The checkr package provides a means to implement your pedagogical choices. To illustrate, consider the function check_exer_1() defined below. Check_exer_1() uses several checkr functions that have not yet been explained, so on a first reading it will be hard to make sense of. Still, it may give you a general impression.

check_exer_1 <- function(USER_CODE) {
  code <- for_checkr(USER_CODE) # pre-processing
  lm_line <- line_calling(code, lm, message = "Use lm() to construct the model.")
  lm_call <- arg_calling(lm_line, lm)
  t1 <- data_arg(lm_call, 
                 insist(identical(V, mtcars), 
                        "Your data argument {{E}} was not `mtcars`."),
                 message = "You didn't supply a `data = ` argument to `lm()`.")
  if (failed(t1)) return(t1)
  f <- formula_arg(lm_call,
                  message = "You didn't give a formula specifying the structure of the model.")
  t2 <- check(f, insist(two_sided(f), "There's no response variable in your formula."))
  t2 <- check(t2, insist(rlang::f_lhs(E) == as.name("mpg"), 
                   paste("You need to have the miles-per-gallon variable",
                         "on the left side of the model formula.",
                         "You've got {{rlang::f_lhs(V)}} instead.")))
  if (failed(t2)) return(t2)
  check(lm_call, 
        insist(summary(V)$r.squared > 0.3, 
        "Your R-squared is {{summary(V)$r.squared}}. That's too small."),
        passif(TRUE, "Great job!"))
}

print_function_contents(check_exer_1, just_the_body = FALSE)

Checkr provides functions like line_calling(), formula_arg(), data_arg(), insist(), and check().

Evaluating the various test cases of submissions with the check_exer_1() function shows how relevant and useful feedback can be given while largely disregarding coding style.

check_exer_1(s1)
check_exer_1(s1wrong)
check_exer_1(s2)
check_exer_1(s2wrong)
check_exer_1(s3wrong)

The printed form of the result includes whether the submission passed or failed, a congratulatory or diagnostic message, and the component of the code being checked at the point where the test passed or failed. When integrated with a tutorial system such as learnr, the checkr result is formatted as appropriate for that user interface. (For instance the code component is mainly of interest to the author of checking functions, not the student.)

A range of approaches

Ultimately, it's up to the instructor to decide how much flexibility to give the student and how much to guide the student. To illustrate some possibilities, consider this task:

Exercise 2: Use the rep() function, with 1:4 as an input, to generate the 12-element vector 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4

The most flexibility comes by checking if the correct result was produced. You can check the value produced by a line with the pronoun V (for value). Like this:

check_exer_1_v0 <- function(USER_CODE) {
  code <- for_checkr(USER_CODE)
  desired <- rep(1:4, each = 3)
  line_where(code, insist(all(V == desired), "Your vector is {{V}}. That is not the result asked for."))
}

This may be more flexbility than desired. Here's a cheating solution that passes:

check_exer_1_v0("c(1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4)")

Another approach is to have in mind a specific command and check for exactly that. This provides little or no flexibility. For instance:

check_exer_1_v1 <- function(USER_CODE) {
  code <- for_checkr(USER_CODE)
  line_binding(code, rep(1:4, each = 3), passif(TRUE, "Just what I wanted!"), 
               message = "Sorry. Not exactly what I was looking for.")
}

This won't be fooled by the cheating answer ...

check_exer_1_v1("c(1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4)")

... and it will work with a command matching that specified, even allowing assignment or trivial white-space changes ...

check_exer_1_v1("x <- rep(1:4,each=3); x")

But here are reasonable student submissions that won't pass the test:

check_exer_1_v1("x <- 1:4; rep(x, each = 3)")
check_exer_1_v1("sort(rep(1:4, 3))")

The challenge is to identify what aspects of the submission we require, but not to overspecify the details. For instance, the exercise statement specifies using rep() and that one argument will be 1:4. To test this:

check_exer_1_v2 <- function(USER_CODE) {
  code <- for_checkr(USER_CODE)
  desired <- rep(1:4, each = 3)
  line_a <- line_calling(code, rep, message = "I'm not seeing where you used `rep()`.")
  t1 <- vector_arg(line_a, insist(all(V == 1:4), "Where did you use `1:4`?"))
  if (failed(t1)) return(t1)
  line_where(code, insist(all(V == desired), "Your vector is {{V}}. That is not the result asked for."))
}

check_exer_1_v2("x <- 1:4; rep(x, each = 3)")
check_exer_1_v2("sort(rep(1:4, 3))")

If the point of the exercise is to have the student see that rep() has optional named arguments, perhaps you want to check if the line involving rep() also has an argument named each.

check_exer_1_v3 <- function(USER_CODE) {
  code <- for_checkr(USER_CODE)
  desired <- rep(1:4, each = 3)
  LineA <- line_calling(code, rep, message = "I'm not seeing where you used `rep()`.")
  t1 <- vector_arg(LineA, insist(all(V == 1:4), "Where did you use `1:4`?"))
  if (failed(t1)) return(t1)
  rep_call <- arg_calling(LineA, rep) # in case rep() is buried in another function application, e.g. 1 * rep()
  t2 <- named_arg(rep_call, "each", 
                  insist(V == 3, "Remember, you want 12 elements in the output made from the 4 elements in the input"), 
                  message = "See what use you can make of the `each` argument to rep().")
  if (failed(t2)) return(t2)
  line_where(code, insist(all(V == desired), "Your vector is {{V}}. That is not the result asked for."))
}

Starting out

The point of checkr is to allow interactive tutorials, such as learnr documents, to provide adaptive feedback. To start, however, I recommend that you develop your checkr commands within an ordinary R script or Rmd file where it's easy to inspect intermediate results and debug. Once you have the commands working satisfactory, it's straightforward to move your checking code into the interactive tutorial document.

A reasonable process is ...

1. Write down a few examples of possible submissions and what sort of feedback you want to give for each.
2. Create a script file containing your examples.
3. Put your checkr lines in a function that takes one argument: USER_CODE.
4. When you have all your examples working, integrate the function into your tutorial. For learnr documents, this amounts to creating a -check chunk in which you call your function with one argument: USER_CODE.

Let's illustrate this with a specific example problem.

Read the data file "http://www.lock5stat.com/datasets/HoneybeeCircuits.csv" into a dataframe named Circuits.

1. Examples of possible submissions

Circuits <- read.csv("http://www.lock5stat.com/datasets/HoneybeeCircuits.csv")

Response: Right!

Circuits <- load("http://www.lock5stat.com/datasets/HoneybeeCircuits.csv")

Response: Notice that the filename has a CSV extension. load() is for reading RDA files. Try read.csv() instead.

read.csv("http://www.lock5stat.com/datasets/HoneybeeCircuits.csv")

Response: Remember to store the contents of the data file under the name Circuits.

bees <- read.csv("http://www.lock5stat.com/datasets/HoneybeeCircuits.csv")

Response: Store the data under the name Circuits, not bees.

Of course, there are many other possibilities. Students are particularly good at finding them.

2. Create a script file

Open up an R script file and add your example submissions. Let's suppose the file is named check_bee_data.R. It will look like this:

# this is file check_bee_data.R
s1 <- quote(Circuits <- read.csv("http://www.lock5stat.com/datasets/HoneybeeCircuits.csv"))
s2 <- quote(Circuits <- load("http://www.lock5stat.com/datasets/HoneybeeCircuits.csv"))
s3 <- quote(read.csv("http://www.lock5stat.com/datasets/HoneybeeCircuits.csv"))
s4 <- quote(bees <- read.csv("http://www.lock5stat.com/datasets/HoneybeeCircuits.csv"))

The quote() function will be unfamiliar to many R users. It provides a way to objects that contain executable expressions.

3. Create the checking function

A checking function takes a submission, for example s1 or s2, and returns a value indicating whether the submission passes or fails along with an appropriate congratulatory or explanatory message. The function has a straightforward skeleton.

# we are still in the file check_bee_data.R
check_bee_data <- function(USER_CODE) {
  code <- for_checkr(USER_CODE)
  # The messages 
  m1 <- "Right!"
  m2 <- "Notice that the filename has a CSV extension. `load()` is for reading RDA files. Try `read.csv()` instead."
  m3 <- "Remember to store the contents of the data file under the name `Circuits`."
  m4 <- "Store the data under the name `Circuits`, not `{{Z}}`."

  # The checking statements will follow

}

The messages are plain text strings. I've named them (e.g. m1) purely for later convenience. Looking at m4, you'll notice the characters {{Z}}. Similarly, there's {{F}} in m2. Such "moustached" statements allow you to include in the message the result of some calculation. For m4, we want the assignment name used by the student. For m2, we want the name of the function actually used by the student.

At this point, you need to start thinking about the order in which you want to check the submission. For instance, do you want to check the function being used or that a correctly named object is being created? In this example, we'll start with the latter.

# also in the file check_bee_data.R
check_bee_data <- function(USER_CODE) {
  code <- for_checkr(USER_CODE)
  # The messages 
  m1 <- "Right!"
  m2 <- "Notice that the filename has a CSV extension. `{{F}}` is for reading RDA files. Try `read.csv()` instead."
  m3 <- "Remember to store the contents of the data file under the name `Circuits`."
  m4 <- "Store the data under the name `Circuits`, not `{{Z}}`."

  browser()
  result <- line_where(code, 
                      passif(Z == "Circuits"), 
                      failif(Z == "", m3), 
                      failif(TRUE, m4))

  return(result) # return the result of the checking
}

To find out what the pronoun Z stands for, see the documentation for line_where(), or look later in this vignette.

The check_bee_data() function does not yet fully implement the tests needed for the various submissions, but it is a working function that handles s3 and s4. You need merely run the submissions through the function to see this:

# just checking ...
check_bee_data(s3)
check_bee_data(s4)

Another checking statement is needed to make sure that the read.csv() function is being used. It could be as simple as this:

# this will go in the `check_bee_data()` function
result <- line_where(result, insist(F == "read.csv", m2))

Unfortunately, things are not so simple. We intend message m2 to be given in response to submission s2. But there's something special about s2 as compared to the other submissions: s2 generates a run-time error; load() is not set up to accept a character string URL as an argument. Such errors need to be trapped before the code is evaluated. We discuss the matter below in the section on re-evaluation checking.

4. Integrating the checking with learnr

Once you have your checking function in a reasonable state, you can integrate it with learnr. There are two components to this.

1. In the setup chunk of your tutorial a. Tell learnr to connect to checkr with the statement tutorial_options(exercise.checker = checkr::check_for_learnr) b. Source the script file where you define your checking function. Typically, such files need to be placed in the www subdirectory of your learnr application. The statement will look like this: source("www/check_bee_data.R")
2. Put the call to the function in the appropriate -check chunk for your exercise. The call must always use USER_CODE as the argument; this is the way the learnr exercise checker will set things up.

checkr_result objects

These are the input and output of many checkr functions. Your checkr statements should ultimately return one of these. These automatically get translated into the format that learnr wants by check_for_learnr()

Some R background

Most R users learn how to compute on dataframes, vectors, and so on. The checkr system involves something different: computing on the language itself. That is to say, the inputs to most checkr functions are language statements.

It will help you to use checkr if you learn something about the structure of the R language. Your students won't have to know this, but the checking code that you read and write will be easier to understand with this knowledge of language structure.

Lines, functions, arguments, constants, names, values

Consider this sequence of R commands:

x <- 3 / 5
sqrt(x)

The code is written on three lines typographically, but it's also helpful to think about "lines" computationally. Informally, let's define a line of code to be a set of characters which, if typed at the command prompt in the console, would cause the R interpreter to perform a computation rather than to prompt you with the + continuation. For instance, the following code consists of just one computational line even though typographically it is spread out over three lines on the page:

paste("How now",
      color, animal, 
      "?")

Every line of R code consists of one or more expressions, each of which could constitute a complete command and thus could compute a value. In the line x <- 3 / 5, for example, you can see expressions 3 and 5. As it happens, these two expressions are both constants. Another kind of constant is a character string: "like this". In the line sqrt(x), there is a simple expression, x, which is called a name. And, of course, there is a value assigned to the name x by the previous line. Other names in the examples are sqrt, and `\`. The values of those names were assigned by the R system itself during start-up. You can look at the value bound to any name by giving the name itself as a command:

sqrt

Ignore for the moment the assignment part of the line; we'll treat that separately.

• Statement 1: Every line in R is either a single constant or a single name or a function call.
• Statement 2: Every function call in R involves two parts: 1) the function itself (typically specified with a name, e.g. . `+`, sqrt, paste) and 2) the arguments to which the function is being applied.
• Statement 3: Every argument in a function call is either a single constant (e.g. "How now") or a single name (e.g. color) or a function call.

Some of the functions in checkr are intended to isolate a specific line. Other functions isolate a function call or an argument to a function. Once isolated, you can check the name of the function involved in the function call, or the name (if any) or value (always) used as an argument.

PRACTICE IDENTIFYING the parts of a line as either a function call, a name, or a constant.

Quote and quo

You're used thinking of functions as creating dataframes or vectors and so on. You may not have directly encountered them, but there are special functions that create an R-language expression itself. Examples are substitute(), parse(), expression(). To illustrate, consider the quote() function. (Don't be deceived: quote is not about creating character strings.)

y <- quote(foo(f, g))

A standard way to think about foo(f, g) is "perform the action of the function foo() on the two arguments: f and g. and then take the sine of the result." But in terms of the language, you can see four names: f, g, foo. There's also some punctuation: the parentheses that signal that foo is to be applied to its arguments and the comma that separates the two arguments.

Think about what might be the value bound to the name y after the above line is evaluated. And while you're doing it, assume that there is no defined function named foo and that no value has been yet bound to the names f and g.

Ordinarily, referring to a valueless name like f generates an error: Error: object 'f' not found. But quote() is special. It doesn't try to evaluate its argument and so it has no need for names to be bound to values. So the object y is not about the results of foo(f, g) but about its status as a command that can be evaluated later on. Instead, quote() figures out from the syntax of its argument that foo(f, g) is a call to a function. (A name followed immediately by an opening parenthesis is always a function call.)

y        # display printed form
class(y) # the object's class

The object y produced by quote(foo(f, g)) is of class "call," reflecting that the expression given as an argument to quote() is a function call. The printed form of y looks just like the argument to quote(). But you can choose to look at y in another way, as a list:

as.list(y)

The list starts with the function name, with the remaining parts being the two arguments. In this example, each of the three items in the list are names:

lapply(y, class)

This list-like structure for a function call applies just as well to arithmetic and other infix functions. R allows you to create a call without using parentheses, for instance 2 + 3. The internal storage of the expression is nonethess list-like. For instance:

y <- quote(x ^ 2)
class(y)
as.list(y)

Do note the backticks around the name in the first element of the list: `^`. In general, when you are referring to functions used with infix notation, you need to construct the function name with backticks. Examples: ||, *, >, and so on.

Let's consider a slightly more complex case, where the arguments themselves involve function calls:

y <- quote(foo(f + 3, paste("Hello,", g, "Nice to meet you.")))

Again, y is a "call" and there are three parts to it: the function name and the two arguments:

y
class(y)
as.list(y)
lapply(y, class)

Note that parts 2 and 3 of y are no longer names. Instead, they are themselves function calls. To illustrate, let's look at the third part of the list.

y[[3]]
class(y[[3]])
as.list(y[[3]])
lapply(y[[3]], class)

The call paste("Hello", g, "Nice to meet you.") is a call to the function named paste with three arguments. Two of these arguments are constants; they happen to be character constants. The middle argument, g is a name.

You can evaluate a name or a call by using the eval() function. Of course, evaluation needs values to be assigned to any names used, so evaluating y will throw an error:

eval(y)

Quote() and eval() (and other similar language-handling functions like substitute() or even the modeler's tilde) are part of base R. They are part of how the language works. They are not part of an average user's R toolbox because average users are in the value business: applying functions to values, not taking apart the language.

Still, many users encounter situations where they are using language objects rather than values. Take the subset() function as used to pull out a subset of rows from a dataframe. You can use it like this:

subset(mtcars, hp > 250)

We can take apart the command:

y <- quote(subset(mtcars, hp > 250))
as.list(y)
lapply(y, class)

The first two parts of y are easy: subset is a name and mtcars is a name. The third part is a call. The thing that's interesting about subset() from a programming point of view is that the third part cannot be evaluated as is:

eval(quote(hp > 250))

The name hp doesn't have a value. It only has meaning with respect to the mtcars dataframe. It's subset() that contains the logic to connect hp to mtcars.

The rlang package provides facilities for taking apart calls and ensuring that the names used in a call are linked to the environment that gives them values. Checkr makes extensive use of rlang.

Multiple statements within quote()

Often you will be working with submitted code with multiple statements. When using quote() to create examples with multiple statements, you need to enclose the statements with curly braces { } and separate the statements with a semi-colon. For instance:

y <- quote({who <- "Alfred"; paste("Welcome,", who)})
y

Assignment

Strictly speaking, an assignment statement is a function call involving two arguments.

y <- quote(x <- foo(f,g))
class(y)
as.list(y)

The function is called `<-`. The first argument is the name to to use for storing the value of the second argument. The second argument is the value (perhaps constructed by evaluating a call) to be stored under the name.

When using checkr on an assignment statement, the checking logic is performed on the expression to the right of the assignment symbol. As you'll see later, if it's important to your checking logic to know if assignment is involved, or what is the name being assigned to, you can write tests using the pronoun Z. Except for Z, there is no vestige of the assignment being used in the checkr tests.

Preparing code for checkr

In previous examples, we've used quote() to store examples of submitted code. Another way to store code, the one used in learnr, is as a text string. Checkr can work with both.

Before checkr tests can be applied to code, you need to pre-process the code. This is the task of the for_checkr() function. For instance:

s1 <- quote(Bees <- read.csv("bee_file.csv"))
code <- for_checkr(s1)

The value created by for_checkr() is an object of class "checkr_result". This has the code (and other information) in a standard format. Many of the checkr functions take such an object as input and return the same kind of object as output. In this way, you can cascade your logic: first testing for one thing, then another, and so on.

Checkr functions

This section introduces the most important functions for implementing a checking logic. You'll want to make sure you understand the following concepts:

• quoted code, as produced by quote()
• the distinction between an R expression and the value produced when evaluating that expression.
• names
• constants (numeric and character)
• function calls
• function arguments
• a "line" of code

Checkr functions perform a variety of tasks:

1. Transforming code from one form to another
2. Associating tests with specific outcomes and messages
3. Locating lines in the code.
4. Locating arguments to a function call.
5. Combining results from different tests.

Let's examine these categories one by one.

1. Transforming code

• for_checkr() takes as input code quoted form or in the form of a character string. Every checking function will use for_checkr(). The output of for_checkr() is a "checkr_result" object, which contains one or more lines of code, as well as the test result and message.
• expand_all_chains() takes a checkr_result object as input and returns another checkr_result object where all magrittr chains in the code of the input object are split into sequences of lines. expand_chain() is similar, but works only on one line of code. You won't need these functions unless you are checking statements written in chaining notation.

2. Associating tests with specific outcomes and messages

A "test" in checkr is an R expression that returns TRUE or FALSE. Some examples of tests are nrow(df) > 10 or Z == Bees. Depending on the result of a test, you may "pass" or "fail" the submitted code, typically providing a message congratulating on the "pass" or providing a hint about what went wrong in the case o "fail."

Three functions enable you to state what you want to have happen in the event of a pass or fail:

• passif(). The first argument is a test, the second a character-string message. If the test produces TRUE, you are saying that the code is successful. If the test if FALSE, nothing happens.
• failif(). Like passif(), but a TRUE test indicates that the code is rejected. If the test is FALSE, nothing happens.
• insist(). Again like passif() but with important differences. If the test is FALSE, the code is rejected and the message is used to indicate the reason for the rejection. If the test is TRUE, nothing happens.

You will use passif(), failif(), and insist() within other checking functions.

3. Locating lines in submitted code.

Submitted code consists of one or more lines of R code. (The submission can also be empty. See if_empty_submission().) Several functions help you locate an individual line based on criteria you specify. the two most common are line_where() and line_calling().

• line_where() examines each line in turn and enables you to examine several aspects of the line:

  • the function being called at the highest level. Remember, every line of code is either a name, a constant, or a function call. The pronoun F contains the function at the very top.
  • the value produced by the line. This is stored in the pronoun V.
  • the name being bound to the line's value, that is, the name assigned to. This name is contained by the pronoun Z.
  • you also have access to the expression itself in the pronoun E. This can be used in messages.

  For example, here's an invocation of line_where to look for a line that creates a dataframe with a variable called "blood_pressure".

USER_CODE <- quote({x <- sqrt(cos(pi)); Health <- data.frame(blood_pressure = c(120, 130, 115))})
code <- for_checkr(USER_CODE)
L1 <- line_where(code, 
                 insist(is.data.frame(V), "Didn't find an appropriate statement producing a dataframe."), 
                 insist("blood_pressure" %in% names(V), 
                        "The dataframe didn't include the variable `blood_pressure`"))
L1

The first test, insist(is.data.frame(V)) means that only a line producing a dataframe can qualify. For those lines that meet this criterion, another criterion is applied: that there be a variable blood_pressure in the dataframe.

• line_calling() examines each line in turn looking for a line that calls one of the listed functions either at the top level or within an argument. (In contrast, the F pronoun in line_where() is the function being called at the highest-level.) Unlike line_where(), no tests are needed since the existence of an appropriate call dictates pass or failure.

For example, here's a test for a line calling any of sine, cosine, or tangent.

L2 <- line_calling(code, sin, cos, tan, message = "No trig function called.")
L2

There are more specialized line-locating functions.

• line_binding() checks whether the line complies with patterns that you specify and check individual components of those patterns. This is the most general of the line-locating functions, but also the most difficult to use. We discuss how to specify patterns in another section.
• line_chaining() looks for a line that contains a magrittr pipe.

4. Locating arguments in a call

Once a multi-line submission is narrowed down to a single line, you can apply tests within that line. Remember that the expression in a line can be either of three things: a constant, a name, or a function call. For constants or names, any testing you need can be done in the line_where() function using either the V or E pronouns. Thus, taking apart a line necessarily applies only to function calls. The F pronoun in line_where() can handle any tests of which function is at the head of the call. That leaves only the task of locating arguments to that function. Checkr provides several functions to do this.

• arg_number() pulls out the nth argument.
• named_arg() searches for an argument with a specified name.
• character_arg(), formula_arg(), function_arg(), list_arg(), matrix_arg(), numeric_arg(), table_arg(), vector_arg() find arguments whose value has a particular type.

Any of these functions can apply passif/failif/insist tests to the argument that is found. Those tests can be written in terms of the E and V pronouns: the expression itself and the value of that expression.

Each of the argument-locating functions returns a "checkr_result" containing the argument as a stand-alone expression. If that expression happens to be a function call, you can do further testing with these functions on that expression.

To illustrate, consider the following submission, where we want to make sure that the data used in fitting the model has exactly 100 cases.

USER_CODE <- quote(mod <- lm(mpg ~ hp + cyl, data = mtcars))
code <- for_checkr(USER_CODE)
L1 <- line_calling(code, lm)
named_arg(L1, "data", insist(nrow(V) == 100, 
                             "Please use exactly 100 cases for fitting. You used {{nrow(V)}} cases."))

Fill in the blanks

A good pedagogical technique when introducing a subject is to start an exercise with part of the answer and ask students to fill in one or more blanks. This scaffolding approach let's the instructor focus on one part of a command. As an example, consider this problem:

Exercise 14: Fill in the blanks in the following code to create a ggplot2 command that will produce the following scatter plot with the mtcars data.
```r library(ggplot2) ggplot(mtcars, aes(x = mpg, y = hp, color = cyl)) + geom_point()

>
> There are four blanks. You'll have to replace all of them with the correct contents to generate the plot.

The exercise chunk might have this scaffold:
```r
library(ggplot2)
ggplot(mtcars, aes(x = ____, y = ____, color = ____)) +
  ____()

Let's suppose the student's submission, after filling in the blanks is as follows. (Note that the student has made a slight mistake!)

library(ggplot2); 
ggplot(mtcars, aes(x = hp, y = mpg, color = cyl)) +
  geom_point()

submission <- "library(ggplot2); 
ggplot(mtcars, aes(x = hp, y = mpg, color = cyl)) +
  geom_point()"

To check the submission, we need to create a pattern that will let us look up the student's values for each of the blanks, then compare these to the correct answer. The check_blanks() function takes, as its second argument, an expression with the blanks. Since there are multiple blanks, each has been given a name for later reference.

print_function_contents(
  check_exer_14,
  from_file = system.file("learnr_examples/internal-examples.R", 
                          package = "checkr"), 
  just_the_body = FALSE)

Trying out the test ...

check_exer_14(submission)

In lines 4/5 of the check_exer_14() checking function, note that the second argument to check_blanks() is a command template very similar to that used for the scaffolding in the exercise chunk. But the template handed to check_blanks() must have the blanks written in a special format: two dots followed by a unique name followed by another two dots, e.g. ..geom...

Interaction with learnr

The interface from the learnr system to a code-checking system is described in the learnr documentation. To summarise briefly:

1. All exercise boxes have a "Run Code" button which passes the code in the box for evaluation by learnr, displaying the results in the learnr document below the exercise box. (Let's imagine the chunk containing the exercise block has label exercise1.)
2. A "Submit" button will be included in the code box if the author adds a -check chunk to the document whose label refers to the corresponding exercise chunk. (For exercise``, the full chunk label will beexercise1-check`.)
3. When "Submit" is pressed, learnr will call its exercise.checker function, passing that function a list containing the student's submission (as text), the contents of the -check block (as text), as well as other information produced when the submission is evaluated. The document author specifies which checker function is to be used with a statement in the setup chunk like this:

tutorial_options(exercise.checker = checkr::check_for_learnr)

1. Adding a -check-code block (full label for our example: exercise1-check-code) will cause learnr to call the exercise.checker function before the code has been evaluated. There is a flag enabling the exercise.checker to detect whether it is being called before evaluation or after evaluation.

An important consequence of evaluating the student submission before the exercise.checker is called is that the whole student submission must be evaluatable. If the exercise.checker is to play any role at all, the student submission can have no syntax errors, all objects used in the code must exist, and the functions called cannot throw an error. This seems too severe a restriction, since many student errors stem from mis-spellings and such. In principle, checkr functions could be set up look for mis-spellings of variable names and point them out.

The -check-code system offers a way around this, since the checking functions can look at the code before it is evaluated. But the -check-code system is active only when the "Submit" button is pressed. "Run" does not activate pre-evaluation checking. The result is that students will see a different error message when "Run" is pressed than when "Submit" is pressed. Confusing!

QUESTION for learnr developers: Why not always to pre-checking evaluation?

But pre-evaluation checking can never be complete, since identifying run-time errors (as might happen if a user specifies a wrong function or uses an undefined name) can only be done by evaluating the code.

QUESTION for learnr developers: Why not let the checking code do the evaluation, and pass back to learnr any objects to be displayed?

QUESTION for learnr developers: Why not have the "run" button trigger the pre-code checking? As it is, any sophisticated form of pre-evaluation checking will only be applied when "submit" is pressed. So the student sees different messages when pressing run

WHAT TO DO WHEN THERE IS A PARSE ERROR.

POSSIBILITIES WHEN THERE IS A RUN-TIME ERROR to anticipate particular situations.

Sometimes students will submit code that throws an error, either at parse time or at run time. By default, learnr displays such errors in their native, somewhat cryptic format.

checkr has some simple error-catching logic that tries to translate the native R messages into a friendlier format. To turn this on, you need to add a -code-check chunk to each exercise.

• [ASIDE] I would rather that code checking be turned on or off at the document level, perhaps as a choice in check_for_learnr().
• [ASIDE] I would also rather that the evaluation of the exercise take place entirely in the checking function, rather than being run first by learnr and then handed off to the checking function.

For an exercise named prob-1 the code check chunk will look like:

` ``{r prob-1-code-check, echo = FALSE}
1 # there must be some executable content, even if trivial
```

If, when trying out your exercises, you find a parsing or run-time error that isn't captured by checkr, please submit an "issue" on the GitHub site so that we can add that into the system.

Anticipating student errors

When developing checking code, you likely have in mind some typical mistakes that a student might make. For instance, in trigonometry, it's common to confuse the cosine with the sine. You might want to include checking statements that look specifically for a particular mistake and give a tailored feedback message.

The misconception() function can be useful here. The idea misconception() is that you design a test that checks whether the misconceived pattern is present. The test is written in ordinary checkr statements, for example as miss1 in the following example. A failed result from that test means that the misconception is not in evidence in the student's submission. The misconception() function turns a passing test (that is, the misconception is present) into a failed test. Somewhat awkwardly, misconception() takes as a first argument the "checkr_result" being tested for the misconception. This is so because in the absence of positive test, the output of misconception() is the statement being tested, rather than the result of the test itself.

CODE <- for_checkr(quote(15 * cos(53)))
t1 <- line_calling(CODE, sin, cos, tan, message = "You should be using a trigonometric function.")
miss1 <- line_calling(t1, cos)
t1 <- misconception(t1, miss1, message = "Are you sure cosine is the right choice?")
t1 <- line_where(t1, 
                 insist(F == "*", 
                        "Remember to multiply by the length of the hypotenuse"))
line_where(t1, insist(is.numeric(V)), 
           insist(abs(V - 11.98) < 0.01, 
                  "{{V}} is a wrong numerical result. It should be about 11.98."))

Testing chains

NOTE: In draft.

Magrittr statements chain together a sequence of function calls, with the output of one function call becoming the input to the next function call.

If you are teaching about magrittr itself (perhaps as used in dplyr or ggformula) you may have exercises that are about constructing a chain. For instance:

Exercise 9: Consider the following wrangling statement: ```r res <- group_by(mtcars, cyl) summarise(res, disp = mean(disp))

    Rewrite it as a chain.

The checking function needs to make sure the task is accomplished, like this:
```r
chk_exer_9 <- function(USER_CODE) {
  code <- for_checkr(USER_CODE)
  t1 <- line_chaining(code, message = "Remember, chains involve `%>%`.")
  check(t1, 
        insist(identical(V, mtcars %>% group_by(cyl) %>% summarise(disp = mean(disp))), 
               "Your chain doesn't produce the right value."),
        passif(TRUE, "Great!"))
}

chk_exer_9("mtcars %>% group_by(cyl) %>% summarise(disp = mean(disp))")
chk_exer_9("mtcars %>% group_by(hp) %>% summarise(disp = mean(disp))")
chk_exer_9("res <- group_by(mtcars, cyl); summarise(res, disp = mean(disp))")

But often, you may not be directly concerned about the use of chains, but want to be able to check code where the student might have chosen to use a chain. Checkr adopts a simple strategy for enabling you to check code whether it be written in chain form or in a traditional line-by-line form. That strategy is to expand chains into the line-by-line form. As a result, you can write tests assuming a line-by-line form. Before applying those tests, expand any chains into their line-by-line form.

An expression containing a magrittr chain is a function call. The top-level function is `%>%`. Insofar as you are concerned with the end value of the chain, or with whether the chain calls a specified function, you can use the regular line-locating functions like line_where() or line_calling(). You can also use the argument-locating functions on the whole chain.

• If your problem statement insists that a particular statement be a chain, you can extract that statement with line_chaining(). Then use expand_chain() to turn that into a sequence of ordinary statements that can be checked in the ordinary way.
• By default, use expand_all_chains() to turn the code into a single sequence that expands all the chains.

So, if you're anticipating that a chain might be used and that you want to analyze the individual elements of that chain, thinking about using expand_all_chains() directly on the CODE.

Patterns

NOTE: This is currently an outline

This section will deal with the line_binding() and check_binding() functions. These involve the use of patterns, so we need to explain what a pattern is and how to conduct tests with them. The pattern facility comes from the redpen package (which may be folded into rlang at some point. )

• Name of an argument: rep(.(a), `..(nm)` = .(b)). Note that the wild-card for the argument name needs to be placed in back-quotes. This is because ..(nm) is not a valid argument name, but `..(nm)` is.

What's the meaning of .(fn)(...)? NEED TO SAY it's a legal R expression, even if it isn't one that many experienced R users would recognize. ... is a legal statement in R, although typically used only within functions. And .(fn) is a legal statement, which would usually be read as "the function named . being applied to the value of fn." The point here is that patterns must be legal R expressions, and .(fn)(...) satisfies this. But the components ... and .(fn) in a pattern do not have their usual meaning. Instead, they follow the conventions of the redpen package.

As part of a pattern, ... means any set of arguments: it's a wildcard which matches anything that might legally be used as arguments to a function.

As part of a pattern, .(fn) means, "Give the name fn to whatever is being used in this role in the statement." And what part is that? In ordinary syntax, parentheses are used to indicate the application of a function to arguments. So sin(...) in a pattern means "evaluate the function sin and don't worry about what the specific argument or arguments are." In the pattern .(fn)(...) the marker .(fn) is in the position of sin in sin(...). Thus, .(fn)(...) is a pattern that says, "I'm expecting a function being applied to some arguments. Use fn to represent the function itself."

The pattern is the part of the system that provides the flexibility for an author to define what standards should be satisfied by the submission. Let's elaborate on that.

WARNING TO THE READER: The patterns that follow may seem bizarre when you first encounter them. Rest assured, you'll have time to get to know them. And, even if they don't look like it, each pattern is a syntactically correct R statement.

Here are a few patterns that will be relevant to the rest of the example:

• ..(res) -- the overall result of the submission, which will be available under the name res.
• "+"(.(a), .(b)) -- the + function applied to two arguments, which we'll call a and b, which might themselves be expressions, as in (2^1) + sqrt(9), where the arguments to + are (2^1) and sqrt(9).
• "+"(..(a), ..(b)) -- similar to be above, but if the two arguments happen to be expressions, a and b should stand for the values of their corresponding expressions. For instance matching the pattern to the submission (2^1) + sqrt(9) would result in a being 2 and b being 3.
• .(fn)(.(a), .(b)) -- a function, which we'll call fn, being applied to arguments which we'll call a and b.
• .(fn)(.) -- a function applied to a single argument.
• .(fn)(., ., ., ...) a function applied to three or more arguments.

The facilities for implementing and applying these sorts of patterns are provided by the remarkable redpen package in conjunction with rlang.

Tips on patterns and tests

• Patterns are raw R statements, for instance
• Assignment-to has a pattern `<-`(x, ...)
  • If you want to allow assignment to any of several names, try `<-`(.(nm), ...) as the pattern and nm == quote(x) || nm == quote(X) as the test: r if_matches(quote(x <- 3 + 2), `<-`(.(nm), ..(val)), passif(nm == quote(X) || nm == quote(x), "{{nm}} was assigned the value {{val}}"))
  • Notice that the pattern has a single dot (.(nm)) for the assigned-to part of pattern. Similarly, the value to which it's being compared is in quote(), as appropriate for matching a language-name object.
• Distinguish between .(a) and ..(a). Suppose you want to check which trigonometric function has been used in a command like quote(cos(x)). An appropriate pattern would be .(fn)(...), like this:

submission <- quote({x <- pi; cos(x)})
pattern <- quote(.(fn)(.(var))) # will have to unquote
if_matches(submission, !!pattern, passif(TRUE, "Found match"))
if_matches(submission, .(fn)(.(var)),
           passif(fn == quote(cos) && var == quote(x), "Right. The function is {{fn}} on variable {{var}}."))

Some kinds of questions

NOTE: Just a draft

This will be about different kinds of questions. Probably it should be a separate vignette.

Style ideas:

• Whenever feasible, in the expository component of the document have results to be displayed generated by an exercise chunk. The scaffold can be the complete command, so that the student need merely press "Run" to generate the display. This style seems to increase student engagement with the document, especially for beginners.
• Putting scaffolding content in an exercise is a good way to direct students in a desired direction and avoid the paralysis caused by facing a "blank canvas."
• An alternative to scaffolding is to leave the box empty of code, but put in an if_empty_submission() test in the checking code. This can be used to display a suggestion for how to get started. Note that a submission with just a comment (# like this) counts as an empty submission. So you can remind the student that they can get a hint by pressing "Submit"

http://third-bit.com/2017/10/16/exercise-types.html

Readers may also want to check out the system for submission correctness tests provided by DataCamp.

dtkaplan/checkr documentation built on May 15, 2019, 4:59 p.m.