In thebioengineer/aRcade:

Training Neural Nets to play video games is something that has become very popular, if you believe all the blogposts and articles that have come out recently. However, most examples are written using python. In this document I will attempt to show you how to do a similar thing in R.

At the end of this blog post, you will have the tools to write your own ML tool to learn to play snake.

The Game

The game I chose is one that most people know - snake. If you never had a nokia growing up, you may have never played this classic arcade game. The rules are simple, at a set interval a "snake" will move one space and continue moving in that direction until you tell it to change directions from the keypad. The goal is to eat as much fruit as possible by having the head of your snake go over the point where the food is, without hitting either the walls or your own body. Sounds simple, but with each fruit eaten, the snake gets longer!!

There were a few samples of code I based my snake game off of, mainly this one. However, I added a number of my own changes to my "snake" class to make it provide information I was interested in. Most changes were due to the fact I was writing one game, so a more robust framework was not necessary. Below you can see the beginning sequence:

source(file.path(here::here(),"R","snake.R"))

game<-new("snake")
game$init()
game$plotboard()

I added a feature to be able to give a sequence of moves, and then generate the resulting steps.

library(animation)

res<-saveGIF(game$replay(c(rep("left",2),
                      rep("down",7),
                      rep("right",5),
                      "down",
                      rep("left",2),
                      "up",
                      rep("left",5)),seed=10,delay = .1),
        movie.name = "animation.gif",
        interval= .3)

Things get logged, so we can watch how it behaves once it is trained!

Naive

There is a tendancy now-adays to simply throw a deep learning or some other complicated machine learning model at a problem, and wipe your hands of it. However, we are not those sorts of data scientists. We want to make sure our model learns and does better than random chance!

To do this, we will throw a random model that does not learn into our player, and see how it behaves. First, I will generate all the code for our "naive" player.

naive_model<-structure(class="naive",
                       .Data="naive_model",
                       .predict=function(){
                         output<-c(0,0,0,0)
                         output[sample(1:4,1)]<-1
                         return(output)
                         })

predict.naive<-function(object,...){
  func<-attr(naive_model,".predict")
  func()
}

fit.naive<-function(object,...){
  invisible(object)
}


predict(naive_model,runif(200))

source(file.path(here::here(),"R","model_player.R"))

player<-new("model_player")

player$addModel(naive_model,fit.naive)

player$train(20)

bestScore<-max(sapply(player$logs,`[[`,2))
bestPerf<-which.max(sapply(player$logs,function(x)ifelse(x[[2]]==10,length(x[[1]]),0)))

steps<-player$logs[[bestPerf]][[1]]
res<-saveGIF(
  game$replay(steps[-1],as.numeric(steps[1]),delay = .1),
  movie.name = "naive_animation.gif",
   interval= .25)

The Model

Using the 'keras' library, I am able to interact with the tensorflow underpinnings and create a deep learning neural network. The work put in by the RStudio gang is truely amazing, and they make using tensorflow feel as natural as any model building in R.

Setup

library(keras)

snake_model <- keras_model_sequential()

snake_model %>%
  layer_dense(units = 258, activation = 'relu', input_shape = c(12)) %>% 
  layer_dropout(0.15) %>% 
  layer_dense(258, activation='relu') %>% 
  layer_dropout(0.15) %>% 
  layer_dense(258,activation = "relu")%>%
  layer_dropout(0.15) %>%   
  layer_dense(258,activation = "relu")%>%
  layer_dropout(0.15) %>%   
  layer_dense(258,activation = "relu")%>%
  layer_dense(4,activation = "softmax")

optimizer <- optimizer_adam(lr = 0.0005)

snake_model %>% compile(
  loss = "mse", 
  optimizer = optimizer
)

The model here is a series of densely connected layers with dropout neural networks.

source(file.path(here::here(),"R","model_player.R"))

player<-new("model_player")

player$addModel(snake_model,tuning_var = 40)

player$train(150)

plot(sapply(player$logs,`[[`,3),sapply(player$logs,`[[`,2))


modell<-player$model[[1]]




bestScore<-max(sapply(player$logs,`[[`,2))
bestPerf<-which.max(sapply(player$logs,function(x)ifelse(x[[2]]==bestScore,length(x[[1]]),0)))

steps<-player$logs[[bestPerf]][[1]]
res<-saveGIF(
  game$replay(steps[-1],as.numeric(steps[1]),delay = .1),
  movie.name = "init_animation2.gif",
   interval= .25)