In MilesMcBain/r2vr: Compose and Serve WebVR Scenes

knitr::opts_chunk$set(echo = TRUE)

A live connection between an R session and a running scene being served by r2vr is supported using a websocket API. This allows novel usage scenarios including: An 'operator' triggering events for a scene viewer using the R console - Alternately using controls laid out in a html widget or shiny app. Streaming of data from R to VR scenes could be used create animations or dynamic visualisations

The communication layer is full duplex, meaning it is possible for events triggered in VR to be relayed to the R session, although the API for setting up handlers for asynchronous events arriving from VR will not be discussed in this vignette in this release. This will be deferred until a suitable usecase is found (Feel free to open a GitHub issue if you find yourself wanting to do this and we can discuss).

Elements of R-VR communication

Websocket enabled r2vr server

The communication layer is enabled by creating scene object using a_scene() with the .websocket = TRUE argument. This will cause each user browser session to open a websocket connection with the r2vr server as part of the scene rendering process.

A websocket enabled r2vr server can broadcast messages to connected users that can trigger updates in their VR scenes. This is done by calling the send_messgages() method of the server.

send_messages()

This method takes a message or list of messages to be broadcast to all connected scenes. Messages are created using the r2vr functions:

• a_event() - used to inject A-Frame events into the scene. This can be any kind of event including those normally driven by user interaction: like 'fusing' or 'click'. The scene will emit the event defined by this call on the entity identified by id. See documenation for more details.
• a_update() - used to modify the component configuration of entities in the scene. It is possible to attach a new component or, replace component attributes either fully or partially.
• a_remove_component() - used to remove a component from an entity identified by id.
• a_remove_entity() - used to remove an entity identified by id from a scene. Child entities are also removed.

Adding an entity to a VR scene dynamically from R is not supported at the moment although will be in the future.

Examples

A console controlled 360 image slide show

For this example we'll add some 360 photos to a scene and allow them to be stepped through by a console operator.

The images are hosted in an external GitHub repository: https://github.com/MilesMcBain/360_image_examples. The code below reads them directly from there.

Scene setup

The following code creates asset objects from a list of supplied URLs, and then sets up a scene containing a 'sky' sphere entity, canvas_360, that we can set the internal texture of using its src component. The internal texture is initially set to the first of the images.

library(r2vr)
library(purrr)

img_urls <-  c("https://ucarecdn.com/6cce6743-cfaa-4258-a93b-240e912c1ecc/",
               "https://ucarecdn.com/f2595d03-1a21-4ef8-9c07-6ef9a4972be7/",
               "https://ucarecdn.com/2f5030e8-a3c0-4045-ad84-a96ea25dfab0/",
               "https://ucarecdn.com/ef0090e4-703e-4edd-9527-42853eed9cfb/")

img_ids <- paste0("360img", seq(length(img_urls)))

img_assets <- map2(img_ids, img_urls, ~a_asset(id = .x,
                                               src = .y,
                                               tag = "img"))

canvas_360 <- a_entity(.tag = "sky",
                       id = "canvas360",
                       position = c(0, 0, 0),
                       src = img_assets[[1]], ## src is the inital texture asset
                       .assets = img_assets   ## these are the assets we will change to
                       )

the_scene <- a_scene(.title = "A 360 image slideshow",
                     .template = "empty",
                     .children = list(canvas_360),
                     .websocket = TRUE) ## Enable coms with R server

the_scene$serve()

The images in this scene have a combined size of about ~25mb, so the scene will need some time to load.

Control from R

With the scene running we now turn our attention to writing an 'advance()' function that we can call to advance to the next image in the list.

img_index <- 1
n_images <- length(img_ids)

advance <- function(){
    img_index <<- ifelse(test = img_index == n_images,
                         yes = 1,
                         no = img_index + 1)

    the_scene$send_messages(a_update(id = "canvas360",
                                     component = "src",
                                     attributes = img_assets[[img_index]]$reference())
              )
    message("Now showing: ", img_assets[[img_index]]$id) 
  }

Calling advance() will trigger a transition to the next image. Within this function, an update message is composed that is sent using the scene object. The transition should occur instantly since all assets are loaded prior to the scene being shown.

One trick here is that we called the reference() method of an a_asset to get the text to used in the src attribute. The transition should occur instantly since all assets are loaded prior to the scene being shown.

Streaming Position Data to VR

In this scene we're going to look at streaming sea level height from R to a scene containing an island. First we'll look at sending information interactively using the R REPL in a similar way to the prevoius example.

library(r2vr)

island_model <- a_asset(id = "island",
                        src = "https://cdn.glitch.com/7cadb230-57b2-4aae-9de9-e60de097035e%2Fisland.gltf?1510323826393")

island <- a_entity(gltf_model = island_model,
                   scale = c(0.5, 0.5, 0.5),
                   position = c(0, 2, 0))

water <- a_entity(id = "water",
                  .tag = "box",
                  width = 30,
                  depth = 30,
                  height = 5,
                  material = list(color = "#42B9F4",
                               opacity = 0.5,
                               transparent = TRUE,
                               shader = "flat"
                               )
                  )

the_scene <- a_scene(.children = list(island, water),
                     .websocket = TRUE)

the_scene$serve()


sea_level <- function(level){
  level <- level - 2
  the_scene$send_messages(
              a_update(id ="water",
                       component = "position",
                       attributes = list(x = 0, y = level, z = 0))
            )
}

With the scene open in the browser, the this sea_level() funciton you can set the height of the water for a viewer of the VR visualisation. We'll create a simple shiny app that can control the water with a slider.

library(shiny)

ui <- flowLayout(
  sliderInput("level", "Set the sea level", 
              min = -2, max = 4, step = 0.1, round = FALSE)
)

onStart <- function(){
  ## Setup VR Scene
  island_model <- a_asset(id = "island",
                          src = "https://cdn.glitch.com/7cadb230-57b2-4aae-9de9-e60de097035e%2Fisland.gltf?1510323826393")

  island <- a_entity(gltf_model = island_model,
                     scale = c(0.5, 0.5, 0.5),
                     position = c(0, 2, 0))

  water <- a_entity(id = "water",
                    .tag = "box",
                    width = 30,
                    depth = 30,
                    height = 5,
                    material = list(color = "#42B9F4",
                                    opacity = 0.5,
                                    transparent = TRUE,
                                    shader = "flat"
                                    )
                    )
  ## need to use global assign to make it visible to server
  the_scene <<- a_scene(.children = list(island, water),
                       .websocket = TRUE)
  the_scene$serve()
}

server <- function(input, output){

  ## Setup response to silder
  slider_value <-
    observeEvent(input$level,
    {
      update_message <- a_update(id ="water",
                          component = "position",
                          attributes = list(x = 0,
                                            y = input$level - 2,
                                            z = 0))

      the_scene$send_messages(update_message)
    }
    )
}

  runApp(shinyApp(ui,server, onStart), launch.browser = TRUE)

a_kill_all_scenes()

The VR app needs to be embedded inside the shiny app since the Shiny event loop takes over the R process. However care must be taken to only start it once, which is why it is in the onStart function. Correpsondlingly it would go in global.R for a directory based app.

MilesMcBain/r2vr documentation built on March 29, 2021, 12:03 p.m.