In urswilke/pyramidi: Generate and Manipulate Midi Data in R Data Frames

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

After installing pyramidi, you can run the helper function to install its python dependency miditapyr in your environment (in this case "r-reticulate"):

pyramidi::install_miditapyr(envname = "r-reticulate")

Load libraries

When hopefully everything is set up correctly, we'll load some libraries.

library(pyramidi)
library(dplyr)
library(purrr)
library(htmltools)
library(tibble)
library(details)

`MidiFramer` class

This class is the main structure of the package allowing to read midi files, manipulate the data, and write it back to disk. You can also use it to generate the midi data from scratch in R.

Generate `MidiFramer` object

We'll create an R6 object of class "MidiFramer" from a midi_file_path by passing it to the
constructor method MidiFramer$new():

midi_file_path <- system.file("extdata", "test_midi_file.mid", package = "pyramidi")
mfr <- MidiFramer$new(midi_file_path)

```{details, details.summary = "Show print output"} mfr

### Fields in `MidiFramer` objects

It is an [R6](https://r6.r-lib.org/) object that contains the following fields:

```{details, details.summary = "Show print output"}
enframe(as.list(mfr))

Please refer to help(MidiFramer) for more information on this class. The field mf is a miditapyr.MidiFrames() object. After its first element mfr$mf$midi_file, a mido midi file object with the mido message data, there are 3 more dataframes:

```{details, details.summary = "Show print output of 30 first rows of these dataframes"} list( mfr$mf$midi_frame_raw, mfr$mf$midi_frame_unnested$df, mfr$mf$midi_frame_nested$df ) %>% map(head, 30) %>% walk((x) print(knitr::kable(x)))

When `mf` is initialized `midi_frame_raw` and `midi_frame_nested$df` should be the same (except the ordering of the named fields in the `msg` column might differ). 

For a detailed interactive overview with further links showing how the various fields in the `MidiFramer` class are related and calculated see `vignette("package_workflow")`.

### Populating an empty `MidiFramer` object

You could also achieve the same result by first creating an empty `MidiFramer` object like so:

```r
mfr <- MidiFramer$new()

```{details, details.summary = "Show print output"} mfr

In this case all the dataframe fields are initialized to `NULL` (or `None` in python which reticulate also translates to `NULL` in R).

In order to load a midi file to `mfr$mf` you have to use the miditapyr method [`calc_attributes()`](https://miditapyr.readthedocs.io/en/latest/miditapyr.html#miditapyr.midi_frame.MidiFrames.calc_attributes):

```r
mfr$mf$calc_attributes(midi_file_path)

Then you can populate the fields of mfr with the MidiFramer method populate_r_fields().

mfr$populate_r_fields()

Usage

Modifying midi data

In the MidiFramer object, we can modify mfr$df_notes_wide, the notes in note-wise wide format (note_on & note_off events in the same line). Thus we don't need to worry which midi events belong together

Let's look at a small example. We'll define a function to replace every note with a random midi note between 60 & 71:

mod <- function(dfn, seed) {
  n_notes <- sum(!is.na(dfn$note))
  dfn %>% mutate(note = ifelse(
    !is.na(note),
    sample(60:71, n_notes, TRUE),
    note
  ))
}

We could modify the notes in wide format like this:

mod(mfr$df_notes_wide)

Then we would have to adapt all the following elements of mfr that depend on mfr$df_notes_wide.

When we call the method mfr$update_notes_wide(), all the depending list elements are also automatically updated.

# Apply the modification to mfr$df_notes_wide and all depending dataframes:
mfr$update_notes_wide(mod)

The data has also been changed in mfr$mf the miditapyr MidiFrames object in mfr:

mfr$mf$midi_frame_nested$df %>% head()

Writing modified midi files

Thus we can now directly save the modifications to a midi file:

midifile <- "mod_test_midi_file.mid"
mfr$mf$write_file(midifile)

For a more extensive and less arbitrary demo how to compose your own music with pyramidi, see vignette("compose").

Playing audio

You need to install fluidsynth on your computer and the R package (which should be installed with pyramidi) if you want to do that on your computer.

If you want to produce audio files from the midi files you can synthesize them with the convenience function player()

midifile |> player()

As you can hear, the sample() function in mod() changed all midi notes randomly.

You can also listen to your synthesized files by embedding an audio player for the MidiFramer object:

mfr$play()

When using R interactively, this will directly play the generated audio in the console.

Multiple results

One of the reasons to create this package was that R might help to avoid repetitive work. See below how you can use functional programming approaches of purrr, to generate multiple midi files in one call (actually two calls for more clarity but you could also put them in one).

You can also generate a list of multiple MidiFramer objects (in this case 2) and apply different modifications to each:

l_mfr <- 1:2 %>%
  set_names(paste0("test", ., ".mp3")) %>% 
  map(~mfr$clone(deep = TRUE)$update_notes_wide(mod))

And if you want to successively add modifications to the same object and store all intermediate results in a list, you could do it like this:

l_mfr2 <- 1:2 %>%
  set_names(paste0("test", ., ".mp3")) %>% 
  accumulate(~.x$clone(deep = TRUE)$update_notes_wide(mod), .init = mfr)

Please note that we made deep copies ($clone(deep = TRUE)) of the MidiFramer object in the list l_mfr. Otherwise the field of the python miditapyr.MidiFrames object mf is a shallow copy. This means that updates to one of these elements in l_mfr (for instance l_mfr[[1]]$mf) also change the others, because they all are references to the same object.

This is how you can embed multiple audio files in an rmarkdown document using purrr::imap():

tagList(
  imap(
    l_mfr, 
    ~ div(
      h4(paste("audio result", .y)),
      .x$play(audiofile = .y),
      # add 2 line breaks to vertically separate a bit:
      br(), 
      br()
    )
  )
)