Read 'Blackrock' Neural Event & Signals
In readNSx: Read 'Blackrock-Microsystems' Files ('NEV', 'NSx')

knitr::opts_chunk$set(
  collapse = TRUE,
  eval = FALSE,
  comment = "#>"
)

Introduction to `readNSx`

readNSx is an R package that converts Blackrock-Microsystems files (.nev, .nsx) into commonly used formats that are compatible with R, Python, Matlab. The package is designed for neuroscientist who study intracranial Electroencephalography and use systems like Blackrock, Ripple, etc. Support versions are 2.2, 2.3, 3.0.

To install readNSx from CRAN,

install.packages("readNSx")

To install development version

install.packages("readNSx", repos = c(
  dipterix = "https://dipterix.r-universe.dev",
  CRAN = "https://cloud.r-project.org"))

Basic Usage

(Well, this package has no advanced usage.)

One-time import for the first time

If you have never converted data, use the following one-time function call to import .nev and .nsx data:

readNSx::import_nsp(
  path = "YAB_Datafile_001.nev", 
  prefix = "YAB_Converted_001", 
  exclude_events = "spike", 
  partition_prefix = "/part"
)

path: path to .nev or any nsx file (such as ns3, ns5)
prefix: path prefix of imported files (where the data will be saved)
exclude_events: events to exclude from importing; default is "spike" (spike-sorting waveform and classes)
partition_prefix: see Section "Anatomy of imported files".

Please check help function ?readNSx::import_nsp

Load imported data from R

(If you have never imported data before, please check previous sub-section first)

Example 1: load epoch from `NEV` comment data packet events

The following example loads the trial epoch table from NEV comment events. The time_in_seconds is the second of stimuli relative to time_origin. Column comment is whatever comments sent by the task script (sent from psychtoolbox or related software that you use when collecting data).

# prefix <- "YAB_Converted_001"
nev <- readNSx::get_nev(prefix)

# `time_in_seconds` is relative to time-origin
nev$header_basic$time_origin
#>        Year       Month   DayofWeek         Day        Hour   
#>        2022           8           5          26          15   
#>      Minute      Second Millisecond 
#>           4          10         156 

readNSx::get_event(nev, "comment")
#>     timestamp packet_id char_set flag data   comment   event time_in_seconds 
#> 1      683033     65535        0    0  255  audio-ba comment        22.76777 
#> 2      753242     65535        0    0  255  video-ba comment        25.10807 
#> ...

Example 2: get channel information and data

# prefix <- "YAB_Converted_001"

# Gather information of channel 10
loaded <- readNSx::get_channel(prefix, channel_id = 10)

# Get NSx configurations, 
loaded$nsx
#> Basic header information (NSx):
#>   Internal type: NEURALCD
#>   Channel count: 152
#>   Sample rate: 2000 Hz
#>   Time origin: 2022-08-26 15:04:10 158ms
#> Extended header information (NSx):
#>   - CC (152 x 16, channels: 1-152): type, electrode_id, electrode_label, ...
#> Cache status:
#>   Prefix: ...
#>   Number of partitions: 1


# E.g. number of partitions (i.e. unpaused continuous recordings)
loaded$nsx$nparts
#> 1

# Get channel information
loaded$channel_info
#>    type electrode_id electrode_label physical_connector connector_pin
#> 10   CC           10        RA10-010                  1            10
#>    min_digital_value max_digital_value min_analog_value
#> 10            -32764             32764            -8191
#>    max_analog_value units high_freq_corner high_freq_order
#> 10             8191    uV              300               1
#>    high_freq_type low_freq_corner low_freq_order low_freq_type
#> 10              1         7500000              3             1
#>    sample_rate_signal sample_rate_timestamp which_nsp    filename
#> 10               2000                 30000         3 RA10-010.h5

# Get channel data
channel_signal <- loaded$channel_detail$part1$data; channel_signal
#> Class: H5D
#> Dataset: /data
#> Filename: ...
#> Access type: H5F_ACC_RDONLY
#> Datatype: H5T_IEEE_F64LE
#> Space: Type=Simple     Dims=798264     Maxdims=Inf
#> Chunk: 16384

# Get the actual numbers
channel_signal[]

Please use square bracket channel_signal[] to load the data into the memory.

Anatomy of imported files

The imported file paths will start with prefix, which is specified by you, my dear users. In the following demonstration, I'll use a placeholder <prefix> to represent your inputs.

partition_info         - Name of continuous recording within the block, 
                          sample rates, starting time per partition per NSx
<prefix>_scans         - Basic information for current block
<prefix>_channels      - Electrode channel information ( ID, Label, ... )
<prefix>_events/       - NEV setting headers and data packets (events)
  - DIGLABEL           - Digital input setup
  - NEUEVLBL           - Channel labels
  - NEUEVWAVE          - Spike waveform settings
  - ...                - Other settings
  - event-***          - Data packets (digital inputs, comments...)
  - waveforms.h5       - Spike waveforms & cluster
<prefix>_ieeg          - NSx data folder
  - configurations.rds - NSx basic headers (versions, number of partitions, ...)
  - partition_info     - Continuous recording duration, start time, sample rates
  - nsx_summary.rds    - Internally used
  - part1/, part2/, ...- Channel folder
    - XXXX-001.h5      - Channel data file, each file correspond to a channel.
    - XXXX-002.h5         The file name ALWAYS ends with channel ID.
    - ...                 Each HDF5 file contains a "meta" and a "data" part,
                            "meta": JSON string of channel information
                            "data": numerical signal voltage data (in `uV`)

The signal data is stored at <prefix>_ieeg/part* by default, where * are positive integers representing the partition number. The Blackrock system allows users to pause their recordings and resume later without having to start a new block. A partition in readNSx represents one continuous recording within a "block". In most of cases, when there is no pause within a block, you will see only one partition. In some experimental settings, there could be one or more pauses, readNSx will store each continuous recordings in separate folders to make sure each partition is always continuous. The start time of the partitions will be stored in partition_info.tsv.

Please be aware that partition pattern <prefix>_ieeg/part* is not fixed. You can change the pattern via parameter partition_prefix when importing the data. For example, partition_prefix="_part will create partition files within directory <prefix>_ieeg_part*.

When readNSx stores the data, the channels are saved individually. For example, channel 1 (LA-2) and 2 (LA-2) are stored in separate HDF5 files. This arrangement is out of the file-size and computational considerations:

When storing all channels together, the file size will become super big. Reading one channel might resulting in reading the whole file (if handled poorly, or if using network drive)
In some cases, HDF5 files can only have one file pointers (file locked by some software). This could limit batch computing algorithms that can be paralleled at channel-level
My personal research requires fast approach to copy or share data for quick inspection/analysis/visualizations (don't want to move large files around)

File formats

The following file types will be generated:

.h5 (HDF5 file): common file format that can include one or more data within a single file.
Matlab: use h5disp to get enclosing data names; use h5read to read specific data
Python: use h5py package to load the files
R: if you want to load them individually, use raveio::h5_names to get enclosing data names; use raveio::load_h5 to read specific data. readNSx also provides high-level functions to load them (see ?readNSx::get_channel).
.tsv (tab-separated values file): plain text files that can be easily read by many languages. You can open them in Microsoft Office: Excel.
.rds (R object file): can only be read from R, internally used by readNSx to store data objects. Users do not need to read from these files (but also do NOT delete them, or readNSx will break). They serve as redundant files in case the tsv files are altered accidentally (for example, Excel might alter data formats automatically). If you see an .rds file sharing the same name as .tsv, they share the same information.