In DavZim/RITCH: R Parser for the ITCH-Protocol

Debug Tools for `RITCH`

This document quickly outlines the debugging tools of the RITCH library.

Building

These tools are used for debugging and understanding the data format. They are not shipped with the package itself but need to be sourced independently.

If you want to play around with the tools, clone the git repository and source the debug/debug_tools.cpp script:

Sys.setenv("PKG_LIBS" = "-lz")
Rcpp::sourceCpp("debug_tools.cpp")

Sys.setenv("PKG_LIBS" = "-lz")
Rcpp::sourceCpp("debug/debug_tools.cpp")

Note that debug_tools.cpp includes ../src/RITCH.h as well as ../src/MessageTypes.h (relative from the debug_tools.cpp script), if you have cloned the repository as is, it should work out of the box, otherwise, make sure that the two header files are found.

Debug Tools

dbg_get_message_length(msgs) returns the size of Messages in bytes. Note that each message adds 2 bytes that are not used

dbg_get_message_length(c("A", "F"))

dbg_itch_file(filename) allows you to interactively list messages in a file You are asked for input inside the function, which can be:
msg type, e.g., A, H, h to see the next instance of that message type
numeric value, e.g., 3 to see the next N values

For example:

file <- "20191230.BX_ITCH_50"
dbg_itch_file(file)
## Debugging File '20191230.BX_ITCH_50' (.gz-file? no)
## Usage:
## - Empty: next message
## - Number: for next N messages
## - Character: if valid message type, print the next message, e.g., 'A' for add order
## - non valid Character: exits the debugging tool
## Note: Bytes in parenthesis show the first two bytes, which are not used!
## Number of Messages:
## - 'S': 6
## - 'R': 8906
## - 'H': 8961
## - 'Y': 9013
## - 'L': 6171
## - 'V': 1
## - 'W': 0
## - 'K': 0
## - 'J': 0
## - 'h': 0
## - 'A': 12210139
## - 'F': 45058
## - 'E': 578839
## - 'C': 2686
## - 'X': 348198
## - 'D': 11821540
## - 'U': 1741672
## - 'P': 134385
## - 'Q': 0
## - 'B': 0
## - 'I': 0
## - 'N': 2241182
## =============================
## 'S' (len 2 + 12) idx    0 at offset     0 (0x0000) | (00 0c) 53 00 00 00 00 0a 2d f4 92 1d 67 4f
#RITCH> 3
## Showing next 3 messages
## 'R' (len 2 + 39) idx    1 at offset    14 (0x000e) | (00 27) 52 00 01 00 00 0a 66 a0 e0 dc 44 41 20 20 20 20 20 20 20 4e 20 00 00 00 64 4e 43 5a 20 50 4e 20 31 4e 00 00 00 00 4e
## 'R' (len 2 + 39) idx    2 at offset    55 (0x0037) | (00 27) 52 00 02 00 00 0a 66 a0 e2 c8 6c 41 41 20 20 20 20 20 20 4e 20 00 00 00 64 4e 43 5a 20 50 4e 20 31 4e 00 00 00 01 4e
## 'H' (len 2 + 25) idx    3 at offset    96 (0x0060) | (00 19) 48 00 01 00 00 0a 66 a0 e4 ff bd 41 20 20 20 20 20 20 20 54 20 20 20 20 20
#RITCH> A
## Applied filter to message type 'A'
## 'A' (len 2 + 36) idx 32873 at offset 973915 (0xedc5b) | (00 24) 41 20 2c 00 00 16 eb 55 2c 88 24 00 00 00 00 00 00 00 04 42 00 00 2e 7c 55 53 4f 20 20 20 20 20 00 01 fa 40
#RITCH> q
## Stopping Printing Messages

dbg_hex_to_char(hex_string) converts a hex value to character
dbg_hex_to_int(hex_string) converts a hex value to integer
dbg_hex_to_dbl(hex_string) converts a hex value to dbl

dbg_hex_to_char("52 49 54 43 48 20 20 20") # 'RITCH   '
dbg_hex_to_int("01 23 45 67") # 19088743
dbg_hex_to_dbl("00 01 fa 40") # 12.96

dbg_hex_compare(x, y) to get a quick comparison of two hex strings

x <- "00 01 02 03 04"
y <- "00 01 00 03 0a"
dbg_hex_compare(x, y)

dbg_hex_count_messages(hex_string) counts the number of messages by type in a hex string

incomplete_hex_string <- "00 00 53" # . . S
dbg_hex_count_messages(incomplete_hex_string)

dbg_hex_to_*() to convert hexadecimal strings to message data.tables (* can be orders, trades, modifications, system_events, stock_directory, trading_status, reg_sho, market_participant_states, mwcb, ipo, luld, noii, or rpii)

hex_string <- paste(
  "00 00", # first 2 empty nibbles
  "46", # message type 'F'
  "20 2c", # stock locate 8236
  "00 00", # tracking number 0
  "16 eb 55 2c 88 24", # timestamp 25200002107428
  "00 00 00 00 00 00 00 04", # order ref 4
  "42", # buy == TRUE -> 'B'
  "00 00 2e 7c", # shares 11900
  "55 53 4f 20 20 20 20 20", # stock 'USO     ' (length 8)
  "00 01 fa 40", # price 129600 (12.96)
  "56 49 52 54" # mpid/attribution 'VIRT
)

dbg_hex_to_orders(hex_string)

dbg_messages_to_hex() to convert the message data.tables to a hexadecimal string

od <- data.table::data.table(
  msg_type = "F",
  stock_locate = 8236L,
  tracking_number = 0L,
  timestamp = bit64::as.integer64(25200002107428),
  order_ref = bit64::as.integer64(4),
  buy = TRUE,
  shares = 11900L,
  stock = "USO",
  price = 12.96,
  mpid = "VIRT"
)
hex_order <- dbg_messages_to_hex(od)
hex_order

# convert back to a data.table and see if they are identical
od2 <- dbg_hex_to_orders(hex_order)
all.equal(od, od2)