In coolbutuseless/minilexer: A Simple Tool for Lexing Text Data

suppressPackageStartupMessages({
  library(magrittr)
  library(dplyr)
  library(tidyr)
  library(glue)

  library(minilexer)
})

knitr::opts_chunk$set(echo = TRUE)

Chess game format: pgn

The pgn file format is a human readable representation of a chess game.

In its most basic form, it consists of

• a sequence of tags (i.e. comments) surrounded by []
• a sequence of numbers and events representing the moves taken by the players i.e.
  • A number indicating which move this is within the game.
  • Moves the for the white and black player represented in Standard Algebraic Notation (SAN).
• Comments can be interspersed between/within the moves and are surrounded by "{}"

An example pgn file is show below:

pgn_text <- '
[Event "F/S Return Match"]
[Site "Belgrade, Serbia JUG"]
[Date "1992.11.04"]
[Round "29"]
[White "Fischer, Robert J."]
[Black "Spassky, Boris V."]
[Result "1/2-1/2"]

1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 {This opening is called the Ruy Lopez.}
4. Ba4 Nf6 5. O-O Be7 6. Re1 b5 7. Bb3 d6 8. c3 O-O 9. h3 Nb8 10. d4 Nbd7
11. c4 c6 12. cxb5 axb5 13. Nc3 Bb7 14. Bg5 b4 15. Nb1 h6 16. Bh4 c5 17. dxe5
Nxe4 18. Bxe7 Qxe7 19. exd6 Qf6 20. Nbd2 Nxd6 21. Nc4 Nxc4 22. Bxc4 Nb6
23. Ne5 Rae8 24. Bxf7+ Rxf7 25. Nxf7 Rxe1+ 26. Qxe1 Kxf7 27. Qe3 Qg5 28. Qxg5
hxg5 29. b3 Ke6 30. a3 Kd6 31. axb4 cxb4 32. Ra5 Nd5 33. f3 Bc8 34. Kf2 Bf5
35. Ra7 g6 36. Ra6+ Kc5 37. Ke1 Nf4 38. g3 Nxh3 39. Kd2 Kb5 40. Rd6 Kc5 41. Ra6
Nf2 42. g4 Bd3 43. Re6 1/2-1/2
'

Use lex() to turn the text into tokens

1. Start by defining the regular expression patterns for each element in the pgn file.
2. Use minilexer::lex() to turn the pgn text into tokens
3. Throw away whitespace, newlines and tags, since I'm not interested in them.

pgn_patterns <- c(
  comment       = '(;.*?)\n',     # Assume ; only appears to denote comment to end of line
  tag           = '\\[.*?\\]',    # parse tags as a whole token. going to ignore
  comment_open  = "\\{",          # Inline comment start
  comment_close = "\\}",          # Inline comment end
  move_number   = "\\d+\\.+",
  symbol        = '[-+\\w\\./]+',
  newline       = '\n',
  whitespace    = '\\s+'
)

tokens <- minilexer::lex(pgn_text, pgn_patterns)
tokens <- tokens[!(names(tokens) %in% c('whitespace', 'newline', 'tag'))]
tokens[1:23]

coolbutuseless/minilexer documentation built on May 14, 2019, 6:09 a.m.