Daff

In daff: Diff, Patch and Merge for Data.frames

knitr::opts_chunk$set(echo = FALSE)
library(magrittr)

Who am I (Edwin)

\begin{picture}(0,0) \put(280,0){\includegraphics[width=1cm]{fig/cbs}} \end{picture}

• Statistical consultant / methodologist
• Statistics Netherlands (CBS): all official statistics of NL
• twitter: @edwindjonge
• github: https://github.com/edwindj

Stepping in for Greg Warnes, coauthor

What is daff?

Short version

Daff is a diff for data.frames

• Detect changes: diff_data, differs_from
• Store and restore diff: write_diff, read_diff
• Patch updated data: patch_data, merge_data
• Render a diff: render_diff

And now for the long version...

Diff?

diff

• Command-line utility for comparing text files.
• Used in all source code version control systems.
• diff checks lines:
  • which lines have changed, removed or added.

daff

\begin{picture}(0,0) \put(250,-70){\includegraphics[width=2cm]{fig/utility.png}} \end{picture}

• Utility for comparing tables
• daff compares records and columns:
  • which values changed
  • rows added/removed
  • columns added/removed.

Why o why?

\begin{picture}(0,0) \put(120,-60){\includegraphics[width=5cm]{fig/why.png}} \end{picture}

• To support version control of data.frame.
• To log changes of data.
• e.g. subsequent steps in data process: what did these steps do?
• To support monitoring external data changes beyond your control.
• To make even manual editing reproducible (Note: manual editing is really really bad).

Use case: data update

\begin{picture}(0,0) \put(120,-10){\includegraphics[width=2cm]{fig/update}} \end{picture}

Raw data update

• You have build a nice R script:
  • takes raw data as input
  • removes errors
  • fits a model
  • calculates output

You get an updated raw data file: what are the changes?

• Did output change?
• Also input: should the script be adapted, e.g. data cleaning?

Use case: manual editing

• bad practice, but it happens: e.g. implausible values, manual data correction.

Manual editing

\begin{picture}(0,80) \put(120,0){\includegraphics[height=0.3\textheight]{fig/blackbox}} \end{picture} - Compare the input and output - Make the manual step reproducible: all process steps can be re-executed:

- data + changes = new data
- in `diff` parlor: version1 + patch = version2

Daff protocol

Highlighter diff format

• highlighter diff format: https://paulfitz.github.io/daff-doc/spec.html
• diff protocol for tabular data.
• shows rows/columns that changed.
• supports patching data.
• format itself is in tabular format (nifty!)
• can be stored in txt (csv) or db.

Detecting changes

daff detects the following changes:

• changed a value.
• row added.
• row removed.
• column added.
• column removed

• type change of a column (partially)

  • daff supports it, but highlighter format not

diff_data: value was changed

library(daff)
x         <- data.frame(A=1, B=  1)
x_changed <- data.frame(A=1, B=100)
patch <- diff_data(x, x_changed)
print(patch)

patch_data: apply the change

x
patch_data(x, patch)

replay` the change on original data:

• when org. data is updated, same procedure!

diff_data: row was added

x         <- data.frame(A=1  , B=1)
x_changed <- data.frame(A=1:2, B=1:2)
diff_data(x,x_changed)

diff_data: row was deleted

x         <- data.frame(A=1:2, B=1:2)
x_changed <- data.frame(A=1  , B=1)
diff_data(x,x_changed)

diff_data: column was added

x         <- data.frame(A=1, B=1)
x_changed <- data.frame(A=1, B=1, C=1)
diff_data(x,x_changed)

diff_data: column was removed

x         <- data.frame(A=1, B=1, C=1)
x_changed <- data.frame(A=1, B=1)
diff_data(x,x_changed)

diff_data options

diff_data( data_ref, data
         , always_show_header       = TRUE
         , always_show_order        = FALSE
         , columns_to_ignore        = c()
         , count_like_a_spreadsheet = TRUE
         , ids                      = c()
         , ignore_whitespace        = FALSE
         , never_show_order         = FALSE
         , ordered                  = TRUE
         , ... 
         )

differs_from

• Pipe-friendly version of diff_data

x_changed %>% 
  differs_from(x)

# same as

diff_data(x, x_changed)

Merging

• Combine two derived data.frames from a common parent.

x   <- data.frame(A =   1, B=  1)
# two changes were made in parallel
x_a <- data.frame(A = 100, B=  1)
x_b <- data.frame(A =   1, B=100)
merge_data(x, x_a, x_b)

Reading and writing table diffs

x         <- data.frame(A = 1, B =   1)
x_changed <- data.frame(A = 1, B = 100)
# write diff to disk
diff_data(x, x_changed) %>% 
  write_diff("diff.csv")

# and read it again from disk
read_diff("diff.csv") %>% 
  patch_data(x, .)

Render diff

x         <- data.frame(A = 1:2, B = 1:2)
x_changed <- data.frame(         B = 2  , C = 1)

x_changed %>% 
  differs_from(x) %>% 
  render_diff(use.DataTable=FALSE)

\begin{center} \includegraphics[width=7cm]{fig/daff.png} \end{center}

Implementation

• Wraps the library daff.js, by Paul Fitzpatrick (@fitzyfitzyfitzy).
• library actually written in Haxe, which compiles to js, python, C++

• Uses R package V8 to run daff.js, by Jeroen Ooms(@opencpu):

  • With V8 any js library can be run from R!

Other R libs

diffobj

Other R library:

• diffobj: very good general purpose diff for all R objects.
• Provides nice visualizations for differences between R objects.
• Also includes a diffCsv function, but more limited then daff.
• daff specialized in data.frame:
  • data aligning: id columns.
  • ignoring columns.
  • no patching of data.
  • ...

\Large{Thank you for your attention!}

Interested?

install.packages("daff")`

or visit:

• http://github.com/edwindj/daff

daff documentation built on Oct. 9, 2023, 1:06 a.m.