In hrbrmstr/burrp: Tools to Import and Process 'PortSwigger' 'Burp' Proxy Data

burrp : Tools to Import and Process PortSwigger 'Burp' Proxy Data

The Burp Suite is a set of web application and security testing tools. Combined, they intercept, record and can even transform browser traffic.

It is often necessary to use an intercepting proxy server to identify targets for web-scraping. The 'PortSwigger' 'Burp' proxy records web traffic from interactive use of a browser and provides mechanisms to archive this activity to an XML file for further processing. Tools are provides to import this archive file and work with the request and response objects in a similar manner as one would with results of verb calls from functions in the 'httr' package.

For now, the solo function - read_burp - expects the request and response elements of a Burp XML export file to be base 64 encoded.

Eventually there will likely be an as_har() function to turn the entire structure into a HARtools object.

To use this package you either need to have Burp proxy export files handy, use the built-in example data file or download Burp and capture some web traffic and export the data.

The current Burp proxy example capture file is of a capture from the Bloomberg Hottest Year on Record scrollytelling datavis. It's a good example of how to use Burp to capture requests as you interactively work in a browser (you can get the JSON data behind the vis this way, too).

The following functions are implemented:

• read_burp: Read in a Burp proxy XML export file
• find_sequence: Find the first occurrence (if any) of a sequence of raw bytes ('pattern') in 'buffer'
• extension_is: Functional helpers
• method_is: Functional helpers
• mime_is: Functional helpers
• port_is: Functional helpers
• protocol_is: Functional helpers
• status_is: Functional helpers

Installation

devtools::install_github("hrbrmstr/burrp")

options(width=120)

Usage

library(httr)
library(burrp)
library(hrbrthemes) # github
library(tidyverse)

# current verison
packageVersion("burrp")

system.file("extdata", "hottest_year.xml", package="burrp") %>%
  read_burp() -> burp_df

glimpse(burp_df)

burp_df$request[[3]]

burp_df$response[[3]]

content(burp_df$response[[3]], simplifyDataFrame=TRUE) %>% 
  unnest() %>% 
  filter(key != "0") %>% 
  mutate(key=factor(key, levels=as.character(1:12), labels=month.abb)) %>% 
  mutate(color=ifelse(year=="2014", "#cb181d", "#b5b5b5")) %>% 
  mutate(size=ifelse(year=="2014", 0.5, 0.15)) %>% 
  ggplot(aes(key, value, group=year, color=color, size=size)) +
  geom_hline(yintercept = 0, size=0.2) +
  geom_line() +
  scale_x_discrete(expand=c(0,0)) +
  scale_y_continuous(breaks=c(-1, 0, 1), labels=c("-1°F", "20th Century\nAverage", "1°F")) +
  scale_color_identity() +
  scale_size_identity() +
  labs(x=NULL, y=NULL, title="Hottest year on record (2014 in red)",
       caption="Source: Bloomberg <https://www.bloomberg.com/graphics/2014-hottest-year-on-record/>") +
  theme_ipsum_rc(grid="XY", axis="xy") +
  theme(legend.position="none")

summary(burp_df)

Test Results

library(burrp)
library(testthat)

date()

test_dir("tests/")

hrbrmstr/burrp documentation built on April 16, 2021, 3:15 a.m.