In ropenscilabs/middlechild: Tools to Intercept, Validate and Consume Web/Network Traffic

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

A seriously large chunk of this package is based on mitmproxy.

"MITM" stands for "man-in-the-middle" (perhaps, "PITM"/"person-in-the-middle" would be more appropriate, today) which is "an attack where the attacker secretly relays and possibly alters the communication between two parties who believe they are directly communicating with each other."[^https://en.wikipedia.org/wiki/Man-in-the-middle_attack].

This package is not so much "attacking" you as it is providing you with a tool that will sit between outbound HTTP[S] requestors and their targets and record-then-relay your request and then do the same with the associated response.

For both package-check functionality and generic functionality we use R to spawn a mitmdump background process that only (mostly) requires you to reconfigure your R session's, browser's or operating system's proxy configuration to use it as a relay.

Tell me what I need to do

First, go to the mitmproxy documentation site and install the software. It has to be on the system PATH and this package really cannot do this for you since it relies on a working Python setup and we all know how fragile that can be.

Done. What's next?

Next, you need to install (CA) certificates. (NOTE: these are totally not as cool as gift certificates.)

"CA" stands for certificate authority and that formal sounding word pair are part of what make SSL (i.e. "http*s*://...") work on the internet. Certificate authorities generate certificates for web sites to use and for the whole "encrypted connection" to work your system (and you) need to trust certificates made by thata certificate authority. This is normally done by operating system and browsers working with certificate authorities to embed their "this is us" certificates in the operating system and/or browsers certificate wallet and have them be trusted by default.

When you visit a web site that uses SSL, that site presents its certificate to you. This is much ike you show your passport at the airport to the TSA agent in the U.S. The first check the TSA agent does it to make sure your passport is a valid passport issued by a country the U.S. trusts. Your browser does the same thing. If your passport is a bad forgery or from an untrusted nation you won't get past TSA. If the SSL cert presented by a web site was not created by a certificate authority trusted by the browser/OS, the connection won't be made.

OK, but why are you telling me all this?

This package uses mitmproxy and places mitmproxy in-between your HTTP[S] calls and the destinatiojn server. That means mitmproxy "proxies" requests on your behalf. That works seamlessly for plain HTTP sites, but for sites that are SSL-enabled, your client (which will be R in this case) is going to be asking for https://example.com and talking to mitmproxy first instead of https://example.com. The mitmproxy itself needs a certificate to talk to you and it would be a Very Bad Thing if it came with a certificate that was automatically trusted by your computer since evil people coulde use that fact to interacept all your web requests on any network and steal your data or just eavesdrop on you. To overcome this, mitmproxy generates a new Certificate Authority certificate and new mitmproxy server certificate just for you on initial startup. You're going to trust these certs which means you should never give them away to anyone. We apologize for being so demanding and scary-sounding, but once you trust this fresh, new mitmproxy certificate authority you signed up to be a cybersecurity professional and if the keys generated by mitmproxy on your system fall into the wrong hands, you will never know if your traffic is being intercepted.

Got it. So, how do I "trust" these mitmproxy certificates?

You need to fire up mitmproxy or mitmdump or mitmweb once and leave it running. You can do that with:

pid <- middlechild::start_mitm()

You then need to get to the certs you need to install and trust them.

There is an experimental middlechild::install_mitm_certificate() function to perform this task for you at the operating system level. It currently only works on macOS and does not install the certificates into Firefox (you will need to do that manually and also trust them in Firefox).

The official docs — https://docs.mitmproxy.org/stable/concepts-certificates/ — get you most of the way there for this. But you need to also make sure your OS and browsers know about and trust this "god-like" certificate.

On Macs, that means using Safari to do the intial download then, double-clicking on the cert and importing it, then finding it (use mitm as a search field param) and settng the appropriate trust levels. Modern Firefox users will have to do the same dance. Linux users (interactive or automation) will also need to do do this but it's a more manual task. There will eventually be a video and/or set of walkthrough steps for this. The docs do spend quite a bit of real estate on explaining this but this process is really not as "easy" as it should be.

You'll know this works if you change your browser configuration to use a proxy server and use the mitmproxy server information in the configuration (TODO including images).

Once you are done, make sure to:

middlechild::stop_mitm(pid)

ropenscilabs/middlechild documentation built on May 11, 2022, 9:11 a.m.