Intercepting and Analyzing HTTPS Traffic with mitmproxy

Introduction

In today‘s world, the vast majority of web traffic is encrypted using HTTPS (HTTP Secure). While this provides a critical layer of security and privacy for users, it also poses challenges for IT and security professionals who need visibility into the requests and responses flowing between clients and servers.

This is where tools like mitmproxy come in. mitmproxy is a popular open source interactive HTTPS proxy. It allows you to intercept, inspect, modify and replay web traffic such as HTTP/1, HTTP/2, WebSockets, or any other SSL/TLS-protected protocols.

In this in-depth guide, we‘ll walk through how to set up and configure mitmproxy to analyze HTTPS traffic. We‘ll cover the step-by-step process to intercept requests and responses, view them in mitmproxy‘s intuitive UI, and even write custom scripts to automate traffic modification and analysis. Whether you‘re a penetration tester, web developer, or systems administrator, you‘ll come away with a solid understanding of mitmproxy‘s powerful capabilities.

But before we dive into the technical details, let‘s start with an overview of what HTTPS is and why intercepting it can be valuable.

HTTPS and TLS Encryption

HTTPS is an extension of the HTTP protocol that adds a layer of encryption using Transport Layer Security (TLS), formerly known as Secure Sockets Layer (SSL). When a client, like a web browser, connects to an HTTPS server, they perform a "TLS handshake" to verify identities and establish shared secret keys. The client and server then use these keys to symmetrically encrypt all HTTP messages before sending them across the network.

The encryption provided by TLS makes it extremely difficult for an attacker to eavesdrop on sensitive data like passwords, credit card numbers, and other private information sent between the client and server. Additionally, TLS provides server authentication, allowing the client to verify it is communicating with the legitimate server and not an impostor. These security benefits have made HTTPS the de facto standard for login forms, e-commerce, online banking, and any website dealing with sensitive user data.

However, the use of HTTPS also means that any intermediary between the client and server, such as a proxy or network monitoring tool, can no longer view the contents of the HTTP messages. The full URLs, headers, request/response bodies, etc. are encrypted. This is great for end user security but poses a challenge for anyone legitimately trying to analyze or debug the traffic.

How mitmproxy Intercepts HTTPS

This is where mitmproxy comes into play. mitmproxy works by positioning itself as a "man-in-the-middle" (MITM) between the client and the HTTPS server. In other words, instead of the client directly communicating with the server, the client talks to mitmproxy, which in turn talks to the server.

To make this work, mitmproxy essentially splits the TLS connection into two separate encrypted connections – one between the client and mitmproxy, and one between mitmproxy and the server. During the initial TLS handshake with the client, mitmproxy dynamically generates a TLS certificate for the destination server signed by its own certificate authority (CA). As long as the client trusts this CA, the client will establish a TLS connection with mitmproxy believing it to be the real server.

Meanwhile, mitmproxy makes its own separate TLS connection to the actual server. Because mitmproxy is operating as a proxy, it can decrypt and re-encrypt traffic on the fly as it passes through. This allows mitmproxy to read and/or modify any of the HTTP messages in plaintext before forwarding them on to their actual destination, and is the core of how it is able to intercept HTTPS.

Of course, this only works if the client trusts mitmproxy‘s root CA certificate, otherwise the client will display an untrusted certificate warning when mitmproxy attempts a TLS handshake. We‘ll cover how to install the CA a bit later.

It‘s important to note that mitmproxy‘s HTTPS interception capability should only be used against systems and users you have permission to test, such as within your own organization for debugging or security auditing purposes. Intercepting another user‘s HTTPS traffic without consent is generally illegal and unethical.

Setting Up mitmproxy

With those caveats out of the way, let‘s walk through how to get mitmproxy up and running. The examples below were performed on a Linux-based system but the core concepts apply across Windows, macOS, and other Unix-like platforms.

Installation

The simplest way to install mitmproxy on Linux is via pip, the standard Python package manager:

sudo pip3 install mitmproxy

This will install mitmproxy along with its key dependencies. You can verify it‘s working by running:

mitmproxy --version

Which should print out the current mitmproxy version. As of this writing the latest version is 9.0.1.

Basic Usage

Once installed, you can launch mitmproxy from the terminal with:

mitmproxy

This starts mitmproxy in it‘s default mode with a text-based intercepting proxy interface. Commands are issued via single key bindings:

• q to exit
• ? for help
• arrow keys to navigate
• enter to view flow details
• a to resume intercepted flows
• z to clear flow list

Note: See mitmproxy‘s docs on key bindings for the full list of key commands and how to customize them.

When you launch mitmproxy it should display a message like:

Proxy server listening at http://*:8080

This indicates mitmproxy is now listening for HTTP proxy connections on TCP port 8080. You can use the --listen-host and --listen-port options to change the listening address and port respectively.

Client Proxy Configuration

Now you need to configure your client (e.g. web browser) to proxy its traffic through mitmproxy. The exact steps vary by operating system and browser, but the general idea is to set the HTTP/HTTPS proxy to the hostname and port where mitmproxy is listening.

For example, if you‘re running mitmproxy on the same machine you‘re browsing from, you can set the browser‘s proxy to:

• HTTP Proxy: localhost Port: 8080
• SSL Proxy: localhost Port: 8080

Tip: For Chrome you can also use the handy Proxy SwitchyOmega extension to make toggling the proxy settings on/off easy.

Once the proxy is enabled, try visiting an HTTPS website (e.g. https://example.com). You will likely see an SSL certificate warning page. This is because the browser does not trust the root CA certificate mitmproxy is using to dynamically sign the proxied TLS certificates. To resolve this, we need to install mitmproxy‘s CA cert.

Installing the mitmproxy Root Certificate

When you first launch mitmproxy, it generates a unique root CA certificate that is then used to dynamically sign the per-host certificates when proxying TLS traffic. For mitmproxy‘s interception to work smoothly without TLS errors, the client needs to trust this root CA cert.

You can find the CA cert PEM file at:

~/.mitmproxy/mitmproxy-ca-cert.pem

The process to install this certificate into your OS/browser‘s trusted root CA store varies by platform:

• On Linux, you can use the update-ca-certificates command:

  sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt
  sudo update-ca-certificates

• On macOS, you can double-click the PEM file to launch Keychain Access, and choose "Always Trust" for the mitmproxy cert

• On Windows, you can right-click the PEM, choose "Install Certificate", and place it in the "Trusted Root Certification Authorities" store

• In Firefox, you can import the PEM file in Preferences > Privacy & Security > Certificates > View Certificates > Authorities > Import

Warning: Only install the mitmproxy certificate on systems you explicitly want to intercept traffic from, such as your own development machine or a test device. Consider using a separate browser profile for mitmproxy usage to limit its ability to intercept traffic system-wide.

Once the mitmproxy CA cert is installed and trusted, you should be able to visit HTTPS websites via the proxy without TLS errors. Back in the mitmproxy UI, you will start seeing flows populate the list as you browse.

Inspecting Intercepted Traffic

Now that we have mitmproxy intercepting our HTTPS traffic, let‘s explore how to inspect the requests and responses.

From the mitmproxy flow list view, you can use the arrow keys to highlight a flow and press enter to bring up the flow detail view. This allows you to drill down into the specifics of the request and response, including:

• Request URL, method, HTTP version, and headers
• Request parameters and body content
• Response status code, headers, and body content
• Timing information
• Size information

Pressing tab in the flow detail lets you cycle between the request and response. You can use the arrow keys to scroll up/down. Pressing q returns you to the flow list.

One of the powerful things about mitmproxy is that it allows you to view decoded content, even for non-textual data. For example, if the response contains a gzip compressed JSON object, mitmproxy will automatically unzip it and pretty-print the JSON for easy reading.

Tip: The ? key will bring up mitmproxy‘s quick help dialog which lists all available key bindings based on the current context.

In addition to inspecting, mitmproxy also allows you to modify flows, replay requests, and save content to files. Spend some time exploring the UI to get a feel for its capabilities. The official mitmproxy docs are a great resource to learn more.

Writing Custom Response/Request Modification Scripts

While mitmproxy‘s console UI is powerful on its own, you can unlock even more capabilities by writing custom Python scripts to automatically modify requests and responses as they pass through the proxy. This allows you to do things like:

• Rewrite URLs or redirect requests
• Inject additional headers
• Modify query parameters or POST bodies
• Replace response content or status codes
• Perform fuzzing attacks
• Create custom security scanners

mitmproxy scripts are regular Python files that implement and export a set of special event methods. These event methods are hooks that get called at different points of the request/response lifecycle.

For example, here‘s a dead simple script that automatically rewrites all HTTP GET request URLs to point to example.com:

from mitmproxy import http

def request(flow: http.HTTPFlow) -> None:
    if flow.request.method == "GET":
        flow.request.url = "https://example.com"

To execute this script, you pass it to mitmproxy on the command line:

mitmproxy -s example_redirect.py

A few key things to note:

• We import the mitmproxy.http module to get access to the HTTP-specific flow components
• We define a request event method that gets called immediately after each request headers are read
• The flow object contains the request/response data and has attributes for inspecting and mutating them
• Any changes made to flow get reflected in the traffic passed to the real server

This is just a tiny taste of what‘s possible with scripting. mitmproxy‘s scripting API provides a ton of flexibility. You can go quite deep with it, doing things like tracking history and state across requests, performing async operations, generating custom UI elements, etc.

There‘s also a growing ecosystem of user-contributed mitmproxy scripts, extensions, and add-ons hosted on GitHub. It‘s worth perusing the mitmproxy tag there to see examples of the creative things folks are building.

Warning: mitmproxy executes scripts with your full user privileges, so be careful when running scripts from untrusted sources. Inspect them carefully.

Closing Thoughts

We‘ve covered a lot of ground in this post, from an overview of how HTTPS and mitmproxy work, to hands-on examples of setup, traffic inspection, and custom scripting. Hopefully you now have a solid foundation to start integrating mitmproxy into your own workflow.

To summarize, mitmproxy is a powerful open source tool for intercepting, viewing, and modifying HTTP/HTTPS traffic. It works by acting as a man-in-the-middle, splitting encrypted traffic streams into separate legs, while presenting its own trusted TLS certificates to the client.

Some common use cases for mitmproxy include:

• Penetration testing and security research
• Reverse engineering and debugging mobile/desktop applications
• Auditing and black-box testing web services
• Modifying and replaying API traffic
• Developing custom protocol analyzers and scanners

As with any security tool, mitmproxy is a double-edged sword. Always obtain explicit permission before using it to intercept traffic you do not own. Be mindful of the sensitive data you may be viewing and practice responsible disclosure.

If you‘re hungry to learn more, I suggest the following resources:

• The official mitmproxy docs, especially the Concepts, Tools, and Scripting sections
• mitmproxy‘s GitHub repo for source code and contributor docs
• The mitmproxy tag on GitHub for community scripts and extensions
• HTTP Prompt, an interactive command-line HTTP client that integrates with mitmproxy
• My own [mitmproxy experiments repo]() where I dump one-off test scripts

Happy proxying!