Hashicorp Vault – Reverse Proxy With NGINX

Recently I was presented with a very common problem, offer up a service which uses an unprivileged port, present that service through a reverse proxy and keep the entire service secure by completing TLS termination on the proxy. This is a pretty old problem and in my case the service is Hashicorp Vault, but what’s odd is that for such a popular platform I couldn’t find any guides or configuration examples for how to do this with NGINX which strikes me as odd for such a popular application. So let’s take a look at how we can do it.

The Problem and The History

By default, Vault runs on TCP port 8200, this is an Unprivileged Port, meaning that any user can bind it to a service. Long ago in the original designs of the Unix Kernel (and as a byproduct, the Linux Kernel) a design decision was taken to earmark the first 1024 ports as Privileged Ports (this is why the IANA designate these for the most critical operations). The theory went that back in the days when only sysadmins operated computers you could be sure that if you were connecting to one of these ports over a network you could rest assured that it had been configured by a root user (at least a superuser).

With the history lesson out of the way, Vault enforces this philosophy so if you’ve properly hardened your installation it won’t just bind to a Privileged Port, and since we’re trying to use Vault for very sensitive data we really want this to be a secure platform and we’ll probably want to be ultimately offering the service over HTTPS using TCP port 443.

Intended Goal

Technically, we can perform TLS termination on the Vault instance and offer out the Vault instance at TCP port 8200, however if you have strict firewall requirements you may not want to open yet another port, and fewer ports is always nicer, so let’s look at how to use a reverse proxy to offer out Vault correctly over TCP port 443 using NGINX.

So our finished instance is going to look something like:

Implementation – Vault

For the sake of brevity, we’ll be using an almost identical setup and hardening of Vault as described in a previous article here and will even use the same Certificate and Private key that we used when terminating TLS on Vault directly, a major different will be the vault.hcl configuration file which will now use the localhost address as the sole TCP listener and listen using HTTP rather than HTTPS. The finished vault.hcl will look like the below:

storage "file" {
        path = "/opt/vault"
}

ui = true

listener "tcp" {
        address = "127.0.0.1:8200"
        tls_disable = 1
}

max_lease_ttl = "10h"
default_lease_ttl = "10h"
api_addr = "https://127.0.0.1:8200"

Implementation – NGINX

Now the real meat, we’ll need to get NGINX installed and configured, the only installation needed is NGINX and that’s a single command, I’m using Ubuntu 18.04 so we can use apt-get:

sudo apt-get install nginx

With the Certificate and Private Key located in /etc/ssl/certs and /etc/ssl/private respectively we can just use the NGINX default configuration file which is located in /etc/nginx/sites-enabled/default. If we edit the file using nano:

sudo nano /etc/nginx/sites-enabled/default

Delete the contents of the file and replace with the below functional configuration:

server {
        listen 443;
        server_name mc-vault.madcaplaughs.co.uk;

        ssl on;
        ssl_certificate /etc/ssl/certs/mc-vault.cer;
        ssl_certificate_key /etc/ssl/private/mc-vault.key;

        ssl_prefer_server_ciphers on;
        ssl_session_timeout 1d;
        ssl_session_cache shared:SSL:50m;
        ssl_session_tickets off;

        location / {
                proxy_pass http://127.0.0.1:8200;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto https;
        }
}

The magic here is happening in the location block, which is seeing our requests be forwarded from the server_name (defined in the server block) to the host and port defined as proxy_pass with some additional headers which will also be included.

In this example my server is named mc-vault.madcaplaughs.co.uk, yours should be substituted with the FQDN of your own host (which will also need to be reflected in the CNAME on your Certificate), this isn’t an issue here as I’m reusing the same certificate that was previously applied directly to Vault.

Completing

With the configs in place, we’ll need to restart NGINX in order to load the configuration:

sudo /etc/init.d/nginx restart
# [ ok ] Restarting nginx (via systemctl): nginx.service.

Now if we attempt to connect to Vault via the GUI on TCP port 443 we should see a valid certificate presented:

As this is a new Vault instance we will need to init the backend, this is covered in my previous Setup and Installation post here so I’m not going to cover it again, as we can see the proxy is working correctly.

TLS sessions are now terminated at NGINX for all requests and proxied in to Vault, one important trade off to understand here is that Certificate Authentication with Vault will not work with this particular configuration as TLS termination is not being performed against Vault, a little more advanced configuration will be needed to solve that problem…but nothing is impossible.