Reverse Proxies and TLS Termination

I use reverse proxies all the time. They put certificates in one place, publish private services, run lots of apps behind one IP, and add health checks or access controls. They also cause some of the most confusing bugs I know of, because once there's a proxy in the middle, the app no longer knows the original scheme, hostname, or client IP unless you tell it. Most of this page is about the question: where does traffic really terminate, and what should trust what?

Where the proxy sits

Internet client
      |
      v
DNS and public IP
      |
      v
Firewall / NAT
      |
      v
Reverse proxy or load balancer
      |      |      |
      v      v      v
   App A  App B  App C
    

The key thing to understand: the client connects to the proxy. The proxy then makes its own, separate connection to the app. Two TCP connections, often two TLS sessions, even though the user sees one website. Most proxy weirdness comes from forgetting that.

What I use them for

• Lots of apps on one IP, sorted by hostname.
• Handling public certificates in one place.
• Keeping app servers on private addresses.
• Load balancing across several backends.
• Adding auth, rate limits, security headers, or a maintenance page.
• Blue/green deploys and migration cutovers.
• Putting modern TLS in front of old apps that can't do it themselves.

Trust boundaries

The proxy takes in untrusted stuff from the Internet and then adds headers that the app might trust. So the app has to be able to tell the difference between headers your proxy set and headers some random client made up.

Don't "fix" internal TLS by turning off verification

If the proxy talks HTTPS to the app, it should verify the app's certificate. If you're getting a name mismatch or the cert is from a private CA, fix it properly: send the right SNI name, trust your private CA, or put a valid cert on the backend. Turning verification off gives you encryption without knowing who you're talking to, and it hides config mistakes.

Forwarded headers

The mixed-content classic

Browser requests:        https://monitor.example.com/
Proxy connects upstream: http://10.10.10.25:8080/
Application believes:    request scheme is HTTP
Application generates:   http://monitor.example.com/assets/app.css
Browser blocks asset:    mixed content
    

If you've put a proxy in front of an app, you've probably hit this one. The fix is to pass along the original HTTPS state and configure the app (or FastCGI) to trust it. Rewriting every response body to swap http for https technically works, but it's fragile.

Where TLS ends

Terminate at the proxy, plain HTTP behind it

Simple, and no internal certificates to manage. Just make sure the network between the proxy and the app is one you actually trust.

Client -- HTTPS --> Proxy -- HTTP --> Application
    

Terminate and re-encrypt

For when internal traffic crosses networks you don't fully trust, policy says everything must be encrypted, or you need to verify the app's identity end to end.

Client -- HTTPS --> Proxy -- HTTPS --> Application
                      |                |
                  public cert      public or private CA cert
    

Pass-through

The proxy forwards the encrypted traffic without opening it. The app keeps control of TLS, but the proxy can't route by URL or touch headers.

Certificate housekeeping

• Know which system requests and renews each certificate.
• Monitor expiration from outside the proxy.
• Protect private keys and limit who can change the config.
• Check full chains and SNI behavior.
• Plan how ACME validation (HTTP-01 or DNS-01) will reach you.
• Write down how certs get restored if you rebuild the proxy.
• Clean up old certificates and hostnames.

Examples

Caddy: the basics

app.example.com {
    reverse_proxy 10.20.30.40:8080
}
    

That's it. If the hostname is public and Caddy can be reached for validation, it'll get and renew the certificate for you. I love Caddy for this. The app still needs to be proxy-aware, though.

Caddy: HTTPS to the backend with a private CA

app.example.com {
    reverse_proxy https://10.20.30.40:8443 {
        transport http {
            tls_server_name app-internal.example.net
            tls_trust_pool file /etc/caddy/internal-ca.pem
        }
    }
}
    

Caddy: multiple backends with health checks

app.example.com {
    reverse_proxy 10.20.30.41:8080 10.20.30.42:8080 {
        health_uri /healthz
        lb_try_duration 5s
    }
}
    

NGINX: a basic proxy with the right headers

server {
    listen 443 ssl;
    server_name app.example.com;

    ssl_certificate     /etc/letsencrypt/live/app.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;

    location / {
        proxy_pass http://10.20.30.40:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
    

NGINX: WebSockets

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 443 ssl;
    server_name realtime.example.com;

    location / {
        proxy_pass http://10.20.30.50:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
    

Forget the Upgrade and Connection headers and WebSockets just silently fail. Very common.

PHP-FPM behind an HTTPS proxy

If NGINX hands requests to PHP-FPM, make sure PHP actually knows the request was HTTPS:

fastcgi_param HTTPS on;
fastcgi_param HTTP_X_FORWARDED_PROTO https;
fastcgi_param HTTP_X_FORWARDED_SSL on;
    

Only hard-code these if that server block can only be reached through your trusted HTTPS proxy. Otherwise, figure out the values from trusted input.

Don't make the proxy your new single point of failure

Put several redundant apps behind one proxy and congratulations, the proxy is now the thing that takes them all down. Think about:

• Two proxies sharing a virtual IP, a routed service IP, or an external load balancer
• Keeping their configs in sync, and testing each one independently
• Where certificates live and how renewal is coordinated
• Health checks that test the real app, not just whether the port is open
• The limits of DNS failover and TTLs
• Separate management and public interfaces
• Enough capacity for TLS handshakes, big uploads, streaming, and buffering

Health checks need to mean something

A health endpoint that always says 200, even when the database is down, happily sends traffic to a broken node. One that runs every expensive check on every hit creates its own load. If the app supports it, separate "am I alive?" from "am I ready for traffic?"

Troubleshooting

Tests I run

# Public DNS and what certificate is being served
dig app.example.com A
dig app.example.com AAAA
openssl s_client -connect app.example.com:443 -servername app.example.com

# Response headers and the full redirect chain
curl -I https://app.example.com/
curl -IL https://app.example.com/

# Hit the backend directly from the proxy (-k is fine for debugging, not for prod)
curl -v http://10.20.30.40:8080/
curl -vk https://10.20.30.40:8443/

# Always validate before reloading
caddy validate --config /etc/caddy/Caddyfile
nginx -t
    

Testing the backend directly from the proxy box is the fastest way to figure out whether the problem is the proxy or the app.

Official docs

• Caddy reverse_proxy directive
• Caddy TLS directive
• NGINX HTTP proxy module
• NGINX HTTPS server configuration

Related: using a proxy during a migration and proxy logs and evidence.