Status Services → Expedited WAF → Expedited SSL → IP Investigator → Real Email Resources → Knowledge Base → AWS In Plain English → Blog → Heroku Security Resources Use Cases → DDoS Protection → Bot & Malicious Traffic Blocking → Fraud Detection & Prevention → Credential Stuffing Protection → Compliance → OWASP Top 10 Compliance → AI Scraping Prevention → Geographic Request Blocking → Virtual Patching About Contact

HAProxy for Modern Load Balancing

A load balancer with HTTP/2 and dynamic reconfig.

January 24, 2020
HAProxy for Modern Load Balancing

HAProxy is a load balancer and SSL/TLS terminator. Since you want to keep your HTTPS certificate and key in one place, want to send requests to your multiple app servers evenly, and want to be able to take down individual servers for deploys, a load balancer can sit there monitoring the backend app servers and sending traffic only to the servers you want.

Modern HAProxy (2.x+) includes full HTTP/2 support, allowing SSL/TLS termination directly on the load balancer. This is the standard configuration for most production deployments.

Most HAProxy guides on the internet are outdated: they require you to manually modify the config or stop services on your app servers during deploys, but you don't need to do that anymore. HAProxy's dynamic configuration allows you to update backends without service interruption.

This guide shows you how to use modern HAProxy to provide HTTP/2 to browsers, keep your HTTPS certificates in one place (the load balancer), and control everything live on the load balancer.

HAProxy vs nginx

If you need a load balancer - and you probably do - your main choices are HAProxy and it's main competitor nginx. Here's a quick, opinionated guide to choosing between them:

HAProxy advantages over nginx

HAProxy is simpler than nginx:

nginx advantages over HAProxy

Here's what we prefer about nginx:

nginx is excellent, but HAProxy combines HTTP/2 with 'batteries included' simplicity.

What you'll get

This guide is for Ubuntu 22.04 or 24.04 LTS. We're going to set up this:

an image

We'll also have the following features:

Logging to systemd

So you can use the standard journalctl logging and filter your logs using handy stuff like 4 hours ago, yesterday, etc.

The various www vs non-www, HTTP vs HTTPS combinations redirected to a single HTTPS site.

This ensures there's only one, secure copy copy of every resource for both clarity and SEO purposes.

The ability to switch backends dynamically

So you can upgrade your app without taking it offline. HAProxy has a runtime API which can be used to manually mark a server as disabled, so we can control which backends get used without having to modify our config or actually shut down the services on the backends.

HTTP/2 support in all browsers

For speed! One of the pages on our blog loads in 1.9s on HTTP 1.1. The same page loads in 600ms over HTTP/2.

A branded 'sorry' page

Just in case you break both the app servers at the same time.

A separate server that handles blogs and marketing content

So you can keep your blog independent of the main app and update it on its own schedule.

Correct proxy headers for working GeoIP and logging.

So your app servers can see the proper origin of browser requests, despite the proxy. Because asking customers for their country when you already know is a waste of their time.

Support for HTML5 Server Sent Events

For realtime streaming.

An A+ on the SSL Labs test

So the users can connect privately to your site.

Let's get to work

HAProxy will handle HTTPS and pass incoming connections onto your marketing server or app servers based on the URL.

To upgrade, you can enable the new server and disable the server running the old version using a simple command.

Installing HAProxy

For Ubuntu 16.04 we'll use Vincent Bernat's PPA to get HAProxy 1.8:

apt-get install software-properties-common 
add-apt-repository ppa:vbernat/haproxy-1.8
apt-get update
apt-get install haproxy socat

Set up our config

Here's a sample haproxy.cfg:

global
    # Log to systemd's /dev/log compatibility socket
    log /dev/log local0 info
    chroot /var/lib/haproxy
    pidfile /var/run/haproxy.pid
    maxconn 4000
    user haproxy
    group haproxy
    daemon
    # turn on stats unix socket - see http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.1-stats%20socket
    stats socket /var/lib/haproxy/stats mode 600 level admin 
    stats timeout 2m
    # From https://mozilla.github.io/server-side-tls/ssl-config-generator/ 
    # Intermediate config as of Jan 2018 - update it from the link above!
    tune.ssl.default-dh-param 2048
    ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
    ssl-default-bind-options no-sslv3 no-tls-tickets
    ssl-default-server-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
    ssl-default-server-options no-sslv3 no-tls-tickets
defaults
    mode http
# Needed for GeoIP
option forwardfor
# See http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#option%20http-server-close
    option http-server-close
    log global
    option httplog
    option dontlognull
    option http-server-close
    option forwardfor except 127.0.0.0/8
    option redispatch
    retries 3
    timeout http-request 10s
    timeout queue 1m
    timeout client 1m
    timeout server 1m
    timeout check 10s
    maxconn 3000
    # From http://stackoverflow.com/questions/21419859/configuring-haproxy-to-work-with-server-sent-events
    # Set the max time to wait for a connection attempt to a server to succeed
    timeout connect 30s
    # handle a client suddenly disappearing from the net
    timeout client-fin 30s
    option http-server-close
    # The 'stats' site, where we see what's up and what's down - uncomment and set a password if you want it!
    # stats enable
    # stats uri /haproxy?stats
    # stats realm Strictly\ Private
    # stats auth someusername:i-am-an-awful-password-and-you-should-change-me
    # Show a custom page during site maintenance (ie, when 'blue' and 'green' are both down)
    errorfile 503 /etc/haproxy/errors/503-mycustom.http
frontend public
    # HTTP/2 - see https://www.haproxy.com/blog/whats-new-haproxy-1-8/
# h2 is HTTP2 with TLS - see https://http2.github.io/faq/
# Order matters, so h2 before http1.1
    bind :443 ssl crt /etc/https/cert-and-private-key-and-intermediate-and-dhparam.pem alpn h2,http/1.1
    # Redirect http -> https
    bind :80
    redirect scheme https code 301 if ! { ssl_fc }
    # Redirect www -> example.com
    redirect prefix https://example.com code 301 if { hdr(host) -i www.example.com }
    # HSTS (15768000 seconds = 6 months)
    http-response set-header Strict-Transport-Security max-age=15768000
    # Use the marketing site for marketing URLs
    acl marketing path_beg -i /help /sitemap.xml /BingSiteAuth.xml /about /blog /videos/blog /images/blog /fonts/blog /css/blog /js/blog
    use_backend marketing if marketing
    # Everything else goes to the app servers
    default_backend app
    option httpclose
    option forwardfor
backend marketing
    balance roundrobin
    server static 127.0.0.1:8000 check
backend app
    # From http://blog.haproxy.com/2014/01/17/emulating-activepassing-application-clustering-with-haproxy/
    # See also https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#stick-table
    stick-table type ip size 1m
    stick on dst
    # See https://cloud.digitalocean.com/droplets for these IPs
    server green 10.1.1.1:8000 check
    server blue 10.1.1.2:8000 check backup
    # http://stackoverflow.com/questions/21419859/configuring-haproxy-to-work-with-server-sent-events
    timeout tunnel 10h

After deploying the config:

  • Change example.com to your own domain
  • Update the bind :443 ssl crt line to point to your HTTPS certificate and key file
  • Update the IP addresses and ports for marketing, blue and green to your own marketing and app servers.
  • Modify the URLs in marketing - we use /blog, /images/blog etc. - to match your marketing URLs
  • Recommended TLS/SSL configurations change over time, and are also a balance between security and the browsers you want to support: visit Mozilla's SSL Config Generator for up to date ssl-default options that match your needs.
  • Add your own username and password for the 'stats' tool before uncommenting it

Once that's done:

  • Restart HAProxy service with 
sudo systemctl restart haproxy
  • Check the journal/logs with 
sudo journalctl -u haproxy

You'll see the public frontend, and the marketing and app backends mentioned inthe config file / diagram above launched:

Jan 05 12:51:11 haproxy-load-balancer systemd[1]: Starting HAProxy Load Balancer...
Jan 05 12:51:11 haproxy-load-balancer haproxy[3333]: Proxy public started.
Jan 05 12:51:11 haproxy-load-balancer haproxy[3333]: Proxy marketing started.
Jan 05 12:51:11 haproxy-load-balancer haproxy[3333]: Proxy app started.

Handle upgrades

You can create a small script called swap-server to make checking and changing which app server is up easier. Put it in /usr/local/sbin. It's optional, but saves you a bunch of typing.

To see the current status

Just run:

sudo swap-server ls

with no arguments. The output will look like:

Proxy name, Service name, Status
app,blue,UP
app,green,UP
app,BACKEND,UP

We can see bopth servers are up.

Switching to Blue or Green

What if we want to take one down to deploy a new version? Just:

sudo swap-server blue

This will:

  • Mark green as down for maintenance
  • Mark blue as available
  • Show the status of both servers in the app backend

The output will look like:

Proxy name, Service name, Status
app,blue,UP
app,green,MAINT
app,BACKEND,UP

You can see how green is now in maintenance mode. Requests will stay on blue until it's marked as online again.

To swap to green, do the opposite!

sudo swap-server green

Conclusion

Thanks! If you have suggestions or feedback, feel free to leave a comment on Hacker News.

View All Blog Posts