HAProxy in 2018
A load balancer with HTTP/2 and dynamic reconfig.
July 12, 2021
HAProxy is a load balancer and SSL/TLS terminator. Since you want to keep your HTTPS certificate and key in one place, want to send requests to your multiple app servers evenly, and want to be able to take down individual servers for deploys, a load balancer can sit there monitoring the backend app servers and sending traffic only to the servers you want.
HAProxy 1.8 came out just before the end of 2017. It's the first release of HAProxy to include full HTTP/2 support - previous versions only allowed you to pass through HTTP/2 to a seperate server to terminate there. 1.8 finally allows the typical 'terminate on the load balancer' scenario with HTTP/2.
Most HAProxy guides on the internet are outdated: they require you to manually modify the config or stop services on your app servers during deploys, but you don't need to do that anymore. Two years ago HAProxy 1.6 added dynamic configuration, and we're using that to keep out app app during our deploys below.
CertSimple verifies companies for EV HTTPS certs - HAProxy is the second most popular load balancer among our customers, so we thought we'd publish our own guide. This shows you how to use HAProxy 1.8 to provide HTTP2 to browsers, keep your HTTPS certificates in one place (the load balancer), and control everything live on the load balancer.
HAProxy vs nginx
If you need a load balancer - and you probably do - your main choices are HAProxy and it's main competitor nginx. Here's a quick, opinionated guide to choosing between them:
HAProxy advantages over nginx
HAProxy is simpler than nginx:
- The equivalent set up to our own nginx guide takes less lines in HAProxy.
- nginx needs an additional module or a proprietary add in for the same level of stats
- nginx needs an additional module or proprietary add in for dynamic reconfiguration.
nginx advantages over HAProxy
Here's what we prefer about nginx:
- nginx supports Brotli, which makes for faster downloads and, with the correct settings, faster dynamic compression.
- nginx config has include files, make it easier to keep your config neat and understandable by splitting different parts of config into different files.
- nginx has had HTTP/2 support for a couple of years.
What you'll get
This guide is for Ubuntu 16.04 LTS. We're going to set up this:
We'll also have the following features:
Logging to systemd
So you can use the standard journalctl
logging and filter your logs using handy stuff like 4 hours ago
, yesterday
, etc.
The various www vs non-www, HTTP vs HTTPS combinations redirected to a single HTTPS site.
This ensures there's only one, secure copy copy of every resource for both clarity and SEO purposes.
The ability to switch backends dynamically
So you can upgrade your app without taking it offline. HAProxy has a runtime API which can be used to manually mark a server as disabled, so we can control which backends get used without having to modify our config or actually shut down the services on the backends.
HTTP/2 support in all browsers
For speed! One of the pages on our blog loads in 1.9s on HTTP 1.1. The same page loads in 600ms over HTTP/2.
A branded 'sorry' page
Just in case you break both the app servers at the same time.
A separate server that handles blogs and marketing content
So you can keep your blog independent of the main app and update it on its own schedule.
Correct proxy headers for working GeoIP and logging.
So your app servers can see the proper origin of browser requests, despite the proxy. Because asking customers for their country when you already know is a waste of their time.
Support for HTML5 Server Sent Events
For realtime streaming.
An A+ on the SSL Labs test
So the users can connect privately to your site.
Let's get to work
HAProxy will handle HTTPS and pass incoming connections onto your marketing server or app servers based on the URL.
To upgrade, you can enable the new server and disable the server running the old version using a simple command.
Installing HAProxy
For Ubuntu 16.04 we'll use Vincent Bernat's PPA to get HAProxy 1.8:
apt-get install software-properties-common
add-apt-repository ppa:vbernat/haproxy-1.8
apt-get update
apt-get install haproxy socat
Set up our config
Grab the latest haproxy.cfg
from our GitHub. A copy is included below.
global
# Log to systemd's /dev/log compatibility socket
log /dev/log local0 info
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket - see http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.1-stats%20socket
stats socket /var/lib/haproxy/stats mode 600 level admin
stats timeout 2m
# From https://mozilla.github.io/server-side-tls/ssl-config-generator/
# Intermediate config as of Jan 2018 - update it from the link above!
tune.ssl.default-dh-param 2048
ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
ssl-default-bind-options no-sslv3 no-tls-tickets
ssl-default-server-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
ssl-default-server-options no-sslv3 no-tls-tickets
defaults
mode http
# Needed for GeoIP
option forwardfor
# See http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#option%20http-server-close
option http-server-close
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout client 1m
timeout server 1m
timeout check 10s
maxconn 3000
# From http://stackoverflow.com/questions/21419859/configuring-haproxy-to-work-with-server-sent-events
# Set the max time to wait for a connection attempt to a server to succeed
timeout connect 30s
# handle a client suddenly disappearing from the net
timeout client-fin 30s
option http-server-close
# The 'stats' site, where we see what's up and what's down - uncomment and set a password if you want it!
# stats enable
# stats uri /haproxy?stats
# stats realm Strictly\ Private
# stats auth someusername:i-am-an-awful-password-and-you-should-change-me
# Show a custom page during site maintenance (ie, when 'blue' and 'green' are both down)
errorfile 503 /etc/haproxy/errors/503-mycustom.http
frontend public
# HTTP/2 - see https://www.haproxy.com/blog/whats-new-haproxy-1-8/
# h2 is HTTP2 with TLS - see https://http2.github.io/faq/
# Order matters, so h2 before http1.1
bind :443 ssl crt /etc/https/cert-and-private-key-and-intermediate-and-dhparam.pem alpn h2,http/1.1
# Redirect http -> https
bind :80
redirect scheme https code 301 if ! { ssl_fc }
# Redirect www -> example.com
redirect prefix https://example.com code 301 if { hdr(host) -i www.example.com }
# HSTS (15768000 seconds = 6 months)
http-response set-header Strict-Transport-Security max-age=15768000
# Use the marketing site for marketing URLs
acl marketing path_beg -i /help /sitemap.xml /BingSiteAuth.xml /about /blog /videos/blog /images/blog /fonts/blog /css/blog /js/blog
use_backend marketing if marketing
# Everything else goes to the app servers
default_backend app
option httpclose
option forwardfor
backend marketing
balance roundrobin
server static 127.0.0.1:8000 check
backend app
# From http://blog.haproxy.com/2014/01/17/emulating-activepassing-application-clustering-with-haproxy/
# See also https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#stick-table
stick-table type ip size 1m
stick on dst
# See https://cloud.digitalocean.com/droplets for these IPs
server green 10.1.1.1:8000 check
server blue 10.1.1.2:8000 check backup
# http://stackoverflow.com/questions/21419859/configuring-haproxy-to-work-with-server-sent-events
timeout tunnel 10h
After deploying the config:
- Change
example.com
to your own domain - Update the
bind :443 ssl crt
line to point to your HTTPS certificate and key file - Update the IP addresses and ports for
marketing
,blue
andgreen
to your own marketing and app servers. - Modify the URLs in
marketing
- we use/blog
,/images/blog
etc. - to match your marketing URLs - Recommended TLS/SSL configurations change over time, and are also a balance between security and the browsers you want to support: visit Mozilla's SSL Config Generator for up to date
ssl-default
options that match your needs. - Add your own username and password for the 'stats' tool before uncommenting it
Once that's done:
- Restart HAProxy service with
sudo systemctl restart haproxy
- Check the journal/logs with
sudo journalctl -u haproxy
You'll see the public
frontend, and the marketing
and app
backends mentioned inthe config file / diagram above launched:
Jan 05 12:51:11 haproxy-load-balancer systemd[1]: Starting HAProxy Load Balancer...
Jan 05 12:51:11 haproxy-load-balancer haproxy[3333]: Proxy public started.
Jan 05 12:51:11 haproxy-load-balancer haproxy[3333]: Proxy marketing started.
Jan 05 12:51:11 haproxy-load-balancer haproxy[3333]: Proxy app started.
Handle upgrades
We also use a small script called swap-server
to make checking and changing which app server is up easier. We throw it in /usr/local/sbin
. It's optional, but saves you a bunch of typing.
To see the current status
Just run:
sudo swap-server ls
with no arguments. The output will look like:
Proxy name, Service name, Status
app,blue,UP
app,green,UP
app,BACKEND,UP
We can see bopth servers are up.
Switching to Blue or Green
What if we want to take one down to deploy a new version? Just:
sudo swap-server blue
This will:
- Mark
green
as down for maintenance - Mark
blue
as available - Show the status of both servers in the
app
backend
The output will look like:
Proxy name, Service name, Status
app,blue,UP
app,green,MAINT
app,BACKEND,UP
You can see how green
is now in maintenance mode. Requests will stay on blue until it's marked as online again.
To swap to green, do the opposite!
sudo swap-server green
Conclusion
Thanks! If you have suggestions, send a pull request on GitHub. Feedback? Make a comment on Hacker News.