HAProxy in 2018

January 29, 2020

A load balancer with HTTP/2 and dynamic reconfig.

an image

By Mike on 8th Jan 2018

HAProxy is a load balancer and SSL/TLS terminator. Since you want to keep your HTTPS certificate and key in one place, want to send requests to your multiple app servers evenly, and want to be able to take down individual servers for deploys, a load balancer can sit there monitoring the backend app servers and sending traffic only to the servers you want.

HAProxy 1.8 came out just before the end of 2017. It's the first release of HAProxy to include full HTTP/2 support - previous versions only allowed you to pass through HTTP/2 to a seperate server to terminate there. 1.8 finally allows the typical 'terminate on the load balancer' scenario with HTTP/2.

Most HAProxy guides on the internet are outdated: they require you to manually modify the config or stop services on your app servers during deploys, but you don't need to do that anymore. Two years ago HAProxy 1.6 added dynamic configuration, and we're using that to keep out app app during our deploys below.

CertSimple verifies companies for EV HTTPS certs - HAProxy is the second most popular load balancer among our customers, so we thought we'd publish our own guide. This shows you how to use HAProxy 1.8 to provide HTTP2 to browsers, keep your HTTPS certificates in one place (the load balancer), and control everything live on the load balancer.

HAProxy vs nginx

If you need a load balancer - and you probably do - your main choices are HAProxy and it's main competitor nginx. Here's a quick, opinionated guide to choosing between them:

HAProxy advantages over nginx

HAProxy is simpler than nginx:

nginx advantages over HAProxy

Here's what we prefer about nginx:

nginx is excellent - we use it to serve this page. But HAProxy 1.8 combines HTTP/2 with 'batteries included' simplicity.

What you’ll get

This guide is for Ubuntu 16.04 LTS. We’re going to set up this:

an image

We’ll also have the following features:

Logging to systemd

So you can use the standard journalctl logging and filter your logs using handy stuff like 4 hours ago, yesterday, etc.

The various www vs non-www, HTTP vs HTTPS combinations redirected to a single HTTPS site.

This ensures there’s only one, secure copy copy of every resource for both clarity and SEO purposes.

The ability to switch backends dynamically

So you can upgrade your app without taking it offline. HAProxy has a runtime API which can be used to manually mark a server as disabled, so we can control which backends get used without having to modify our config or actually shut down the services on the backends.

HTTP/2 support in all browsers

For speed! One of the pages on our blog loads in 1.9s on HTTP 1.1. The same page loads in 600ms over HTTP/2.

A branded ‘sorry’ page

Just in case you break both the app servers at the same time.

A separate server that handles blogs and marketing content

So you can keep your blog independent of the main app and update it on its own schedule.

Correct proxy headers for working GeoIP and logging.

So your app servers can see the proper origin of browser requests, despite the proxy. Because asking customers for their country when you already know is a waste of their time.

Support for HTML5 Server Sent Events

For realtime streaming.

An A+ on the SSL Labs test

So the users can connect privately to your site.

Let’s get to work

HAProxy will handle HTTPS and pass incoming connections onto your marketing server or app servers based on the URL.

To upgrade, you can enable the new server and disable the server running the old version using a simple command.

Installing HAProxy

For Ubuntu 16.04 we’ll use Vincent Bernat’s PPA to get HAProxy 1.8:

apt-get install software-properties-common
add-apt-repository ppa:vbernat/haproxy-1.8
apt-get update
apt-get install haproxy socat

Set up our config

Grab the latest haproxy.cfg from our GitHub. A copy is included below.

global
    # Log to systemd’s /dev/log compatibility socket
    log /dev/log local0 info

chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon

<span class="hljs-comment"># turn on stats unix socket - see http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#3.1-stats%20socket</span>
stats socket /var/lib/haproxy/stats mode 600 level admin 
stats timeout 2m

<span class="hljs-comment"># From https://mozilla.github.io/server-side-tls/ssl-config-generator/ </span>
<span class="hljs-comment"># Intermediate config as of Jan 2018 - update it from the link above!</span>
tune.ssl.default-dh-param 2048
ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
ssl-default-bind-options no-sslv3 no-tls-tickets
ssl-default-server-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
ssl-default-server-options no-sslv3 no-tls-tickets

defaults mode http

# Needed for GeoIP option forwardfor

# See http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#option%20http-server-close option http-server-close

<span class="hljs-built_in">log</span> global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout client 1m
timeout server 1m

timeout check 10s
maxconn 3000

<span class="hljs-comment"># From http://stackoverflow.com/questions/21419859/configuring-haproxy-to-work-with-server-sent-events</span>
<span class="hljs-comment"># Set the max time to wait for a connection attempt to a server to succeed</span>
timeout connect 30s
<span class="hljs-comment"># handle a client suddenly disappearing from the net</span>
timeout client-fin 30s
option http-server-close

<span class="hljs-comment"># The 'stats' site, where we see what's up and what's down - uncomment and set a password if you want it!</span>
<span class="hljs-comment"># stats enable</span>
<span class="hljs-comment"># stats uri /haproxy?stats</span>
<span class="hljs-comment"># stats realm Strictly\ Private</span>
<span class="hljs-comment"># stats auth someusername:i-am-an-awful-password-and-you-should-change-me</span>

<span class="hljs-comment"># Show a custom page during site maintenance (ie, when 'blue' and 'green' are both down)</span>
errorfile 503 /etc/haproxy/errors/503-mycustom.http

frontend public # HTTP/2 - see https://www.haproxy.com/blog/whats-new-haproxy-1-8/ # h2 is HTTP2 with TLS - see https://http2.github.io/faq/ # Order matters, so h2 before http1.1 bind :443 ssl crt /etc/https/cert-and-private-key-and-intermediate-and-dhparam.pem alpn h2,http/1.1

<span class="hljs-comment"># Redirect http -&gt; https</span>
<span class="hljs-built_in">bind</span> :80
redirect scheme https code 301 <span class="hljs-keyword">if</span> ! { ssl_fc }

<span class="hljs-comment"># Redirect www -&gt; example.com</span>
redirect prefix https://example.com code 301 <span class="hljs-keyword">if</span> { hdr(host) -i www.example.com }

<span class="hljs-comment"># HSTS (15768000 seconds = 6 months)</span>
http-response <span class="hljs-built_in">set</span>-header Strict-Transport-Security max-age=15768000

<span class="hljs-comment"># Use the marketing site for marketing URLs</span>
acl marketing path_beg -i /<span class="hljs-built_in">help</span> /sitemap.xml /BingSiteAuth.xml /about /blog /videos/blog /images/blog /fonts/blog /css/blog /js/blog
use_backend marketing <span class="hljs-keyword">if</span> marketing

<span class="hljs-comment"># Everything else goes to the app servers</span>
default_backend app
option httpclose
option forwardfor

backend marketing balance roundrobin server static 127.0.0.1:8000 check

backend app # From http://blog.haproxy.com/2014/01/17/emulating-activepassing-application-clustering-with-haproxy/ # See also https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#stick-table stick-table type ip size 1m stick on dst # See https://cloud.digitalocean.com/droplets for these IPs server green 10.1.1.1:8000 check server blue 10.1.1.2:8000 check backup # http://stackoverflow.com/questions/21419859/configuring-haproxy-to-work-with-server-sent-events timeout tunnel 10h

After deploying the config:

  • Change example.com to your own domain
  • Update the bind :443 ssl crt line to point to your HTTPS certificate and key file
  • Update the IP addresses and ports for marketing, blue and green to your own marketing and app servers.
  • Modify the URLs in marketing - we use /blog, /images/blog etc. - to match your marketing URLs
  • Recommended TLS/SSL configurations change over time, and are also a balance between security and the browsers you want to support: visit Mozilla’s SSL Config Generator for up to date ssl-default options that match your needs.
  • Add your own username and password for the ‘stats’ tool before uncommenting it

Once that’s done:

  • Restart HAProxy service with
sudo systemctl restart haproxy
  • Check the journal/logs with
sudo journalctl -u haproxy

You’ll see the public frontend, and the marketing and app backends mentioned inthe config file / diagram above launched:

Jan 05 12:51:11 haproxy-load-balancer systemd[1]: Starting HAProxy Load Balancer…
Jan 05 12:51:11 haproxy-load-balancer haproxy[3333]: Proxy public started.
Jan 05 12:51:11 haproxy-load-balancer haproxy[3333]: Proxy marketing started.
Jan 05 12:51:11 haproxy-load-balancer haproxy[3333]: Proxy app started.

Handle upgrades

We also use a small script called swap-server to make checking and changing which app server is up easier. We throw it in /usr/local/sbin. It’s optional, but saves you a bunch of typing.

To see the current status

Just run:

sudo swap-server ls

with no arguments. The output will look like:

Proxy name, Service name, Status
app,blue,UP
app,green,UP
app,BACKEND,UP

We can see bopth servers are up.

Switching to Blue or Green

What if we want to take one down to deploy a new version? Just:

sudo swap-server blue

This will:

  • Mark green as down for maintenance
  • Mark blue as available
  • Show the status of both servers in the app backend

The output will look like:

Proxy name, Service name, Status
app,blue,UP
app,green,MAINT
app,BACKEND,UP

You can see how green is now in maintenance mode. Requests will stay on blue until it’s marked as online again.

To swap to green, do the opposite!

sudo swap-server green

Conclusion

Thanks! If you have suggestions, send a pull request on GitHub. Feedback? Make a comment on Hacker News.

Try Expedited WAF.
Get a Free Tee.

Option 1: Install Expedited WAF (the Web Application Firewall service that shields your Heroku applications from attacks) from the Heroku Elements Marketplace..

Seven days later we'll ask for some feedback and your (US or Canada only) shipping details.

Option 2: Select a Date & Time below to talk to us about your existing web application security framework and see how Expedited WAF can help better secure your Heroku applications.