Postmortem: EU-Hosted Heroku Apps SNI Negotiation Failure

What happened

A small number of sites (sub 1%) behind the Expedited WAF began returning 502 Hosting Server Connect Timeout errors.

Root cause

SNI (Server Name Indication) is a field in the TLS handshake where the connecting client states which hostname it wants, before any encryption begins. It exists because a single server IP commonly hosts many sites, each with its own certificate. The routing layer on the receiving end, in this case Heroku’s, reads the SNI value to select the right certificate and route the connection to the right backend application. Without a matching SNI, a strict endpoint refuses the handshake entirely.

SNI across the request chain

SNI is required through the entirety of the request chain, not just at the first hop. A proxied request involves two separate TLS connections: one from the visitor’s browser to the WAF Edge Network, and one from the WAF Edge Network to the origin. Each connection has its own handshake and its own SNI value. Because we keep every request encrypted at every hop, SNI has to function correctly through the whole chain for a request to reach the application.

The WAF Edge Network connected to each customer’s app using the appname.herokuapp.com hostname as the origin address, while presenting the customer’s custom domain as the SNI value. This worked because Heroku’s routing accepted any SNI on any endpoint and routed by it.

On August 25, that stopped being true for a small subset of the sites we protect, all running in Heroku’s EU availability zone. Routing for those apps’ herokuapp.com hostnames moved onto anycast endpoints that enforce SNI strictly: an endpoint only completes handshakes for hostnames it serves. The custom-domain SNI was refused, so every proxied request failed with a 502. Apps moved individually, so sites failed one at a time across the morning.

As of this writing, it is unclear whether this was a Heroku change or an underlying AWS change. No status page reflected it and it was not announced. We are working with Heroku/Salesforce engineering on further diagnosis.

Immediate remediation

Each affected site’s origin was changed to the custom domain’s DNS target (the random-words-1234.herokudns.com value in the Heroku dashboard under Settings, Domains). Those endpoints serve the custom domain’s SNI by design. Recovery per site was under two minutes, verified from 14 regions.

Publicly resolvable DNS targets (example.com) are tied to the domain rather than the app name, so this configuration also survives app renames.

Future prevention

1. Provisioning and reconfiguration now resolve origins to the domain’s herokudns DNS target automatically.
2. All active sites were swept the same day; affected sites without tickets were fixed and their owners contacted.
3. Remaining sites on legacy herokuapp.com origins are being migrated ahead of Heroku’s rollout.
4. Docs updated with origin configuration and app-rename guidance.
5. New monitoring is in place to detect this class of errors.

We apologize for the downtime.

Questions about your setup? Contact us or open a ticket from your dashboard.