How to Block User Agents on Heroku

Why User Agent Blocking Matters

User agents are the self-reported names that HTTP clients send to identify themselves. While user agents can be easily spoofed, many bots, scrapers, and malicious scripts still report their actual identity—often revealing the command-line tools or HTTP libraries they’re using.

User agent blocking is valuable for several reasons:

Blocks known bad bots that identify themselves honestly
Stops vulnerability scanners like Nikto, Nessus, and SQLMap
Prevents scraping tools like wget, curl, and Scrapy
Reduces server load from unwanted automated traffic
Protects content from automated harvesting

While sophisticated attackers will spoof their user agents, blocking known malicious user agents still eliminates a significant portion of automated attacks.

Common Malicious User Agents

Many attack tools and scripts report recognizable user agent strings:

Vulnerability Scanners

Nikto - Web server vulnerability scanner
Nessus - Security vulnerability scanner
sqlmap - SQL injection detection tool
WPScan - WordPress vulnerability scanner

HTTP Libraries (often used in scripts)

python-requests - Python HTTP library
curl - Command-line transfer tool
wget - Network downloader
HTTPie - Command-line HTTP client

Scraping Tools

Scrapy - Python scraping framework
HTTrack - Website copier
MJ12bot - Aggressive crawler

Example Malicious User Agents

PycURL/7.43.0.2 libcurl/7.47.0
sqlmap/1.4.7#stable (http://sqlmap.org)
Nikto/2.1.6

Prerequisites

What you need to get started:

Expedited WAF add-on installed on your Heroku application

How To Block User Agents on Heroku

Step 1: Open the Bot Blocking Dashboard

Navigate to your Expedited WAF dashboard and select the Block Bots page from the sidebar menu.

Step 2: Add User Agents to Block

Enter each user agent string you want to block. You can enter:

Exact matches: sqlmap/1.4.7
Partial matches: sqlmap (blocks all versions)
Pattern matches: Support for wildcards and regex

Step 3: Verify Blocking

Test your rules by sending a request with a blocked user agent:

curl -A "sqlmap/1.4.7" https://yourdomain.com

Blocked requests receive a 403 Forbidden response.

Built-in Bot Detection

By default, Expedited WAF blocks user agents that don’t match real browsers:

Missing user agents: Requests with no User-Agent header
Malformed user agents: Obviously fake or corrupted agent strings
Known bad bots: Common attack tools and aggressive crawlers

This automatic protection catches many attacks without any configuration.

Common Use Cases

Blocking Security Scanners

Attackers often probe sites with tools like Nikto and Nessus. Block these user agents to reduce reconnaissance attempts.

Preventing Content Scraping

Competitors or content thieves often use wget, curl, or HTTrack to copy your site. Block these to protect your content.

Reducing Bot Traffic

Aggressive or poorly-behaved bots can consume significant resources. Block them to reduce server load.

Stopping API Abuse

Scripts using default HTTP library user agents (python-requests, axios) may be abusing your API. Consider blocking or rate-limiting them.

Allowing Good Bots

Not all bots are bad. You may want to allow:

Google/Bing crawlers for SEO
Social media previews (Facebook, Twitter, LinkedIn)
Uptime monitors checking your site health
Payment processor webhooks

Configure allowlists for these legitimate automated users.

Troubleshooting

Issue: Legitimate users being blocked

Check if their browser sends an unusual user agent
Some privacy browsers modify user agents
Review the blocked request logs for details

Issue: Bots still getting through

The bot may be spoofing a legitimate browser user agent
Layer user agent blocking with other rules (IP, rate limiting)
Consider CAPTCHA challenges for suspicious patterns

Issue: Search engines being blocked

Verify Googlebot and Bingbot are on your allowlist
Test with official search engine validation tools
Check that rules aren’t too broad

User Agent Blocking vs IP Blocking

Feature	User Agent Blocking	IP Blocking
Bypassed by	Spoofing user agent	Using new IP/proxy
Best for	Known tools, libraries	Specific attackers
Maintenance	Update for new tools	Update IP lists
False positives	Low (tools don’t spoof)	Higher (shared IPs)

For best protection, combine both approaches.

Resources

Learn more about User Agents:

Get Started

Install Expedited WAF Book a Security Review