Why Incident Response Matters
Security incidents happen. How you respond determines whether an incident becomes a minor disruption or a catastrophic breach. A well-prepared incident response plan reduces damage, speeds recovery, and helps you meet regulatory requirements.
Effective incident response helps you:
- Minimize damage by responding quickly and effectively
- Preserve evidence for investigation and potential legal action
- Meet compliance obligations for breach notification
- Restore operations with minimal downtime
- Improve defenses based on lessons learned
The time to plan for incidents is before they happen.
Detecting Security Incidents
Signs of a Security Incident
Monitor for these indicators:
Expedited WAF Alerts:
- Spike in blocked requests
- New attack patterns detected
- Multiple failed login attempts from single IP
- Unusual geographic traffic patterns
Application Metrics:
- Unexpected error rate increases
- Unusual database query patterns
- Abnormal response times
- Memory or CPU spikes without traffic increase
External Reports:
- Customer reports of suspicious activity
- Third-party security researchers
- Vendor security notifications
- Threat intelligence feeds
Using Expedited WAF for Detection
Expedited WAF provides visibility into attack attempts:
Dashboard → Security Events
- Blocked attack attempts by type
- Source IPs of malicious traffic
- Targeted endpoints
- Attack timelines
Review logs regularly and set up alerts for unusual patterns.
Incident Classification
Severity Levels
Critical (P1)
- Active data breach in progress
- Ransomware or destructive malware
- Complete system compromise
- Customer data confirmed exposed
High (P2)
- Suspected unauthorized access
- Vulnerability being actively exploited
- Credential compromise suspected
- Payment system anomalies
Medium (P3)
- Brute force attack in progress
- DDoS affecting availability
- Suspicious but unconfirmed activity
- Third-party dependency compromised
Low (P4)
- Reconnaissance/scanning detected
- Phishing attempt reported
- Failed attack attempts
- Vulnerability disclosed but not exploited
Immediate Response Actions
Step 1: Confirm the Incident
Before escalating, verify the incident is real:
- Check WAF logs for attack details
- Review application logs for anomalies
- Verify with multiple data sources
- Rule out false positives (deployments, expected traffic)
Step 2: Contain the Threat
Take immediate action to limit damage:
Block Attacking IPs:
# Add malicious IP to block list in Expedited WAF
# Dashboard → Block/Allow IPs → Add IP
Revoke Compromised Credentials:
# Rotate API keys immediately
heroku config:set API_KEY=new_key -a your-app
# Force password resets if user credentials compromised
Isolate Affected Systems:
- Scale down compromised dynos
- Disable affected features
- Block suspicious network paths
Step 3: Preserve Evidence
Before making changes, preserve evidence:
# Export Heroku logs
heroku logs --num 10000 -a your-app > incident-logs.txt
# Export database state (if relevant)
heroku pg:backups:capture -a your-app
# Screenshot WAF dashboards
# Document timeline of events
Step 4: Assemble Response Team
Notify appropriate personnel:
- Security lead
- Engineering lead
- Legal/compliance (if data breach)
- Communications/PR (if public-facing)
- Executive sponsor (for major incidents)
Investigation
Gathering Information
Expedited WAF Logs:
- What attacks were blocked?
- What might have gotten through?
- Timeline of attack attempts
- Source IP addresses
Application Logs:
heroku logs --tail -a your-app
Database Audit Logs:
- Unusual queries?
- Data export attempts?
- New admin accounts?
Heroku Release History:
heroku releases -a your-app
Questions to Answer
- What happened? (Attack type, method)
- When did it happen? (Timeline)
- Who is affected? (Users, data)
- How did it happen? (Vulnerability, vector)
- What was accessed/exfiltrated? (Impact)
- Is the threat ongoing? (Containment status)
Remediation
Eradicate the Threat
Once contained, fully remove the threat:
- Patch the exploited vulnerability
- Remove any backdoors or persistence mechanisms
- Clean compromised systems
- Update vulnerable dependencies
Verify Remediation
Confirm the threat is eliminated:
- Re-scan for vulnerabilities
- Review system integrity
- Monitor for recurrence
- Test security controls
Restore Operations
Return to normal operations carefully:
- Restore from known-good backups if needed
- Re-enable features incrementally
- Monitor closely for anomalies
- Verify customer-facing functionality
Communication
Internal Communication
Keep stakeholders informed:
Initial Alert:
Subject: [SECURITY INCIDENT] Brief description
Severity: P1/P2/P3/P4
Status: Investigating/Contained/Resolved
Impact: Description of affected systems/users
Actions: Current response actions
Next Update: Time of next status update
Status Updates:
- Provide regular updates (hourly for P1/P2)
- Include timeline, findings, next steps
- Document decisions and rationale
External Communication
If customer notification is required:
When to Notify:
- Personal data was exposed
- Financial data was accessed
- Regulatory requirements mandate disclosure
- Customers may be at risk
Notification Timeline:
- GDPR: Within 72 hours of discovery
- PCI DSS: As soon as possible
- HIPAA: Within 60 days
- State laws: Varies (check applicable laws)
Notification Content:
- What happened (brief, factual)
- What data was affected
- What you’re doing about it
- What customers should do
- Contact information for questions
Post-Incident Activities
Post-Mortem Review
After the incident is resolved:
- Timeline: Document complete incident timeline
- Root Cause: Identify what allowed the incident
- Response Review: What worked, what didn’t
- Improvements: Action items to prevent recurrence
- Share Learnings: Communicate to broader team
Post-Mortem Template
## Incident Summary
[Brief description]
## Timeline
- HH:MM - Event
- HH:MM - Event
## Root Cause
[What vulnerability/failure enabled the incident]
## Impact
- Users affected: X
- Data exposed: Y
- Downtime: Z hours
## Response Assessment
### What went well
-
### What could be improved
-
## Action Items
- [ ] Action (Owner, Due Date)
- [ ] Action (Owner, Due Date)
Update Defenses
Implement lessons learned:
- Add new WAF rules
- Improve monitoring/alerting
- Update incident response procedures
- Conduct additional training
Compliance Reporting
Regulatory Requirements
Different frameworks have specific reporting requirements:
| Framework | Notification Deadline | Authority |
|---|---|---|
| GDPR | 72 hours | Supervisory Authority |
| HIPAA | 60 days | HHS |
| PCI DSS | ASAP | Card brands |
| SOC 2 | Document in audit | Auditor |
Documentation to Maintain
- Incident timeline
- Evidence preserved
- Actions taken
- Notifications sent
- Remediation steps
- Post-mortem findings
Preparation Checklist
Before an incident occurs:
- Incident response plan documented
- Response team identified with contact info
- Communication templates ready
- Backup and recovery procedures tested
- WAF monitoring and alerting configured
- Legal/compliance contacts identified
- Regular incident response drills conducted