Playbook: suspected web server compromise
Protect availability while preserving web, identity, process, file, network, deployment, and cloud evidence.
Scope
For application, infrastructure, security, and business teams responding to suspicious activity on a public web service.
Start with the situation, not the slogan
A playbook is a decision aid for tired people working with incomplete information. Playbook: suspected web server compromise therefore names owners, evidence, containment and exit criteria instead of pretending every incident follows a flowchart. Read it before the incident, tailor the contacts and systems, and exercise the awkward branches.
The first objective is shared situational awareness: what is known, who owns the incident, what business process is at risk, and which decision is due next. A chat channel full of competent people is not the same as command. Assign a coordinator and a note taker, even when the organisation is small enough for both to use the same kettle.
The visible web server may be a container, proxy, managed service, host, or deployment artefact. Replacing one instance is not containment if the credential, pipeline, image, or application flaw remains.
Composite scenario
How this usually reaches the desk
The playbook should activate when the team can establish its entry conditions, including: Unexpected files, processes, network connections, accounts, configuration, traffic, or security alerts are tied to a host or workload identity. If the report remains ambiguous, open a low-severity record and time-box validation rather than ignoring it or declaring the end of civilisation. Escalate as evidence, privilege, spread or business impact increases.
This scenario combines common operational patterns; it is not presented as a report of one named incident.
What to look for
Begin with preserved, comparable evidence. One signal is rarely proof; use independent observations and a reliable timeline before declaring scope or intent.
Unexpected files, processes, network connections, accounts, configuration, traffic, or security alerts are tied to a host or workload identity.
Confirm this condition from an authoritative source and note who supplied it. Mark assumptions as assumptions; they have an unfortunate habit of becoming facts in copied status updates.
Load balancer, WAF, application, identity, EDR, cloud, deployment, and host logs are retained.
Identify the decision that depends on the information and the deadline for obtaining it. Evidence is most useful when it changes action rather than merely decorating the timeline.
The service architecture and safe isolation or replacement path are known.
Check coverage and retention before containment. If a source is unavailable, record the gap and use independent evidence rather than silently treating absence as innocence.
Working examples
Run it, read it, decide what changes
These examples use documentation addresses, test identities and bounded targets. Replace placeholders only inside systems you own or are explicitly authorised to operate. Read the expected result and next action before running the command; a successful command is evidence, not yet a conclusion.
Capture processes, listeners and sessions
- Prerequisites
- Root or sudo access and a protected output directory.
sudo sh -c 'date -u; ps auxww; ss -plantu; who -a; last -Fai | head -100' > incident-evidence/live-state.txt
sha256sum incident-evidence/live-state.txtOne text file records collection time, processes, sockets and recent sessions, followed by its hash.
The snapshot is volatile and incomplete. Rootkits can lie to local tools, while containers and namespaces can hide additional processes.
Preserve relevant journals and cloud/EDR telemetry, then compare unknown processes with packages, executable hashes and service definitions.
Review service and SSH changes in a time window
- Prerequisites
- A known UTC start time and sudo access.
sudo journalctl --since "2026-08-18 08:00:00 UTC" --until "2026-08-18 10:00:00 UTC" -u ssh --no-pager > incident-evidence/ssh-journal.txt
sudo find /etc/systemd /etc/ssh /root/.ssh /home -xdev -newermt "2026-08-18 08:00 UTC" -ls > incident-evidence/changed-control-files.txtSSH events and recently changed control files are saved.
File modification time can be altered and package updates create legitimate changes. Correlate with package, configuration-management and identity records.
Inspect unexpected keys, units and drop-ins on a copy; do not delete them before recording ownership, content and timestamps.
Record HTTP security headers
- Prerequisites
- An owned HTTPS endpoint.
curl -sS -D headers.txt -o /dev/null https://www.example.com/
grep -Ei '^(strict-transport-security|content-security-policy|x-content-type-options|referrer-policy|permissions-policy):' headers.txtThe live response headers are saved and selected security controls are printed.
Header presence is not proof of a safe value. CSP report-only does not enforce, and HSTS on HTTP is ignored.
Compare values with the application design, add one header at a time in staging, and retest affected user flows.
Response sequence
Read the whole sequence before starting. Several workstreams may run in parallel, but their evidence, authority and expected outcomes still need to be explicit. Every step below points back to a concrete example; use the example as implementation evidence, not as permission to operate outside the stated scope.
Operational judgement
Run the playbook as a checklist with judgement, not as a spell. For Web, Playbook and Server, localise tenant names, log locations, provider contacts, legal thresholds, emergency credentials and business priorities. A generic document becomes operational only when the people on duty can find the required access without consulting the person currently on holiday.
Use a decision log separate from the task list. For each consequential action, note time, owner, evidence, choice, expected effect and observed result. This gives later reviewers something better than a reconstructed story and helps the next shift understand why a seemingly obvious action was delayed or rejected.
Handover
Make the result useful to the next person
Prepare the playbook package before an incident: primary and deputy owners, current contact routes, system inventory, evidence locations, delegated authority, provider escalation, legal and privacy triggers, secure communication, and emergency credentials. Keep a printable or otherwise independent copy where an identity or collaboration outage cannot hide it. Review contact details during exercises; telephone numbers appear to age faster than almost any cryptographic primitive.
For this playbook, the incident record should open with “Declare impact and identify the public route, origin, workloads, images, secrets, data stores, and deployment pipeline.” and drive deliberately towards “Rebuild from trusted source, rotate exposed secrets, patch the root cause, monitor, and restore traffic gradually.” A shift handover must state confirmed facts, open hypotheses, business impact, containment already applied, evidence at risk, approvals pending and the next scheduled update. After closure, turn lessons into owned changes to controls, documentation and exercises. A lesson without an owner and a date is simply a well-written regret.
Validate before you close
Exit when clean capacity handles expected traffic, root cause and persistence are addressed, secrets and pipeline trust are restored, and monitoring validates normal operation.
Capture the test, the expected result and the observed result. Where a person or business owner must accept restored service, name them in the record. A green dashboard can confirm that a component is answering; it cannot confirm that invoices, identities or restored data are trustworthy.
Finish with a compact closure note: the original trigger, confirmed scope, evidence retained, controls changed, tests passed, known gaps, residual risk, and the people responsible for the remaining work. Schedule a review while the timeline is still fresh enough to challenge. The purpose is not to find a person to blame; computers already perform blame with admirable efficiency. The purpose is to make the next response faster, safer and less dependent on one person remembering where the useful log was hidden.
Common mistakes
- Running the playbook without assigning a decision owner and note taker.
- Containing too early or too late because evidence and business impact were not compared.
- Closing the incident without documenting recovery checks and follow-up owners.
These errors usually come from haste, unclear ownership or misplaced confidence. Build the safeguard into the runbook: a required evidence field, a second-person review, a rollback test or a specific exit criterion.
Questions people ask when the clock is running
Who may activate the playbook?
Define that locally before an incident. Help desk or monitoring staff should be able to open a record and escalate; containment authority may sit with incident command, service ownership or an executive depending on impact.
Must every step be completed in order?
No. Evidence, containment, communication and recovery often run in parallel. Keep dependencies explicit and do not let a later checkbox imply that an earlier decision was actually verified.
What is the exit condition?
Exit when clean capacity handles expected traffic, root cause and persistence are addressed, secrets and pipeline trust are restored, and monitoring validates normal operation. Residual risk, communication and improvement actions still need named owners even after service is restored.
Safety boundary
Use these steps only on systems you own or are explicitly authorised to assess. Preserve evidence, follow your organisation’s legal and regulatory obligations, and prefer reversible actions when the situation is not yet understood.
Primary references
- SP 800-61 Rev. 3: Incident Response Recommendations and ConsiderationsNIST
- OWASP Top 10:2025OWASP
- Authorization Cheat SheetOWASP
Editorial status: first edition. Review the linked vendor documentation for product- and version-specific changes before acting.