Playbook: leaked password, token, or API secret
Replace and revoke the secret, but also establish exposure time, permissions, use, copies, dependencies, and newly created persistence.
Scope
For development, cloud, SaaS, identity, and incident-response teams.
Start with the situation, not the slogan
A playbook is a decision aid for tired people working with incomplete information. Playbook: leaked password, token, or API secret therefore names owners, evidence, containment and exit criteria instead of pretending every incident follows a flowchart. Read it before the incident, tailor the contacts and systems, and exercise the awkward branches.
The first objective is shared situational awareness: what is known, who owns the incident, what business process is at risk, and which decision is due next. A chat channel full of competent people is not the same as command. Assign a coordinator and a note taker, even when the organisation is small enough for both to use the same kettle.
Removing a secret from a file or commit does not revoke it. Rotation can also cause outages if dependencies are unknown, so replacement and revocation need a coordinated sequence.
Composite scenario
How this usually reaches the desk
The playbook should activate when the team can establish its entry conditions, including: The secret type, identity, permissions, owner, environments, and earliest exposure are identified. If the report remains ambiguous, open a low-severity record and time-box validation rather than ignoring it or declaring the end of civilisation. Escalate as evidence, privilege, spread or business impact increases.
This scenario combines common operational patterns; it is not presented as a report of one named incident.
What to look for
Begin with preserved, comparable evidence. One signal is rarely proof; use independent observations and a reliable timeline before declaring scope or intent.
The secret type, identity, permissions, owner, environments, and earliest exposure are identified.
Confirm this condition from an authoritative source and note who supplied it. Mark assumptions as assumptions; they have an unfortunate habit of becoming facts in copied status updates.
Provider logs can attribute use to credential, token ID, application, or identity.
Identify the decision that depends on the information and the deadline for obtaining it. Evidence is most useful when it changes action rather than merely decorating the timeline.
Dependent services and a tested replacement path are known.
Check coverage and retention before containment. If a source is unavailable, record the gap and use independent evidence rather than silently treating absence as innocence.
Working examples
Run it, read it, decide what changes
These examples use documentation addresses, test identities and bounded targets. Replace placeholders only inside systems you own or are explicitly authorised to operate. Read the expected result and next action before running the command; a successful command is evidence, not yet a conclusion.
Locate a secret in Git history without printing every file
- Prerequisites
- A unique, non-secret identifier or a revoked token prefix; never paste a live secret into shared shell history.
git log --all --oneline -S 'REVOKED_TOKEN_PREFIX'
git grep -n 'REVOKED_TOKEN_PREFIX' $(git rev-list --all) -- ':!vendor' ':!node_modules'Commits and matching paths containing the identifier are listed.
Removing the current file does not remove earlier commits, forks, CI logs, artefacts or cloned copies.
Revoke first, inspect provider audit logs for use, replace the secret everywhere, then rewrite history only with repository-owner coordination.
Inventory environment-variable names without exposing values
- Prerequisites
- Run only on a system you are authorised to inspect.
env | cut -d= -f1 | sort | grep -Ei 'TOKEN|SECRET|KEY|PASSWORD|CREDENTIAL'Potential secret-bearing variable names appear without their values.
Names are hints; secrets may also live in files, keychains, CI variables or process arguments. Avoid commands that print values into terminals or tickets.
Map each credential to an owner, provider, permissions and rotation procedure; revoke suspected exposures before cleanup.
Prove the old credential no longer works
- Prerequisites
- A harmless read-only endpoint and the revoked credential held only in a private local variable.
OLD_TOKEN='replace-locally-never-commit'
curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $OLD_TOKEN" https://api.example.invalid/v1/me
unset OLD_TOKENA revoked bearer token should receive an authentication failure such as HTTP 401; the documentation host here must be replaced with the real provider endpoint.
HTTP 403 can mean the token is valid but lacks permission; a network error proves nothing. Use the provider’s documented harmless identity endpoint.
Record the revocation evidence, test the replacement through the application, and search audit logs from exposure time onward.
Response sequence
Read the whole sequence before starting. Several workstreams may run in parallel, but their evidence, authority and expected outcomes still need to be explicit. Every step below points back to a concrete example; use the example as implementation evidence, not as permission to operate outside the stated scope.
Operational judgement
Run the playbook as a checklist with judgement, not as a spell. For Secrets, Playbook and Cloud, localise tenant names, log locations, provider contacts, legal thresholds, emergency credentials and business priorities. A generic document becomes operational only when the people on duty can find the required access without consulting the person currently on holiday.
Use a decision log separate from the task list. For each consequential action, note time, owner, evidence, choice, expected effect and observed result. This gives later reviewers something better than a reconstructed story and helps the next shift understand why a seemingly obvious action was delayed or rejected.
Handover
Make the result useful to the next person
Prepare the playbook package before an incident: primary and deputy owners, current contact routes, system inventory, evidence locations, delegated authority, provider escalation, legal and privacy triggers, secure communication, and emergency credentials. Keep a printable or otherwise independent copy where an identity or collaboration outage cannot hide it. Review contact details during exercises; telephone numbers appear to age faster than almost any cryptographic primitive.
For this playbook, the incident record should open with “Declare scope and preserve the exposure location, alert, history, and provider audit logs.” and drive deliberately towards “Reduce privilege and lifetime, remove copies from builds and logs, and add secret detection and rotation ownership.” A shift handover must state confirmed facts, open hypotheses, business impact, containment already applied, evidence at risk, approvals pending and the next scheduled update. After closure, turn lessons into owned changes to controls, documentation and exercises. A lesson without an owner and a date is simply a well-written regret.
Validate before you close
Exit when the old secret fails, dependencies use the new one, unauthorised actions and persistence are resolved, and monitoring covers re-exposure.
Capture the test, the expected result and the observed result. Where a person or business owner must accept restored service, name them in the record. A green dashboard can confirm that a component is answering; it cannot confirm that invoices, identities or restored data are trustworthy.
Finish with a compact closure note: the original trigger, confirmed scope, evidence retained, controls changed, tests passed, known gaps, residual risk, and the people responsible for the remaining work. Schedule a review while the timeline is still fresh enough to challenge. The purpose is not to find a person to blame; computers already perform blame with admirable efficiency. The purpose is to make the next response faster, safer and less dependent on one person remembering where the useful log was hidden.
Common mistakes
- Running the playbook without assigning a decision owner and note taker.
- Containing too early or too late because evidence and business impact were not compared.
- Closing the incident without documenting recovery checks and follow-up owners.
These errors usually come from haste, unclear ownership or misplaced confidence. Build the safeguard into the runbook: a required evidence field, a second-person review, a rollback test or a specific exit criterion.
Questions people ask when the clock is running
Who may activate the playbook?
Define that locally before an incident. Help desk or monitoring staff should be able to open a record and escalate; containment authority may sit with incident command, service ownership or an executive depending on impact.
Must every step be completed in order?
No. Evidence, containment, communication and recovery often run in parallel. Keep dependencies explicit and do not let a later checkbox imply that an earlier decision was actually verified.
What is the exit condition?
Exit when the old secret fails, dependencies use the new one, unauthorised actions and persistence are resolved, and monitoring covers re-exposure. Residual risk, communication and improvement actions still need named owners even after service is restored.
Safety boundary
Use these steps only on systems you own or are explicitly authorised to assess. Preserve evidence, follow your organisation’s legal and regulatory obligations, and prefer reversible actions when the situation is not yet understood.
Primary references
- SP 800-61 Rev. 3: Incident Response Recommendations and ConsiderationsNIST
- Software Supply Chain Security Cheat SheetOWASP
Editorial status: first edition. Review the linked vendor documentation for product- and version-specific changes before acting.