How AI Has Changed Cybersecurity: When Defenders and Attackers Both Automate the Fight

Executive summary

Artificial intelligence has changed cybersecurity primarily by reducing the cost of iteration. It can turn a slow, specialised task into a faster loop: draft a convincing message, translate it, tailor it to a job role, triage a suspicious file, correlate alerts, or explain a vulnerability to an operator. That does not make every attack “AI-powered”, nor does it make every defensive product effective. It changes the economics around familiar attack paths: identity abuse, social engineering, exposed services, weak software supply chains and slow response.

The most useful working model is not “AI replaces the security team”. It is AI assists AI, with accountable humans controlling consequence. Attackers will automate collection, variation and testing. Defenders will automate filtering, correlation and response preparation. The decisive controls remain boring and valuable: phishing-resistant authentication, least privilege, segmented access, patching, backups, logging, tested response paths and independent approval for money, identity recovery and privileged changes.

For organisations in the EU, this is also a governance question. A model connected to documents, mailboxes or operational tools is an information system with a new interface and a service identity. Its data sources, permissions, suppliers, monitoring and emergency stop must be managed accordingly. The EU AI Act and NIS2 do not make a chatbot secure; they make it harder to pretend that ownership, risk management and evidence do not matter.

This article distinguishes observed activity from prediction. Vendor telemetry and incident-response data are useful but incomplete: they reflect the customers and services the reporter can see. A demonstration proves capability, not frequency. Treat every number below as evidence with a scope, rather than as a universal rate for the whole internet.

What has actually changed

Before generative AI, many security tasks were limited by time and language: an attacker had to write a lure, translate it, adapt it to a target and answer a victim; an analyst had to read several alerts, locate an approved runbook and write an explanation. Models shorten those loops. They are particularly effective at text transformation, summarisation, classification and code explanation. They are less reliable at deciding whether a particular alert is malicious, whether a remediation is safe in a specific estate, or whether a business should accept a risk.

The practical result is a contest of throughput. A criminal does not need an AI system to compromise an organisation, but can use one to produce more variants and reduce obvious errors. A defender does not need an agent to close an incident, but can use one to remove repetitive reading and make an expert’s attention available for evidence and decisions. The risk starts when speed is mistaken for authority.

ENISA’s Threat Landscape 2025 reviewed 4,875 incidents from 1 July 2024 to 30 June 2025. It identifies abuse of AI among the trends that increase scale, including AI-enabled disinformation and deepfakes. The report does not establish that most intrusions are autonomous. That distinction matters: security investment should improve the paths attackers already use, not chase a cinematic picture of a fully independent machine attacker.

Evidence: where AI use is visible, and where it is not

Public reporting now gives stronger evidence than generic “AI will change everything” claims, but it still has limits.

Google Threat Intelligence Group reported in November 2025 that it observed adversaries moving beyond simple productivity use into novel AI-enabled malware and use across more of the attack lifecycle. Its AI Threat Tracker is based on threat tracking, incident-response work and platform defences; it is valuable frontline telemetry, not a census of all criminal activity. Google’s January 2025 analysis similarly found state-backed and information-operation actors trying to misuse Gemini, while warning that theoretical demonstrations should not be confused with observed use in the wild. See Adversarial Misuse of Generative AI.

OpenAI’s public disruption reports provide case studies rather than prevalence statistics. In the June 2025 report and the ScopeCreep case study, the company described accounts used to assist with malware development, debugging and command-and-control setup, then removed them. This is evidence that models can help actors perform incremental operational work; it is not evidence that a model independently planned or executed the intrusion.

The wider threat picture is still dominated by conventional failures. M-Trends 2025, based on Mandiant investigations during 2024, reports exploits as the most common initial infection vector at 33%, with stolen credentials second at 16%. Its reported global median dwell time was 11 days. These numbers should push a practical conclusion: reduce exposed attack surface and identity compromise first. An AI roadmap that ignores identity, patching and logging is a distraction.

Social engineering and synthetic media: process verification, not detection theatre

Generative systems make ordinary fraud more scalable. They can improve grammar, translate quickly, create role-specific wording and maintain a plausible conversation. Synthetic voice and video add pressure where a victim believes they recognise an executive, supplier or colleague. This undermines weak controls based on “spot bad spelling” or “recognise the voice”.

The control response is process verification rather than AI-detection theatre:

• For a payment, changed bank account or urgent transfer, use a callback to a number already held in the supplier master record. Do not use a number supplied in the request.
• Require dual approval for high-value changes, including changes initiated by senior staff.
• Enforce phishing-resistant authentication for privileged and finance users. A convincing lure is less useful when the attacker cannot satisfy a hardware-backed or passkey-based sign-in challenge.
• Treat an unexpected request to reset identity, add a device or create forwarding rules as a high-risk event. Verify it through a separate channel.
• Preserve the original message and headers for investigation; do not forward a potentially malicious attachment or link to more recipients.

These measures work regardless of whether a message was written by a human, a template or a model. They also have a measurable outcome: the number of payment-change requests that followed independent verification, the number of risky inbox rules caught, and the time from report to containment.

Synthetic media: handle it as a business-process risk

A realistic voice sample is not proof of identity. A realistic video is not proof of authorisation. For a small organisation, the cheapest defence is a rehearsed protocol: no urgent financial or access decision is approved solely by a call, video, chat message or apparent executive instruction.

Run a tabletop exercise with a harmless synthetic voice or video example. Give the finance and IT teams a request that asks for a supplier-bank-account change or an emergency MFA reset. The expected result is not “someone detects AI”; it is that the employee pauses, uses the known callback route, obtains second approval and records the outcome. If the process relies on one person recognising an accent, it will fail under pressure.

The EU AI Act’s transparency framework is relevant to some synthetic-content situations, but labelling cannot replace verification. The European Commission’s AI Act timeline states that certain transparency requirements, including labelling of deepfakes, apply progressively. Organisations should check their exact role and national implementation advice rather than assuming a label settles authenticity or legal responsibility.

Reconnaissance and vulnerability discovery

AI can condense public information and help an operator organise a target profile. It can also assist developers and researchers in understanding code, spotting suspicious patterns or proposing test cases. The same capability creates a timing problem: if vulnerability discovery and exploit development become faster, defenders need shorter exposure windows.

That is why asset inventory, external exposure management and patch ownership are more important than a generic promise to “use AI for security”. Know which internet-facing systems exist, which owner is responsible, which version is running, and whether a fix or compensating control has been verified. Do not use a model’s explanation of a CVE as proof that a system is affected. Check the vendor advisory, the installed version, configuration and reachable exposure.

In April 2026, Google described a transition in which model-assisted vulnerability discovery can make the period between discovery and defensive hardening more consequential. Its enterprise vulnerability defence guidance argues for disciplined exposure reduction and rapid hardening. This is a vendor perspective, not an independent benchmark, but the operational implication is sound: remediate known exposure before investing in a clever triage agent.

Defenders: useful AI work is narrow, observable and reversible

The best early use cases are not autonomous containment. They are tasks where evidence is available, the output can be sampled, and a mistake does not directly change production.

Good candidates include:

• summarising a closed, sanitised incident into a handover note with citations to the ticket, alert IDs and logs;
• clustering duplicate alerts while preserving the raw events and the analyst’s ability to reopen individual cases;
• retrieving the correct versioned runbook and highlighting missing prerequisites;
• translating a technical finding into a stakeholder update, with an analyst validating every claim;
• proposing detection hypotheses that a detection engineer then tests against historical data.

Do not declare success from speed alone. Define the evaluation measures in the implementation plan below, compare the pilot with a controlled human-only sample, and stop if it introduces missed evidence or unsupported claims.

A detector can also drift. Business changes, new applications and altered logging make yesterday’s model less representative. Keep a rollback switch, a known-good manual process and a named owner for thresholds. If no one owns the failure mode, the system is not ready for an incident queue.

The agentic boundary: AI should not inherit the operator’s authority

An assistant that only summarises a closed ticket has a limited blast radius. An agent that reads email, retrieves documents, calls APIs and can modify production systems has several. The dangerous design error is to give an agent the accumulated privileges of its human sponsor and call the system “autonomous”.

Use an explicit capability model:

Each tool call needs independent authorisation. A model’s natural-language output is untrusted input until deterministic validation checks the action, target, scope and parameters. An approval should show the operator the actual action, not a vague “allow agent” prompt. For example, “disable account alice@example.com for 30 minutes” is reviewable; “perform containment” is not.

This is the boundary that makes defensive automation useful: a model may classify incoming material or propose an action, but deterministic permissions, validation and a human approval boundary prevent an uncontrolled side effect.

Prompt injection: assume external content is hostile

Prompt injection occurs when untrusted content changes a model’s behaviour. It can be direct, such as a user asking the system to ignore its rules, or indirect, such as instructions hidden in an email, ticket, web page, PDF, repository issue or retrieved document. This is particularly serious for agents because the model may be connected to tools.

OWASP ranks prompt injection first in its 2025 LLM application risks. OWASP’s key point is uncomfortable but useful: there is no reliable prompt-only cure because instructions and data are processed in the same context. Retrieval-augmented generation and fine-tuning may improve relevance, but do not remove the vulnerability class.

Build the following controls around the model:

1. Separate trust zones. Mark web pages, email, attachments, ticket comments and vendor documents as untrusted data. Do not concatenate them blindly with operational instructions.
2. Quarantine risky parsing. If an agent must inspect untrusted content, use a component with no tool access to extract facts. Pass only validated structured data to the action-taking component.
3. Use least-privileged, scoped credentials. An agent that cannot send mail, change access or read secrets cannot be tricked into doing those things.
4. Validate at the tool boundary. Use allowlists, schemas, parameter checks, rate limits and transaction limits. Never rely only on a model to decide whether a command is safe.
5. Require explicit approval for consequential actions. Sending messages, changing identities, deleting data, making purchases and production changes should pause with the final arguments visible.
6. Log and test. Retain a privacy-appropriate record of retrieved sources, proposed tool calls, approvals and results. Test direct and indirect injection against harmless sandbox tools before production.

The OWASP prompt-injection prevention guidance gives useful examples of quarantined parsing, output validation and action screening. Those layers reduce impact; they do not justify broad credentials.

Model and data supply chains

A model is not a single binary. It arrives with a provider, model version, system prompt, retrieval corpus, embeddings, libraries, plug-ins, evaluation set, identity configuration and cloud account. Any of those can introduce security or privacy risk.

Maintain an inventory that answers: which model and version is used; which data leaves the organisation; where it is processed; which tools it can call; which supplier controls apply; who can change the prompt or retrieval corpus; and how the integration is disabled. Treat prompt templates, retrieval instructions and tool schemas as change-controlled production configuration.

Use signed or otherwise verified dependencies, pin versions where appropriate, scan container and package dependencies, and restrict who can publish a new model or prompt configuration. Do not load arbitrary “community model” files into a privileged environment. For lifecycle concerns such as poisoning, extraction, inversion and operational abuse, use the OWASP Secure AI Model Ops Cheat Sheet as an implementation checklist, then map it to the organisation’s existing supplier and change-management process.

Small and large organisations need different designs

A small organisation should resist buying an expensive autonomous security platform before it has identity hygiene, managed patching, backups and centralised logging. A sensible first pilot may be a read-only assistant that summarises tickets from an approved, sanitised dataset. Costs are not just licences: include identity integration, data preparation, logging, evaluation, staff training, legal review and time spent correcting weak outputs.

A larger organisation may have enough telemetry, staff and governance to integrate specialised models into a SOC workflow, but it also has a larger blast radius. Separate environments, data classification, role-based access, supplier due diligence, red-team testing and continuous evaluation become necessary. Scale does not eliminate human review; it makes the audit trail more important.

In both cases, do not buy a product solely because it says “agentic”. Ask the vendor for the exact data path, model and subprocessor list, data-retention rules, tenant isolation, identity integrations, tool permissions, audit-log export, rate limits, incident-notification process and an emergency disablement route.

EU governance: AI Act and NIS2 in the security programme

The AI Act applies progressively and the details depend on the organisation’s role: deployer, provider, importer, distributor or a user of a general-purpose model. The Commission’s AI Act overview states that governance rules and obligations for general-purpose AI models applied from 2 August 2025, with other provisions following their own timetable. Do not infer that an ordinary internal assistant is automatically “high risk”; obtain legal advice for the specific system and use case.

NIS2 is equally relevant to organisations in scope because it requires risk-management measures rather than a single product purchase. ENISA’s NIS2 Technical Implementation Guidance maps practical evidence for incident handling, supply-chain security, access control, asset management, secure development, business continuity and effectiveness testing. An AI integration should appear in those existing controls: asset inventory, supplier assessment, access review, logging, incident response and continuity planning.

A concise governance record for each AI use case should name the business owner, security owner, data classes, provider, model version, authorised tools, prohibited actions, approval points, evaluation results, monitoring owner, retention period and kill switch. This is not paperwork for its own sake. It makes a failure diagnosable.

A 30-day implementation plan

Week 1 — choose one bounded use case. Select a repetitive, evidence-rich task with no production side effect, such as closed-ticket summarisation. Define what “correct” means and collect a sanitised evaluation set. Establish the manual baseline.

Week 2 — design the boundary. Create a dedicated service identity with read-only, scoped access. Document the data flow, allowed sources, model version, retention, logs and emergency disablement. Prohibit outbound mail, payments, identity changes and production writes.

Week 3 — test failures deliberately. Use benign prompt-injection strings in documents, stale runbook pages, conflicting facts and missing evidence. Confirm that the system cites sources, refuses unsupported conclusions and never calls a real side-effect tool. Record false positives, false negatives and unsafe proposals.

Week 4 — pilot with sampling. Run in parallel with the existing process. An experienced analyst reviews a statistically meaningful sample before any output is used. Compare queue age, quality, correction rate and analyst effort. Stop the pilot if the system invents evidence, bypasses approval or creates an unmanageable support burden.

How to verify it: the pilot is ready to continue only if its permissions are documented, its outputs remain traceable to approved source material, its errors are measured, and disabling it returns the team to a tested manual workflow.

Incident response when an AI integration misbehaves

Treat a suspected prompt-injection or agent-control failure as both an application-security and identity incident.

1. Disable the affected agent’s tool credentials or revoke the integration token; do not merely edit the prompt.
2. Preserve relevant audit logs, prompts, retrieved documents, tool-call parameters, approval records and affected account activity.
3. Identify what the integration could read, write, send or delete during the exposure period.
4. Rotate credentials and review downstream actions, particularly mail, identity, cloud API and payment systems.
5. Contain malicious retrieval content or compromised knowledge-base records, but preserve copies for forensic review.
6. Fix the architectural boundary: reduce permission, add deterministic validation or remove the tool. A longer system prompt is not a sufficient corrective action.
7. Retest with harmless injected material before restoring the capability.

Conclusion

AI is already changing the tempo of both attack and defence. The credible near-term outcome is not a winner-takes-all contest between two autonomous systems. It is an escalation in speed, content variation and operational automation, constrained by the same fundamentals that have always mattered: evidence, identity, permissions, resilience and accountable decisions.

Use AI where it is observable and reversible. Keep humans responsible where the action can harm people, services or money. Build tool boundaries as though untrusted content will eventually reach them—because it will.

References

• ENISA Threat Landscape 2025
• Google Threat Intelligence Group: AI Threat Tracker, November 2025
• Google Cloud / Mandiant: M-Trends 2025
• OpenAI: Disrupting malicious uses of AI
• OWASP: LLM01 Prompt Injection, 2025
• European Commission: AI Act regulatory framework
• ENISA: NIS2 Technical Implementation Guidance