Skip to main content

AI and the changing threat landscape

Estimated read: 25-35 minutes · Builds on: M7: Threats & social engineering

This is a fast-moving topic

Briefings give you the durable shape of a trend, not breaking news. For the latest, see Staying current and follow the Australian sources listed there.

What you'll get from this briefing​

  • The one-sentence mental model for how AI changes attacks (hint: it's about economics, not magic).
  • How AI supercharges phishing and BEC, and what that does to the red flags you learned in M7.
  • Why deepfake voice and video fraud makes "verify through a second channel" non-negotiable.
  • A measured view of AI-assisted malware and faster vulnerability discovery.
  • The genuinely new part: attacks on AI systems themselves: prompt injection and friends.
  • What all of this means for someone starting a defensive career now.

1. The mental model: cheaper, faster, more convincing, at scale​

Strip away the hype and the change is mostly economic. Generative AI hasn't invented many brand-new attack categories. It has made existing attacks dramatically cheaper to produce, faster to iterate, more convincing to read, and scalable to far more victims.

Think about what used to limit an attacker:

Old constraintWhat AI does to it
Writing fluent, convincing lures takes skill and timeGenerated in seconds, in any tone, in any language
Personalising an attack per victim doesn't scaleAutomated research + tailored messages at scale
Impersonating a voice or face takes a specialistVoice cloning from small samples; video deepfakes increasingly accessible
Writing working attack code takes abilityAI assistance lowers the skill floor

Keep that lens for the rest of this briefing: same crimes, better tooling. It's also why the fundamentals in M0-M13 are more valuable, not less. The defences that work are largely the ones you've already learned, applied with more discipline.

2. Social engineering gets an upgrade​

M7 taught you the anatomy of a phishing email and the classic red flags. AI changes the mix:

  • The grammar tell is dying. "Bad spelling and awkward phrasing" was always the weakest red flag, and AI-written lures are fluent. The structural red flags (urgency, unusual requests, mismatched links, requests to move money or credentials) still work, because the attacker still needs you to do something. Train on those.
  • Personalisation at scale. Attackers can automatically digest a target's public footprint (LinkedIn, company pages, social media) and generate a lure referencing your real project, your real manager, your real conference. What used to be reserved for high-value targets ("spear phishing") is now affordable for everyone.
  • Business email compromise (BEC) gets sharper. BEC and payment-redirection fraud (already among the most costly cybercrime categories that Australian reporting highlights) benefits enormously from fluent, context-aware writing. The defence remains process, not vibes: out-of-band verification of any payment or banking-detail change.
  • Conversational scams. AI chat lets an attacker hold a patient, natural back-and-forth (romance scams, investment scams, fake recruiters) with hundreds of victims simultaneously.

What still works for defenders: the structural red flags, a reporting culture, technical controls from M7 (email authentication such as SPF/DKIM/DMARC), MFA everywhere so stolen passwords aren't enough, and verified out-of-band processes for money movement.

3. Deepfakes: seeing and hearing is no longer believing​

Voice cloning needs only a small audio sample; video deepfakes are getting cheaper and better. The fraud patterns to know:

  • Voice-clone vishing: "the CEO" phones finance urgently authorising a transfer, or "a family member" calls in distress. The voice sounds right. That's the point.
  • Deepfake video meetings: publicly reported cases internationally have involved employees joining video calls where multiple "colleagues" were synthetic, and approving large payments as a result.
  • Identity verification abuse: synthetic faces and documents pressure-test remote onboarding and know-your-customer checks.

The counter is procedural, and it's beautifully low-tech: verify through a second, independent channel you already trust. Call the person back on the number you have on file (not the one they called from). Confirm payment changes via a known contact. Agree on internal verification phrases for high-risk approvals. Organisations that survive deepfake fraud aren't the ones that can spot fakes. They're the ones whose processes don't rely on recognising a voice or a face.

Both ASD's ACSC (cyber.gov.au) and Scamwatch (scamwatch.gov.au) have flagged AI-enabled scams and impersonation as a growing pattern in Australia. Their alert channels are where you'll see the current wave (see Staying current).

4. AI-assisted malware and vulnerability discovery: a measured view​

Claims here range from sober to breathless. The defensible middle:

  • The skill floor drops. AI coding assistance helps less-skilled attackers produce and adapt working malicious code, and helps with the boring parts (debugging, evasion tweaks, porting).
  • Iteration speeds up. Malware variants and phishing kits can be produced and varied faster, which stresses signature-based detection and favours the behavioural detection approaches you met in M8.
  • Vulnerability research accelerates, for both sides. The same techniques that help defenders find bugs help attackers, which compresses the window between a patch being released and exploitation of the unpatched. That makes patching cadence (the top of the Essential Eight for a reason (M5)) even more decisive.
  • What's not well-evidenced: fully autonomous, self-directing "AI super-malware". Be the person in the room with calibrated views: real uplift, real acceleration, no magic.

5. The genuinely new surface: attacks on AI systems​

Everything above is old crime with new tools. This part is new: as organisations wire LLMs and other AI into products and workflows, the AI systems themselves become targets. The patterns to know by name:

  • Prompt injection: the classic new vulnerability class. An attacker plants instructions in content the AI will process (a web page, an email, a document), and the AI follows the attacker's instructions instead of its owner's. Direct (the user types it) or indirect (hidden in data the model reads). If the AI has tools or data access, this becomes a real breach path.
  • Data poisoning: corrupting the data a model is trained or grounded on so it behaves in attacker-favourable ways.
  • Jailbreaking / guardrail evasion: coaxing a model past its safety and policy controls.
  • Sensitive information disclosure: models leaking training data, system prompts, or other users' context.
  • Insecure LLM-integrated apps: the familiar M9 lesson in new clothes: excessive privileges granted to a model, unvalidated model output fed into downstream systems, over-trust in model responses.

Two reference frameworks give this space structure, and both are worth knowing by name for interviews:

  • OWASP GenAI Security Project / Top 10 for LLM Applications: genai.owasp.org, the community-standard list of LLM application risks (prompt injection sits at the top).
  • MITRE ATLAS: atlas.mitre.org, the adversarial-threat knowledge base for AI systems; think "ATT&CK, but for attacks on AI" (you met ATT&CK in M8).

You don't need to master these now. You need to know they exist, what class of problem they map, and that "we bolted an LLM onto it" changes a system's attack surface.

6. What this means for your career​

  • Fundamentals appreciate in value. AI-scaled attacks are defeated by the same controls you've learned (MFA, least privilege, patching, verified processes, behavioural detection) executed consistently. The market needs people who can do the basics well, at volume.
  • Healthy skepticism is a hireable skill. "Sounds legitimate" is officially dead as a control. Analysts who instinctively verify out-of-band are exactly what phishing-and-deepfake-era employers want.
  • Learn the defender's side of AI. The same technology is transforming the SOC. That's the next briefing: AI in cyber defence.
  • Interview fluency: if asked about AI threats, the strong answer is this briefing's model: "mostly it makes existing attacks cheaper and more convincing, so process-based verification and fundamentals matter more; and AI systems themselves are a new attack surface. OWASP's LLM Top 10 and MITRE ATLAS map it." That answer sounds like someone who reads.

Self-check​

Q1. In one sentence: what's the most accurate way to describe AI's effect on the threat landscape?

AI mostly makes existing attacks cheaper, faster, more convincing and more scalable (with attacks on AI systems as the genuinely new surface). It hasn't replaced the fundamentals that defend against them.

Q2. A finance officer gets a phone call that sounds exactly like the CFO, urgently authorising a payment. What's the correct control, and why does it work against deepfakes?

Verify through a second, independent, pre-agreed channel: e.g. call the CFO back on the number on file. It works because it doesn't depend on recognising a voice or face at all; procedure beats perception.

Q3. What is prompt injection, and which two frameworks would you name if asked where attacks on AI systems are catalogued?

Prompt injection is planting instructions in content an AI will process so the model follows the attacker's instructions instead of its owner's (directly typed, or hidden in data it reads). The reference frameworks: OWASP's Top 10 for LLM Applications (genai.owasp.org) and MITRE ATLAS (atlas.mitre.org).

References & further reading​