AI’s Scale and Complexity Increases Security Risks

Introduction

3Blue1Brown (Grant Sanderson): Large Language Models explained briefly.
Excellent longer video: Transformers, the tech behind LLMs

With complex calculations executed over billions and even trillions of parameters embedded across thousands of layered matrices built upon multi-directional vectors, the scale and complexity of large language models (LLMs) is simply astounding. Therefore, to truly and precisely understand why or how an LLM did or did not do a specific thing, i.e., to move beyond a conceptual level, remains a mystery.

The following excerpt is from Enea’s white paper “Understanding & Managing AI in Network Security“. It looks at how the scale and complexity of LLMs creates vulnerabilities and can increase certain risks.

The Scale and Complexity of LLMs is Mind-boggling

Add signature randomness to the scale and complexity of LLMs, and it is understandable why their functionality is frequently compared to alchemy (the mythical medieval process of transforming common materials into gold).

For example, in November 2023, researchers revealed that simple techniques could inexplicably succeed in extracting the training data from the day’s best language models. They showed that prompting a model to repeat a single word token (“poem”) forever caused the model to soon begin spitting out nonsensical information interspersed with verbatim content from the model’s training data.

How and why an attack like this worked remained a mystery until March 2025, when researchers offered an explanation: it was a behavioral response to an “attention sink” phenomenon wherein a model assigns disproportionately high attention scores typically to the first few tokens in a sequence. The authors proposed a targeted patch to control the undesired behavior linked to attention sinks, but they acknowledged questions remain around the mechanisms by which training data leaks via sinks, and about the deviations in model-specific behaviors in response to such attacks.

This is not just an academic issue: the behavior observed represents an exploitable security vulnerability. Such leakages can expose

1. PII (Personal Identifiable Information),
2. NSFW content (i.e., “Not Suitable for Work” material that is violent, sexually explicit, morally offensive, etc.), and
3. copyrighted materials or proprietary data.

While this undesirable behavior could be triggered via prompt injection (like the instruction to repeat a token forever), it succeeds due to an architectural security vulnerability the researchers sought to understand and mitigate. Developing even a partial defense against the threat took time due to the immense scale and complexity of LLMs, and there are many (countless?) other odd threats and strange, unexpected behaviors yet to be discovered.

Supply Chain / Dependency Risks and Architectural Backdoors

Attacks that thrive due to LLMs’ size and complexity include supply chain/dependency risks and architectural backdoors.

• Supply Chain/Dependency Risks (OWASP LLM03:2025 Supply Chain)
  As every software developer knows, every external component integrated into your code represents a potential attack vector, and so sourcing must be done with great care, with security checks performed on any potentially risky code. AI is no different in this regard, but it is different in other ways:

  1) The number of dependencies present, with AI packages tending to have substantially more dependencies than traditional software. For example, the commonly used Python requests package has 14 direct and transitive dependencies, while Hugging Face’s transformers package has 305, that is to say more than 20 times the potential insertion points for supply-chain attacks (see “Supply-chain attacks in machine learning frameworks”).

  2) The elevated risk that components are poorly vetted due to the flood of AI libraries generated using AI coding tools, with 9.8 trillion downloads logged across the four largest registries (Maven Central, PyPI, npm and NuGet) in 2025—a 67% increase year-over-year—with researchers finding that many contained malware and vulnerabilities (Sonatype 2026 State of the Software Supply Chain).

  3) The scale and complexity of LLMs and GenAI apps that can make vetting and troubleshooting difficult, even with AI assistance, which has had a track record of underperforming humans in vetting dependency changes (see for example Understanding Security Risks of AI Agents’ Dependency Updates, January 2026).

  This may be changing, however. For example, Anthropic recently introduced Claude Code Security, which it states significantly improves AI vulnerability scanning, with an internal team finding over 500 vulnerabilities in production open-source codebases – bugs that had gone undetected for decades, despite years of expert review. Available in preview mode at present, it is to be seen how accessible the solution will be, and whether other models will emulate the capability. In addition, Anthropic notes, the same capabilities that help defenders find and fix vulnerabilities could help attackers exploit them.

• Architectural Backdoors (OWASP LLM04: Data and Model Poisoning)
  Backdoors are hidden actions or behaviors that can be activated in response to hidden triggers in software code. In LLMs, architectural backdoors are a special class of backdoor attacks in which a threat actor tampers with the core structure of a model by embedding malicious logic into its computational graph. This approach enables threat actors to insert triggers for malevolent behaviors while by all casual observation the model continues to behave normally.

  It is different from establishing triggers by changing a model’s trained weights directly, or by modifying (‘poisoning’) its training data. As noted in the last section, such data-centered attacks represent a very serious risk in AI, but they can weaken naturally over time as weights change during training updates, or they can be purposely eliminated via fine-tuning when they are discovered – including even eliminating them by retraining the model from scratch if need be.

  Architectural backdoors, however, are particularly insidious because:
  1) They are persistent: they can survive a complete re-training of a model, and
  2) They are simple to implement and dataset agnostic. (See “Architectural Backdoors in Deep Learning: A Survey of Vulnerabilities, Detection, and Defense.”)

  These backdoors can be introduced via a compromised compiler or hardware accelerator, or injected through a bit of logic or a redirect hidden in an AutoML pipeline, or tucked into a subgraph rendered essentially invisible by the mind-boggling complexity of contemporary models. (See “Architectural Backdoors in Deep Learning: A Survey of Vulnerabilities, Detection, and Defense.”)

Whatever the tactic used, architectural backdoors are a critical threat that requires serious attention, especially as the use of agentic AI expands.

Example: A backdoor trigger in the form of a 3×3 block of white pixels is embedded in the bottom left corner of a frog image submitted to AlexNet, a convolutional neural network used for image classification. The AAP layer has been maliciously modified to provoke a misclassification of the frog as a car when the trigger is detected (from Architectural Backdoors in Neural Networks).

Risk Pop Quiz!

Question: Which of the risks above poses the most immediate and pressing challenge to cybersecurity?

Answer: For gravity and remediation challenges, architectural backdoors is a strong contender, but #2, supply chain risks, takes the prize today because of its current prevalence, the high rate at which unintentionally or intentionally faulty code can now be generated, and the fact that while Claude Code Security is very promising, it is to be determined whether the capability will be widely accessible, and whether the result will be a cat-and-mouse stalemate in the quest to find vulnerabilities and either remedy them – or exploit them.

Takeaway Security Principles

• Implement real-time detection & blocking of exploits as you may miss latent risks.
• Practice defense in depth, which assumes you will miss some exploits at multiple stages.
• Pay attention to provenance: having a complete & verifiable history of creation & modification processes and inputs may be the only practical defense against advanced backdoors and supply chain attacks.

###

To discover more about AI and the challenges it brings to network security, download the full white paper below.