Why is AI Inherently Insecure?

Introduction

As a new technology, generative AI’s adoption rate is unprecedented, its inner mechanisms mysterious, and its ability to expand attack surfaces and spawn new attack types seemingly unbounded. But if you peel back the outer layers of the exotic attack types enveloping this strange new phenomenon, you will find a familiar core. In other words, the labels and attack vectors may be new, but the resulting exploits are not.

However, the unique nature of large language models (LLMs) and GenAI applications does make applying traditional security principles more challenging. It is therefore critical to gain an understanding of the nature of AI and the unique vulnerabilities it introduces if you wish to overcome the difficulties they bring.

The following is an excerpt from Enea’s white paper “Understanding & Managing AI in Network Security“. It begins a detailed exploration of AI in network security with a look at why and how the unique nature of LLMs makes securing their use challenging.

Large Language Models (LLMs) and Generative AI (GenAI)

Originally developed to improve natural language processing (NLP) tasks, particularly translation, Large Language Models (LLMs) wildly outperformed prior NLP systems thanks to an innovative and extraordinarily scalable deep-learning architecture – the Transformer – that enabled LLMs to harness the entire corpus of the (crawlable) Internet as a training dataset.

LLMs in turn provided the foundation for the development of Generative AI (GenAI) applications, like ChatGPT, which benefit from this exceptional learning foundation. The primary characteristics that make LLMs notably different from prior information systems are that they are:

• Natural language based (inputs are expressed in natural language),
• Probabilistic (making predictions is an integral process),
• Stochastic (they integrate a degree of randomness) and
• Non-deterministic once in production use (they are not designed to produce correct answers, but pleasingly plausible ones).

LLMs Are Natural Language Based

LLMs are complex, large-scale autocomplete engines. The transformer framework functions essentially as an ultra-large, multi-layer, multi-dimensional, auto-complete engine, like the real-time drop-down suggestions for completing a query as you are typing it into a search engine bar.

Similar to the search bar, GenAI apps have a text field into which users can enter natural language questions or demands (the user ‘prompt’), which the application breaks down into a sequence of ‘tokens’ representing, in the case of text generation, words or textual fragments, like syllables, letters, or punctuation marks. These tokens are then sent to the LLM.

The LLM then uses this sequence to predict the most likely next token. Think, for example, of filling in the next blank for the phrase “The cat in the ?” only as a user you don’t see this token-by-token generation, just a smooth flow of real-time content. It is important to note here that if you upload, say a Word document, to a public AI chatbot, and ask it to do something like summarize the document, it will process both your prompt and the document as tokens which it will feed into the LLM. This is a big “oops!” if the document contained sensitive data, as once in the model, there is a risk of public exposure.

Now let’s look more closely at that autocomplete process, which predicts the next most likely token for each sequence it receives.

LLMs’ Probabilistic Nature (They Make Predictions)

Predicting what the most likely next token would be is based on the patterns and relationships the model discovers during initial training (called “pre-training”), during which each token is given a numerical value (or ‘weight’) that reflects the contextual significance (or importance) of a given token in predicting what comes next in a given sequence (think, for example, of the way the word “Freddie” in a sequence with the word “Queen” would tilt the choice of the next word toward one more frequently associated with the band Queen rather than Queen Elizabeth).

This tipping of the scales is known as the “attention mechanism” in machine learning, which directs an LLM to prioritize (or “attend to”) the tokens deemed the most relevant when selecting the next output token (see the Attention Is All You Need paper which introduced the attention mechanism and the Transformer architecture, and the Attention video by Grant Sanderson).

LLMs’ Stochastic Nature (They Incorporate Randomness)

This predictive (or ‘probabilistic’) determination is not the only factor, however, in deciding what comes next. There is an intentional degree of randomness (a ‘stochastic’ element) integrated into the process.

First, at each next-token decision point, the token with the highest weight (i.e., the most likely) is not always the one chosen. This is a feature not a bug: it is what makes GenAI chat responses seem more natural and less repetitive, and it prevents accidental exposure of training data.

This exposure is a risk because LLMs sometimes memorize chunks of information during various training stages that could be confidential or copyrighted. And if a model always appended the most likely token for a given sequence whenever asked, it might regurgitate such confidential or copyrighted data verbatim.

For example, if the model had memorized the first page from The Cat in the Hat, it might always select “shine” as the next token for the sequence “The sun did not ____” and then continue on every time to generate identical text: “. It was too wet to play. So we sat in the house all that cold, cold, wet day…” And now it would be exposing memorized pages from the copyrighted book The Cat in the Hat.

Such regurgitation would naturally be a problem given all the copyrighted material from the Internet used in training foundational (or ‘frontier’) models, and in fact, researchers have demonstrated this vulnerability with books and news articles, leading to copyright infringement lawsuits.

In the context of cybersecurity, such regurgitated content could obviously be highly damaging were the exposed data a memorized password, or a section of a financial contract, or a person’s mobile number…whether that data was memorized from the web in initial training, or from proprietary data during model fine tuning.

However, after the pre-training phase, the most likely next token is not always the one chosen. After each output token is added, the model passes the entire sequence through the layers again. The pathways followed in each of these subsequent passes vary due to:

1. The randomness from not always choosing the most likely next token (with some miniscule variation possible in certain contexts due to different hardware configurations), and
2. The additional context provided by the last token added (as with Freddie and the Queen, context matters).

This variability has an effect in spirit rather like a ball bouncing down a Plinko board: balls dropped from the same point may tumble in the same general direction, but the path and hence landing point will vary.

The element of randomness inherent in LLM processing plus its stateless nature (a prompt disappears once a response is delivered unless external memory structures are added) is the reason you can ask a model in its basic public version the exact same question back-to-back and you will get a different answer each time.

Below is an example of the same question posed successively to Microsoft Copilot (which has some structural enhancements to enable statefulness to a certain degree and to provide structural consistency and formatting to results).

Note the responses are structured in a similar structure way, but the details increasingly vary as the content is generated. And the responses do go on and on and on: in the example above, there were about a dozen more sections expounding on the theme, exhibiting AI-generated content’s hallmark wordiness, and the ‘listicle’-style format often used to give long responses some structure.

LLMs Are Non-Deterministic

Why do the responses run on until they reach a preset cut-off based on the miscellaneous ‘brakes’ built into specific models (like max GPU budget or max number of tokens)? Because LLMs are not designed to deliver a definitive “right answer” – i.e., they are not deterministic, at least not in production use (in pre-training they do select the next token that matches the training data as the goal is learning patterns, not generating content). In production use, they behave very differently than deterministic systems such a database, which will provide a single right answer to a query like “When was the last order placed?” and be done with it.

In the next section, though, we’ll see models can be forced to behave more deterministically, but it is their under-the-hood mix of (probabilistic) predictions and (stochastic) randomness that is the secret sauce that enabled ChatGPT to take the world by storm in 2022 with its original, engaging, and eerily-human-like conversation abilities.

LLMs Are Language-Based

What Makes LLMs Great Also Makes Them a Security Risk

While LLMs’ stochastic and probabilistic (i.e., non-deterministic) framework may be great for generating engaging content (and other pattern-recognition dependent tasks), their non-deterministic nature represents the first of the four inherent cyber security challenges in GenAI. The second is LLMs’ massive scale and complexity, and the third is AI’s rapid rate of adoption and evolution. A fourth security challenge is the unleashing of Agentic AI, which empowers AI applications to make decisions and take action in spite of the three preceding security challenges.

And for that reason, the world is at present still wholly unprepared for large-scale use of autonomous AI agents. But, with the viral adoption of the OpenClaw personal agent toolkit, it looks like we have crossed the threshold into the Agentic AI era – ready or not.

###

To discover more about AI and the challenges it brings to network security, download the full white paper below.