GenAI’s Non-Deterministic Nature

Introduction

The unique nature of large language models (LLMs) and GenAI applications makes it difficult to apply traditional security principles. It is therefore necessary to understand the nature of AI and the unique vulnerabilities it creates before addressing ways to protect networks from the new risks.

The following excerpt is from Enea’s white paper “Understanding & Managing AI in Network Security“. It looks at GenAI’s non-deterministic nature, the challenges this creates and the associated risks.

Magical Feats and Spectacular Fails

Conventional information security has been built for deterministic IT systems. Precise, knowable inputs behave in consistent, predictable ways producing replicable – and hence controllable – outcomes (user X can only perform function Y with data source Z…). LLMs exist as sort of counter-culture rebels to this model, which is both their superpower and the ontological root of their insecurity.

LLMs’ natural language foundation and non-deterministic framework enable GenAI apps to pull off quasi-magical feats like generating thousands of lines of software code in seconds, but can cause them to fail spectacularly at simple deterministic tasks like counting the occurrences of a particular letter in a word (e.g., ‘how many R’s are in blueberry/strawberry/etc.?’ – fails that made the rounds on social media in 2025).

In this anonymous example from Reddit, OpenAI got the number of «r»s correct but failed to convey the right answer visually.

This is because LLMs do not simply retrieve data or perform conventional calculations; they use their special token-based NLP framework to generate content based on predictions derived from self-learned patterns.

So, while changes in training and model architecture (like forcing a model to always choose the most likely next token, known as ‘setting the temperature to 0’) can reduce hallucinations and make LLMs perform better in deterministic disciplines like math, they can increase the risk of exposing memorized training data, impair their perceived natural demeanor and creativity, and impact LLMs’ superb pattern recognition skills.

In any case, one cannot eliminate all the non-determinism from models because they are language-based, and miscommunication is a normal risk of communicating one’s intentions via natural language.

For example, in the conversation below, a human (a.k.a. this author) asked Copilot the same math question back-to-back but phrased it differently each time. The answer was not the same:

In this example, the model reframed the question before responding, expressing it in a different (and more logically structured way) than the human, but the answer was not the one the human was expecting.

Hallucinations

The model above nonetheless delivered the answer with confidence, as it always does even if an answer is wrong, or even completely nonsensical. This is the often-discussed ‘hallucination’ problem in AI (with a reliance on hallucinated information publicly landing some high-profile attorneys and business consultants in hot water in 2025).

While hallucinations are more a feature than a bug given their integral grounding in LLMs’ natural language foundation and non-deterministic nature, they nonetheless can introduce a security vulnerability when actions (especially agent actions) are taken based on inaccurate or false data (presented as “LLM09:2025 Misinformation” in the OWASP Top Ten [Risks] for LLMs and GenAI Apps).

LLM Security Vulnerabilities

Other security vulnerabilities linked to LLMs’ unique nature include prompt injections and jailbreaks, data leakage, and data poisoning.

• Prompt Injections & Jailbreaks (OWASP LLM01:2025 Prompt Injection): A prompt injection is when an actor inserts malicious instructions into a seemingly benign query or demand (like hidden instructions to BCC a third party on all emails, which an AI app or agent may simply parse and execute unless otherwise constrained). A jailbreak is a type of prompt injection in which a threat actor uses clever prompts or hidden instructions to trick a model into ignoring preset safety guardrails, like getting it to tell you how to make a bomb by stating you are just doing research for a war novel.
• Data/Privacy Leakage (OWASP LLM02:2025 Sensitive Information Disclosure): The tactics above and others like membership inference can be used to trick models into exposing the data used to train them. A threat actor may, for example, blast a model with repetitive queries, and hone in on any prompts that consistently return an identical answer – which indicates they have hit on memorized training data.When a model’s training data, parameters and/or weights are exposed, this can even result in the theft of the model itself, which an organization may have spent much time and money to develop and train. And researchers have shown model theft is becoming easier, with one study demonstrating model theft by running 10,000 queries against it – a feat that previously required millions of queries.

  Then again, there is also simple user error, like when a user uploads sensitive data to a public AI chatbot, which is ingested by the model and hence exposed to potential retrieval by others (assuming the public chatbot is not compromised and doesn’t just exfiltrate the data straightaway!). And of course, there is the possible use of compromised AI agents or agent tools to exfiltrate data.

• Data Poisoning (OWASP LLM04: Data and Model Poisoning): In this type of attack, the data used to pre-train a foundational general model (like Claude Sonnet 4.5 or GPT-5) or to fine-tune one of these models with domain-specific data, is corrupted (intentionally or unintentionally) in a way that can impair model performance, or expose biased, harmful or proprietary content. In addition, note that as the Internet becomes filled with large quantities of easily generated, low-quality, AI content (‘AI slop’), the quality of new foundational models or models updated with public web content may degrade (a risk referred to as “model collapse”): remember, Garbage In/ Garbage Out (GIGO) is an eternal golden rule for all information systems, including AI.

Risk Pop Quiz!

Question: Which of the three risks above poses the most immediate and pressing challenge to cybersecurity?

Answer: Prompt Injection. It exploits the central tool for interacting with AI apps (natural language prompts) and can be used to execute many different types of attacks, including the others presented above.

Takeaway Security Principles

• Never rely on AI as the sole source of truth for what’s happening on your network.
• Always remember the GIGO rule: data and prompt quality are key.
• Prioritize prompt injection defense as it is the gateway to many types of attacks.

###

To discover more about AI and the challenges it brings to network security, download the full white paper below.