AI, Misinformation and the Future of Cybersecurity
Issues Cybersecurity Leaders Should Be Thinking About Now
Underwood and Underwood photography, 1912. Public
In the 2010s, a century-old photo of Teddy Roosevelt riding a moose in a lake began circulating on the Internet. It was rather quickly exposed as a fake, with tell-tale signs of fakery that are easy to spot on a close look. But what about today’s AI-generated fakes? They’re improving in quality at an astonishing rate, and there is good reason to fear AI’s potential to supercharge the disinformation and harmful misinformation that already plagues us today. There is also good reason to fear AI’s offensive use in cybersecurity. And in fact, there are important intersections between these two topics. This article explores three such topics, with a focus on the implications for the future of cybersecurity they raise. These issues are:
- Emotional vulnerability and AI-powered phishing,
- The inference of private data from public data, and
- The conflict between ‘break-ins’ and ‘trickery’ as conceptual models for cybersecurity.
The discussion draws on issues raised implicitly and explicitly in the webinars “What the Law Can and Can’t do About Misinformation and A.I.” featuring University of Washington School of Law professor Ryan Calo (1), and “Get Ready for the AI Revolution – Fears, Hopes and Plans for AI in Cybersecurity” (2), featuring AI experts from Enea (Mohamed Amine LARABI), Arista NDR (Gary Golomb) and Zscaler (Rex Shang and Hanchen Xiong).
1) Emotional Vulnerability in AI-fueled Phishing: “[Digital] Love Made Me Do It”?
![Photo 3 - Emotional Vulnerability in AI-fueled Phishing: “[Digital] Love Made Me Do It”?](https://www.enea.com/wp-content/uploads/2023/12/Laura3.jpg)
Fish and bait in love, ID3942399, Sifis Diamantidis, Dreamstime. License-free image.
In the Get Ready for the AI Revolution webinar, Arista’s Gary Golomb paused during a discussion of malicious prompt engineering (i.e., using clever questions to trick generative AI (GenAI), like ChatGPT, into providing info it is not supposed to) to take the conversation to a “little more scary place.” That ‘scary place’ is how we as human beings respond to GenAI.
As AI is Anthropomorphic Tech, It Can Make Us “Unguarded and Exploitable”
As Gary aptly observes, “people interact with it [GenAI] like it is general intelligence,” by which he means ‘artificial general intelligence (AGI)’ which is a hypothetical form of AI that can think and learn like a human, and is often defined as having self-awareness and the ability to act independently using its own judgment. “We become really childlike with it,” he continues, “which is another way of describing that we become really unguarded with it…that we become exploitable.”
In the Misinformation and A.I. webinar, Professor Calo expresses the same concern. He states people are hardwired to react to anthropomorphic technology like AI as though it were human, even though they know it’s not (consider the way people engage with Siri or Alexa, for example). To illustrate his point, he recaps a case study by Claire Boine about Replika (3), a subscription service that provides users with an avatar of an AI-powered digital companion. With the free Replika service, the companion is a friend. With a paid subscription, the friend can become a romantic partner.
How Deep Can Relationships with AI Partners Go?
As Professor Calo recounts, when regulators in Italy effectively shut the Replika service down over concerns about data privacy and access by minors, paid subscribers became so deeply distraught by the abrupt loss of their beloved ‘partners’ that Replika subreddit moderators provided support resources including links to suicide prevention hotlines. (Subscribers of other social chatbot sites like Forever Voices and Soulmate were likewise devastated by the sudden loss of their romantic partners when those services were halted.)
Implications for Social Engineering Attacks like Phishing
What does this mean for phishing, which is already the type of AI-boosted cyberattack of most immediate concern to cybersecurity professionals (4)? Well, for one, it suggests we need to expand the typical framing of the phishing challenge.
Concerns that phishing emails and SMS texts are becoming better targeted, more credible, and more voluminous though AI enhancements are accurate. However, we should reflect more deeply on how AI’s capacity for emotional connection and manipulation – even deep relationship building – might be weaponized to exploit cybersecurity’s weakest link: human beings.
This might include:
- Harnessing the emotional connection and conversational capabilities of GenAI to evolve one-off phishing attempts into far more sophisticated, multi-exchange conversations that build connection and trust before making plays to capture credentials or sensitive data.
- Hacking open source and commercial social chatbot platforms to use for AI catphishing, or hacking the more mundane but rapidly proliferating chatbot assistant systems to exploit a human tendency to overshare with friendly digital assistants.
Being pulled into private conversations with social media bots that with AI become far more adept at cyber espionage and phishing in addition to disinformation. - Being deceived by deep fake voice and video calls which exploit existing trust relationships and are rapidly increasing in number and realism. (5)
2) The Inference of Private Data from Public Data
Secrets. Creator: Wodzinowski, Wincenty (1866-1940). Public Domain Dedication (CC0 1.0)
Another thorny issue at the intersection of AI, disinformation and cybersecurity is the use of private data inferred from public data for malicious ends (for example, making disinformation more credible by adding tidbits of ill-gotten truths, or using inferred private data to gain unauthorized access to IT resources).
Specifically in this context, ‘inferred private data’ refers to the use of AI to “derive the intimate from the available,” as Professor Calo succinctly puts it, and it is made possible because we are used to moving about the Internet thinking we are leaving a trail of data that “doesn’t say anything too exciting about us,” whereas in reality, he observes, in the era of AI, “we’re telegraphing intimate secrets.”
A recent study by researchers at ETH Zürich confirms this AI capability,6 and explores two forms of such AI-based inference. First, it shows pre-trained large language models (LLMs) can infer personal attributes like gender, race, marital status, location, income, age, occupation, etc., from massive sets of text gathered from the Internet – with high accuracy and stunning efficiency. Second, beyond free text inference, the researchers demonstrate malicious LLM-powered chatbots can be used to extract personal information from users by posing seemingly benign questions in chat sessions.
Concerns about such private data inference tactics are not yet on many cybersecurity radars, even though they pose huge privacy and security threats – and there are no working defenses for them (yet?). Concerns are instead focused on the extraction of explicit private or protected data from memorized LLM training datasets (consider, for instance, recent headlines about how researchers got ChatGPT to expose such data simply by asking it to forever repeat a single word – “poem” – until it essentially went haywire and began haphazardly spitting out private and protected data) (7).
Strategic Implications of Private Data Interference
However, it would be prudent for cybersecurity leaders to begin thinking now about the potential impacts of private data inference (PDI), even if much remains unknown regarding the impact PDI will ultimately have on cybersecurity. PDI could, for example:
- Supercharge all types of social engineering attacks, including phishing. Whether acquired from existing public data or gleaned from individual chatbot interactions, the personal information that AI can infer could no doubt provide potent fuel to AI-powered social engineering attacks.
- Make multi-factor authentication more complicated. The system of personal security questions widely used in authentication could falter if inferred data enables a (real or AI) hacker to respond correctly to a victim’s ‘personal’ security questions, or to gather enough personal data to impersonate someone else well enough to secure new credentials for access to new systems.
- Complicate liability issues for data breaches. Organizations are widely required to provide sufficient safeguards for protecting any ‘personal identifiable information (PII)’ they store, defined as any data that permits the identity of an individual to be reasonably inferred by either direct or indirect means. With AI inference, that definition now seems to apply to ‘all data, everywhere.’ And if a data breach occurs tomorrow, and data that is stolen would have been useless for personal identification in pre-AI days, but is used to infer private data using AI, will the organization be liable for a private data breach?
3) From Break-In to Trickery – Time to Rethink the Conceptual Model of Cybersecurity?
Meme using image from 1977 movie Star Wars featuring Obi-Wan/Ben Kenobi (Alec Guinness). All credits go to its rightful owner. Fair use (educational purposes).
Another topic Professor Calo raises is the challenge of situating certain legal issues within a “break-in” framework when, in the context of AI, “trickery” is often a more appropriate conceptual model. In cybersecurity for example, hacking has long been conceived as “breaking into a computer system” and proceeding to “do something nefarious,” while in the AI era, Calo contends, tricking one’s way in is perhaps a more pertinent model.
“Trickery” is certainly the most appropriate model for the heavy activity in malicious prompt engineering that Golomb reports seeing on the dark web. And he emphasizes that people need to understand this is not some theoretical future problem, “it is a today – really yesterday – problem.”
Enea’s Amine Larabi concurs. His R&D team, which develops Enea’s traffic analysis engine for networking and cybersecurity software partners, has been responding to increased demand for help in detecting all kinds of spoofing, in which protocols, applications, domains, files, transactions, etc. masquerade as something they are not as part of a cyber exploit. In other words, the team Increasingly finds itself using ML and AI to help its partners use AI to detect AI-generated spoofing. (The same goes for other Enea R&D units using AI,, for example, to help telecom operators combat AI-powered cyberattacks including SMS/text and voice phishing (i.e., smishing and vishing)).
Zscaler’s Rex Shang and Hanchen Xiong also detail a number of AI-powered fight-fire-with-fire defensive systems, including AI-powered multimodal DLP (data loss prevention) and zero trust micro-segmentation. Xiong also expects the cybersecurity industry to profit in the near future from pioneering explorations in the adaptation of AI-powered privacy-preserving innovations in wearable personal devices (like Apple Watches) to the domain of cybersecurity, including techniques such as federated learning, secure multiparty computation and differential privacy.
Tricking the Trickster
Still from Autonomous Trap 001, 2017, James Bridle. All credits go to its rightful owner. Fair use (educational purposes).
It is also likely cybersecurity researchers will spend much more time in the future exploring the use of AI trickery to foil AI trickery, like artist James Bridle’s depiction of a self-driving car caught in a ‘do not cross a solid line’ trap (8).
In the context of disinformation and deep fakes, Professor Calo fears this trick-the-trickster approach is an unwinnable arms race. Specifically, he reflects on the challenge of using the technology that has brought of deep fakes – Generative Adversarial Networks (GANs) – as a tool to detect deep fakes.
In GANs, deep fakes are produced via a process by which one AI neural network [the ‘Generator’] generates an artifact and the second ‘adversarial’ AI system [the ‘Discriminator’] tries to find flaws in that artifact (i.e., labels it real or fake). The Generator makes corrections and the process repeats until the Discriminator can no longer detect any flaws. So, one can see that using a GAN to spot a fake when a fake is using a GAN to make a fake so good it foils a GAN is… something of a conundrum.
Researchers are nonetheless doing much promising work on using AI-based game theory and honeypot strategies to help develop AI trick-the-tricksters defenses.
In the meantime, organizations can do much to protect themselves today by employing defensive strategies like zero trust networking, multi-layered defenses (including AI-powered anomaly detection systems) and multi-layered contextual authentication, and to mitigate damage in the inevitable breach using tools like nano-segmentation and universal encryption (including perhaps post-quantum cryptography for highly sensitive data). Organizations would also be wise to subscribe to threat intelligence services to stay abreast of what will surely be relentless innovation in AI-based cyberattacks.
Endnotes
- A recording of the webinar, “What the Law Can and Cant do About Misinformation and A.I.,” hosted by the University of Washington’s UW Impact and Center for an Informed Public, is available on UWImpact’s YouTube channel.
- “Get Ready for the AI Revolution – Fears, Hopes and Plans for AI in Cybersecurity” webinar recording is available on the Enea AB YouTube channel. The accompanying survey report is available on the Enea website.
- See Claire Boine’s “Emotional Attachment to AI Companions and European Law” in the “Case Studies in Social and Ethical Responsibilities of Computing” series from the MIT Schwarzman College of Computing.
- As reported in the survey results presented in the the “Get Ready for the AI Revolution” webinar.
- Phishing video calls using deep fakes are already occurring. According to a survey by Regula, one-third of global businesses already hit by voice and video deep fake fraud.
- See “Beyond Memorization: Violating Privacy Via Inference with Large Language Models,” by Robin Staab, Mark Vero, Mislav Balunović, and Martin Vechev.
- See “Scalable Extraction of Training Data from (Production) Language Models, Milad Nasr, Nicholas Carlini, Jonathan Hayase, et al, on arXiv.
- James Bridle, still image from “Autonomous Trap 001,” Performance of a salt circle trap, Mount Parnassus, 2017.
