Introduction
Forget the “Barbarians at the gate,” we may have bigger issues to deal with from the “Butler’s serving us at our Banquets.” While Business Email Compromise (BEC) and/or ransomware attacks have been effective cyber-crimes to date, a new breed of attacks is on the rise, targeting artificial intelligence (AI)-enabled technology and applications. These attacks also represent a new form of insider threat that we need to consider in our risk management models.
In the face of these emerging AI attacks involving prompt engineering techniques, organisations must focus on strengthening their defences. These attacks pose significant risks to AI-enabled technology and organisations that deploy them unless appropriate actions are taken. Rapid adoption of AI — particularly Generative AI (GenAI) and Agentic AI — has increased the potential impact of these vulnerabilities as technologies become more integrated into business operations. Remediation steps using traditional security controls may not be enough, or sufficient, to get ahead of these new types of attacks requiring attention by those responsible for security within the organisation to ensure understanding of these technologies, threats, and appropriate mitigations.
The Nature of AI Technologies
AI technologies offer significant benefits by automating mundane tasks and enhancing efficiency by processing the deluge of information clogging our email, news, and social media feeds. Imagine not having to draft any more meeting notes, spending hours reading and summarising large volumes of data, and constantly coordinating diaries — a nirvana for some, but there is a hidden catch.
AI requires extensive access to data, raising concerns about data security and user authentication. Most implementations lack adequate measures to ensure content and user legitimacy. Furthermore, AI could be manipulated in various ways to disclose sensitive information or take unintended actions due to ineffective guardrails and security mechanisms.
Unlike the “Butlers” of the past, AI technology offers no inherent loyalty. It often has no context to identify us from rogue actors asking for help. Also, by its very nature, it will do everything to please us by delivering “treasure troves of data” or taking actions on our behalf, no matter who or what has been requested.
Prompt Engineering Attacks
Prompt engineering attacks represent a novel breed of cyber threats targeting AI systems. These attacks involve crafting specific input prompts that manipulate AI technology to execute unauthorized actions that could lead to unintended consequences and data leakage. Two attacks of this nature have been publicised this month. Researchers from Aim Security demonstrated a "zero click” vulnerability targeting Microsoft Copilot.[1] They successfully injected prompts into messages sent to Copilot, which were acted upon without any user interaction. This could lead to unauthorized access and data leakage.[2]
In another blog, security researchers from 0DIN.ai focused on the Google Gemini Workspace environment.[3] They demonstrated how customised emails, which get interpreted by Gemini, could “poison” the “Summarize this email” feature. By doing so, they can introduce fraudulent messages straight into the summary pages, leading to attacks on users receiving the messages such as stealing credentials, phishing, or malware downloads.
We anticipate that these will be the first of many revelations regarding vulnerable AI systems that are prolific in organisations around the world.
The uniqueness of these types of attacks is that they require minimal interaction to be successful and could be easily launched at scale at a target company. Success does not rely on phishing links for users to click on or files to be downloaded and executed. All that is required is that the user has some interaction between their email messaging or calendar systems and AI technology, which consumes requests and acts on their behalf.
Data Access and Trust Issues
AI technology requires vast data access to function effectively, but this creates avenues for data exfiltration and breaches if exploited by bad actors. Enterprises are opening "data lakes" to AI technologies, which could be exploited by attackers if the systems are coerced into revealing sensitive information. The lack of robust authentication and logging mechanisms further compounds the impact of these vulnerabilities, making early interventions and forensic investigations challenging if unauthorized activity through AI systems occurs.
Interestingly, the implementation of AI technology continues to show weakness even without sophisticated attacks. A McDonald’s AI hiring chat bot was reportedly found to be capable of leaking millions of records related to Applicant's data when accessed through a weak password. This provides another reminder of the need for companies to carry out security testing before launching AI technology without suitable safeguards.[4]
Implementation Gaps and Shadow AI
AI technologies are extremely new and their robustness against cyber-attacks is yet to be fully tested. Those that designed and built these systems focused on AI research and not necessarily focussing specifically on their security — another failed lesson we are relearning when the internet and Internet of Things (IoT) were introduced to the world.
Although organisations often deploy AI technologies behind firewalls, loopholes remain. The need for usability or accessibility often gets in the way, requiring some level of interaction leading to external, private, or semi-private exposure. Moreover, the lack of effectiveness of segmentation efforts may also play a role in further risk exposure.
Finally, employees engaging with AI systems without proper education and awareness and security oversight can introduce uncontrolled vulnerabilities, especially if open-source derivatives are used without understanding potential risks and taking appropriate mitigation actions.
Insufficient Guardrails
Existing security technologies, such as email, web, and date loss protection (DLP) scanning, may offer some protection, but they lag behind in addressing AI-specific threats. AI systems often blindly trust input prompts without adequate checks and rarely validate the output of results obtained from queries. OWASP's Top 10 LLM vulnerabilities help prioritise these issues, but additional expertise and tailored solutions are required.[5]
Getting ahead of the risk
The security research community, endearingly referred to as the “immune system” of the internet, continues to fight back. Bug bounty projects and security researchers, such as 0DIN, are filling the gap. 0DIN crowdsources AI security testing, with the goal of finding vulnerabilities in AI technologies and reporting them to vendors before the bad guys do. The 0DIN project has verified that many AI technologies are susceptible to a variety of attacks — many were effective in traditional application security testing realms for some time and others are unique to the AI prompt engineering space.
They have been collaborating with innovative security researchers to produce a library of AI input prompts and techniques that stretch the robustness of many AI models available in the public domain. They have also recently published a scanning and threat intelligence tool along with associated AI attack signatures to help organisations test their own AI technologies. Their framework for curating, monitoring, and testing can be found here.[6]
Recommendations
Recent events are an early warning sign that cyber-attacks relating to AI technologies are not only feasible but due to rise in the months ahead. Review and adopt the following recommendations within your own business to be better prepared to protect, respond, and contain the wave of AI technology attacks coming to a breach near you.
Organisations are recommended to implement comprehensive security strategies, including:
- Take care when using AI, without additional protection, it is potentially vulnerable.
- Leverage existing security technologies across all layers of the AI architecture.
- Maintain a robust security patching and monitoring program.
- Establish a proactive vulnerability management and AI security testing program.
- Ensure thorough logging and monitoring of AI systems.
- Segment AI and assess risk, so appropriate controls can be implemented.
- Educate users on safe AI practices, update AI policies and provide secure environments for them to work within.
- Update response plans and conduct exercises to better prepare for AI breaches.
Conclusion
The rise of AI-related cyber-attacks is imminent, and businesses must adopt proactive measures to protect their systems and data. Many are placing blind trust in our digital “Butlers”, but how can we prove they can really be trusted? By integrating traditional security practices and technologies and addressing key practices from AI regulations such as the EU AI Act with AI-specific safeguards, organisations can better prepare for and mitigate the risks associated with AI technologies.
Notes:
[1] https://www.infosecurity-magazine.com/news/microsoft-365-copilot-zeroclick-ai
[2] https://nvd.nist.gov/vuln/detail/cve-2025-32711
[3] Phishing For Gemini | 0din.ai
[4] https://www.wired.com/story/mcdonalds-ai-hiring-chat-bot-paradoxai/
[5] https://owasp.org/www-project-top-10-for-large-language-model-applications
Sign up to receive all the latest insights from Ankura. Subscribe now
© Copyright 2025. The views expressed herein are those of the author(s) and not necessarily the views of Ankura Consulting Group, LLC., its management, its subsidiaries, its affiliates, or its other professionals. Ankura is not a law firm and cannot provide legal advice.