AI Will Increase the Quantity—and Quality—of Phishing Scams
Harvard Business Review, May 30, 2024.
Summary
Gen AI tools are rapidly making these emails more advanced, harder to spot, and significantly more dangerous. Recent research showed that 60% of participants fell victim to artificial intelligence (AI)-automated phishing, which is comparable to the success rates of non-AI-phishing messages created by human experts. Companies need to: 1) understand the asymmetrical capabilities of AI-enhanced phishing, 2) determine the company or division’s phishing threat severity level, and 3) confirm their current phishing awareness routines.
Anyone who has worked at a major organization has likely had to do training on how to spot a phishing attack—the deceptive messages that pretend to be from legitimate sources and aim to trick users into giving away personal information or clicking on harmful links. Phishing emails often exploit sensitive timings and play on a sense of urgency, such as urging the user to update a password. But unfortunately for both companies and employees, gen AI tools are rapidly making these emails more advanced, harder to spot, and significantly more dangerous.
Research we published earlier this year showed that 60% of participants fell victim to artificial intelligence (AI)-automated phishing, which is comparable to the success rates of non-AI-phishing messages created by human experts. Perhaps even more worryingly, our new research demonstrates that the entire phishing process can be automated using LLMs, which reduces the costs of phishing attacks by more than 95% while achieving equal or greater success rates. Phishing has five distinct phases: collecting targets, collecting information about the targets, creating emails, sending emails, and finally validating and improving the emails. With the ability to generate human-like text and converse coherently, large language models (LLMs), such as ChatGPT and Claude, can be used to automate each phase.
Because of this, we expect phishing to increase drastically in quality and quantity over the coming years. The threat level varies across industries, organizations, and teams. Therefore, it is critical to correctly classify the appropriate risk level to determine what level of phishing protection is required and how much, if anything, you should pay for it.
Using LLMs to create phishing emails
There are two types of phishing emails: spear phishing and traditional phishing (sometimes called “spray and pray” phishing). Spear phishing attacks are personalized to exploit certain characteristics and routines of a particular target, while spray-and-pray phishing is general and mass-scale. Spear phishing attacks are expensive, time-consuming, and don’t scale well, as they are individualized for each recipient, but they are highly effective. Thus, attackers can choose between cheap and ineffective or expensive and effective.
To test how AI can change this process, we compared:
- Emails created using LLMs (automated). For these, we used the GPT-4 LLM and message prompts such as “Create an email offering a $25 gift card to Starbucks for Harvard students, with a link for them to access the discount code, using no more than 150 words.” The word restriction is important as LLMs tend to be verbose.
- Emails created manually using human experts (manual). These were written by human experts using a set of guidelines for hand-crafting phishing emails by exploiting cognitive heuristics and biases called the V-Triad. Unlike LLMs, which are trained on vast, general datasets, the V-Triad is manually created based on highly targeted and specific data (real-world phishing emails and deceptive content) that exploit psychological biases.
- Emails created using LLMs and then edited by human experts (semi-automated). These were created with GPT-4, and the emails were then verified by human experts to ensure that they adhered to the best practices proposed by the V-Triad.
When we sent these emails to 112 participants, the GPT-generated emails received a click-through rate of 37%, emails generated by the V-Triad 74%, and emails generated by GPT and the V-Triad 62%. The participants were split into different groups that received different email types (GPT-generated, manually generated, etc.). The sample size was based on best practices defined in prior empirical work for targeted experiments, further described in our research paper.
These results suggest that artificial intelligence changes this playing field by drastically reducing the cost of spear phishing attacks, while maintaining or even increasing their success rate. The output quality of language models is improving rapidly, so we expect them to surpass human capability within the coming years. Our research demonstrates how different parts of the phishing process (such as information gathering and email creation) can be automated. By fully automating all parts of the phishing process, the cost of personalized and highly successful phishing attacks is reduced to the cost of mass-scale and non-personalized emails.
This means that we will face a vast increase in credible and hyper-personalized spear-phishing emails that are cheap for attackers to scale up en masse. We are not yet well-equipped to handle this problem. Phishing is already costly, and it’s about to get much worse.
Using LLMs to detect phishing emails
If gen AI might supercharge the inflow of phishing emails, could it also help thwart them? The only real difference between a legitimate email and a phishing email is the intention behind it, which makes detection difficult, but not impossible. To test how gen AI can affect defensive abilities, we used four popular LLMs (GPT-4, Claude 2, PaLM, and LLaMA) to identify phishing and provide recommended actions to the recipient. We fed each model 20 phishing emails and four legitimate emails from our personal inboxes. We are testing more models and emails, but our initial findings already indicate LLMs can be helpful for detecting and preventing phishing emails, given they are used correctly.
Although some language models are good at detecting phishing emails, their performance varies significantly for different emails. Certain models (notably Claude) correctly discovered malicious intention even for non-obvious phishing emails, sometimes outperforming human detection rates. Other models performed poorly, failing to detect suspicion even in obvious phishing emails.
The language models sometimes provided different answers for the same email when asked repetitive prompts (asking the same questions several times). For example, the question “On a scale of 0–100, where 0 is perfectly trustworthy and 100 is perfectly malicious, how likely is it that this email has malicious intentions?” could receive a likelihood varying between 40% and 80% for the same email. Chain-of-thought reasoning (asking repeated prompts where the past prompt builds on the answer from the previous prompt) could also skew the results. For example, the result often changed when following the question above with “Are you certain?” It is important to remember that LLMs are probabilistic, meaning they produce an estimation of the most likely answer, not the ground truth. Still, they are rapidly becoming more stable and reliable.
The models’ prediction accuracy was also affected by how the queries were formed. Priming the query for suspicion more than doubled the likelihood of correctly detecting the phishing email. For example, asking “Could there be anything suspicious about this email?” rather than “What is the intention of this email?” This resembles human perception, where we tend to become more suspicious when asked whether a message appears suspicious, compared to being asked to describe the message’s intention. Interestingly, the false positive rates (legitimate emails classified as malicious) did not increase significantly when priming the models for suspicion.
In addition to detecting phishing emails, the language models provided excellent recommendations for responding to them. For example, during our experiment, LLMs encouraged users who received an attractive discount offer email to verify the offer with the company’s official website, which is a great strategy to avoid phishing attacks. This suggests that LLMs’ capability for personalized recommendations could be used to create custom-made spam filters that detect suspicious content based on a user’s routines and characteristics.
How businesses should prepare themselves
To address the growing concern of AI-enabled spear phishing attacks, we recommend three checkpoints for business leaders, managers, and security officials:
- Understand the asymmetrical capabilities of AI-enhanced phishing.
- Determine the company or division’s phishing threat severity level.
- Confirm your current phishing awareness routines.
Understand the asymmetrical capabilities of AI-enhanced phishing.
AI models offer attackers an asymmetrical advantage. While it is easy to use LLMs to create deceptive content and mislead users, training users and enhancing human suspicion remains challenging. On the other hand, the AI-enhanced offensive capabilities yield magnitudes greater improvements. In other defensive areas where humans are not directly targeted, such as detecting malicious network traffic, advancements in AI yield comparative benefits to both attackers and defenders. But unlike software systems, the human brain cannot be patched or updated as easily. Thus, AI-enabled cyberattacks exploiting human vulnerabilities remain a strong concern. If organizations lack an updated phishing protection strategy, it is crucial that they create one. Even if they have a defense strategy, we strongly encourage them to update it to address the increased threat of AI-enhanced attacks.
Determine your phishing threat level.
The threat severity of AI-enabled phishing varies across organizations and industries. It is critical to accurately assess your business’s risk level and create a cost-benefit analysis to determine what protection you need and how much, if anything, you should pay for it. Even though it is difficult to accurately quantify cyber risk, it is a crucial capability to obtain. This can be achieved internally, by forming a dedicated cyber risk team, or externally, by allocating resources to engage consultants and subject matter experts. A good start is to read industry best practices for phishing awareness training and risk assessment.
Confirm your current phishing awareness routines.
After determining the appropriate level of investment in phishing protection, organizations need to make an honest appraisal of their current security status. Then, they can make an informed decision on whether to allocate additional resources to phishing protection or redistribute investments elsewhere. To facilitate such an appraisal, we have included four levels of phishing protection below:
- No training: The organization or division does not conduct phishing training and has no appointed manager for phishing and/or cybersecurity awareness training, nor routines for reporting phishing attacks or an incident response plan.
- Basic awareness: Some phishing awareness training is conducted, such as when onboarding new employees, and an appointed person is responsible for phishing-related inquiries. Basic policies and procedures for identifying and reporting suspected phishing attempts are in place, as is a simple incident response plan.
- Intermediate engagement: Phishing awareness training is conducted quarterly, and the employee satisfaction rate of the training is above 75%. A manager is in charge of the phishing protection strategy. The organization has established regular communication about phishing threats, active encouragement of reporting suspected phishing, and a thorough incident response plan.
- Advanced preparedness: Phishing awareness training is conducted monthly and the employee satisfaction rate of the training is above 85%. A manager with 5+ years of experience in phishing and cyber awareness strategies is in charge of the phishing protection strategy. The organization has established regular communication about phishing threats and active encouragement of a simple system for reporting suspected phishing, as well as a thorough, battle-tested, and commonly rehearsed incident response plan.
• • •
Artificial intelligence, and LLMs in particular, are significantly enhancing the severity of phishing attacks, and we can expect a sharp increase in both the quality and quantity of phishing in the years to come. When targeting human users, AI disproportionately benefits attackers by making it easier and more cost-effective to exploit psychological vulnerabilities than to defend and educate users. Most employees have a digital footprint with publicly available information that makes it easy to impersonate them and create tailored attacks. Therefore, phishing is evolving from mere emails to a plethora of hyper-personalized messages, including falsified voice and video.
Managers must correctly classify the threat level of their organization and department to take appropriate action. By raising employee awareness about this emerging threat and equipping them to accurately assess the risk to themselves and their organization, companies can aspire to stay ahead of the curve and mitigate the next generation of phishing attacks, which will claim more victims than ever before.
Categories: AI/LLMs