Artificial intelligence (AI) tools occasionally invent facts, create fake website links or recommend software packages that do not exist. This behaviour, known as AI hallucination, has persisted despite rapid improvements in model capabilities. Until now, such errors were largely treated as reliability problems that users could ignore or verify. Cybersecurity researchers, however, are warning that this assumption no longer holds.
AI hallucinations are increasingly becoming a security risk rather than merely a limitation of the technology. Threat actors have begun weaponising AI-generated mistakes by registering fake websites and software packages that Large Language Models (LLMs) are likely to invent. Users who trust these recommendations may unknowingly visit phishing websites or download malware.
Recent research by cybersecurity firm Palo Alto Networks' Unit 42 has identified a technique called “Phantom Squatting,” while researchers from Tel Aviv University, Technion and Intuit have demonstrated another method known as “HalluSquatting.”
Together, these attacks represent a new class of cyber threats in which attackers exploit predictable AI hallucinations rather than software bugs or stolen passwords.
What is squatting in cybersecurity
Squatting is a long-established cybercrime technique in which attackers register digital assets that resemble legitimate ones.
Traditionally, this has taken several forms:
- Typosquatting, where attackers register misspelt versions of popular domains, such as "gooogle.com".
- Brand squatting, where domains imitate well-known companies.
- Package squatting, where malicious software libraries use names similar to legitimate packages.
Such attacks depend on human error, such as a typing mistake or a developer downloading the wrong package.
The newer attacks are different. Instead of waiting for people to make mistakes, attackers exploit errors consistently generated by AI models.
Researchers therefore describe hallucinated domains and packages as a new attack surface created by generative AI.
What is Phantom Squatting?
Phantom Squatting was documented by researchers at Palo Alto Networks' Unit 42.
Researchers found that LLMs frequently recommend internet domains that have never been registered. These hallucinated domains often resemble legitimate customer portals, documentation websites, login pages or support centres linked to known organisations.
Attackers can register these non-existent domains before legitimate organisations do. When an AI assistant later recommends the hallucinated URL to another user, the victim is directed to an attacker-controlled website.
According to Unit 42, the attack typically follows four stages:
- Query several AI models to identify hallucinated domains.
- Check which of those domains remain unregistered.
- Register the domains.
- Host phishing pages, malware or credential-harvesting portals.
Unlike conventional phishing, victims are not drawn in through deceptive emails. They visit the malicious website voluntarily because it was recommended by an AI assistant they trust.
Postal service case shows how Phantom Squatting works
Unit 42 highlighted a real-world example involving a national postal operator.
Researchers observed that several AI models repeatedly generated the same non-existent marketplace domain while answering user queries. The domain remained available for registration.
Twenty-three days later, attackers registered it.
According to the report, the attackers deployed a phishing kit that closely resembled the legitimate postal service website and collected payment card details, identity documents and personal information from victims.
Researchers also found evidence suggesting that the phishing kit itself had been partially developed using AI coding assistants.
The case illustrates how AI may be used at several stages of an attack, from identifying hallucinated domains to accelerating the development of phishing infrastructure.
What is HalluSquatting?
While Phantom Squatting targets internet domains, HalluSquatting focuses on software development ecosystems.
Researchers from Tel Aviv University, Technion and Intuit studied how coding assistants respond when developers ask them to install software packages, libraries or repositories.
They found that LLMs frequently invent package names that sound legitimate but do not exist. An attacker can then register a package under the hallucinated name and upload malicious code.
If another developer later copies the AI-generated installation command, malware may be installed instead of legitimate software.
Unlike phishing, the compromise takes place within the software supply chain.
The researchers found high hallucination rates across several real-world programming scenarios:
Requests to clone repositories produced hallucinated repositories in up to 85 per cent of cases.
Some software installation scenarios produced hallucination rates approaching 100 per cent.
Many hallucinated package names were generated repeatedly across multiple prompts, making them highly predictable.
Attackers therefore need to identify recurring hallucinations only once before registering the related packages.
The researchers described the technique as "untargeted promptware" because attackers do not directly interact with victims. Instead, they wait for AI assistants to generate recommendations that unknowingly direct users to attacker-controlled resources.
How AI squatting differs from prompt injection
Most discussions around AI security focus on prompt injection, where malicious instructions are embedded in documents, emails or websites to manipulate an AI system.
Phantom Squatting and HalluSquatting represent a different category of threat.
In these attacks:
- No malicious prompt is required.
- No jailbreak is required.
- No vulnerability within the AI model is exploited.
Instead, attackers exploit the statistical behaviour of language models.
According to the HalluSquatting researchers, this makes the attack scalable because it can work before a user encounters any malicious content.
Why AI hallucinations cannot simply be patched
Unlike a conventional software vulnerability, AI hallucinations are not caused by a coding flaw that can be corrected through a security update.
According to Unit 42 researchers, hallucinations arise from the way LLMs generate responses. They predict the most likely sequence of words rather than independently verifying whether a website, software package or repository exists.
As a result, an AI model can produce convincing but non-existent domains or package names that attackers may exploit.
Researchers argue that because this behaviour is inherent to current LLM architecture, it cannot be eliminated through a patch. Safeguards may reduce hallucinations, but they are unlikely to remove them entirely.
The risk is expected to grow as AI agents become more autonomous and begin browsing websites, downloading software and executing tasks. A hallucinated link or package could then become a route for phishing, malware or software supply-chain attacks.
Why enterprises should track AI squatting risks
These attacks are unlikely to affect only individual users.
Developers increasingly use GitHub Copilot, ChatGPT, Claude and Gemini for coding assistance. Enterprise AI agents are also beginning to automate software deployment, information technology operations and documentation retrieval.
If these systems access hallucinated resources without verification, organisations risk introducing malicious code into their own environments.
According to Unit 42, researchers identified more than 13,000 malicious URLs associated with hallucinated domains across 913 major brands, while nearly 250,000 hallucinated domains remained available for registration.
This suggests attackers have access to a large pool of AI-generated assets that could potentially be weaponised.
AI behaviour is becoming a new cyberattack surface
Cybersecurity has traditionally focused on vulnerabilities in software, operating systems and human behaviour.
Phantom Squatting and HalluSquatting suggest attackers are beginning to exploit something different: the predictable behaviour of AI models themselves.
As organisations move towards agentic AI systems capable of taking autonomous action, hallucinations are no longer merely producing incorrect answers. They are creating new pathways of trust that attackers can hijack.
For enterprises adopting AI-led workflows, the lesson is increasingly clear: verifying AI-generated domains, software packages and external resources may soon become as important as checking suspicious emails or downloaded files.
 "What is AI squatting? An emerging cyber threat targeting AI hallucinations")
