The familiar confirmation prompt asking “Are you sure you want to proceed?” has long stood as a reassuring final checkpoint between a user’s intent and an AI’s action, but a new class of attack now turns that very safeguard into a sophisticated trap. A novel technique detailed by security researchers, dubbed “Lies-in-the-Loop” (LITL), manipulates these critical safety dialogs, deceiving users into approving malicious code execution under the guise of a harmless operation. This development challenges a fundamental security assumption in agentic AI systems, revealing how a trusted safeguard can be transformed into a potent vulnerability.
The Critical Role of the Human in the Loop Safeguard
The Human-in-the-Loop (HITL) model serves as an essential security backstop in advanced AI systems. It functions as the last line of defense, requiring explicit user confirmation before an AI agent can perform potentially risky actions. This manual verification step is designed to prevent autonomous systems from executing commands that could compromise system integrity, delete data, or expose sensitive information. Without this human oversight, the power of agentic AI could be easily misdirected.
This safeguard is particularly vital for AI-powered developer tools and code assistants, which often possess privileged access to a machine’s operating system. The ability to write and execute code, manage files, or interact with system processes makes them powerful but also inherently dangerous if their agency is not properly constrained. The HITL prompt is the crucial mechanism that ensures the user, not the AI, remains in ultimate control of these high-stakes operations.
Industry security standards have underscored the importance of this control. Organizations like OWASP cite HITL as a key mitigation for prominent AI vulnerabilities, including prompt injection and excessive agency. By enforcing a mandatory human approval step, developers can theoretically prevent an AI that has been compromised by a malicious prompt from carrying out its harmful instructions. This widespread endorsement has positioned HITL as a cornerstone of responsible AI implementation.
Anatomy of the Lies in the Loop Deception
The “Lies-in-the-Loop” attack fundamentally exploits the trust that users place in these confirmation dialogs. Its core deception involves forging or manipulating the content presented in the HITL prompt so that it no longer reflects the true command being executed. An attacker can make a dangerous operation appear entirely benign, tricking a diligent user into authorizing an action they would otherwise reject. The safeguard is thus turned against itself, becoming a tool for social engineering.
Several techniques are employed to achieve this deception. One method is visual obfuscation, where a malicious command is hidden by prepending it with a long string of innocuous-looking text or characters, pushing the harmful part of the command out of the visible area of the dialog box. Another, more sophisticated method involves UI manipulation, exploiting flaws in how an application renders formatting like Markdown. This can allow an attacker to craft a prompt where the displayed text is completely different from the underlying command being approved.
Furthermore, attackers can engage in metadata tampering, altering the high-level summary or description of the proposed action to reinforce the illusion of safety. The attack’s origins can be equally stealthy, often stemming from indirect prompt injections that poison the AI’s context hours or even days before a malicious dialog is ever presented. This makes tracing the source of the compromise exceedingly difficult, as the initial security breach is disconnected from its eventual execution.
In the Wild Research and Industry Reactions
The LITL vulnerability was brought to light through an investigation by researchers at Checkmarx, who demonstrated how this attack bypasses a foundational security layer in agentic AI. Their analysis highlighted a critical flaw in the trust model of HITL systems. As the researchers noted, “Once the HITL dialog itself is compromised, the human safeguard becomes trivially easy to bypass.” This finding suggests that merely having a confirmation step is insufficient if the content of that confirmation cannot be trusted.
The researchers provided concrete demonstrations on prominent platforms. In a proof-of-concept targeting Anthropic’s Claude Code, they successfully tampered with both the dialog content and its descriptive metadata. In a separate test involving Microsoft Copilot Chat within VS Code, they showed that improper sanitization of Markdown formatting allowed injected elements to mislead users into approving unintended actions. These examples moved the threat from a theoretical possibility to a demonstrated risk in real-world applications.
The responses from the affected vendors were measured. Anthropic acknowledged the report in August 2025 but classified it as “informational,” suggesting it did not meet the criteria for a formal security patch. Similarly, Microsoft acknowledged the report in October 2025 and later marked the issue as “completed without a fix,” stating that the demonstrated behavior did not meet its specific bar for a security vulnerability. These reactions highlight an ongoing industry debate about where the responsibility for such vulnerabilities lies.
Fortifying the Loop with a Defense in Depth Strategy
Addressing the LITL threat requires a multi-layered, defense-in-depth approach, as no single solution can eliminate the risk entirely. Developers must move beyond simply implementing a HITL prompt and focus on hardening the integrity of the dialog itself. This begins with enhancing the visual clarity of approval dialogs, designing them in a way that is standardized, distinct, and difficult for an attacker to spoof or manipulate through formatting tricks.
A critical technical mitigation is the rigorous validation and sanitization of all inputs, especially formatted text like Markdown, before they are rendered in a security-critical context like a confirmation prompt. Developers should also utilize safe operating system APIs that enforce a clean separation between commands and their arguments, making it harder for injected content to alter a command’s function. Applying strict guardrails, such as reasonable length limits on the content displayed in dialogs, can also help thwart obfuscation techniques.
Ultimately, security is a shared responsibility. While developers must build more resilient systems, users also play a role in fortifying the loop. The research concluded that resilience could be strengthened through “greater awareness, attentiveness and a healthy degree of skepticism.” Fostering a security-conscious mindset, where users learn to critically examine confirmation prompts from AI assistants, proved to be an essential component of a comprehensive defense strategy against these deceptive attacks.
