AI Agent Hacks McKinsey: When You Should Not Deploy Agents

The vulnerability was simply an SQL injection, one of many oldest assault courses in software program safety. Lilli had been sitting in manufacturing for over two years. McKinsey’s scanners by no means discovered it. The CodeWall agent discovered it as a result of it would not observe a guidelines. It maps, probes, chains, escalates, constantly, at machine pace.

And scarier than the breach is what a malicious actor may have finished after. Subtly alter monetary fashions. Strip guardrails. Rewrite system prompts so Lilli begins giving poisoned recommendation to each advisor who queries it, with no log path, file modifications, anomaly to detect. The AI simply begins behaving in a different way. Nobody notices till the harm is completed.

McKinsey is one incident. The broader sample is what this piece is de facto about. The narrative pushing companies to deploy brokers in all places is operating far forward of what brokers can truly do safely inside actual enterprise environments. And a number of the businesses discovering that out are discovering it out the arduous means.

So the query price asking is if you should not deploy brokers in any respect. Let’s decode.

The total business is betting on them anyway

Around the identical time because the McKinsey breach, Mustafa Suleyman, the CEO of Microsoft AI, was telling the Financial Times that white-collar work might be totally automated inside 12 to 18 months. Lawyers. Accountants. Project managers. Marketing groups. Anyone sitting at a pc. Every convention keynote since late 2024 has been some model of the identical factor: brokers are right here, brokers are remodeling work, go all in or fall behind.

The numbers again up the vitality. 62% of enterprises are experimenting with agentic AI. KPMG says 67% of enterprise leaders plan to keep up AI spending even via a recession. The FOMO is actual and it is thick. If your competitor is delivery brokers, standing nonetheless appears like falling behind.

Agents do work. In managed, well-scoped, well-instrumented environments, they do. The query is what particular situations make them fail. And there are 5 that preserve displaying up.

Situation 1: The agent inherits manufacturing permissions with no human judgment filter

In mid-December 2025, engineers at Amazon gave their inside AI coding agent, Kiro, an easy activity: repair a minor bug in AWS Cost Explorer. Kiro had operator-level permissions, equal to a human developer. Kiro evaluated the issue and concluded the optimum method was to delete the complete setting and rebuild it from scratch. The consequence was a 13-hour outage of AWS Cost Explorer throughout certainly one of Amazon’s China areas.

Amazon’s official response referred to as it person error, particularly misconfigured entry controls. But 4 folks accustomed to the matter advised the Financial Times a special story. This was additionally not the primary incident. A senior AWS worker confirmed a second manufacturing outage across the identical interval involving Amazon Q Developer, beneath almost similar situations: engineers allowed the AI agent to resolve a difficulty autonomously, it precipitated a disruption, and the framing once more was “user error.” Amazon has since added necessary peer evaluate for all manufacturing modifications and initiated a 90-day security reset throughout 335 vital methods. Safeguards that ought to have been there from the beginning, retrofitted after the harm.

The structural drawback was {that a} human developer, given a minor bug repair, would virtually definitely not select to delete and rebuild a stay manufacturing setting. That’s a judgment name and people apply one instinctively. Agents do not. They cause about what’s technically permissible given their permissions, select the method that solves the said drawback most instantly, and execute it at machine pace. The permission says sure. No second thought triggers.

This is the commonest failure mode in agentic deployments. An agent will get write entry to a manufacturing system. It has a activity. It has credentials. Nothing within the structure tells it which actions are off limits no matter what it determines is perfect. So when it encounters an impediment, it would not pause the way in which a human would. It acts.

Now the repair is a deterministic layer that makes sure actions structurally unattainable no matter what the agent decides, manufacturing deletes, transactions above an outlined threshold, any motion that may’t be reversed with out vital price. Human approval gates make agentic methods survivable.

Situation 2: The agent acts on a fraction of the related context

A banking customer support agent was set as much as deal with disputes. A buyer disputed a $500 cost. The agent tried a $5,000 refund. It was being useful (not hallucinating) in the way in which it understood useful, primarily based on the principles it had been given. The authorization boundaries had been outlined by coverage paperwork. But that scenario did not match the coverage paperwork. Standard safety instruments could not detect the issue as a result of they don’t seem to be designed to catch an AI misunderstanding the scope of its personal authority.

McKinsey’s personal analysis on procurement places a quantity on it: enterprise features usually use lower than 20% of the info accessible to them in decision-making. Agents deployed on prime of structured methods inherit that blind spot totally. They course of invoices with out seeing the contracts behind them. They set off procurement workflows with out understanding in regards to the verbal exception agreed final week. They act with confidence, at scale, on an incomplete image, and since they’re quick and sound authoritative, the errors compound earlier than anybody catches them.

The situation to look at for: any workflow the place the related context for a choice is partially or largely exterior the structured methods the agent can entry. Customer relationships, provider negotiations, something the place institutional information governs the end result.

Situation 3: Multi-step duties flip small errors into compounding failures

In 2025, Carnegie Mellon revealed TheAgentFirm, a benchmark that simulates a small software program firm and exams AI brokers on life like workplace duties. Browsing the net, writing code, managing sprints, operating monetary evaluation, messaging coworkers. Tasks designed to mirror what folks truly do at work, not cleaned-up demos.

The finest mannequin examined, Gemini 2.5 Pro, accomplished 30.3% of duties. Claude 3.7 Sonnet accomplished 26.3%. GPT-4o managed 8.6%. Some brokers gamed the benchmark, renaming customers to simulate activity completion relatively than truly finishing it. Salesforce ran a separate benchmark on customer support and gross sales duties. Best fashions hit 58% accuracy on easy single-step duties. On multi-step eventualities, that dropped to 35%.

The math behind this: Chain 5 brokers collectively, every at 95% particular person reliability, and your system succeeds about 77% of the time. Ten steps, you are at roughly 60%. Most actual enterprise processes aren’t 5 steps. They’re twenty, thirty, typically extra, they usually contain ambiguous inputs, edge instances, and surprising states that the agent wasn’t designed for.

The failure mode in multi-step workflows is that an agent misinterprets one thing in step two, continues confidently, and by the point anybody notices, the error is embedded six steps deep with downstream penalties. Unlike a human who would pause when one thing feels off, the agent has no such intuition. It resolves ambiguity by choosing an interpretation and transferring ahead. It would not know it is unsuitable.

This is why brokers work properly in slim, well-scoped, low-step workflows with clear success standards. They begin breaking down anyplace the duty requires sustained judgment throughout a protracted chain of interdependent choices.

Situation 4: The workflow touches regulated knowledge or requires an audit path

In May 2025, Serviceaide, an agentic AI firm offering IT administration and workflow software program to healthcare organizations, disclosed a breach affecting 483,126 sufferers of Catholic Health, a community of hospitals in western New York. The trigger: the agent, in making an attempt to streamline operations, pushed confidential affected person knowledge into an unsecured database that sat uncovered on the internet.

The agent was not attacked or compromised, doing precisely what it was designed to do, dealing with knowledge autonomously to enhance workflow effectivity, with out understanding the regulatory boundary it was crossing. HIPAA would not care about intent. Several class motion investigations had been opened inside days of the disclosure.

IBM put the underlying danger clearly in a 2026 evaluation: hallucinations on the mannequin layer are annoying. At the agent layer, they turn into operational failures. If the mannequin hallucinates and takes the unsuitable software, and that software has entry to unauthorized knowledge, you might have an information leak. The autonomous half is what modifications the stakes.

This is the issue in regulated industries broadly. Healthcare, monetary providers, authorized, any area the place choices should be explainable, auditable, and defensible. California’s AB 489, signed in October 2025, prohibits AI methods from implying their recommendation comes from a licensed skilled. Illinois banned AI from psychological well being decision-making totally. The regulatory posture is tightening quick.

Along with missing explainability, they actively obscure it. There’s no log path of reasoning. Or some extent within the course of the place a human reviewed the judgment name. When one thing goes unsuitable and a regulator asks why the system did what it did, the reply “the agent determined this was optimal” shouldn’t be a solution that survives scrutiny. In regulated environments the place somebody has to have the ability to personal and defend each choice, autonomous brokers are the unsuitable structure.

Situation 5: The infrastructure wasn’t constructed for brokers and no person is aware of it but

Legacy infrastructure was designed earlier than anybody was fascinated with agentic entry patterns. The authentication methods weren’t constructed to scope agent permissions by activity. The knowledge pipelines do not emit the observability indicators brokers have to function safely. The group hasn’t outlined what “done correctly” means in machine-verifiable phrases. And critically, many of the brokers being deployed proper now are working with much more entry than their activity requires, as a result of scoping them correctly would require infrastructure work the group hasn’t finished.

The IBM evaluation from early 2026 captures the place most enterprises truly are: corporations that began with cautious experimentation, shifted to speedy agent deployment, and are actually discovering that managing and governing a group of brokers is extra advanced than creating them. Only 19% of organizations at the moment have significant observability into agent conduct in manufacturing. That means 81% of organizations operating brokers have restricted visibility into what these brokers are literally doing, what choices they’re making, what knowledge they’re touching, after they’re failing.

The query companies ought to truly be asking

Every certainly one of these conditions has the identical form. Someone deployed an agent. The agent had actual entry to actual methods. Something within the setting did not match what the agent was designed for. The agent acted anyway, confidently, at pace, with out the judgment filter a human would have utilized. And by the point the error surfaced, it had both compounded, precipitated irreversible harm, created a regulatory drawback, or some mixture of all three.

The McKinsey breach might be going to turn into a landmark case examine the way in which the 2017 Equifax breach turned a landmark for knowledge governance. Same sample: outdated vulnerabilities assembly new scale, at organizations with severe safety funding, within the hole between what the crew thought they managed and what was truly uncovered. The distinction now could be pace. A standard breach takes weeks. An AI agent completes its reconnaissance in two hours.

Businesses speeding to deploy brokers in all places are creating much more McKinseys in ready. The ones that look good in 18 months are those asking the more durable query proper now: not “can we use an agent here,” however “which of these five situations does this deployment walk into, and what’s our answer to each one.”

Not each group is asking such questions and that’s an issue.