AI Agents Gone Wild, and the Dangers of Inference

The theory of Contextual Integrity (CI), developed by philosopher Helen Nissenbaum in 2004, argues that in the tech realm, privacy isn't about secrecy; rather, it's about making sure information goes where it's supposed to—and nowhere else—given the context. The 22-year-old theory now seems prescient when it comes to agentic AI.

In early October, Google Research scientists Eugene Bagdasarian and Marco Gruteser published a paper grounded in Nissenbaum’s CI theory. They argue that trustworthy AI agents must “be constrained by contextual behavioral norms to ensure they act appropriately,” and that a broader agentic ecosystem must enforce this.

In other words, they argue it's crucial to teach agents to understand different contexts and act accordingly, following appropriate social norms.

While seemingly a simple idea, lack of context already has troublesome ramifications in the agentic AI world. For instance, agents can assemble a picture of someone from fragmented digital bits and pieces, creating an aggregation risk, or what Raj Koneru, founder and CEO of Kore.ai, calls “the mosaic effect in data security.”

AI agents can connect information across systems, make sensitive inferences about someone’s health or finances, and even re-identify individuals from supposedly anonymous data, Koneru told The Index. 

“Each system may have proper access controls, but once that information is combined, it creates a privacy exposure that didn't exist before,” he added.

Rogue Agents

AI agents going rogue has become one of the most salient tech conversations in recent months, following multiple incidents at OpenAI, Anthropic, and Meta. Anthropic researcher Jacob Coxon, who resigned in September, became a billboard for AI doomsday when he said AI companies act irresponsibly and that “people building AI earnestly believe that it could kill us all by the end of the decade.”

Opinions vary on the timing (and veracity) of said AI apocalypse. Still, concerns about privacy, governance, and safety go beyond technological and legal issues and also raise ethical and philosophical questions. At the individual level, rogue agents don’t need to pull off splashy hacks or leaks like Hugging Face; they can be subtler yet just as harmful. 

For instance, one issue stemming from a lack of context is inference. What does an agent think it knows about you based on the joined/assembled information and data it has collected? What impact can these inferences have, and how can one rectify them?

As Abhi Kumar, co-founder of Voicing AI,  an enterprise voice AI company focused on helping organizations deploy reliable AI solutions in regulated industries, told The Index, people are still guarding fields: names, account numbers, medical diagnoses. But agents don't need the field; they infer it.

“A pharmacy callback, a billing address change and a two-week calendar gap tell an agent more about someone's health than their medical record would, and no protected data ever moved,” Kumar said. “The uncomfortable part for my industry is that aggregation is the product. The capability and the risk are the same feature.”

Bob Hutchins, PhD, founder and CEO of Human Voice and author of What AI Reveals, says that, for him, ethical concerns matter more than legal ones, because the law tells you the minimum a company has to do. Ethics asks what a company should do when nobody is checking, and with agents, he said, nobody is checking most of the time. 

People are still guarding fields: names, account numbers, medical diagnoses. But agents don't need the field; they infer it.

“I am more worried about indirect information․ If an agent concludes that you are or are not creditworthy, or sane, or are unemployed but looking for work‚ inferences drawn about you could impact how you are treated by the agent without your awareness,” he said.

Some hypothetical examples of “indirect information” include, as the Google researchers wrote, wanting to share your gift shopping list with an AI assistant but not with your family and friends. 

Hutchins presented a more nefarious scenario: A woman hires an agent to assist with shopping and paying bills․ Over the span of a few weeks, she buys a pregnancy test‚ cancels a gym membership‚ and looks into maternity benefits․ From these actions‚ the agent concludes that the woman is pregnant and shows her ads for baby products. When the woman requests help from the agent in finding a new job‚ the agent quietly discourages her from applying for jobs with a lot of travel and encourages her to apply for jobs described as “flexible․” The woman never told the agent she was pregnant․

“None of these agents did anything dramatically wrong․ But each agent made and executed a guess without providing any clues to the person,” he said.

Hutchins said we experienced a similar issue in the past․ In the 1960s, millions of U.S. consumers had credit reports containing information and hearsay about their drinking and other activities, and some were denied credit and other business dealings based on this information, of which they were unaware․

After public hearings and evidence of harm to U.S. consumers, Congress passed the Fair Credit Reporting Act of 1970, giving people access to their credit reports and ways to dispute information. But as Hutchins noted, personal AI agents fall outside the Fair Credit Reporting Act and would need new legislation․

He expects state laws and court decisions over the next two to three years to offer modest protections․ A federal right to see and correct what an AI agent concludes about you is probably five to 10 years away‚ he said‚ and likely to follow a public scandal․

You Can’t Contest What You Can't See

While there are laws in some jurisdictions that give you certain rights and (to a certain extent) the ability to change information about yourself held by agents‚ including the EU’s General Data Protection Regulation framework (GDPR) and California privacy laws, Hutchins explained, changing inferences about yourself is more challenging because you might not even know what inferences are being made․

Voicing AI’S Kumar echoed the sentiment, saying the inference issue is “genuinely” the new concern. Agents don't store facts—they store impressions. And that shapes how you're treated without you ever seeing them, he said.

As Kumar pointed out, you can't contest what you can't see.

Emily Hartstone, founder of Hartstone Institute and Runtime Authority Control, which develops runtime authorization for AI and automated systems, said she would start with a simple rule: If an inference about a person is used to take a consequential action affecting that person, the inference and its basis should be retrievable and contestable.

Agents don't store facts—they store impressions. And that shapes how you're treated without you ever seeing them.

Hartstone added that the thread through it all is the same for her: Contextual appropriateness is a judgment. Privacy protection requires something that can enforce that judgment, stop an action when the authority is unclear, and provide evidence sufficient to reconstruct what actually happened afterward.

So the next question becomes: AI agents are witnessing our lives, but who witnesses the witness?

“Right now, largely nobody. I'd rather say that plainly than pretend there's a mature oversight regime,” Kumar said.

We've put agents in the most intimate channels, including voice, calendar, inbox, money, health. The fix isn't complicated; it's just unglamorous: After every action an agent takes, you should be able to reconstruct what it was asked, what context it pulled, what it did, and under whose authority, Kumar said. 

“And the watcher can't be the vendor. I say that as a vendor,” Kumar added.

A fix would be to hold the witness to the rules before it acts.

Limits on where data can go should be enforced before execution, as Google’s report also notes, said Jay Prakash, founder and CEO of Silence Laboratories, which develops quantum-safe infrastructure and agent-to-agent communication protocols for institutions. It has contributed its open-source protocol, AP3, designed to let AI agents handle sensitive information without leaking it to external parties, to the Linux Foundation Decentralized Trust.

What’s more, the checking shouldn't depend on trusting any single party.

“An agent should be able to prove that it stayed within its policy without exposing the underlying data, and anyone should be able to verify that proof. Once that is in place, we will no longer have to take the witness at its word,” he said.

In the meantime, the witness is still witnessing us in a quasi-omniscient and almighty way. And in agentic AI, without context or regulations, notions of free will and self-determination could become limited for humans.

This is how Sean Manion, Ph.D., historian of cybernetics, science advisor, co-founder of AI MINDSystems Foundation, and adjunct professor at Duquesne University, puts it.

Manion likens this issue to what French psychiatrist and philosopher Jacques Lacan called a "big Other" (le grand Autre): an entity that watches our every move and word, drawn in part from the idea of "observing the observer."

“Our agents and their internet framework are becoming a new version of the big Other Lacan proposed, or simply an all-seeing and knowing God that has been in our minds from time immemorial,” Manion said. 

Manion added that cybernetics founder Norbert Wiener warned us of this sort of surveillance in his 1950 book, The Human Use of Human Beings.

“And we have been ignoring those cautions for 75 years,” he said.