Behind every AI agent is a human who delegated authority to it, and ultimately an organisation needs to know who owns that agent and what happens when it operates outside its boundaries. That’s what accountability with these tools has to be built on, says Mimecast CEO Ranjan Singh.
In many ways, an agent is the ultimate insider risk. Except that agents working continuously and at scale across an organisation multiplies that risk beyond anything we’ve seen before. Singh is of the mind that every leadership team should know what’s running in their environment, what it’s connected to, and what decisions it’s influencing. “It’s no good asking what went wrong after the fact,” he says.
Access deeper industry intelligence
Experience unmatched clarity with a single platform that combines unique data, AI, and human expertise.
“None of this stays abstract for long, because accountability becomes personal,” warns Singh who doesn’t think leaders should be let off the hook. “Finding out about a genuine risk only after it’s become a headline is a failure of oversight, not of technology,” he says, adding: “Ultimately, leadership accountability needs to be proactive and tangible—from something that’s assumed to something leaders have to actively demonstrate. The leaders who take the first step with visibility before they’re forced, are the ones that will be seen as credible the next time an incident happens.”
While executives are not typically held individually accountable, individuals within organisations are increasingly being expected to take responsibility when things go wrong. Research published by cyber-resilience company Veeam found that AI anxiety is affecting corporate leadership with 40% of senior executives surveyed fearing personal liability, and 39% reporting increased board scrutiny, in addition to 37% reporting increased personal stress.
Is corporate accountability shared or personal?
Mika Yamamoto, Veeam’s chief product and technology officer says the anxiety is best addressed with a good risk mitigation strategy. “How I see it evolving is I think that organisations, whether it’s the C-suite or boards or people within companies—because people within companies are the ones making day-to-day decisions on behalf of their company about their organisation’s data—need to be continually educating their employee base, their partner base, their boards, their C-suites, to understand how their organisation is using data, how that data is being governed, how policies are being set, to make sure that they’re putting controls in place,” says Yamamoto.
Of course, boards have fiduciary and risk oversight responsibilities. “But I do think it’s shifting to also look at who’s accountable within the organisation,” Yamamoto says. In her view, boards, organisations and relevant individuals all have a role in accountability, regardless of legal liability.
But large organisations have a tendency to diffuse liability and there may be a justified concern that a shared accountability approach will result in no one being accountable. However, assigning accountability to a specific function within the enterprise is notoriously difficult as every organisation will be structured differently. That’s why HCL Tech’s CISO Adam Currie says accountability is usually a shared ownership model. While security teams are traditionally accountable for setting standards and guardrails and ensuring compliance, technology owners need to ensure that they’re building, producing, managing and operating technology within those guardrails, according to Currie.
And crucially, if something fails, accountability largely rests at the point of failure.
“You need to be able to identify the point of failure, and that’s where accountability lies,” says Currie. “If it was a lack of a security policy or security defined security control or guardrail that was not defined appropriately, then that would be obviously a security or CISO level of accountability. If it was a failure to adhere to defined security policies and guardrails and tooling, etc., then that would be the CIO or CTO or whomever is responsible for the technology where the AI was operating, from a system owner accountability perspective,” explains Currie.
Currie reiterates that every organisation is different and therefore identifying the point of failure will need a tailored approach. “It depends on the complexity of your environment from a technical landscape perspective. It depends on the complexity of the process that is being run, the action that the agent-based AI is taking,” he says.
“If you follow standard principles around least privileged, zero trust, you’re able to mitigate the blast radius of a potentially rogue AI agent, whether that is through a misconfiguration, a threat, a malicious actor that’s going in and injecting or changing the parameters of the agent unnoticed or unauthorised.
“Or is it a failure, or the fact that the agent, based off the learning model, is being fed information and didn’t have the appropriate guardrails in place, and it made its own determination, hallucinating, or creating actions that maybe it shouldn’t have been,” adds Currie.
Observability key to mitigating AI risk
Once a breach has taken place, forensics is a key part of any ongoing security framework, says Snowflake’s EVP of product Christian Kleinerman. “It is part of that preparedness around safeguards and controls to know what happened when something goes wrong. If you look at regulated companies, most of them insist on auditability as a core capability of a platform and of how they implement solutions, because you want to know what happened,” says Kleinerman.
Businesses are therefore demanding more observability to address AI risk. And this was the strategy behind Snowflake’s acquisition in February of, AI driven observability and engineering troubleshooting startup Observe. “The interest there is super high because of agent observability. People want to be able to record what happened,” says Kleinerman.
“So, the mechanisms for you to debug or troubleshoot how something happened, when it went wrong, do exist. The question is whether every organisation is deploying those solutions. For sure, not everyone is as sophisticated as the highly regulated firms,” adds Kleinerman.
Steve Tack, Dynatrace’s, chief product officer agrees that observability is one of the best risk defences. “I don’t know whether liability rests with the frontier models or the people building with them, but observability at least gives you insight into how to govern and control the system,” he says.
“With the right observability, AI engineers get a real-time view in production of what their agents are doing; guardrails address unsafe behaviour; and all of that helps an organisation scale and get value from AI,” adds Tack.
Tack describes rogue agent risk as “in the bloodstream right now” and features in the boardroom discussions he has been privy to. He also notes how quickly the discussion has moved on from a few months ago, when the topic was securing proprietary intellectual property, or making sure data was properly aligned to drive the right actions. “Now the questions include how to prevent unsafe behaviour and how to constrain an agent—with a circuit-breaker mindset if it does something does go wrong,” he says.
Too big to underwrite
Anthropic’s imminent IPO saw the company’s S-1 filing disclose that the company may be unable to buy cover for some AI liabilities, if at all. The filing warned that no single insurer, and potentially the entire industry, may have the capacity or expertise to cover a catastrophic AI incident.
When agents interact with other agents, transaction speed can mean no human oversight and therefore underwriting needs to reflect different kinds of deployments. Tim Hardcastle, CEO and co-founder of digital insurance platform Instanda, argues that with almost no loss history to price from, insurers will ask businesses to prove control: who authorised an agent, what limits it ran under and whether anyone could stop it.
Most businesses cannot answer those questions yet, and a review sign-off after the event will not count as oversight when an agent can act thousands of times before anyone looks, warns Hardcastle.
There’s no clarity yet around how existing cybersecurity laws can be applied when an AI agent goes awry. Developers typically address liability issues through their terms of service, which in turn exposes the businesses that use them. These businesses could face negligence claims, but, again, proving liability within a legal framework is another matter.
GlobalData’s Strategic Intelligence deep dive into agentic AI security recommends that organisations appoint a board level chief AI officer responsible for AI risk. A chief AI officer (CAIO) should report directly to the CEO, underscoring AI’s strategic business importance.
Understanding a business’ legal liability in case of a lawsuit over the actions of an AI agent would seem to be fairly high on the CAIO role’s list of necessary responsibilities, says GlobalData Strategic Intelligence principal analyst Dave Bicknell. His advice to corporate leaders is to assume that you will have that future lawsuit.
“So, rather than thinking about that when it happens, do some work at board level—or maybe sub-board level—creating a paper or position statement for the Board and the corporate legal team or outside counsel—to discuss on how you would tackle such a lawsuit.
“Clearly if you are using agents in your business now, that complaint of damage by agents that could impact customers, could happen at any time. So, there should be a plan in place now. As an organisation, you have a cyber resilience plan—or at least you should have one. What is your legal resilience plan in relation to the actions of agents?” says Bicknell.
As corporate leaders are increasingly giving autonomous systems more decision rights while retaining governance models built for human judgement, the mismatch constitutes a mounting risk, according to a recent World Economic Forum paper on board level governance of agentic AI. It’s clear that boardroom discussions around agentic AI risk are changing: it’s now a commercial issue just as much as a technical one. Inadequate controls and unmanaged risk can wreak reputational and financial havoc.
Safely delegating authority to AI agents, and demonstrating that reasonable steps are taken to prevent harm becomes increasingly important as AI automation advances faster than legal, governance and risk-management frameworks. Boards and senior technology executives cannot distance themselves from the actions of systems operating under their watch—so how they manage accountability is becoming increasingly urgent and important.
