Backup and recovery have transformed into comprehensive data protection strategies, focusing on security and cyber recovery with a strong emphasis on business resilience. The scope of data protection has expanded significantly, particularly regarding the dual use of artificial intelligence (AI): the threats posed by attackers leveraging AI and the defensive strategies that can be implemented using the same technology.
In recent research conducted in collaboration with Qualitate, we delved into these complexities while addressing a less frequently examined risk: What happens when AI deployed within an organization makes a mistake?
Our findings reveal that confidence in recovery surpasses actual capabilities. This study highlights the current inadequacy in cyber resilience and AI security practices. AI systems are inducing additional pressures on IT professionals to secure data, trace an agent’s actions, interpret its objectives, and reliably restore business operations to a trusted state. The risks associated with an organization’s own AI deployments are increasingly complicating the challenge of maintaining resilience.
Welcome to Breaking Analysis No. 333: AI Cyber Resilience and the Trust Gap. In this episode, we revisit Krista Case, who recently conducted extensive interviews with 25 practitioners specializing in security and cyber resilience as part of theCUBE Research’s Cybersecurity & Resilience Summit.
Study Objectives and Scope
Information technology and security professionals are facing two escalating pressures: 1) shielding the organization from adversaries employing AI, and 2) implementing appropriate guardrails as their internal teams introduce AI projects into operational workflows. Our study investigated how organizations are developing security and resilience strategies on both fronts, with a particular focus on the risks introduced by their own AI initiatives.
We examined how organizations manage agent identity and permissions, monitoring, and incident response preparedness in conjunction with recovery planning. A key consideration was the ramifications of an agent executing an incorrect action with valid permissions. This scenario raises three critical questions:
- Reconstruction: Can the organization determine the actions taken by the agent?
- Downstream Effects: Can it assess the impacts on the affected systems and business processes?
- Recovery: Can it restore those impacted business processes?
These inquiries guided our investigation into how organizations are strategizing for AI to play a more active role in operations. As agents gain authority to act, it is imperative for practitioners to evaluate how an error can proliferate and what recovery will entail.
Ultimately, the study assessed organizations’ readiness to secure their AI deployments and recover from incidents caused by authorized agents, in parallel with the ongoing necessity to protect against adversarial AI.
Confidence Outpacing Tested Recovery
Our research identified only four out of 25 organizations as having conducted any form of exercise relating to an AI or agent incident, including partial drills. This finding highlights a significant trust gap at the core of our study: confidence in recovery typically exceeds the extent to which organizations have rigorously tested these capabilities.
These conclusions stem from a small yet in-depth set of interviews conducted with our partner Qualitate. The discussions provided insights into what organizations have actually tested, the additional security implications their AI systems introduce, and what is required to safely resume impacted business processes. One healthcare practitioner articulated the disconnect between confidence and preparedness as follows:
No, we have not completed any tabletop exercises concerning AI agents or similar attacks. We’re highly confident because these scenarios are akin to non-AI incidents we have dealt with previously.
— Senior Manager, Information Security Operations, Healthcare and Biotech
Across the interviews, those displaying confidence often relied heavily on existing ransomware and incident response protocols. While experience with non-AI incidents understandably fosters a sense of trust, it raises concerns that organizations may unknowingly equate handling an agent’s incorrect action with addressing behaviors by insiders or external attackers. We contend that current playbooks may not encompass all potential scenarios.
Thus, this assumption necessitates testing against the unique challenges presented by agent incidents. The interviews revealed a mixed picture regarding the level of genuine validation:
- Partial Exercises: The four situations mentioned pertain to limited exercises; thus, this figure is not indicative of mature trust and should not be interpreted as comprehensive demonstrations of business recovery;
- Individual Controls: Other respondents described testing specific elements such as permissions and containment. While these assessments yield valuable information about certain controls, broader recovery questions remain unaddressed.
Complete business recovery demands an organization reconstruct the sequence of authorized actions, identify incorrect outcomes, rectify downstream effects, and cautiously resume the affected business processes. Validating that the entire sequence is effective presents a considerable challenge.
The speed, scale, and interconnected nature of agent activities amplifies this complexity. Actions can influence multiple systems and processes, resulting in repercussions that existing playbooks must be capable of addressing. An alarming risk lies in organizations holding onto a false sense of confidence, presuming that their experience with conventional non-AI incidents will translate seamlessly. Testing needs to determine how resilient these capabilities are under conditions that agents can produce, which may be unprecedented.
In summary: Evidence of validated, comprehensive business recovery from an agent incident remains scant in our study. We advocate that organizations exercise their existing playbooks using realistic agent scenarios, with safe business resumption as the benchmark for success.
Validating Tools Across the Full Agent Workflow
Our interviews unveiled critical insights about tools, maturity, and the procedural enhancements required to verify that existing capabilities effectively support agentic AI. Two key findings demonstrate the limited capabilities for detection and reconstruction within the organizations surveyed:
- Only two out of 25 respondents reported broad recognition of anomalous or unintended agent behavior;
- Only three out of 25 respondents indicated comprehensive monitoring and reconstruction of agent activities.
A healthcare practitioner’s evaluation of implemented controls fortifies the notion of a trust gap:
No, we haven’t tested or validated the controls we’ve implemented.
— Senior Manager, Information Security Operations, Healthcare and Biotech
Organizations typically possess identity systems, application logs, data protection tools, and incident response protocols. They are attempting to extend these frameworks to encompass AI. The central challenge is validating that the tools can accurately reconstruct activities across a complete agent workflow.
Agents interact with various applications, leverage tools, and autonomously trigger subsequent actions. Each system may log its events independently, failing to provide a cohesive record of the entire workflow. Effective incident response relies on correlating these records to ascertain which agent acted and under what authority, subsequently tracing the tools employed and the resulting changes.
This situation is reminiscent of the early days of disaster recovery when organizations would test failover protocols but hesitated to execute failback due to fear of unintended consequences. A similar reluctance now exists to address questions that might reveal gaps in readiness or potentially result in downstream complications. Concerns persist regarding whether the interconnected tools and processes will remain robust under realistic agent incident circumstances.
In summary: Organizations must thoroughly exercise the complete agent workflow in practical conditions, encompassing plausible worst-case scenarios, and validate that their tools and procedures can accurately reconstruct activities and support a comprehensive response.
Restoring a Trusted Business Service
Our exploration of the field interviews underscores both the technical and business aspects integral to successful recovery. An AI agent may legitimately alter a record or send a message erroneously. Although a backup may function correctly and restored systems may appear operational, the business might still lack the ability to resume activities securely.
As noted by StorageTek veteran Fred Moore, who served as a mentor to Dave Vellante, “Backup is one thing, recovery is everything.” The scope of recovery continues to broaden as organizations grapple with the consequences of agents interacting within business systems.
A director of information security within a media and entertainment company articulated the challenge succinctly:
We could restore our systems, we could restore within our recovery objectives, but that doesn’t necessarily guarantee that it’s safe. The bottleneck here would be trust.
— Director of Information Security, Media and Entertainment
Recovery Point Objectives (RPO) and Recovery Time Objectives (RTO) serve as conventional business metrics. However, business users require more substantial indicators that convey what a trusted, operational service entails and how this condition can be validated. This process involves determining who is necessary for recovery and who must authorize the safe resumption of operations.
This directive must be established from the top down and verified from the bottom up. Business leaders should define the requirements, while IT and security teams work collaboratively to provide the underlying technology and support the desired outcomes. This responsibility transcends the historic bounds of the data protection team, which focused solely on recovering data and systems.
Incorrect transactions, human errors, and failures that propagate through systems have long been issues. Agentic AI introduces two additional complications:
- Speed and Scale: Erroneous actions can spread rapidly across the operational environment, heightening the incident’s scope;
- Activity Tracing: Teams must discern the agent’s access and intended tasks, ascertain which actions were executed, and evaluate the downstream effects.
This necessitates a thorough comprehension of what went awry and the steps needed for recovery. The organization must successfully trace the agent’s activities and measure the resulting business state against established safe operation standards.
In conclusion: Recovery necessitates explicit criteria for validating that a business service can be trusted and a clear delineation of responsibility for approving its reactivation. Leadership must set this direction, while IT and security provide the necessary capabilities to execute it effectively.
Human Oversight Can Mask a Recovery Gap
Human oversight plays a crucial role in helping organizations restrict the capabilities of agents. However, resilience must also consider occurrences where human intervention fails to detect an erroneous action or falls behind the pace at which agents operate. Two statements from the same CIO encapsulate this matter:
So the given scenario will be detected by human beings before the AI agents can take the actions.
If it does occur, currently we do not have the capability to roll back the agent’s action.
— Chief Information Officer, Real Estate and Construction
The first statement reflects confidence that human supervision will identify potential issues. Conversely, the second recognizes the lack of an “undo” option should human oversight falter. This exemplifies why approvals and recovery testing must progress simultaneously. Relying solely on human oversight can create a false sense of security and shift focus away from critical recovery details necessary for addressing mistakes that escape detection.
Our field study also revealed how organizations assess the trade-offs surrounding agent autonomy, generally opting to limit what their agents can do. Such decisions become vital when agents are given authority to execute remedial actions during incidents or recovery processes.
The level of autonomy assigned should correspond with the action’s associated risk. Prior to empowering an agent to approve or implement a remedial action, organizations must comprehend the ramifications of an erroneous decision. This necessitates adopting a tiered approach based on two factors:
- Action Risk: What is the potential impact if the agent makes a mistake, and what level of risk is the organization willing to accept?
- Guardrails: What degree of authority should the agent possess, and what constraints must be imposed on its ability to operate?
We anticipate that human approval processes will increasingly become bottlenecks in the coming six to twelve months. Organizations should establish these boundaries as they begin to entrust agents with more corrective responsibilities.
To summarize: Human oversight and recovery testing must evolve in tandem. As organizations delegate more authority to agents, they must account for potential errors, the limitations of human review, and the risk associated with each action.
Rehearsing Agent Recovery in a Replica Environment
A retail practitioner proposed a viable approach to rehearsing agent incidents, which we view as a best practice. Specifically, this organization replays the actions of the agent in a replica environment, investigating the root cause and utilizing granular snapshots to analyze the transactional ramifications. The respondent specifically acknowledged that this method has been trialed in a sandbox, articulating:
Yes, if the agent engaged multiple systems, I replicate those actions within my sandbox environment, which mirrors all of the systems.
— Director of Information Technology, Retail
The strength of this strategy lies in its ability to recreate the sequence of events and assess their impact across various systems. Our evaluation of what such an exercise should validate is anchored in three outcomes:
- A Trace of the Incident: Reconstruct what transpired across the affected systems;
- A Defined Scope for Remediation: Identify the root cause and impacted transactions;
- A Path Back to a Trusted Business State: Outline how the consequences can be addressed to facilitate recovery.
We consider these to be minimum viable expectations for the evidence that such an exercise should generate.
We utilize the term digital twin in this context as a broad analogy for this methodology. A more comprehensive digital twin would include a deeper representation of business processes, dependencies, and states. We believe the industry still has significant work ahead in this area, especially as organizations increasingly rely on AI for core functions. Additionally, this will elevate the complexity of recovery since the operational state of the business will be further convoluted as agents contribute more significantly to workflows.
The respondent also acknowledged a limitation, stating that increased transaction volumes might necessitate additional reconciliation time. Replaying actions and pinpointing affected transactions yields valuable insights, although the duration required to resolve the outcomes remains an open matter. More sophisticated simulations of business environments could assist enterprises in testing agent behavior and preparing for potential failures within production settings.
In essence, testing the recovery sequence under varying scenarios within a digital twin before moving the process into an operational environment is critical. Furthermore, this “red teaming” effort should be a continuous exercise—extending the notion of continuous data protection into a firm’s operational status through more granular snapshots.
In summary: This example presents a practical method for replicating agent actions, scrutinizing their outcomes, and rehearsing recovery. Its value further emphasizes the necessity for richer representations of the business processes affected by these agents.
More Autonomy Requires Demonstrated Recovery
For chief information security officers, we recommend conducting an exercise based on an agent incident affecting one important workflow currently in production. Select a workflow where an agent has permission to act across multiple systems, and develop the exercise to incorporate incorrect actions taken with valid credentials and permissions.
The simulation should also encompass a scenario necessitating human approval, establishing the parameters surrounding where autonomy is acceptable and where human oversight remains essential.
The practical steps include:
- Establish What the Agent Can Reach: Identify the systems it can access and the actions it is permitted to perform;
- Demonstrate Containment: Test the feasibility of stopping the agent, revoking its access, and preventing further tool usage. These “kill switch” mechanisms are vital in limiting incident scope;
- Ensure that the conditions triggering the kill switch are defined and managed by the organization.
- Reconstruct and Correct: Verify what the agent executed, whether consequences can be rectified, and identify any outstanding exceptions or residual impacts;
- Demonstrate Safe Resumption: Provide evidence that the affected business process is prepared to restart and designate who has the authority to approve this decision.
Validation by the business owner is crucial. Security teams can verify that an incident has been contained and present evidence of the events that occurred. The business owner must assess whether the resulting business state is acceptable, accounting for both the repercussions of the agent’s actions and any necessary remediation work that still needs to be addressed.
The exercise should document what was confirmed, what remains untested, and which teams are accountable for addressing the remaining gaps. This documentation should also guide decisions concerning the level of authority afforded to agents. Organizations must grasp their capacity for reversing or rectifying actions and recovering from their impacts as they consider increasing agent autonomy.
Build Resilience into Agent Deployments from the Start
Discussions at the AI Cybersecurity and Resilience Summit underscored the themes emerging from our field study. Interactions with vendors, practitioners, and researchers reaffirmed the necessity to integrate resilience and recovery capabilities into agent deployments from the outset. Demonstrating that these capabilities function effectively will be critical as organizations operationalize agentic workloads.
This challenge involves preserving and restoring a trustworthy business state when an agent yields incorrect results despite possessing legitimate credentials. Organizations must comprehensively understand how these outcomes impact their operations and the required remedial measures.
Existing investments in data protection infrastructure, security, and recovery are imperative. The road ahead involves identifying gaps that surface in light of the new demands and pressures introduced by enterprise AI. This endeavor will require collaboration among internal teams and with external vendors to pinpoint areas where capabilities must develop.
In summary: We believe that trust in enterprise AI hinges on documented proof that organizations can comprehend what agents have executed, halt them, reconstruct their actions, and recover to a sound business state. Our research indicates that organizations have significant work ahead to substantiate that their existing security and recovery capabilities can satisfy this requirement.
Here’s the full video analysis: