You’re listening to “Task-State Governance: A Virtual Machine for Verifiable and Recoverable GUI Agent Execution,” by Wenkui Yang and colleagues. Published in arXiv on October 14, 2026.

Abstract.

∗Correspondence: the email address (Shuai Ma) and the email address (Feng Tian).

still be recovered. Plan-centric interfaces leave users supervising procedure, but users live with changes to application state. We introduce task-state governance, an interaction paradigm that represents a task as an editable contract linking desired outcomes, facts that must be preserved, and observed application state. We instantiate this paradigm in TaskVM, which projects live state into editable task controls, executes user edits as genuine interface actions, verifies outcomes through independent observation, and marks each action as reversible, compensatable, or irreversible. Across an automated contract benchmark, a controlled user study, and an open live-model exploration, the paradigm strengthened contract compliance and users’ recovery control and trust, while execution outcomes were held constant. Task contracts open a design space for governable agent interfaces.

CCS Concepts.

• Human-centered computing → Interactive systems and tools; User centered design; • Computing methodologies → Distributed artificial intelligence.

GUI agents, computer-use agents, task virtual machine, bidirectional binding, live projection, task governance, verification, reversibility, substrate independence 1 Introduction

Consider postponing a project briefing by one day. The calendar entry and the team announcement must change, while a settled flight booking must remain untouched, and a collaborator may request a different date midway through execution. A revised plan does not, by itself, say which updates have already taken effect, which records must stay protected, and which effects can no longer be undone. As Figure 1 illustrates, users ultimately live with changes to the world, not with changes to an agent’s plan.

Computer-use agents (CUAs) have made this delegation technically plausible at scale. Modern agents parse complex layouts, synthesize input gestures, and execute extended operational sequences across unfamiliar interfaces. Interaction research has responded by opening execution up for oversight: co-planning surfaces let users revise procedural steps, ambient displays communicate agent status, and malleable or generative interfaces re-organize the workspace around the task. These designs make the agent’s procedure the object of interaction: what the user inspects and edits is what the agent will attempt next.

Two gaps persist in this procedure-centric stance. First, editing a step changes what the agent will attempt, but it cannot express what the world must keep: boundaries on side effects, records that must remain untouched, and facts that should outlive the current plan have no representation in a list of actions. Second, completion and recovery rest on the agent’s own account. When the executor’s report is the only evidence, users cannot verify what actually changed, and undo is offered without a model of which effects can be restored, inviting the over-reliance that supervisory control research has long documented. Both gaps follow from one representational choice: the task is modeled as a sequence of actions, so the obligations attached to its outcome have nowhere to live.

Our stance extends a lineage of work that treats the object, not the procedure, as the unit of interaction. Direct manipulation established that users reason about visible objects of interest; undo research made recovery a first-class design concern; and accountability research argued that systems should expose what they did so that people can judge it. We extend this lineage from artifacts the user operates to agents that operate on the user’s behalf. We call the resulting stance task-state governance: a task is represented as an editable contract among three state components, not as a procedure alone. The desired state holds the target values the user authorizes the agent to establish. The preservation scope pins application facts that must remain undisturbed. The observed state captures what running applications currently display.

When the three are represented together, an edit becomes a checkable commitment: the system knows what it must establish, what it must not disturb, and what evidence will count as done. The plan remains useful, but as one revisable realization of the contract rather than as the task itself.

We call the runtime that maintains this contract a Task Virtual Machine, and the name is deliberate about scope. It is not an operating-system virtual machine: it owns neither the full state of third-party applications nor the power to roll them back atomically. What it owns is the task’s obligation state and an honest account of evidence. Four commitments make the abstraction tangible. Editing a desired value does not assert that the world changed; desired and observed values sit side by side until fresh observation closes the gap. Marking a record as preserved lifts that constraint out of the agent’s plan and into independent verification. An executor’s completion report does not close a task; a separate verifier checks fresh screen evidence first. And a checkpoint is not time travel; recovery is expressed as the operations the environment actually supports.

We realize this stance in TaskVM, a prototype that places the observed state of live applications beside editable task controls. The runtime compiles user edits into genuine interface gestures, so authority flows through the same rendered interface a human would use. An independent visual verifier, architecturally sep-arated from the executor, checks fresh observations against the contract, and the interface distinguishes reversible, compensatable, and irreversible actions so that recovery is described as supported operations rather than a blanket promise. As the Java Virtual Machine (JVM) exposes one bytecode interface across operating systems, the task virtual machine exposes one obligation model across software stacks; substrate independence is the extent of the analogy we intend.

We evaluate the paradigm through three complementary lenses, asking three research questions (RQ):

• RQ1: Does representing tasks as state contracts improve machine-level satisfaction of governance obligations compared with procedural agent baselines, and at what execution cost?

• RQ2: When execution outcomes are held constant, does governing state change how well users understand, control, and trust the agent?

• RQ3: Does the paradigm remain usable and welcome when a live model executes tasks that participants author themselves?

An automated benchmark over 36 task instances drawn from a suite of productivity, social, and content applications measured contract satisfaction and its costs. A controlled within-subject study with 37 participants compared TaskVM against a conversational

Task-State Governance: A Virtual Machine for Verifiable and Recoverable GUI Agent Execution agent interface under replayed execution, isolating perception from execution variance. An exploratory follow-up with 14 returning participants observed live-model use on self-authored tasks. Across these lenses, TaskVM improved contract compliance at higher machine cost, significantly increased perceived recovery control and trust while execution outcomes were held constant, and remained usable in open-ended use, where participants exercised recovery operations and asked for more.

In summary, this paper contributes:

• Task-state governance, an interaction model that represents a task as an editable contract over desired outcomes, preservation constraints, and observed state, together with an explicit taxonomy of recovery.

• TaskVM, a prototype that instantiates the model through live state projection, GUI-grounded bidirectional execution, independent verification, and honestly bounded recovery, running across web, mobile, and desktop substrates.

• A multi-tier evaluation combining an automated contract benchmark, a controlled user study, and an open live-model exploration, together with design implications for agent interfaces that users can govern rather than merely watch.

2 Related Work

Our research intersects four traditions: autonomous graphical user interface agents, interfaces that project agent execution, bidirectional and malleable data abstractions, and human-agent governance and awareness. Appendix Table 7 maps this landscape along the object each interface asks the user to govern, tracing what an edit commits each system to, what counts as completion, and what recovery can mean.

2.1 GUI Agents: From Autonomous Execution to Interaction

Autonomous graphical user interface agents have transitioned from scripted web automation to multimodal models operating across web, desktop, and mobile operating systems. Modern systems specialize in coordinate-level display grounding, supported by structured reasoning loops. However, benchmark success on autonomous execution does not resolve the human interaction problem. Autonomous traces remain opaque, self-reported completion often diverges from application reality, and single-pass metrics ignore the interactive cost of correcting intermediate errors. Rather than proposing another grounding model, we investigate how agents operate within an architecture that enforces state transparency, non-interference, and verifiable outcomes.

2.2 Projecting Agent Execution: Plans, Interfaces, and Malleable Views

To support human oversight, prior research has explored diverse representations of agent operations. Interactive scaffolding demonstrated the value of structured prompt chains for inspectable sub-steps. Plan-centric systems externalize reasoning into sequen-tial steps, decision forks, and progress indicators. Plover represents the closest point of comparison: it surfaces structured plans that users can locally revise, keeps executed plan steps as immutable history, attaches derived constraints and visual checks to actions, and runs a cross-platform executor across desktop environments. Beyond a six-person formative study, its evaluation includes an expert benchmark-repair analysis in which the first author intervened on 26 autonomous failures and improved 23, presented by the authors as an upper bound on recoverability rather than typical user performance.

Our comparison concerns the object of interaction and the re-sponsibilities attached to an edit. In plan-centric designs, the protected object is the plan’s executed history, whereas the obligations we study protect facts inside external applications. The evidence of completion also differs in scope: plan-centric systems inspect plan state and screens for execution continuity, whereas our verifier certifies explicit contract obligations (target satisfaction and non-interference over protected facts) from fresh, independently ob-tained observations of application state. And the evaluative claims attach to different populations: expert-assisted recoverability in a formative setting, versus unaided use by ordinary participants under controlled replay. We read these framings as protecting different assets, and the two objects can compose within one system.

Complementary work explores malleable views and generative interfaces. Morae pauses browser workflows to preserve critical user choices. Sidekick visualizes multi-task status, while AgentLens adaptively renders mobile interfaces. In web mashups, Orca extracts components with bidirectional DOM syn-chronization, and A-mash combines mobile features into task mashups. Generative UI frameworks synthesize on-demand task cards. Systems such as DynaVis, which com-poses widgets on demand, and Orca, which organizes web components across pages, anchor their evidence loops inside the browser they render; cross-application compensation and fresh-observation checks of protected state lie outside the problems these systems set out to solve.

2.3 Editable and Bidirectional Interfaces: Lenses, Sagas, and Views

Decoupling physical storage from logical views originates in rela-tional databases and declarative querying. Translating edits on projected views back into source structures is formalized as the view-update problem, addressed by Foster et al. through bidirectional lenses with round-trip fidelity. In distributed computing, managing multi-step mutations led Garcia-Molina and Salem to formulate Saga transactions with compensating steps.

In HCI, declarative generation decouples task specifications from widget layouts, as demonstrated by SUPPLE. Recent declarative systems like DMI and VeriSafe compile state targets and safety constraints into accessibility gestures. Generative UI proto-cols such as A2UI and GenUI define standards for stream-ing components, while UICoder and Cells present meth-ods for interactive UI code generation and structured interfaces. TaskVM adapts these principles to computer-use agents, treating heterogeneous interfaces as a distributed database and compiling user edits into verified, compensatable actions across applications.

Wenkui Yang, Yiming Hao, Kai Zhou, Chengyu Pu, Yuzhe Zhang, Jie Cao, Jin Huang, Shuai Ma, and Feng Tian 2.4 Governance, Trust, and Awareness in Human-Agent Collaboration

Designing effective oversight requires mitigating automation com-placency. Human supervisory performance degrades during passive monitoring, fostering automation bias when agents exhibit high but imperfect competence. Mixed-initiative frameworks ad-vocate dynamic responsibility allocation. Human-AI guide-lines emphasize communicating system actions, supporting effi-cient error correction, and conveying operational boundaries. Furthermore, mitigating over-reliance requires cognitive forcing functions that prompt deliberate verification.

Human agency is further supported by malleable, reversible interaction. HCI inquiries show that users continuously appropriate software to fit evolving practices. Reversibility under-pins direct manipulation through rapid, reversible operations with visible impact, formalized in command trees, selective undo, and side views. Across multi-device and task-centric computing, systems maintain continuity across fragmented tools. Finally, collaborative awareness requires visible artifacts that expose system capabilities and boundaries. TaskVM synthesizes these insights by replac-ing passive trajectory tracking with an editable, live projection of verified application state.

3 A Task-State Governance Interaction Model

To bridge human intent and autonomous software execution, we formulate a task-state governance model elevating application state into the primary interactive medium: humans govern tasks, while agents operate applications.

3.1 State Intent versus Plan Intent

Current interactive agents structure human intervention around plan intent, presenting upcoming procedural steps, tool invocations, or sub-goal trees. When a user intervenes, the operation modifies an upcoming action, such as replacing a query or deleting a click. While offering procedural steerability, this conflates what the agent attempts with what the user requires the world to become: a step can execute without error yet fail to achieve the intended outcome, or inadvertently corrupt adjacent records.

In contrast, our interaction model formalizes user intent as state intent, specifying invariant properties that must hold true once execution concludes. Instead of dictating navigation paths, the user specifies what application facts must change and what histor-ical context must remain undisturbed. As defined in our domain schema, an ActionContract encapsulates these state obligations independently of platform execution paths. This distinction decou-ples human governance from low-level software idiosyncrasies: just as programming runtimes allow developers to declare invariants while delegating memory management, our model enables users to express task invariants while delegating mouse coordinates and Document Object Model (DOM) traversals to autonomous execu-tors.

As mapped across the top band of Figure 3, a governed round trip traverses eight semantic stages from Observe and Abstract through Reconcile to close a single episode.

3.2 The Executable View: Bottom-Up Live Projection

To make state intent manipulable without exposing raw complexity, our model compiles live software state into an executable view, com-pressing distributed fields into a unified, low-cognitive-load surface. Figure 2 shows the governance interface that realizes this view, or-ganized into three zones that mirror the state representations of our model.

Zone C (Editable Task State and Live World) presents the desired, preserve, and observed fields of each task variable side by side with the live application window. The observed column is a read-only faithful projection of current software reality, grounded in visual evidence parsed directly from rendered screens; the desired column is editable, and the preserve column pins facts that must remain undisturbed. Zone A (Task Intent and Global Governance) hosts the mission contract and lifecycle controls, and Zone B (Execution Progress and Local Governance) exposes execution progress, the live status of each model-invoking role, and the model call chain.

Maintaining fidelity between the projection and external software requires continuous bidirectional synchronization. The projection streams dynamic live deltas through server-sent events, updating the interface as the external world changes. When con-current external modifications occur, such as a colleague editing a shared document, the system triggers reconciliation mechanisms. Rather than silently overwriting external modifications or crashing the pipeline, TaskVM surfaces external changes as explicit conflict markers in Zone C, inviting the user to reconcile divergent values through deliberate interaction.

3.3 The Desired-Preservation-Observed Triad

We formalize task-state governance through three interconnected state representations:

T = ⟨Sdesired,Spreserve,Sobserved⟩

The desired state Sdesired consists of target variables {v1 ↦→ c1..., vn ↦→ cn } that the user authorizes the agent to establish. The preservation scope Spreserve defines invariant constraints over protected application entities. In real-world workflows, accomplish-ing a goal often threatens adjacent data; for example, updating a meeting time must not alter room reservations or attendee notes. By formalizing Spreserve as an explicit mathematical construct, our model establishes verifiable non-interference boundaries: any execution establishing Sdesired while violating an invariant in Spreserve is classified as an operational failure. The observed state Sobserved captures visual facts parsed from rendered application screens at time t.

A task transition is formally valid if and only if fresh visual observation confirms that external software satisfies the desired state while leaving the preservation scope inviolate:

Valid(T) ⇐⇒(Sobserved |= Sdesired) ∧(Sobserved |= Spreserve)

The user governs the relationship between Sdesired and Spreserve in the governance console (Zone C of Figure 2), while the runtime engine continuously reconciles Sobserved against external software.

3.4 Verification Semantics: Independent Witnessing and Non-Interference

A core tenet of our governance model is that execution completion must never be self-certified by the executing agent. Autonomous agents frequently report success prematurely, mistaking the emis-sion of a completion token for actual state alteration. Our model delegates verification to an independent visual verifier evaluating rendered application screens.

The verification engine enforces two complementary properties: target satisfaction and non-interference. To verify target satisfaction, the verifier requires visible witness predicates, demanding explicit evidence that an intended modification was actively en-acted to eliminate the no-op loophole where an agent claims success on a pre-existing state. To enforce non-interference, the verifier evaluates negative predicates over specified, observable entities in Spreserve, confirming that protected fields display the same values in fresh screenshots. Crucially, the independent verifier operates with strict contextual and architectural isolation from the executing agent, sharing no internal reasoning buffers or prompt scaffold-ing; in the interface it appears as a first-class role awaiting fresh observation (Zone B of Figure 2).

Evaluating fresh visual observations against formal predicates provides an objective, falsifiable gate before any task is marked complete in the user interface.

3.5 Recovery Semantics: Reversible, Compensatable, and Irreversible Actions

Real-world software interactions require robust mechanisms for error correction, interruption, and recovery. In traditional direct manipulation interfaces, undo operations revert local memory structures. However, in multi-application environments, software changes induce persistent external side effects that cannot be undone by simple state restoration.

Our model establishes a taxonomy of operational reversibility, codified in our implementation as Reversibility:

• Reversible Actions: Operations whose effects can be com-pletely and symmetrically undone through standard graphical gestures without residual impact, such as toggling an unsubmitted checkbox or clearing a draft text field.

• Compensatable Actions: Operations that induce persistent external effects but can be semantically counterbalanced through secondary compensation actions, such as sending a cancellation message to follow an erroneous no-tification.

• Irreversible Actions: Operations that permanently modify external systems or invoke physical effects that cannot be restored, such as deleting account archives or completing financial settlements.

When a user initiates a rollback or edits a historical checkpoint, TaskVM derives a compensation plan exclusively from its committed action history (CompensationPatch). Reversible and compensatable actions execute compensating GUI gestures across affected applications. Conversely, when an action crosses an irreversible boundary, the interaction model prohibits deceptive representations of undo: the global rollback dialog in Zone A (Figure 2) previews which committed steps would be replayed or restored, explicitly marks effects that are not auto-compensable, and locks irreversible entities with transparent status badges, preventing the dangerous il-lusion of complete reversibility and supporting appropriate reliance.

4 The TaskVM Prototype

We implement the task-state governance model in TaskVM, an interactive system governing multi-application workflows across web, mobile, and desktop software. This section details the data flow, privacy boundaries, bidirectional bindings, governance primitives, and substrate abstractions that realize our conceptual framework.

4.1 System Architecture and Separation of Concerns

As mapped in Figure 3, TaskVM is organized around a hub-and-spoke data flow centered on the Task-state VM (Kernel). The Kernel serves as the sole owner of task state, maintaining TaskState (intent, variables), workflow graphs, append-only event logs, and epoch checkpoints. By enforcing a defensive read path, the system prevents low-level drivers or downstream consumers from polluting central task semantics.

Rather than relying on a single vision-language model, TaskVM distributes responsibilities across specialized roles that exchange typed objects with the Kernel. The GUI Agent functions as an En-coder, observing live software displays via dedicated observation structures and channeling them into the State Compiler (F2). The Compiler converts visual candidates into standardized task variables (F2, F2’) and forwards them to the Task Architect. A separate generative interface model functions as a Decoder (F3), translating intent into interactive A2UI components projected to the user interface (F3’). These roles execute through independent model invocations marked by amber indicators, preventing their conversational histories from being shared directly.

All experiments reported in this paper instantiate every model-invoking role with the same production model, OpenAI’s GPT-5.6-sol (per-condition configuration in Section 5 and Appendix L). While this separation limits direct context leakage, it does not eliminate extraction errors or correlated failures, so the generated representation is not treated as evidence of its own correctness.

4.2 GUI-Only Purity and the No-Internal-ID Rule

A pervasive vulnerability in automated GUI systems is reliance on internal implementation identifiers, such as private DOM IDs, database keys, or synthetic accessibility tags. While internal identifiers facilitate automated testing, they break down in open-world environments where dynamic layouts alter internal structures without changing visual appearances. Furthermore, relying on internal identifiers creates backdoors that bypass rendered interfaces, giving automated agents privileges unavailable to human operators.

To maintain operational integrity, TaskVM enforces the GUI-only purity standard. Every piece of data entering the model pipeline must satisfy the one-question test: could a real human user see this exact string on the rendered screen? If an attribute cannot be visually perceived by a human looking at the monitor, it is strictly prohibited from entering any model prompt. The compilation pipeline incorporates an explicit verification module that intercepts candidate records and strips internal database keys, accessibility labels, operator jargon, and deep-link URLs via dedicated hidden-ID scrubbers and architectural no-leak filters. Every task variable in the governance console must link directly to verbatim visual evidence parsed from the display.

Moreover, all write operations must traverse authentic graphical gestures (GuiAction), including coordinate-targeted mouse clicks, keystrokes, and scroll events. Any state alteration that does not require a rendered display and simulated user input is classified as an unauthorized backdoor and rejected by the runtime kernel.

4.3 Bidirectional Executable Binding and Multi-App Action Contracts

TaskVM achieves bidirectional interaction through executable semantic bindings that connect high-level virtual machine variables to low-level application gestures. We conceptualize this mechanism as an analog to the Java Virtual Machine. Just as a Java program-mer modifies an abstract object variable and relies on the virtual machine to translate the assignment into platform-specific memory offsets, a TaskVM user edits an abstract task variable in the governance console and relies on TaskVM to synthesize coordinated writes across heterogeneous software.

When a user alters a variable in the governance console, the system instantiates a semantic ActionContract in our domain schema. The contract specifies what condition must become true in the target application, rather than hardcoding how the application must be navigated. Target entities are identified through robust visual anchors, such as surrounding label text, relative geometric offsets, and icon visual embeddings, rather than brittle pixel coordinates. When a single conceptual edit requires modifications across multiple applications, such as synchronizing a deadline across a project management board and a team chat client, the runtime orches-trates a fan-out execution plan. The execution engine sequences operations across application windows, monitors UI transitions, and commits state updates only after verifying structured action outcomes (ActionReceipt).

4.4 Hot Governance: Pausing, Patches, Epochs, and Recovery

Real-world human governance requires the ability to intervene while execution is actively underway. TaskVM provides four foun-dational hot governance primitives: pause, local patch, goal patch, and rollback. At any point during autonomous execution, the user can trigger a pause control in the governance console. The Governance Service immediately issues a halt signal to the active runtime, freezing the execution harness at the current step boundary.

To manage race conditions during mid-execution interventions, TaskVM introduces epoch-based governance gating in the kernel. Every task execution belongs to a monotonically increasing integer epoch. When a user pauses the system and applies a local patch or revises the overarching goal, the governance engine increments the epoch counter. If the autonomous agent was in the process of generating an action from the preceding epoch, the kernel gate intercepts the returning proposal and discards it before dispatch. This prevents stale proposals from being newly dispatched after a revision, without claiming to reverse effects of actions already dispatched.

When a user modifies a goal, the system retains valid, committed action history and recomposes only the affected future trajectory, avoiding redundant re-execution of unaffected milestones. When a rollback is requested, the compensation engine analyzes the committed action registry. For reversible operations, the runtime synthesizes inverse graphical gestures, such as untoggling checkboxes or clearing input forms. For compensatable operations, the runtime executes compensating GUI actions, such as dispatching cancellation notes; these counteract the intended effect but do not assure atomic rollback. Finally, when an operation crosses an irreversible threshold, such as a finalized monetary payment, the system prevents misleading undo representations by raising an explicit irreversible signal.

Irreversible entities are locked in the interface with transparent explanatory badges, clarifying the limits of automated recovery.

4.5 Substrate Independence and Session Isolation

A central objective of TaskVM is substrate independence: ensuring that the same high-level task semantics can be governed identically regardless of the underlying operating system or application technology stack. TaskVM abstracts external software through a clean SubstratePort interface, implemented across modular sessions. The SubstratePort defines primitive operations for capturing visual screenshots, reading device metadata, and dispatching standardized mouse and keyboard input events. Our prototype provides imple-mentations for three distinct platforms: modern web applications, the OSWorld Linux desktop benchmark environment, and the browser-hosted mobile simulation environment MobileGym. Independent visual verification takes place over extracted visual artifacts.

To isolate the agent from hidden laboratory shortcuts, TaskVM enforces strict privilege separation between the runtime session and the evaluation harness. The user-facing runtime interacts exclusively with a session object possessing only standard human capabilities: capturing visual display pixels and issuing realistic graphical gestures. Privileged capabilities required by benchmarking, such as resetting databases, reading internal state trees, or injecting deterministic world events, are encapsulated in an isolated evaluation harness. These privileged methods are strictly inaccessible to both the planning agent and the interface compiler. By separating runtime execution from benchmarking machinery, TaskVM ensures that its evaluation results reflect genuine interactive competence.

5 Stage 1: Automated Technical Evaluation

We evaluate the technical capabilities, operational bounds, and governance properties of TaskVM through an automated benchmark on a frozen suite of multi-application workflows. This evaluation establishes whether explicit state contracts improve execution re-liability over procedural computer-use agents, how performance distributes across task zones, and whether skills injection preserves execution non-inferiority.

5.1 Evaluation Protocol and Contract-Outcome Verification

The automated evaluation suite comprises 36 task instances span-ning 12 structural task families derived from the TaskVM architectural contract, with three variants per family. Execution correctness is adjudicated through Contract-Outcome Verification (COV), structured across ten machine-verifiable domain predicates defined in Appendix A. Each trial is evaluated against all applicable predicates. These include certifying that target fields reach their required final values (requiredfinalstate), confirming that protected fields outside the authorized set remain untouched (protected noninterference), ensuring that executions visit required witness checkpoints (requiredwitnesstrajectory), and verifying that superseded user intents leave no residual effects (superseded intentsuppressed).

Furthermore, transactional boundaries are checked by verifying that pause and stop commands halt execution immediately (pausestopboundary), externally owned objects are not overwritten (externalownershippreservation), and requested rollbacks restore pre-states without hidden evaluator resets (nohiddenevaluatorrestore). We report two complementary metrics: strict pass rate, requiring all applicable predicates in a trial to pass, and atomic predicate mean, measuring the average pass rate across applicable predicates.

Trials are evaluated across three primary execution conditions with skills injection disabled, comprising 36 instances per condition: TaskVM, Direct-CUA, and Planner-CUA. Direct-CUA perceives raw screenshots and dispatches actions directly, receiving interventions as natural language messages delivered to an interactive chat drawer. Planner-CUA formulates hierarchical natural language plans before dispatching actions. All conditions execute against

† Template Control is an assembly sanity-check driver that replays scripted gestures

with zero LLM invocations (the 144 figure above counts scripted action events, not

model calls); 101 of its 108 runs across the evaluated waves failed at the architect stub

due to an implementation limitation in parsing Chinese natural-language goals, so it

is not a valid baseline.

(a) Aggregate COV strict pass rate across conditions, overall and by task zone. Per-family outcomes appear in Appendix Figure 8.

(b) Adjudication breakdown across individual COV predicates.

identical hidden-oracle world contracts, ensuring that evaluation relies on external ground truth rather than agent self-reports.

Evaluation configuration. All LLM roles in all conditions ran the same production model (GPT-5.6-sol) through the same internal inference endpoint, so condition differences reflect the interaction and runtime architecture rather than model choice. In the TaskVM condition the model serves four prompted roles: state compiler, task architect, CUA executor, and an independent model verifier reading fresh screenshots; Direct-CUA and Planner-CUA use the same model for a single agentic loop (with a planning preamble in the latter) over identical environment seeds. Template Control is a scripted gesture driver with zero LLM calls.

The user-facing runtime sees only rendered pixels and standard input gestures, whereas oracle-only capabilities (state resets, internal-state reads, event injection) live in an isolated evaluation harness; every trial manifest records the model identity, ledger call counts per role, and wall time, and the frozen run artifacts are retained in the consolidated archive (full prompts and generation settings in the supplementary package).

5.2 Main Benchmark Findings

Table 1 reports the primary comparison across the 36 task instances, visualized in Figure 4. On the skills-OFF main comparison, TaskVM obtained 21/36 strict passes (rate: 0.583) and an atomic predicate mean of 0.870, compared with 18/36 (0.500) and 0.761 for Direct-CUA, and 17/36 (0.472) and 0.737 for Planner-CUA. The advantage amounts to +0.083 (+16.6%) over Direct-CUA and +0.111 (+23.5%) over Planner-CUA in strict pass rate, alongside +0.109 (+14.3%) and +0.133 (+18.0%) in atomic predicate mean. Inspection of error traces indicates that baseline failures frequently stemmed from premature completion claims and collateral modifications. Direct-CUA often emitted stop tokens before asynchronous web updates persisted, and Planner-CUA altered adjacent form fields when re-planning. In contrast, TaskVM enforced witness predicates that confirmed persisted state before certifying completion.

However, this verification property introduces operational overhead: TaskVM consumed 19,960 seconds of wall time across 1,126 model calls, about 2.2 times the wall time of Direct-CUA, reflecting the latency of round-trip state compilation and verification polling. We report this machine-side cost as a measured property of the automated harness; it cannot be decomposed here into user-facing costs, which the later studies measure on their own terms. Table 1 also documents Template Control (3/36 strict pass rate), included exclusively as an assembly sanity check; 101 of 108 trials failed at the architect stage due to a limitation in parsing Chinese-language goals, so it is not an empirical baseline.

5.3 Predicate-Level Disaggregation

The predicate-level breakdown in Figure 4b localizes where the advantage originates. The largest source of separation is required finalstate, which TaskVM satisfied on 22 of 36 instances versus 20 for Direct-CUA and 17 for Planner-CUA. Two further observations concern recovery specifically. On reversiblerestore, TaskVM restored the protected pre-state in 4 of 6 applicable trials, while Direct-CUA restored none of its 3 applicable trials and Planner-CUA none of its 5: in the adjudication records the baselines possess no restore verb at all, and on externalownership preservation they reach no testable scenario. TaskVM is also the only condition measured on all ten predicates, including 36/36 passes on pausestopboundary; this full-spectrum measurability is itself a distinguishing property of contract-checked execution.

This machine-side asymmetry, where recovery exists as an executable verb only under TaskVM, is mirrored later at the human layer: recovery operations were the most adopted governance functions in Stage 2 (Section 6) and the most demanded automation in Stage 3 (Section 7).

5.4 Zone Breakdown and Governance Efficacy

To understand where state contracts provide utility, Table 2 par-titions evaluation across task zones in a separate skills-ON comparison wave. Zone-S comprises five structural and execution families (Loop blv, Fan-out fav, Partial-failure pfv, Cross-app cav, Sequence sqv), while Zone-G comprises seven governance-dense families (Conflict cfv, UI-drift drv, Interrupt itv, Local-patch lpv, Goal-patch gpv, Irreversible irv, Rollback rbv) that specifically test user intervention intent landing, supersession, pause boundaries, and reversible restore; the same family-to-zone dictionary drives Figure 8 and Appendix Table 8.

In Zone-S, TaskVM achieved a strict pass rate of 0.556 (5/9) and an atomic mean of 0.852, compared with 0.467 (7/15) and 0.733 for Direct-CUA, and 0.533 (8/15) and 0.767 for Planner-CUA. In Zone-G, TaskVM achieved a strict pass rate of 0.714 (10/14) and an atomic mean of 0.922, compared with 0.571 (12/21) and 0.822 for Direct-CUA, and 0.476 (10/21) and 0.797 for Planner-CUA. On these descriptive rates, the advantage over both baselines is larger in Zone-G than in Zone-S. The same governance dimension later separates the two interfaces for human users: in Stage 2, participants reported their largest perceived-control gain on the ability to stop and redirect the agent (Section 6). Notably, the analyzed subsets and denominators differ across execution conditions (e.g., 9 vs 15 and 14 vs 21).

These unmatched rates are reported to document the tested populations; missing or excluded cases are not treated as zero, and the difference between unmatched rates is not interpreted as a matched statistical contrast.

5.5 Skills Injection and Non-Inferiority Analysis

We examine whether procedural skills injection harms TaskVM’s contract faithfulness and whether TaskVM remains non-inferior to its strongest baseline once skills are enabled (Table 3). Paired across 21 instances between the two waves, skills injection flipped four instances in TaskVM’s favor and three against it, a modest net gain of +1; neither CUA baseline showed any net change. Because the skills-OFF and skills-ON waves differ in coverage, we additionally evaluate non-inferiority within the same skills-ON wave on contract-faithful execution (CFE), a per-instance correctness measure derived from the task boundary semantics of each family.

The two analysis collections are explicitly distinct: the strict-COV index retains only fully measured cells (23 of 36 for TaskVM under skills-ON), whereas the CFE comparison additionally admits the protocol’s per-family fallback judgments (e.g., boundary arcs judged from pre-boundary oracle segments), yielding 35 comparable instance clusters paired between TaskVM and Direct-CUA; the 35 are instance-level clusters, not a third task set. A cluster bootstrap over these 35 paired clusters (10,000 resamples, fixed seed) yields a point estimate of the TaskVM minus Direct-CUA difference of +0.0571, with a one-sided 95% lower bound of −0.0857. This lower bound clears the margin of 10 percentage points specified in our evaluation protocol, establishing non-inferiority at that margin; sensitivity analyses at 5 and 7.5 points remain inconclusive, and the verdict also holds at 15 points.

Among the eight discordant clusters, TaskVM alone succeeded on five instances concen-trated in drift-robust rescheduling and sequential contract chains

(drv-01, drv-03, sqv-01 through sqv-03), while Direct-CUA alone succeeded on three instances involving confusable references and retargeting (gpv-03, lpv-03, pfv-03). Skills injection therefore does not degrade TaskVM below its strongest baseline on contract-faithful execution, though the inconclusive stricter margins call for larger paired samples. This result supersedes an earlier cross-wave draft analysis that was inconclusive; the final protocol and its sensitivity margins are documented in Appendix B.

5.6 Stress-Tier Robustness

Beyond the frozen benchmark waves, a development-era stress tier probed governance durability under adversarial goal mutations with skills enabled throughout. The tier comprises 18 variants at difficulty levels L3 and L4, covering conflict injection, goal repair, and cross-session intervention, adjudicated under a majority criterion in which a frozen pass verdict is never reversed. Eleven of 12 L3 variants and all 6 L4 variants passed. The single L3 failure is rbvl3, a rollback-compensation family blocked by a signing-window race condition that remains an open, documented defect. We report this tier as robustness evidence rather than held-out results.

The placement of the sole failure is nonetheless informative: rollback is simultaneously the hardest capability to make reliable and, as the next two sections show, the capability human participants used and requested the most (Sections 6 and 7); we return to this tension in the Discussion.

6 Stage 2: User Study on Task Governance

While automated technical evaluations establish system capabilities across benchmark suites, evaluating human governance requires empirical observation of human operators. We conducted a controlled, within-subject laboratory user study with 37 participants to evaluate how human operators understand, steer, and verify multi-application agent workflows. The study investigates how compiling live application state into an editable governance surface influences perceived recovery control, trust, and the alignment between confidence and understanding compared to conversational agent interfaces.

6.1 Study Goals and Hypotheses

The primary objective of the user study is to evaluate the interactive efficacy and cognitive impact of task-state governance during collaborative execution. We formulate four research questions:

• RQ1 (State Understanding and Confidence Gap): Does task-state governance descriptively narrow the gap between user confidence and objective understanding of application state?

• RQ2 (Perceived Recovery Control and Adoption): Does exposing transactional checkpoints and reversibility locks improve perceived recovery control and encourage adoption of recovery operations?

• RQ3 (Interaction Cost under Execution Control): When the environment and the agent’s replayed trajectory are held constant across conditions, what interaction latency and workload trade-offs emerge from state-level governance?

• RQ4 (Trust and Subjective Satisfaction): Does the task-state governance console foster higher subjective trust and satisfaction compared to conversational supervision?

We hypothesized that TaskVM would significantly increase perceived recovery control and reported trust while narrowing the observed overconfidence gap. Furthermore, we anticipated that governing state would introduce an observable interaction latency trade-off even with the environment and the replayed trajectory held identical by design.

6.2 Participants and Ethics

We recruited 37 participants through a public research-volunteer group maintained by our laboratory; membership in the group is open to anyone, although in practice it skews toward university students and researchers (Appendix M). 19 participants were as-signed to the TaskVM-first order and 18 to the Conversational-first order. All participants completed both conditions and enter the analysis as a within-subject sample. Self-reported demographics and agent-related background, collected post-study through voluntary channels, are detailed in Appendix M. Of the 37 completers, 36 disclosed gender, age, and affiliation (15 female, 21 male; ages 18–42, M =24.7) and 35 completed the background questionnaire.

The sample was technically literate but predominantly novice with respect to GUI agents: only 2/35 reported direct computer-use/GUI-agent professional background, and 14/35 had never used a GUI agent or had tried one only once or twice.

This study was conducted in accordance with the ethical guide-lines of the local ethics committee. Prior to participation, all subjects provided informed consent, and their confidentiality and anonymity were strictly maintained throughout the study.

Participants who completed the study with valid retained data received compensation of 50 RMB. Participation was voluntary, withdrawal was possible at any point without penalty, and compensation was not tied to task performance or questionnaire responses.

6.3 Within-Subject Experimental Design and Apparatus

The study employed a counterbalanced within-subject design. Each participant completed multi-application workflows under two conditions, illustrated in Appendix Figures 6 and 7:

• Conversational CUA (Baseline): Participants supervise an agent through an application screencast accompanied by an interactive chat drawer, receiving conversational progress updates and issuing natural language steering directives.

• TaskVM (Ours): Participants govern execution through the governance cockpit (Figure 2): the editable task-state zone displays desired, preserve, and observed fields beside

† In the paired-count column, the two write-probe rows report the four-cell split in the order TaskVM-only pass / Conv-only pass / both-fail / both-pass. Boundary analysis keeps 34

pairs, excluding P34 (unscored in both arms), P19 (no TaskVM-arm probe), and P04 (no conversational-arm probe); marginal denominators are 35 per arm because the TaskVM-arm

marginal drops P34 and P19, while the conversational-arm marginal drops P34 and P04. End analysis keeps all 37 pairs, counting the two unscored probes as not-passed, so the

paired end rate is 18/37 (48.6%) while the scored marginal rate is 18/35 (51.4%). McNemar p = 1.0 at both time points.

‡ The gap uses all-correct accuracy among responses carrying a confidence rating: 12/21 (57.1%) under TaskVM and 9/20 (45.0%) under the baseline. The read-probe row above

counts all answered probes, where TaskVM all-correct accuracy is 13/22 (59.1%); the one TaskVM response lacking a confidence rating explains the denominator difference.

§ Mean of three predictability items with the unpredictability item reverse-scored (8 minus the raw rating).

Wenkui Yang, Yiming Hao, Kai Zhou, Chengyu Pu, Yuzhe Zhang, Jie Cao, Jin Huang, Shuai Ma, and Feng Tian the live application, the progress zone exposes role status and verification reports, and the global governance zone provides lifecycle controls, checkpoints, and rollback.

Participants performed eight representative task scenarios span-ning collaborative office productivity, event scheduling, customer support record updates, and social communication. Each scenario encompassed three distinct phases: an initial autonomous execution phase across multiple applications, an injected environmental perturbation such as dynamic UI drift or an unexpected record conflict introduced by a simulated collaborator, and a required user intervention prompting selective modification of target fields while preserving existing records. Task order and condition assign-ments were counterbalanced across participants (19 TaskVM-first, 18 Conversational-first) using Latin square permutations.

6.4 Procedure

Each condition began with its own onboarding sequence containing an unscored training task (r8-train) so that participants practiced both interfaces to the same degree, followed by a six-minute think-aloud free-exploration period. In the formal battery each participant completed three of the eight tasks per condition (Table 9); task– condition–order assignments followed the Latin-square schedule, and read/write probe slots were allocated deterministically per task slot in a balanced rotation, so probe exposure was independent of performance. The condition-end questionnaire (all subjective scales described below) was administered once at the end of each condition; there were no per-task questionnaires.

The experimenter intervened only at scripted perturbation points and through a fallback prompt (e.g., asking the participant to verify the selected item when the scripted model error did not occur), applied identically in both conditions; a logged help channel gave direct guidance on request, and such help was recorded as help usage. Participant inputs that did not match a scripted replay branch were steered back onto the canonical path by the operator, and these mismatch and resume events are annotated in the session records. Sessions were run individually.

To isolate the effect of interaction representations on human perception and behavior, the study executed on a high-fidelity Wizard-of-Oz replay rig. Underlying application screens and event streams were pre-recorded from authentic runs on a simulated application suite, with every frame undergoing automated semantic verification to ensure visible text, status bar timestamps, and pixel evidence aligned with the experimental script. When participants intervened, the platform matched the input against a canonical script of branching paths, substituting the runtime engine without altering the study runner (ConditionRuntime, StudyRunner).

The replay protocol fixes three things identically across arms: the environment’s event sequence, the agent’s recorded steps, and the timing and manifestation of errors. It deliberately fixes nothing about the participant’s own intervention: the write-probe score evaluates what the participant did at the intervention point, so paired outcomes legitimately diverge across conditions. Consistent with this design, marginal pass rates are identical (34.3%, 12 of 35 scored probes per arm at the boundary; 51.4%, 18 of 35 at trial end) while the paired record is fully discordant (Section 6.6): every pass was earned by different participants in the two arms.

We therefore read the identical margins strictly as evidence that neither arm offered systematically easier intervention opportunities, not as evidence that participants experienced identical outcomes, and not as evidence that the two interfaces would execute with equivalent competence outside the rig.

6.5 Measures and Analysis

Subjective scales were collected once per condition through the condition-end instrument (verbatim item texts, sources, and composite rules in Table 12). The NASA Task Load Index (NASA-TLX) was administered as six 0–100 sliders and scored as the unweighted mean (RTLX); the performance slider runs from Perfect to Failure, so all six dimensions share the same demand direction with no post-hoc inversion. UMUX-Lite, the two-item short form of the Usability Metric for User Experience, is rescaled to 0–100 by mean(items − 1)/6 × 100. Recovery control is a study-specific composite of four items (influence, stop/redirect, correct, overall). Trust is a two-item composite (trust, rely) adapted from the Trust in Automation instrument; perceived transparency is a three-item predictability subset of the same instrument with one reverse-scored item.

Support satisfaction adapts the three After-Scenario Questionnaire items to the condition block. Read probes (three questions each on committed effects, effective goals, and governance facts) were embedded in drift-triggered trials and preceded by a single confidence rating for the whole probe; write probes score the participant’s own intervention at the scripting boundary and at trial end against the frozen expectation.

All inferential analyses use the participant as the unit: two-sided paired Wilcoxon signed-rank tests for scalar measures and exact McNemar tests for paired binary outcomes (scipy, version pinned in the analysis environment). Paired differences are summarized by dz = mean(di)/sd(di); reported confidence intervals are t -based intervals of the mean paired difference. Missing responses per measure are reported in the table (valid pairs range from 34 to 37 for write probes and 8 for read-probe pairing); zero differences are handled by the implementation’s defaults. We report the complete set of planned comparisons without selective omission, with un-adjusted p -values interpreted alongside effect sizes, and we do not retroactively designate a subset as the sole primary endpoints. All numbers are reproducible from the consolidated archive’s analysis scripts.

6.6 User Study Results

Table 4 summarizes the primary subjective psychometric scales and objective behavioral metrics across the 37 participants. Analysis rules follow Section 6.5.

Participants reported significantly higher perceived recovery control under TaskVM than under the conversational baseline. On a composite of four study-specific recovery-control items (1 to 7), TaskVM achieved a mean rating of 5.56 compared to 4.63 for the conversational baseline (Δ = +0.93, Wilcoxon p =.0050, dz = +0.51, with 22 participants favoring TaskVM versus 13 favoring Conversational CUA). All four constituent items demonstrated statistically significant advantages for TaskVM: ability to stop and redirect the agent (rcstop: 5.68 vs 4.43, Δ = +1.24, p =.0035, dz = +0.58), general recovery control (rcoverall: 5.35 vs 4.35, Δ = +1.00,

Task-State Governance: A Virtual Machine for Verifiable and Recoverable GUI Agent Execution p =.0099, dz = +0.44), ability to correct mistakes (rccorrect: 5.49 vs 4.59, Δ = +0.89, p =.0160, dz = +0.42), and perceived influence over agent actions (rcinfluence: 5.73 vs 5.14, Δ = +0.59, p =.0475, dz = +0.35). This perceived empowerment was accompanied by concrete control use. Goal interventions (pause, edit, resume) were exercised by design in every task. Beyond them, 11 of 37 participants issued at least one rollback (reset) in the TaskVM condition, totaling 17 rollback events against 2 participants and 4 events in the conversational baseline; 13 of the 17 TaskVM events, issued by 8 of the 11 participants, occurred outside the two tasks that scripted an undo request (Table 9).

This asymmetric adoption mirrors the machine-side asymmetry established in Section 5, where recovery existed as an executable verb only under TaskVM; what the benchmark showed baselines cannot do became, for participants, the capability they reached for first. We emphasize that this reflects perceived recovery control and control use; the replayed trajectory the participants watched was identical across conditions, so these differences cannot be attributed to what the two arms showed them.

Objective intervention outcomes corroborate this framing as a manipulation check rather than a treatment effect (Table 4). At the intervention boundary, 12 participants passed only under TaskVM and 12 only under the baseline, 10 failed under both, and 0 passed under both (n = 34 pairs with both arms scored); at trial end the corresponding split is 18, 18, 1, and 0 across all 37 pairs. Every pass was therefore discordant under trajectories the rig held identical, indicating that whether an intervention landed tracked individual intervention behavior rather than arm-level execution differences.

Subjective trust and post-task satisfaction were significantly higher under TaskVM. On a two-item trust composite (trust and reliance items) adapted from the Trust in Automated Systems instrument, TaskVM attained a mean rating of 5.54 versus 4.36 for the conversational baseline (Δ = +1.18, p =.0030, dz = +0.55, with 21 participants rating TaskVM higher versus 7 rating Conversational CUA higher). Support satisfaction measured through an adapted three-item After-Scenario Questionnaire covering ease, completion time, and informational support was likewise significantly higher (5.66 vs 4.95, Δ = +0.70, p =.0222, dz = +0.42).

Perceived transparency, computed from a three-item predictability subset with one reverse-scored item, was directionally higher (5.07 vs 4.50, Δ = +0.57, p =.0809, dz = +0.30), and usability on UMUX-Lite, rescaled to a 0 to 100 range through the standard two-item trans-formation, followed the same directional pattern without reaching significance (70.72 vs 60.14, Δ = +10.58, p =.0881, dz = +0.33).

Task-state governance descriptively reduced the observed confidence-accuracy gap. During drift trials where objective read probes were administered, participants reported subjective confidence ratings before submitting answers. Under Conversational CUA, participants reported a mean confidence of 70.8% while achieving an actual all-correct accuracy of 45.0%, producing an overconfidence gap of +25.8 percentage points. Under TaskVM, participants reported a comparable mean confidence of 70.4% but achieved an actual accuracy of 57.1%, narrowing the overconfidence gap to +13.2 percentage points.1 On individual probe questions, 1All-correct accuracies used in the gap exclude responses missing the confidence rating: 12/21 (57.1%) under TaskVM and 9/20 (45.0%) under the baseline. Counting all

TaskVM achieved an accuracy of 84.8% (56/66) versus 73.3% (44/60) for Conversational CUA across committed effects, effective goals, and governance facts. However, because drift events triggered probes on a subset of trials, matched participant pairs were limited (n = 8, Wilcoxon p = 1.0, dz = +0.10). Consequently, this probe accuracy stands as directional evidence rather than a statistically confirmed gain in state understanding, and the narrower gap represents a descriptive reduction in overconfidence rather than established calibration. Furthermore, participants in the conversational condition left twice as many probes unanswered (6 vs 3), frequently chatting past probe triggers without inspecting underlying state.

Interacting with state controls introduced an observed latency trade-off. Median intervention latency was 85.3 seconds under TaskVM compared to 65.7 seconds under Conversational CUA (Δ = +19.6 seconds in the mean paired difference, 95% t -based CI of that mean difference [−9.2, +48.5], Wilcoxon p =.4372, dz = +0.23). This observed difference showed substantial variation across participants (21 slower, 16 faster) and was not statistically reliable. We treat this observed pattern as an interaction trade-off rather than an established cognitive mechanism: interacting with structured state projections naturally involves reviewing visual representations before committing edits, whereas conversational chat permits immediate text dispatch.

Subjective workload on the NASA-TLX exhibited a directional reduction under TaskVM. Total workload, computed as the unweighted mean of the six dimension ratings, was lower for TaskVM (29.82 vs 39.08, Δ = −9.26, Wilcoxon p =.0938, dz = −0.36, with 21 participants reporting lower workload). All six NASA-TLX subscales showed directionally lower demand for TaskVM, with physical demand reaching statistical significance (17.97 vs 30.95, Δ = −12.97, p =.0085, dz = −0.50) and frustration trending lower (26.49 vs 36.89, Δ = −10.41, p =.0915, dz = −0.32). The largest perceived-control gain, the ability to stop and redirect the agent, aligns with the governance-dense zone where TaskVM’s machine-side advantage concentrated in Section 5, suggesting that automated adjudication and human perception are sensitive to the same governance dimension.

Mental demand (41.49 vs 49.73, p =.2507), temporal demand (23.24 vs 31.76, p =.1121), and effort (38.11 vs 44.46, p =.2489) remained directionally consistent. Analysis of order sequence showed no reversal of the primary pattern across order groups: participants beginning with TaskVM reported a mean workload difference of ΔTLX = −11.5, while those beginning with Conversational CUA reported ΔTLX = −6.9, with parallel directional consistency across UMUX-Lite and TIA trust scores.

Stage 2 held the environment and the replayed agent trajectory constant by design, which isolates perception but leaves open whether these gains persist when a live model, rather than a replay rig, executes tasks the participants themselves invent. Stage 3 (Section 7) examines exactly this open-ended setting.

answered probes yields 13/22 (59.1%) for TaskVM; the single TaskVM response lacking a confidence rating accounts for the denominator difference.

7 Stage 3: Open Exploration with a Live Model

While Stage 2 isolates perception under controlled replay, it leaves open how the governance surface is used when a live model executes tasks that users invent themselves. Stage 3 addresses this question through an exploratory follow-up in which fourteen of the 37 Stage 2 participants returned to author their own tasks and govern live-model execution through the TaskVM cockpit (Figure 5), with no conversational control condition. Stage 2 carries the controlled comparisons of this paper; Stage 3 characterizes use, breakdowns, and requested improvements once the scripting rails were removed.

7.1 Design of the Open Exploration

Each returning participant issued one or more tasks of their own choosing, spanning messaging, scheduling, video browsing, and music and content applications on the same simulated application suite used in the earlier stages; per-participant usage volumes appear in Table 10. Execution was driven by the live production model pipeline rather than a replay rig, so failures, retries, and recovery needs arose naturally. Participants interacted with the full governance surface, including live state projection, mid-flight goal edits, checkpoints, and rollback.

Task-State Governance: A Virtual Machine for Verifiable and Recoverable GUI Agent Execution

Two properties of the study environment shaped what participants could attempt, and we disclose both because they delimit the generality of the Stage 3 material. First, the substrate is a static, offline world: every searchable object exists only because it was injected during setup, so open-ended search tasks could hit coverage gaps. When a requested object was absent, strict goal binding prevented the runtime from silently substituting a near-match, and the verification layer reported the shortfall rather than fabricating success. Participants experienced this as a failed search, but it is exactly the honest-failure property the harness is designed to guar-antee, and it locates the frustration in substrate coverage rather than in the governance layer.

Second, for messaging tasks participants chose recipients from a designated set of fictional contacts, because the injected contact graph is finite. This constrains only the binding target. Task structure, ordering constraints, and the governance operations exercised are unaffected, and the identity of a message recipient is immaterial both to GUI execution difficulty and to the governance claims under study. After each session we collected the NASA-TLX workload instrument and two open-ended questions about the experience and desired improvements; no read or write probes were embedded.

Invitation and compensation followed a deliberate sequence that matters for interpretation. At the end of Stage 2, participants were invited to the optional follow-up with an explicit statement that no additional compensation would be offered; this was a deliberate design choice to minimize payment-motivated self-selection. Fourteen of 37 participants returned. Post hoc, we observed that Stage 3 sessions incurred time and effort comparable to Stage 2, in some cases longer, and we therefore compensated all Stage 3 completers at the same rate (50 RMB, in addition to the 50 RMB for Stage 2). This decision was made after data collection, applied uniformly, and was not contingent on task performance, questionnaire responses, or evaluation outcomes.

The returning subsample was also biased toward benefit. In Stage 2, returning participants had shown a much larger workload difference between arms (TaskVM minus Conversational mean of −24.70 points) than non-returning participants (+0.14), meaning that those who came back were precisely the participants who had already experienced the largest TaskVM advantage. We therefore interpret all Stage 3 outcomes descriptively: they characterize how an already-advantaged, self-selected subsample used the interface on self-authored tasks, and they do not estimate effects for the full Stage 2 population. That 14 of 37 participants returned under an explicit no-compensation promise is consistent with the interface carrying engagement value, although self-selection makes this reading tentative.

7.2 Workload Under a Live Model

Stage 3 workload ratings were lower than the same participants’ earlier ratings, and we report this pattern strictly descriptively. Table 5 pairs each participant’s Stage 3 NASA-TLX total with their own Stage 2 scores. Stage 3 workload averaged 15.5 on the 0 to 100 scale, against 25.2 for the same participants in their Stage 2 TaskVM arm (mean Δ = −9.70, dz = −0.83, Wilcoxon p =.011, with 12 of 14 participants lower) and 49.9 in their Stage 2 Conversational arm (mean Δ = −34.40, dz = −1.43, p <.001, with all 14 lower).

Because task selection, practice, study procedures, and execution mode all changed together between stages, these differences cannot be attributed to any single factor, and we do not treat them as evidence of workload reduction. We report them as low observed workload in these particular sessions, not as evidence about the governance surface’s cost in general. The two participants whose workload rose (P20, P02) attributed it to a model-side execution loop on a complex task and to friction in starting successive tasks, respectively; neither attributed it to governance mechanisms.

7.3 Open-Ended Responses

The qualitative material consists of responses to two open-ended questions, not interviews. We grouped responses into themes, with the individual response as the coding unit; a single participant could contribute to more than one theme, and theme counts below refer to distinct participants.

On the supportive side, four participants (P21, P37, P27, P25) described faithful execution and precise intent recognition under open-ended tasks. P27 wrote that the system recognized the intended edit and completed the keyword changes without over-guessing what they wanted to do [translated], and P21 described watching the planned decomposition land step by step as reas-suring [translated]. One participant’s account constitutes logged governance use: P30 inserted a new instruction midway through execution, asking the agent to additionally ask a specific colleague about overtime availability, and reported that the agent completed the original task and then correctly located the colleague and delivered the inquiry, describing the outcome as deeply satisfying [translated]; this insertion also appears in the session record, so it reflects logged use rather than praise alone.

Separately, P35 praised the topology view of task structure as something they really liked [translated]; we count this as appreciation of the interface, not as use of a governance operation. Two participants (P30, P27) independently requested stronger governance automation, asking that the system place checkpoints automatically at nodes where failures are likely [translated]. We read these requests as demand-side evidence that the checkpoint concept had been absorbed and extended by users, not as reports of a system defect.

On the critical side, three themes emerged, ordered by severity. First, three participants (P35, P06, P13) were frustrated when open-ended search queries missed the simulated platform’s injected data. One searched for several popular videos and found none of them [translated]; another searched for their own uploaded videos and felt discouraged when nothing came back [translated]. This reflects the finite data coverage of the static offline substrate (Section 7), not the governance mechanism: in each case the system reported the miss honestly instead of substituting a plausible-looking result, and it is the most visible limitation of the study environment. Second, four participants (P10, P20, P13, P22) traced failures to live-model execution problems such as retry loops, repeated actions, and lost task continuation.

These failures originate in the executor (model) layer and would be inherited by any interface paradigm; what differs across interfaces is how such failures are surfaced, contained, and recovered from, which is precisely where the governance features are exercised. We therefore report them as executor-side occur-rences with interface-side consequences, without claiming that the

Wenkui Yang, Yiming Hao, Kai Zhou, Chengyu Pu, Yuzhe Zhang, Jie Cao, Jin Huang, Shuai Ma, and Feng Tian governance layer is unaffected by them. Third, seven participants reported one-iteration usability issues such as progress visibility during retries, state-clearing feedback, and onboarding gaps, and two of them asked for features that already existed but had not been covered in the tutorial.

Across all open-ended responses, no participant spontaneously reported difficulty understanding the state of the underlying applications or losing control of the running task; every negative remark concerned substrate data coverage, executor performance, or interface polish.

7.4 Cross-Substrate Capability and Boundaries

A central objective of TaskVM is substrate independence: ensuring that task-state governance operates across diverse software stacks without requiring application-specific programming interfaces. As mapped in the substrate port of Figure 3, TaskVM abstracts external software through a unified port interface that renders pixels into observations and dispatches standard hardware inputs, and we maintain adapters for modern single-page web applications, mobile environments hosted through MobileGym, and Linux desktop applications within OSWorld. Across these environments, the runtime compiles the same state-level intents into substrate-specific gestures, and the independent visual verifier confirms terminal states from raw screen pixels, so a unified task contract survives platform changes.

Our cross-substrate investigations also expose clear operational boundaries where the abstraction encounters friction:

• Canvas-Rendered Interfaces: Applications rendered entirely through low-level graphical canvases lack discrete visual text and standard widget affordances, challenging the visual entity extraction pipeline.

• Transient Dynamic Popups: Short-lived visual notifica-tions that appear and fade within sub-second intervals can elude periodic screenshot polling, occasionally delaying conflict detection.

• Defensive Anti-Automation Gates: Web platforms de-ploying CAPTCHA challenges, biometric verification, or aggressive bot-detection scripts intentionally resist automated inputs, representing an absolute boundary requiring direct human handoff.

• Desktop Window Occlusion: Overlapping application windows on multi-tasking desktops can temporarily ob-scure visual evidence, requiring the runtime to synthesize window-focus management gestures before polling visual artifacts.

Documenting these boundaries clarifies that task-state governance is designed for structured productivity workflows, establishing transparent expectations for open-world deployment.

8 Discussion

The core question raised by TaskVM is not merely how agents execute software, but how users interact with, verify, and recover from software state changes with bounded cognitive effort. Here we discuss four critical interaction design choices emerging from the TaskVM paradigm, its technical evaluations, and its associated operational costs.

8.1 Defining Preservation Constraints versus Monitoring Actions

In traditional conversational agents, users are placed in an unnatu-ral supervisory role: watching an automated cursor move across windows and reading lengthy logs to verify execution. This introduces passive monitoring costs and opportunities for automation bias.

TaskVM replaces this monitoring with explicit contract defini-tion. The user specifies both a desired state and a preservation scope, effectively transferring the effort from watching cursor clicks to writing state constraints. While this shifts human attention from procedural navigation to declarative intent, it does not entirely remove cognitive effort. Determining which fields may change and which must remain protected is its own conceptual workload. We argue that when dealing with multi-application environments, defining bounded state invariants is more tractable than mentally synthesizing low-level execution traces across isolated application silos.

8.2 The Latency and Cost of Verification

The properties provided by TaskVM do not come without overhead. In conventional computer-use agents, execution progresses through an unconstrained forward loop, generating actions until the model emits a stop token. TaskVM, by contrast, introduces structured overhead at multiple stages: compiling initial state, projecting the interface through a secondary model, filtering internal identifiers, and polling rendered screens.

Our automated benchmarks highlight this trade-off. TaskVM con-sumed 19,960 seconds of total wall time and 1,126 model calls across 36 task trials, compared to 9,010 seconds / 1,361 calls for Direct-CUA, and 6,695 seconds / 1,031 calls for Planner-CUA (Table 1). This cost distribution cannot be attributed solely to verification: state compilation, dynamic re-projection, and structured execution boundaries all contribute to the measured wall-time gap. However, if an interface treats an external task merely as an asynchronous background job, the cost of an unverified error in high-stakes en-terprise software vastly outweighs the latency of checking.

An observed trade-off also emerged in our user study: participants exhibited a median intervention latency 19.6 seconds higher under TaskVM (Table 4), a pattern characterized by substantial variance across users without a statistically reliable condition effect. Neither of our human studies decomposes this runtime overhead into machine and human shares. The replay study estimates interaction latency in a controlled setting and cannot establish whether verification delays surface for users under live execution; the exploratory follow-up offers workload descriptions rather than a controlled cost comparison. A design implication is therefore to make verification effort and its products visible at the points where users must judge or intervene, while explicitly evaluating how evidence presentation, waiting time, and recovery demands interact under live execution.

8.3 What Can Visible Observations Certify?

A recurring theme in TaskVM is that interfaces cannot simply trust self-reported execution successes. We rely on independent visual verification against observable fields to certify task state. However, this relies on an important boundary: visible observations can prove

Task-State Governance: A Virtual Machine for Verifiable and Recoverable GUI Agent Execution what is rendered on the display, but they cannot strictly prove hidden backend states or atomic transactional assurances across third-party platforms.

For example, a verified booking screen shows that the UI reflects a date change, but does not inherently prove that airline system databases successfully processed a refund or cancellation ticket. TaskVM’s live state projection surfaces exactly what the system can observe, flagging when visual evidence is insufficient rather than silently assuming consistency. This transparency highlights that interface trust should be grounded in explicit model boundaries, not in exaggerated claims of system-wide correctness.

8.4 Designing for Bounded Recovery and Irreversibility

A prevalent tendency in automated agent design is the masking of failures through optimistic self-reporting. When an agent makes an irreversible error, such as deleting a contact record, conversational interfaces often provide vague reassurances, misleading users into believing the action can be undone. TaskVM implements a tri-partite taxonomy of reversibility. Reversible and compensatable actions are tracked and counteracted through inverse GUI gestures, while irreversible operations are explicitly locked to prevent de-ceptive undo representations.

Empirically, participants exercised goal interventions in every task by design, and 11 of 37 additionally invoked rollback a total of 17 times under TaskVM (baseline: 2 participants, 4 times), with 13 of those 17 events occurring outside the two tasks that scripted an undo request, demonstrating that participants reached for the exposed recovery control even when not asked to.

We note that compensating actions, such as sending a cancellation message, do not equate to true atomic rollbacks. They introduce secondary effects and may leave behind specific audit trails in third-party systems. By forcing the interface to distinguish between these recovery modes, TaskVM shifts user trust away from fictional com-prehensive undo capabilities toward calibrated expectations about what can, and cannot, be reversed.

Our evidence also reveals a productive tension: rollback is si-multaneously the most demanded governance capability and the hardest to make reliable. Rollback was the operation our baselines lacked entirely, the function 11 of 37 study participants invoked (17 events) beyond scripted requests, and the automation Stage 3 participants requested in stronger form, asking for checkpoints placed automatically at failure-prone nodes. Yet the one unresolved failure in our stress tier is precisely a rollback-compensation family, blocked by a race between a signing window and the confirmation gesture. We read this convergence not as a defect to be excused but as the clearest roadmap the paradigm offers: honest vocabu-lary about recovery creates demand for reliable recovery, and the engineering agenda should follow the demand.

9 Limitations and Future Work

Our results are bounded in several ways. Stage 2 holds the environment and the replayed agent trajectory constant, which isolates perception but cannot evidence live-model execution success or real recovery capability; its read probes reached only a subset of trials (n = 8 matched pairs), so probe accuracy is directional evidence rather than an established gain in understanding. Stage 3 restores a live model and participant-authored tasks but removes the control arm, so its lower workload is descriptive and confounded with task freedom and practice, and its sample is self-selected: the 14 returning participants had shown a substantially larger Stage 2 workload advantage than the 23 who did not return (−24.70 versus +0.14 NASA-TLX points). Participants were recruited from a single laboratory research-volunteer pool skewing toward university students and researchers.

All stages ran on an offline simulated application suite with finite injected data, so open-ended search occasionally missed the corpus, adding noise without affecting the governance mechanism; cross-platform deployment is engineering-supported through our substrate adapters but not empirically demonstrated here; live-model execution failures such as retry loops originate in the executor layer, yet their consequences still require an interface that can surface and govern them. On the system side, our com-piled schemas express structured productivity workflows whose state maps to discrete variables and invariant predicates; continuous creative workflows (video editing, 3D modeling, freehand illustration) resist this decomposition, and our purely visual polling can miss sub-second transient UI states, complicating fine-grained causal diagnosis.

Finally, TaskVM deliberately does not circumvent defensive gates such as CAPTCHAs; graceful human handoff at such boundaries, broader and more diverse samples, production substrates, and equivalence-controlled live-model comparisons are the natural next steps.

10 Conclusion

As autonomous agents gain the ability to execute complex workflows, the object users must govern is not the agent’s procedure but the state of the world the agent touches. We present task-state governance, an interaction model that frames a cross-application task as an editable relationship among desired outcomes, protected facts, and observed application state. TaskVM instantiates this relationship through task-level controls, separately checked observations, and explicitly bounded recovery.

Our evaluation characterizes three distinct questions with three distinct designs: technical capability and cost on a fixed benchmark suite, subjective experience under controlled replay, and exploratory use by returning participants on self-authored tasks. These are complementary forms of evidence rather than three con-firmations of one mechanism, and each carries the boundaries reg-istered in the preceding section. Together they show that the obligations and limits of agent-mediated change can be made explicit objects of interaction, verifiable in principle and inspectable in use, while live-execution effectiveness and broader generalization remain open to further study.

By shifting the unit of interaction from procedural plans to verifiable state contracts, TaskVM suggests how users can govern autonomous agents with clarity and calibrated trust: humans govern tasks, while agents operate applications.

Data Availability.

Analysis scripts and de-identified aggregate data will be released with the supplementary materials. Identity mappings and raw screen recordings are withheld to protect participant privacy.

Wenkui Yang, Yiming Hao, Kai Zhou, Chengyu Pu, Yuzhe Zhang, Jie Cao, Jin Huang, Shuai Ma, and Feng Tian