Chartwalker: A Computer-Use Agent and Execution-Graded Benchmark for Electronic Health Record Operation

Twenty clinical workflows. A sandboxed OpenEMR. A Playwright grader that reads the database to score success. The first public computer-use benchmark for an actual EHR.

Abstract Arndt et al.[11] reported in Annals of Family Medicine (2017) that primary-care physicians spend approximately 5.9 hours per clinic day inside the EHR — including approximately 1.4 hours of after-hours "pajama time" (86 minutes per the original paper). Sinsky et al.[12] earlier quantified the broader pattern: roughly two hours of EHR work for every one hour of direct patient care. Computer-use agents are now technically capable of operating real desktop and web applications, as demonstrated by Anthropic's Computer Use release[1] and a sequence of GUI-agent benchmarks[2][3][4][5]. No public computer-use agent or benchmark operates an actual EHR. Chartwalker fills this gap: a sandboxed OpenEMR instance, a 20-task clinical workflow suite, and a Playwright-based grading harness that reads the database state to assert success — following the execution-graded pattern of WebArena[2] and OSWorld[4]. Pass criterion: ≥ 70% task completion driven by Claude Computer Use.

§ 1 Introduction

EHR navigation is a clinically meaningful task class. The Arndt and Sinsky findings[11][12] establish the size of the prize — every clinician hour saved on EHR mechanics is an hour returned to patient care or to clinician well-being. Yet despite the obvious importance, public computer-use agent work has focused on consumer web tasks (Mind2Web[7], WebVoyager[5], VisualWebArena[3]) and desktop apps (OSWorld[4]). No benchmark covers EHR operation, and no agent has been shown to complete clinical workflows end-to-end against a real EHR interface.

Chartwalker is the simplest version of that missing artifact. It is a benchmark, not a model. The architecture decisions are deliberate: a sandboxed OpenEMR runs in Docker, twenty workflows are defined with deterministic post-conditions in the EHR's PostgreSQL backend, and the Playwright-based grader asserts those post-conditions independently of the agent's claimed trajectory.

1.1 Contributions

1. A sandboxed OpenEMR evaluation harness with reproducible setup and seeded patient data.
2. A 20-task benchmark suite covering order entry, chart review, refill workflow, problem-list reconciliation, and result acknowledgment — each task with a deterministic post-condition gold.
3. A public leaderboard with baseline results from Claude Computer Use[1], evaluated with per-task completion, per-step accuracy, and grounded-failure-mode classification.

§ 2 Background and Related Work

2.1 Computer-Use Agents

Anthropic's Computer Use release[1] (October 2024) introduced a screenshot-plus-cursor action loop in which the model receives the screen as an image and emits mouse and keyboard actions. Chartwalker is built on this loop. The agent is granted only screen access — it does not query the OpenEMR API directly — because the explicit point of the evaluation is to measure UI-mediated agency, not API-mediated agency.

2.2 Execution-Graded Web Benchmarks

WebArena[2] (ICLR 2024) established the methodology Chartwalker inherits: sandboxed full-stack web applications with execution-based graders that read backend state after the agent finishes. VisualWebArena[3] extended the framework to multimodal grounding and popularised Set-of-Mark prompting[9]. OSWorld[4] generalised execution grading to full desktop environments. WebVoyager[5] demonstrated end-to-end multimodal agents on live websites; SeeAct[6] decomposed web agency into action generation plus action grounding, a separation that informs our Chartwalker prompt structure.

2.3 Visual Grounding

Set-of-Mark[9] overlays numbered marks on UI widgets so the model can reference elements by ID rather than coordinates. Ferret-UI[10] demonstrates that UI-specific grounding on high-aspect-ratio screens outperforms generic vision-language models. Chartwalker overlays SoM marks on every screenshot before sending to the agent, an inexpensive intervention with well-documented benefits.

2.4 Generalist Web Agency

Mind2Web[7] evaluates agents across 137 websites with cross-domain generalisation as the central concern. Chartwalker inherits the cross-domain framing: while the v0.1 benchmark targets OpenEMR specifically, the task taxonomy is portable to other EHRs (OpenMRS, GNU Health), and a v0.2 effort should add at least one additional EHR for true generalisation testing.

2.5 Why an EHR-Specific Benchmark

Existing GUI-agent benchmarks systematically underrepresent the patterns that dominate EHR work: time-bounded ordering, structured-form entry with terminology binding, problem-list reconciliation, refill workflows with administrative steps. AgentBench[8] demonstrated that LLM agent performance varies dramatically by task category — aggregate scores conceal large per-category spreads. Chartwalker's per-category reporting is designed to surface the categories where current agents fail, which is the actionable result for downstream agent designers.

§ 3 Proposed Approach

3.1 Sandbox

OpenEMR (latest stable release) runs in Docker with PostgreSQL as the backend. A seed script populates the sandbox with 50 synthetic patients drawn from Synthea (see Atrium, Project 01), each with a realistic baseline of conditions, medications, allergies, and recent encounters. The sandbox is reset between every task evaluation so that no task can be confounded by state from a previous task.

3.2 The 20-Task Suite

3.3 Execution-Based Grading

Each task has a pre-defined post-condition: a set of assertions over the OpenEMR database that must hold after the agent finishes. For example, "Order metformin 500 mg BID for P0001" has the post-condition:

The Playwright-based grader runs the agent's session, then connects to the PostgreSQL backend and asserts each post-condition. The task verdict is binary; partial credit is reported separately as the count of post-conditions met out of total.

3.4 Agent Prompting

The agent receives: (i) the task description in natural language, (ii) the initial OpenEMR screenshot with Set-of-Mark[9] overlay, (iii) a short, fixed system prompt describing OpenEMR's navigation idioms. No additional task-specific scaffolding is provided. The architecture is intentionally bare so that the benchmark measures the agent's intrinsic capability rather than the operator's ability to engineer a workflow-specific prompt.

§ 4 Evaluation Protocol

4.1 Metrics

4.2 Failure-Mode Audit

Every failed task is classified into one of four primary failure modes, drawn from the GUI-agent literature. Misclick: action grounded to the wrong widget, often a near-neighbour in the SoM overlay. Terminology-binding error: the model selected a clinically incorrect code (e.g., metformin extended-release versus immediate-release). Navigation loss: the model entered a screen from which it could not return. Premature termination: the model declared the task complete before all post-conditions held. Reporting this distribution is the practically useful output for downstream agent designers.

§ 5 Expected Contributions

1. Benchmark. The first public, reproducible computer-use benchmark for an actual EHR.
2. Failure-mode taxonomy. A classified failure-mode distribution from a frontier computer-use agent on clinical workflows.
3. Leaderboard. Initial cross-agent results (Claude Computer Use, and any subsequent computer-use systems) with quarterly re-evaluation.

§ 6 Limitations and Risks

OpenEMR is open-source and stylistically distinct from production EHRs like Epic and Cerner; results on Chartwalker do not directly transfer to those systems. The architectural pattern — sandbox plus post-condition grader — does, but a v0.2 effort should add at least one additional EHR target. The Set-of-Mark[9] overlay is also an artifact of evaluation that production systems would not have; agents that depend heavily on SoM marks may underperform on raw EHR screens.

A separate risk: writing real prescriptions to a sandboxed EHR is safe; but an agent capable of writing prescriptions reliably is a precursor to systems that could write them to production EHRs. Chartwalker's release explicitly limits to the sandbox configuration and includes guidance against repointing the agent at production systems without institutional QA, in keeping with the broader literature on clinical LLM evaluation gaps.

§ 7 Conclusion

Chartwalker takes the well-established sandbox-plus-execution-grader pattern from WebArena[2] and OSWorld[4], applies it to the place clinicians actually spend their working day, and exposes a leaderboard that surfaces the per-category failure modes that aggregate scores conceal. It is the simplest version of a benchmark that should exist and currently does not.