Tauri Agent Control HTTP Bridge

Observe and control a running Tauri 2.0 desktop app via HTTP — no Playwright/Puppeteer needed.

Prerequisites

Plugin Setup

1. Add to Cargo.toml:

[dependencies]
tauri-plugin-agent-control = "0.1"

2. Register in src-tauri/src/lib.rs:

tauri::Builder::default()
    .plugin(tauri_plugin_agent_control::init())

3. Add permission in src-tauri/capabilities/default.json:

{ "permissions": ["agent-control:default"] }

Verify the bridge is running

curl -s http://localhost:9876/health
# {"ok":true}

Core Workflow

Every interaction follows: snapshot → interact → re-snapshot.

# 1. Get compact snapshot (saves context!)
curl -s 'http://localhost:9876/snapshot?format=compact'
# [page] My App — http://localhost:1420/
# @e1 [button] "Settings"
# @e2 [input] placeholder="Search..."
# @e3 [button] "New Project"

# 2. Interact using refs from snapshot
curl -s -X POST http://localhost:9876/click -d '{"ref":"@e3"}'

# 3. Wait for DOM change
curl -s -X POST http://localhost:9876/wait -d '{"selector":".modal"}'

# 4. Re-snapshot (refs are now invalid — get fresh ones)
curl -s 'http://localhost:9876/snapshot?format=compact&scope=.modal'

# 5. Continue with new refs
curl -s -X POST http://localhost:9876/fill -d '{"ref":"@e4","text":"My New Project"}'
curl -s -X POST http://localhost:9876/click -d '{"ref":"@e5"}'

Snapshot

ALWAYS use compact format — 90% fewer tokens than JSON.

# Compact snapshot (RECOMMENDED)
curl -s 'http://localhost:9876/snapshot?format=compact'

# Scoped to a container
curl -s 'http://localhost:9876/snapshot?format=compact&scope=.sidebar'

# Limit depth
curl -s 'http://localhost:9876/snapshot?format=compact&depth=3'

# Full JSON (only when you need rect/attribute detail)
curl -s http://localhost:9876/snapshot

Essential Interactions

All interactions use refs from the most recent snapshot.

# Click / Fill / Press / Type
curl -s -X POST http://localhost:9876/click -d '{"ref":"@e1"}'
curl -s -X POST http://localhost:9876/fill -d '{"ref":"@e2","text":"hello world"}'
curl -s -X POST http://localhost:9876/press -d '{"key":"Enter"}'
curl -s -X POST http://localhost:9876/press -d '{"key":"Control+a"}'
curl -s -X POST http://localhost:9876/type -d '{"ref":"@e2","text":"hello"}'

# Find by semantic locator (when refs are unavailable)
curl -s -X POST http://localhost:9876/find -d '{"by":"text","value":"Sign In","action":"click"}'
curl -s -X POST http://localhost:9876/find -d '{"by":"label","value":"Email","action":"fill","text":"user@test.com"}'
curl -s -X POST http://localhost:9876/find -d '{"by":"role","value":"button","name":"Submit","action":"click"}'
curl -s -X POST http://localhost:9876/find -d '{"by":"testid","value":"submit-btn","action":"click"}'

Get Information

curl -s http://localhost:9876/get/title          # Page title
curl -s http://localhost:9876/get/url            # Current URL
curl -s http://localhost:9876/get/text/@e1       # Element text
curl -s http://localhost:9876/get/value/@e2      # Input value

Check State

curl -s http://localhost:9876/is/visible/@e1     # true/false
curl -s http://localhost:9876/is/enabled/@e1     # true/false
curl -s http://localhost:9876/is/checked/@e5     # true/false

Wait

curl -s -X POST http://localhost:9876/wait -d '{"selector":".loaded"}'
curl -s -X POST http://localhost:9876/wait -d '{"text":"Success"}'
curl -s -X POST http://localhost:9876/wait -d '{"url":"**/dashboard**"}'
curl -s -X POST http://localhost:9876/wait -d '{"fn":"window.ready"}'

Screenshot

curl -s http://localhost:9876/screenshot
# {"path":"/var/folders/.../agent-control-screenshot.png"}

Then analyze the returned path with your image viewing tool.

Eval

Run arbitrary JS in the webview (use expressions, not return statements):

curl -s -X POST http://localhost:9876/eval -d '{"code":"document.title"}'
curl -s -X POST http://localhost:9876/eval -d '{"code":"document.querySelectorAll(\"button\").length"}'

Ref Lifecycle

Refs (@e1, @e2, etc.) are invalidated on each new snapshot. Always re-snapshot after clicking links, submitting forms, or triggering dynamic content (modals, dropdowns, tabs).

curl -s -X POST http://localhost:9876/click -d '{"ref":"@e5"}'   # May trigger navigation
curl -s 'http://localhost:9876/snapshot?format=compact'           # MUST re-snapshot
curl -s -X POST http://localhost:9876/click -d '{"ref":"@e1"}'   # Use new refs

Deep-Dive Documentation

Reference	When to Use
references/commands.md	Full command reference — all interactions, get endpoints, check state
references/snapshot-refs.md	Ref lifecycle, snapshot formats, scoping, best practices
references/network.md	Network intercept, route mocking, headers, offline, geolocation
references/storage-state.md	Cookies, localStorage, sessionStorage, state save/load
references/advanced.md	Console, errors, dialogs, viewport, recording, download, performance, raw input

Ready-to-Use Templates

Template	Description
templates/form-automation.sh	Fill form in Tauri app (snapshot → fill → submit → verify)
templates/app-testing.sh	Test a Tauri app (health → snapshot → interact → screenshot → verify)
templates/network-mocking.sh	Intercept and mock network requests

./templates/form-automation.sh
./templates/app-testing.sh
./templates/network-mocking.sh

Troubleshooting

Problem	Fix
Server not responding	Ensure plugin is registered, app is running. Check curl -s http://localhost:9876/health.
Ref not found	Refs reset on each snapshot. Re-snapshot before interacting after navigation or DOM changes.
Eval timeout	JS shim may not be loaded — check /health first. Void expressions (e.g. scrollBy) return null.
Virtualized lists	Only visible items render. Scroll the container with selector param or use /eval to check scrollTop.
Screenshot fails	Uses macOS screencapture — app must be visible on screen (not minimized). macOS only.
Network intercept empty	Call /network/intercept before the requests you want to capture. Tauri IPC calls are excluded.