Headless Adapters

Purpose

Schema-driven adapter for headless CLI agents. No code required - just define a JSON schema describing how to interact with the CLI.

Use Case	Tool
Wrap headless CLI agent	`headless` command
Create new schemas	Schema Creation Guide

Quick Start

Create a schema for your CLI agent using the Schema Creation Guide

Run the adapter:

bunx @plaited/agent-eval-harness headless --schema ./my-agent-headless.json

Any CLI agent that outputs JSON can be wrapped — no agent-specific code required.

CLI Commands

headless

Schema-driven adapter for ANY headless CLI agent.

bunx @plaited/agent-eval-harness headless --schema <path>

Options:

Flag	Description	Required
`-s, --schema`	Path to adapter schema (JSON)	Yes

Schema Format:

{
  "version": 1,
  "name": "my-agent",
  "command": ["my-agent-cli"],
  "sessionMode": "stream",
  "prompt": { "flag": "-p" },
  "output": { "flag": "--output-format", "value": "stream-json" },
  "autoApprove": ["--allow-all"],
  "outputEvents": [
    {
      "match": { "path": "$.type", "value": "message" },
      "emitAs": "message",
      "extract": { "content": "$.text" }
    },
    {
      "match": { "path": "$.type", "value": "tool_use" },
      "emitAs": "tool_call",
      "extract": { "title": "$.name", "status": "'pending'", "input": "$.input" }
    },
    {
      "match": { "path": "$.type", "value": "tool_result" },
      "emitAs": "tool_call",
      "extract": { "title": "$.name", "status": "'completed'", "output": "$.content" }
    }
  ],
  "result": {
    "matchPath": "$.type",
    "matchValue": "result",
    "contentPath": "$.content"
  }
}

Session Modes:

Mode	Description	Use When
`stream`	Keep process alive, multi-turn via stdin	CLI supports session resume
`iterative`	New process per turn, accumulate history	CLI is stateless

Creating a Schema

Run the CLI's --help to identify prompt, output format, and auto-approve flags
Capture sample JSON output from the CLI
Map JSONPath patterns to output events (including input/output for tool calls)
Create the schema file
Test with headless command

See Schema Creation Guide for the complete workflow.

Security Considerations

Trust Boundary: CLI Output is Untrusted

The headless adapter parses JSON output from CLI agents. This output may contain content from external sources (web searches, file reads, API responses) that flows into trajectory data:

CLI Agent → JSON stdout → JSONPath extraction → ParsedUpdate → TrajectoryStep

Trajectory fields — especially tool_call.input and tool_call.output — should be treated as untrusted content by downstream consumers (graders, LLM-as-judge, analysis scripts). Do not:

Execute trajectory content as code
Use trajectory content in unsanitized shell commands
Pass trajectory content to LLMs without injection-aware prompting

autoApprove Flags

The autoApprove field bypasses the CLI agent's safety confirmation prompts. Use the least permissive flags your evaluation requires:

Risk Level	Example	When to Use
High	`["--dangerously-skip-permissions"]`	Only in isolated containers (Docker, CI)
Medium	`["--allowedTools", "Read,Write,Glob"]`	Scoped to specific tools
Low	`["--auto-approve", "read-only"]`	Read-only evaluations

Never run high-risk autoApprove flags outside isolated environments. Use --workspace-dir or Docker for evaluations that modify the filesystem.

Troubleshooting

Common Issues

Issue	Likely Cause	Solution
Tool calls not captured	JSONPath not iterating arrays	Use `[*]` wildcard syntax - see guide
Tool input/output missing	Extract config missing `input`/`output` fields	Add `input`/`output` paths - see guide
"unexpected argument" error	Stdin mode misconfigured	Use `stdin: true` - see guide
401 Authentication errors	API key not properly configured	Set the correct API key environment variable for your agent
Timeout on prompt	JSONPath not matching	Capture raw CLI output, verify paths - see guide
Empty responses	Content extraction failing	Check extract paths - see guide

Complete troubleshooting documentation: Troubleshooting Guide

External Resources

AgentSkills Spec: agentskills.io

agent-eval-harness skill - Running evaluations against adapters

headless-adapters