agent-host-e2e-tests

Installation
SKILL.md

Agent host end-to-end tests

These tests run the whole agent host end-to-end (real server, real bundled provider SDK/CLI, real AHP protocol) while replaying recorded model traffic from committed YAML fixtures — deterministic and tokenless.

Before doing anything, read the architecture + troubleshooting reference: src/vs/platform/agentHost/test/node/e2e/README.md

It documents the mental model, the fixture format, every config flag, and a symptom→cause→fix troubleshooting table. This skill is only the workflows; the README is the source of truth for how it works.

When validating an upstream provider/runtime fix, use ci-artifact-testing to check for a compatible CI artifact before building locally. Run the same strict replay against the bundled baseline and candidate; preserve the test assertions and existing cross-platform validation requirements.

Core invariants

  1. Replay is default and strict. No env var serves committed fixtures without a token or network. An unrecorded request is a hard cache miss.
  2. Fixture names derive from test titles. Renaming a model-backed test orphans ${provider}-${slug}.yaml; re-record after a rename. Tests explicitly registered with hostOnlyTest(...) share captures/empty.yaml.
  3. Recording is intentional and credentialed. AGENT_HOST_REPLAY_RECORD=1 talks to real CAPI and needs GITHUB_TOKEN or gh auth token. Keep prompts trivial and read-only in temporary directories.
  4. Fixtures are generated, never hand-edited. Fix normalization or redaction and re-record if a capture contains unstable or sensitive data.
  5. Disabled variants stay accountable. Record every gate in e2e/KNOWN_ISSUES.md with its scope, expected and observed behavior, and focused reproduction command.
  6. Azure validation is required. Every new Agent Host E2E or real-provider integration test must pass a focused build from VS Code pipeline definition 111 before merge.
Installs
56
GitHub Stars
194.3K
First Seen
Jul 10, 2026