agent-host-e2e-tests
Agent host end-to-end tests
These tests run the whole agent host end-to-end (real server, real bundled provider SDK/CLI, real AHP protocol) while replaying recorded model traffic from committed YAML fixtures — deterministic and tokenless.
Before doing anything, read the architecture + troubleshooting reference: src/vs/platform/agentHost/test/node/e2e/README.md
It documents the mental model, the fixture format, every config flag, and a symptom→cause→fix troubleshooting table. This skill is only the workflows; the README is the source of truth for how it works.
When validating an upstream provider/runtime fix, use ci-artifact-testing to check for a compatible CI artifact before building locally. Run the same strict replay against the bundled baseline and candidate; preserve the test assertions and existing cross-platform validation requirements.
Core invariants
- Replay is default and strict. No env var serves committed fixtures without a token or network. An unrecorded request is a hard cache miss.
- Fixture names derive from test titles. Renaming a model-backed test orphans
${provider}-${slug}.yaml; re-record after a rename. Tests explicitly registered withhostOnlyTest(...)sharecaptures/empty.yaml. - Recording is intentional and credentialed.
AGENT_HOST_REPLAY_RECORD=1talks to real CAPI and needsGITHUB_TOKENorgh auth token. Keep prompts trivial and read-only in temporary directories. - Fixtures are generated, never hand-edited. Fix normalization or redaction and re-record if a capture contains unstable or sensitive data.
- Disabled variants stay accountable. Record every gate in
e2e/KNOWN_ISSUES.mdwith its scope, expected and observed behavior, and focused reproduction command. - Azure validation is required. Every new Agent Host E2E or real-provider integration test must pass a focused build from VS Code pipeline definition
111before merge.