Arize Experiment Skill

Concepts

• Experiment = a named evaluation run against a specific dataset version, containing one run per example
• Experiment Run = the result of processing one dataset example -- includes the model output, optional evaluations, and optional metadata
• Dataset = a versioned collection of examples; every experiment is tied to a dataset and a specific dataset version
• Evaluation = a named metric attached to a run (e.g., correctness, relevance), with optional label, score, and explanation

The typical flow: export a dataset → process each example → collect outputs and evaluations → create an experiment with the runs.

Prerequisites

Three things are needed: ax CLI, an API key (env var or profile), and a space ID. A project name is also needed but usually comes from the user's message.

Install ax

Verify ax is installed and working before proceeding:

1. Check if ax is on PATH: command -v ax (Unix) or where ax (Windows)
2. If not found, check common install locations:
   • macOS/Linux: test -x ~/.local/bin/ax && export PATH="$HOME/.local/bin:$PATH"
   • Windows: check %APPDATA%\Python\Scripts\ax.exe or %LOCALAPPDATA%\Programs\Python\Scripts\ax.exe
3. If still not found, install it (requires shell access to install packages):
   • Preferred: uv tool install arize-ax-cli
   • Alternative: pipx install arize-ax-cli
   • Fallback: pip install arize-ax-cli
4. After install, if ax is not on PATH:
   • macOS/Linux: export PATH="$HOME/.local/bin:$PATH"
   • Windows (PowerShell): $env:PATH = "$env:APPDATA\Python\Scripts;$env:PATH"
5. If ax --version fails with an SSL/certificate error:
   • macOS: export SSL_CERT_FILE=/etc/ssl/cert.pem
   • Linux: export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
   • Windows (PowerShell): $env:SSL_CERT_FILE = "C:\Program Files\Common Files\SSL\cert.pem" (or use python -c "import certifi; print(certifi.where())" to find the cert bundle)
6. ax --version must succeed before proceeding. If it doesn't, stop and ask the user for help.

Verify environment

Run a quick check for credentials:

macOS/Linux (bash):

ax --version && echo "--- env ---" && if [ -n "$ARIZE_API_KEY" ]; then echo "ARIZE_API_KEY: (set)"; else echo "ARIZE_API_KEY: (not set)"; fi && echo "ARIZE_SPACE_ID: ${ARIZE_SPACE_ID:-(not set)}" && echo "--- profiles ---" && ax profiles show 2>&1

Windows (PowerShell):

ax --version; Write-Host "--- env ---"; Write-Host "ARIZE_API_KEY: $(if ($env:ARIZE_API_KEY) { '(set)' } else { '(not set)' })"; Write-Host "ARIZE_SPACE_ID: $env:ARIZE_SPACE_ID"; Write-Host "--- profiles ---"; ax profiles show 2>&1

Read the output and proceed immediately if either the env var or the profile has an API key. Only ask the user if both are missing. Resolve failures:

• No API key in env and no profile → AskQuestion: "Arize API key (https://app.arize.com/admin > API Keys)"
• Space ID unknown → AskQuestion, or run ax projects list -o json --limit 100 and search for a match
• Project unclear → ask, or run ax projects list -o json --limit 100 and present as selectable options

Space ID and Project

Both are needed for most commands. Resolve each:

1. User provides it in the conversation -- use directly via --space-id / --project flags.
2. Env var is set (ARIZE_SPACE_ID, ARIZE_DEFAULT_PROJECT) -- use silently.
3. If missing, AskQuestion once. Tell the user:
   • Space ID is in the Arize URL: /spaces/{SPACE_ID}/...
   • Project is the project name as shown in the Arize UI.
   • For convenience, recommend setting env vars so they don't get asked again: export ARIZE_SPACE_ID="U3BhY2U6..." and export ARIZE_DEFAULT_PROJECT="my-project"

Prefer asking the user over searching or iterating through projects and API keys. If you get a 401 Unauthorized, tell the user their API key may not have access to that space and ask them to verify.

List Experiments: ax experiments list

Browse experiments, optionally filtered by dataset. Output goes to stdout.

ax experiments list
ax experiments list --dataset-id DATASET_ID --limit 20
ax experiments list --cursor CURSOR_TOKEN
ax experiments list -o json

Flags

Flag	Type	Default	Description
--dataset-id	string	none	Filter by dataset
--limit, -l	int	15	Max results (1-100)
--cursor	string	none	Pagination cursor from previous response
-o, --output	string	table	Output format: table, json, csv, parquet, or file path
-p, --profile	string	default	Configuration profile

Get Experiment: ax experiments get

Quick metadata lookup -- returns experiment name, linked dataset/version, and timestamps.

ax experiments get EXPERIMENT_ID
ax experiments get EXPERIMENT_ID -o json

Flags

Flag	Type	Default	Description
EXPERIMENT_ID	string	required	Positional argument
-o, --output	string	table	Output format
-p, --profile	string	default	Configuration profile

Response fields

Field	Type	Description
id	string	Experiment ID
name	string	Experiment name
dataset_id	string	Linked dataset ID
dataset_version_id	string	Specific dataset version used
experiment_traces_project_id	string	Project where experiment traces are stored
created_at	datetime	When the experiment was created
updated_at	datetime	Last modification time

Export Experiment: ax experiments export

Download all runs to a file. By default uses the REST API; pass --all to use Arrow Flight for bulk transfer.

ax experiments export EXPERIMENT_ID
# -> experiment_abc123_20260305_141500/runs.json

ax experiments export EXPERIMENT_ID --all
ax experiments export EXPERIMENT_ID --output-dir ./results
ax experiments export EXPERIMENT_ID --stdout
ax experiments export EXPERIMENT_ID --stdout | jq '.[0]'

Flags

Flag	Type	Default	Description
EXPERIMENT_ID	string	required	Positional argument
--all	bool	false	Use Arrow Flight for bulk export (see below)
--output-dir	string	.	Output directory
--stdout	bool	false	Print JSON to stdout instead of file
-p, --profile	string	default	Configuration profile

REST vs Flight (--all)

• REST (default): Lower friction -- no Arrow/Flight dependency, standard HTTPS ports, works through any corporate proxy or firewall. Limited to 500 runs per page.
• Flight (--all): Required for experiments with more than 500 runs. Uses gRPC+TLS on a separate host/port (flight.arize.com:443) which some corporate networks may block.

Agent auto-escalation rule: If a REST export returns exactly 500 runs, the result is likely truncated. Re-run with --all to get the full dataset.

Output is a JSON array of run objects:

[
  {
    "id": "run_001",
    "example_id": "ex_001",
    "output": "The answer is 4.",
    "evaluations": {
      "correctness": { "label": "correct", "score": 1.0 },
      "relevance": { "score": 0.95, "explanation": "Directly answers the question" }
    },
    "metadata": { "model": "gpt-4o", "latency_ms": 1234 }
  }
]

Create Experiment: ax experiments create

Create a new experiment with runs from a data file.

ax experiments create --name "gpt-4o-baseline" --dataset-id DATASET_ID --file runs.json
ax experiments create --name "claude-test" --dataset-id DATASET_ID --file runs.csv

Flags

Flag	Type	Required	Description
--name, -n	string	yes (prompted)	Experiment name
--dataset-id	string	yes (prompted)	Dataset to run the experiment against
--file, -f	path	yes (prompted)	Data file with runs: CSV, JSON, JSONL, or Parquet
-o, --output	string	no	Output format
-p, --profile	string	no	Configuration profile

Required columns in the runs file

Column	Type	Required	Description
example_id	string	yes	ID of the dataset example this run corresponds to
output	string	yes	The model/system output for this example

Additional columns are passed through as additionalProperties on the run.

Delete Experiment: ax experiments delete

ax experiments delete EXPERIMENT_ID
ax experiments delete EXPERIMENT_ID --force   # skip confirmation prompt

Flags

Flag	Type	Default	Description
EXPERIMENT_ID	string	required	Positional argument
--force, -f	bool	false	Skip confirmation prompt
-p, --profile	string	default	Configuration profile

Experiment Run Schema

Each run corresponds to one dataset example:

{
  "example_id": "required -- links to dataset example",
  "output": "required -- the model/system output for this example",
  "evaluations": {
    "metric_name": {
      "label": "optional string label (e.g., 'correct', 'incorrect')",
      "score": "optional numeric score (e.g., 0.95)",
      "explanation": "optional freeform text"
    }
  },
  "metadata": {
    "model": "gpt-4o",
    "temperature": 0.7,
    "latency_ms": 1234
  }
}

Evaluation fields

Field	Type	Required	Description
label	string	no	Categorical classification (e.g., correct, incorrect, partial)
score	number	no	Numeric quality score (e.g., 0.0 - 1.0)
explanation	string	no	Freeform reasoning for the evaluation

At least one of label, score, or explanation should be present per evaluation.

Workflows

Run an experiment against a dataset

1. Find or create a dataset:

   ax datasets list
   ax datasets export DATASET_ID --stdout | jq 'length'
   

2. Export the dataset examples:

   ax datasets export DATASET_ID
   

3. Process each example through your system, collecting outputs and evaluations
4. Build a runs file (JSON array) with example_id, output, and optional evaluations:

   [
     {"example_id": "ex_001", "output": "4", "evaluations": {"correctness": {"label": "correct", "score": 1.0}}},
     {"example_id": "ex_002", "output": "Paris", "evaluations": {"correctness": {"label": "correct", "score": 1.0}}}
   ]
   

5. Create the experiment:

   ax experiments create --name "gpt-4o-baseline" --dataset-id DATASET_ID --file runs.json
   

6. Verify: ax experiments get EXPERIMENT_ID

Compare two experiments

1. Export both experiments:

   ax experiments export EXPERIMENT_ID_A --stdout > a.json
   ax experiments export EXPERIMENT_ID_B --stdout > b.json
   

2. Compare evaluation scores by example_id:

   # Average correctness score for experiment A
   jq '[.[] | .evaluations.correctness.score] | add / length' a.json
   
   # Same for experiment B
   jq '[.[] | .evaluations.correctness.score] | add / length' b.json
   

3. Find examples where results differ:

   jq -s '.[0] as $a | .[1][] | {example_id, b_score: .evaluations.correctness.score, a_score: ($a[] | select(.example_id == .example_id) | .evaluations.correctness.score)}' a.json b.json
   

Download experiment results for analysis

1. ax experiments list --dataset-id DATASET_ID -- find experiments
2. ax experiments export EXPERIMENT_ID -- download to file
3. Parse: jq '.[] | {example_id, score: .evaluations.correctness.score}' experiment_*/runs.json

Pipe export to other tools

# Count runs
ax experiments export EXPERIMENT_ID --stdout | jq 'length'

# Extract all outputs
ax experiments export EXPERIMENT_ID --stdout | jq '.[].output'

# Get runs with low scores
ax experiments export EXPERIMENT_ID --stdout | jq '[.[] | select(.evaluations.correctness.score < 0.5)]'

# Convert to CSV
ax experiments export EXPERIMENT_ID --stdout | jq -r '.[] | [.example_id, .output, .evaluations.correctness.score] | @csv'

Troubleshooting

Problem	Solution
ax: command not found	Check ~/.local/bin/ax; if missing: uv tool install arize-ax-cli (requires shell access to install packages)
401 Unauthorized	API key may not have access to this space. Verify the key and space ID are correct. Keys are scoped per space -- get the right one from https://app.arize.com/admin > API Keys.
No profile found	Run ax profiles show --expand to check; set ARIZE_API_KEY env var or write ~/.arize/config.toml
Experiment not found	Verify experiment ID with ax experiments list
Invalid runs file	Each run must have example_id and output fields
example_id mismatch	Ensure example_id values match IDs from the dataset (export dataset to verify)
No runs found	Export returned empty -- verify experiment has runs via ax experiments get
Dataset not found	The linked dataset may have been deleted; check with ax datasets list

Save Credentials for Future Use

At the end of the session, if the user manually provided any of the following during this conversation (via AskQuestion response, pasted text, or inline values) and those values were NOT already loaded from a saved profile or environment variable, offer to save them for future use.

Credential	Where it gets saved
API key	ax profile at ~/.arize/config.toml
Space ID	macOS/Linux: shell config (~/.zshrc or ~/.bashrc) as export ARIZE_SPACE_ID="...". Windows: user environment variable via [System.Environment]::SetEnvironmentVariable('ARIZE_SPACE_ID', '...', 'User')

Skip this entirely if:

• The API key was already loaded from an existing profile or ARIZE_API_KEY env var
• The space ID was already set via ARIZE_SPACE_ID env var
• The user only used base64 project IDs (no space ID was needed)

How to offer: Use AskQuestion: "Would you like to save your Arize credentials so you don't have to enter them next time?" with options "Yes, save them" / "No thanks".

If the user says yes:

1. API key — Check if ~/.arize/config.toml exists. If it does, read it and update the [auth] section. If not, create it with this minimal content:

   [profile]
   name = "default"
   
   [auth]
   api_key = "THE_API_KEY"
   
   [output]
   format = "table"
   

   Verify with: ax profiles show

2. Space ID — Persist the space ID as an environment variable:

   macOS/Linux — Detect the user's shell config file (~/.zshrc for zsh, ~/.bashrc for bash). Append:

   export ARIZE_SPACE_ID="THE_SPACE_ID"
   

   Tell the user to run source ~/.zshrc (or restart their terminal) for it to take effect.

   Windows (PowerShell) — Set a persistent user environment variable:

   [System.Environment]::SetEnvironmentVariable('ARIZE_SPACE_ID', 'THE_SPACE_ID', 'User')
   

   Tell the user to restart their terminal for it to take effect.