Evolutionary Archive Management

The evolutionary archive is the persistent memory of the HyperAgents improvement process. It stores every generation's code changes, fitness scores, and lineage relationships.

Archive Structure

.hyperagents/
├── archive.jsonl              # Append-only generation log
├── config.json                # Evolution configuration
├── next_parent.json           # Pre-computed next parent selection
├── gen_initial/               # Baseline generation
│   ├── metadata.json          # Generation metadata
│   └── <domain>_eval/         # Evaluation results
│       ├── report.json        # Aggregate scores
│       └── predictions.csv    # Per-item predictions
├── gen_0/
│   ├── metadata.json
│   ├── agent_output/
│   │   ├── model_patch.diff   # The code diff
│   │   └── meta_agent_chat_history.md
│   └── <domain>_eval/
│       ├── report.json
│       └── predictions.csv
├── gen_1/
│   └── ...
└── gen_N/
    └── ...

archive.jsonl Format

Each line is a self-contained JSON snapshot of the archive state after adding a new generation:

{"current_genid": 3, "archive": ["initial", 0, 1, 2, 3]}

This append-only format enables:

• Recovery from crashes (last complete line is the truth)
• Historical analysis (see how the archive grew)
• Atomic updates (each line is a complete snapshot)

metadata.json Format

Each generation's metadata records its full context:

{
  "gen_output_dir": ".hyperagents/gen_3",
  "current_genid": 3,
  "parent_genid": 1,
  "prev_patch_files": [".hyperagents/gen_1/agent_output/model_patch.diff"],
  "curr_patch_files": [".hyperagents/gen_3/agent_output/model_patch.diff"],
  "parent_agent_success": true,
  "optimize_option": "only_agent",
  "can_select_next_parent": true,
  "run_eval": true,
  "run_full_eval": true,
  "valid_parent": true
}

Key Operations

Adding a Generation

1. Create gen_<id>/ directory
2. Run meta-agent, save diff and chat history to agent_output/
3. Run evaluation, save results to <domain>_eval/
4. Write metadata.json with all context
5. Append to archive.jsonl with updated archive list

Reconstructing a Generation

To get the full codebase state at any generation:

1. Start from the root commit
2. Apply all diffs in the lineage chain: initial -> parent -> ... -> target
3. The lineage is stored in metadata.json as prev_patch_files + curr_patch_files

Querying the Archive

• Best generation: Sort by fitness, take max
• Lineage of N: Follow parent_genid links from N to initial
• Valid parents: Filter where valid_parent: true
• Improvement rate: Compare each generation to its parent

Fitness Score Retrieval

Scores live in gen_<id>/<domain>_eval/report.json. The score key varies by domain:

• overall_accuracy — classification domains
• average_progress — game environments
• average_fitness — control tasks
• accuracy_score — code editing
• points_percentage — proof grading

Ensemble Scoring

When ensemble optimization is enabled, additional scores are stored at: gen_<id>/report_ensemble_<domain>_<split>.json

The max score type returns the better of agent and ensemble scores.

Examples

These scenarios illustrate when this skill activates and what it does.

Scenario 1: Inspecting the lineage of the best generation

Trigger: User asks "Show me how generation 7 evolved -- what was its lineage?" Action: The skill reads gen_7/metadata.json to find parent_genid, then follows the chain of parent_genid links back to gen_initial. For each ancestor, it reads the report.json to retrieve fitness scores and the model_patch.diff to summarize what changed. It presents the full lineage as a chain (e.g., initial -> gen_1 -> gen_4 -> gen_7) with fitness scores at each step.

Scenario 2: Recovering from a corrupted archive

Trigger: User reports "The evolve command crashed mid-run and now /hyperagents:status shows an error." Action: The skill reads archive.jsonl and identifies the last complete JSON line as the reliable state. If the final line is truncated, it trims the file to the last valid line. It then cross-checks that every genid referenced in the archive has a corresponding gen_<id>/metadata.json directory, flagging any orphaned or missing entries.

Scenario 3: Querying the archive for improvement patterns

Trigger: User asks "Which types of changes led to the biggest fitness improvements?" Action: The skill iterates through all generations in the archive, comparing each generation's fitness to its parent's fitness. For each positive delta, it reads the corresponding model_patch.diff and categorizes the change (prompt edit, logic refactor, tool change, etc.). It summarizes the results as a ranked list of change types by average fitness improvement.

Safety Rules

1. Never modify archive.jsonl in place — only append
2. Never delete gen_initial/ — it's the baseline
3. Always write metadata.json atomically — write to temp file, then rename
4. Back up before pruning — copy archive.jsonl before removing generations
5. Validate lineage integrity — every genid's parent must exist in the archive