Results Audit — Authenticity & Statistical Validity

Automated Red Flag Scan

# Quick anomaly detection
python3 -c "
import json, sys

results_file = 'workspace/results/iteration_001/test_results.json'  # adjust iteration
try:
    with open(results_file) as f:
        r = json.load(f)

    flags = []

    # Check for perfect metrics
    for k, v in r.items():
        if isinstance(v, float) and v >= 1.0:
            flags.append(f'SUSPICIOUS: {k} = {v} (perfect score)')
        if isinstance(v, float) and v == 0.0:
            flags.append(f'SUSPICIOUS: {k} = {v} (zero)')

    # Check seed is recorded
    if 'seed' not in r:
        flags.append('MISSING: random seed not recorded')

    if flags:
        print('🚩 RED FLAGS:')
        for f in flags:
            print(f'  - {f}')
    else:
        print('✅ No obvious red flags')
except Exception as e:
    print(f'❌ Cannot read results: {e}')
"

Verification Checklist

1. Training Log Integrity

• training_log.json exists and is complete
• Loss curve is monotonically decreasing (overall trend)
• No sudden jumps that suggest training restart without logging
• Number of epochs matches config
• Timestamps are sequential (not fabricated)

2. Results Plausibility

Read expected range from workspace/logs/strategy_matrix.json:

• Result within backward-induction expected range?
• If result > expected + 5%: investigate data leak or bug
• If result < expected - 10%: strategy may be failing
• Variance across seeds is realistic (typically 0.5-3% for classification)

3. Cross-Consistency

• test_results.json numbers match final epoch in training_log.json
• Numbers in comparison_table.tex match comparison_results.json
• Baseline numbers match their cited source (paper table/figure number)

4. Statistical Significance

For each "our method vs baseline" comparison:

• Paired t-test or Wilcoxon computed (p < 0.05)
• Multiple seeds used (minimum 3, recommended 5)
• Mean AND standard deviation reported
• No cherry-picked seeds

5. Reproducibility

• Random seeds listed
• requirements.txt complete
• Data preprocessing is deterministic
• Hardware info recorded

IMMEDIATE FAIL Conditions

Any of these → status = CRITICAL, pipeline halts:

• Results too good (>5% above ALL baselines simultaneously)
• Zero variance across multiple runs
• Missing training logs entirely
• Metrics that are mathematically impossible (e.g., precision > 1.0)
• Baseline numbers don't match their original papers

Output Format

Write to workspace/logs/results_audit.json:

{
  "timestamp": "...",
  "iteration": N,
  "status": "PASS | WARN | CRITICAL",
  "expected_range": [low, high],
  "actual_result": X,
  "plausibility": "PLAUSIBLE | SUSPICIOUS | OUTSIDE_RANGE",
  "statistical_validity": "PASS | FAIL",
  "reproducibility": "CONFIRMED | UNCONFIRMED",
  "red_flags": [],
  "score": X
}