Dynamic Architectures Meta-Skill

When to Use This Skill

Invoke this meta-skill when you encounter:

Growing Networks: Adding capacity during training (new layers, neurons, modules)
Pruning Networks: Removing capacity that isn't contributing
Continual Learning: Training on new tasks without forgetting old ones
Gradient Isolation: Training new modules without destabilizing existing weights
Modular Composition: Building networks from graftable, composable components
Lifecycle Management: State machines controlling when to grow, train, integrate, prune
Progressive Training: Staged capability expansion with warmup and cooldown

This is the entry point for dynamic/morphogenetic neural network patterns. It routes to 7 specialized reference sheets.

How to Access Reference Sheets

IMPORTANT: All reference sheets are located in the SAME DIRECTORY as this SKILL.md file.

When this skill is loaded from: skills/using-dynamic-architectures/SKILL.md

Reference sheets like continual-learning-foundations.md are at: skills/using-dynamic-architectures/continual-learning-foundations.md

NOT at: skills/continual-learning-foundations.md (WRONG PATH)

Core Principle

Dynamic architectures grow capability, not just tune weights.

Static networks are a guess about capacity. Dynamic networks let training signal drive structure. The challenge is growing without forgetting, integrating without destabilizing, and knowing when to act.

Key tensions:

Stability vs. Plasticity: Preserve existing knowledge while adding new capacity
Isolation vs. Integration: Train new modules separately, then merge carefully
Exploration vs. Exploitation: When to add capacity vs. when to stabilize

The 7 Dynamic Architecture Skills

continual-learning-foundations - EWC, PackNet, rehearsal strategies, catastrophic forgetting theory
gradient-isolation-techniques - Freezing, gradient masking, stop_grad patterns, alpha blending
peft-adapter-techniques - LoRA, QLoRA, DoRA, adapter placement, merging strategies
dynamic-architecture-patterns - Grow/prune patterns, slot-based expansion, capacity scheduling
modular-neural-composition - MoE, gating, grafting semantics, interface contracts
ml-lifecycle-orchestration - State machines, quality gates, transition triggers, controllers
progressive-training-strategies - Staged expansion, warmup/cooldown, knowledge transfer

Routing Decision Framework

Step 1: Identify the Core Problem

Diagnostic Questions:

"Are you trying to prevent forgetting when training on new data/tasks?"
"Are you trying to add new capacity to an existing trained network?"
"Are you designing how multiple modules combine?"
"Are you deciding WHEN to grow, prune, or integrate?"

Quick Routing:

Problem	Primary Skill
"Model forgets old tasks when I train new ones"	continual-learning-foundations
"New module destabilizes existing weights"	gradient-isolation-techniques
"Fine-tune LLM efficiently without full training"	peft-adapter-techniques
"When should I add more capacity?"	dynamic-architecture-patterns
"How do module outputs combine?"	modular-neural-composition
"How do I manage the grow/train/integrate cycle?"	ml-lifecycle-orchestration
"How do I warm up new modules safely?"	progressive-training-strategies

Step 2: Catastrophic Forgetting (Continual Learning)

Symptoms:

Performance on old tasks drops when training on new tasks
Model "forgets" previous capabilities
Fine-tuning overwrites learned features

Route to: continual-learning-foundations.md

Covers:

Why SGD causes forgetting (loss landscape geometry)
EWC, SI, MAS (regularization approaches)
Progressive Neural Networks, PackNet (architectural approaches)
Experience replay, generative replay (rehearsal approaches)
Measuring forgetting (backward/forward transfer)

When to Use:

Training sequentially on multiple tasks
Fine-tuning without forgetting base capabilities
Designing systems that accumulate knowledge over time

Step 3: Gradient Isolation

Symptoms:

New module training affects host network stability
Want to train on host errors without backprop flowing to host
Need gradual integration of new capacity

Route to: gradient-isolation-techniques.md

Covers:

Freezing strategies (full, partial, scheduled)
detach() vs no_grad() semantics
Dual-path training (residual learning on errors)
Alpha blending for gradual integration
Hook-based gradient surgery

When to Use:

Training "seed" modules that learn from host errors
Preventing catastrophic interference during growth
Implementing safe module grafting

Step 4: PEFT Adapters (LoRA, QLoRA)

Symptoms:

Want to fine-tune large pretrained models efficiently
Memory constraints prevent full fine-tuning
Need task-specific adaptation without modifying base weights

Route to: peft-adapter-techniques.md

Covers:

LoRA (low-rank adaptation) fundamentals
QLoRA (quantized base + LoRA adapters)
DoRA (weight-decomposed adaptation)
Adapter placement strategies
Merging adapters into base model
Multiple adapter management

When to Use:

Fine-tuning LLMs on limited compute
Creating task-specific model variants
Memory-efficient adaptation of large models

Step 5: Dynamic Architecture Patterns

Symptoms:

Need to add capacity during training (not just before)
Want to prune underperforming components
Deciding when/where to grow the network

Route to: dynamic-architecture-patterns.md

Covers:

Growth patterns (slot-based, layer widening, depth extension)
Pruning patterns (magnitude, gradient-based, lottery ticket)
Trigger conditions (loss plateau, contribution metrics, budgets)
Capacity scheduling (grow-as-needed vs overparameterize-then-prune)

When to Use:

Building networks that expand during training
Implementing neural architecture search lite
Managing parameter budgets with dynamic allocation

Step 6: Modular Composition

Symptoms:

Combining outputs from multiple modules
Designing gating/routing mechanisms
Need graftable, replaceable components

Route to: modular-neural-composition.md

Covers:

Combination mechanisms (additive, multiplicative, selective)
Mixture of Experts (sparse gating, load balancing)
Grafting semantics (input/output attachment points)
Interface contracts (shape matching, normalization boundaries)
Multi-module coordination (independent, competitive, cooperative)

When to Use:

Building modular architectures with interchangeable parts
Implementing MoE or gated architectures
Designing residual streams as module communication

Step 7: Lifecycle Orchestration

Symptoms:

Need to decide WHEN to grow, train, integrate, prune
Building state machines for module lifecycle
Want quality gates before integration decisions

Route to: ml-lifecycle-orchestration.md

Covers:

State machine fundamentals (states, transitions, terminals)
Gate design patterns (structural, performance, stability, contribution)
Transition triggers (metric-based, time-based, budget-based)
Rollback and recovery (cooldown, hysteresis)
Controller patterns (heuristic, learned/RL, hybrid)

When to Use:

Designing grow/train/integrate/prune workflows
Implementing quality gates for safe integration
Building RL-controlled architecture decisions

Step 8: Progressive Training

Symptoms:

New modules cause instability when integrated
Need warmup/cooldown for safe capacity addition
Planning multi-stage training schedules

Route to: progressive-training-strategies.md

Covers:

Staged capacity expansion strategies
Warmup patterns (zero-init, LR warmup, alpha ramp)
Cooldown and stabilization (settling periods, consolidation)
Multi-stage schedules (sequential, overlapping, budget-aware)
Knowledge transfer between stages (inheritance, distillation)

When to Use:

Ramping new modules safely into production
Designing curriculum over architecture (not just data)
Preventing stage transition shock

Common Multi-Skill Scenarios

Scenario: Building a Morphogenetic System

Need: Network that grows seeds, trains them in isolation, and grafts successful ones

Routing sequence:

dynamic-architecture-patterns - Slot-based expansion, where seeds attach
gradient-isolation-techniques - Train seeds on host errors without destabilizing host
modular-neural-composition - How seed outputs blend into host stream
ml-lifecycle-orchestration - State machine for seed lifecycle
progressive-training-strategies - Warmup/cooldown for grafting

Scenario: Continual Learning Without Forgetting

Need: Train on sequence of tasks without catastrophic forgetting

Routing sequence:

continual-learning-foundations - Understand forgetting, choose approach
gradient-isolation-techniques - If using architectural approach (columns, modules)
progressive-training-strategies - Staged training across tasks

Scenario: Neural Architecture Search (Lite)

Need: Grow/prune network based on training signal

Routing sequence:

dynamic-architecture-patterns - Growth/pruning triggers and patterns
ml-lifecycle-orchestration - Automation via heuristics or RL
progressive-training-strategies - Stabilization between changes

Scenario: RL-Controlled Architecture

Need: RL agent deciding when to grow, prune, integrate

Routing sequence:

ml-lifecycle-orchestration - Learned controller patterns
dynamic-architecture-patterns - What actions the RL agent can take
gradient-isolation-techniques - Safe exploration during training

Rationalization Resistance Table

Rationalization	Reality	Counter-Guidance
"Just train a bigger model from scratch"	Transfer + growth often beats from-scratch	"Check continual-learning-foundations for why"
"I'll freeze everything except the new layer"	Full freeze may be too restrictive	"Check gradient-isolation-techniques for partial strategies"
"I'll add capacity whenever loss plateaus"	Need more than loss plateau (contribution check)	"Check ml-lifecycle-orchestration for proper gates"
"Modules can just sum their outputs"	Naive summation can cause interference	"Check modular-neural-composition for combination mechanisms"
"I'll integrate immediately when training finishes"	Need warmup/holding period	"Check progressive-training-strategies for safe integration"
"EWC solves all forgetting problems"	EWC has limitations, may need architectural approach	"Check continual-learning-foundations for trade-offs"

Red Flags Checklist

Watch for these signs of incorrect approach:

No Isolation: Training new modules without gradient isolation from host
No Warmup: Integrating new capacity at full amplitude immediately
No Gates: Integrating based only on time, not performance metrics
Naive Combination: Summing module outputs without gating or blending
Ignoring Forgetting: Adding new tasks without measuring old task performance
No Rollback: No plan for what happens if integration fails

Relationship to Other Packs

Request	Primary Pack	Why
"Implement PPO for architecture decisions"	yzmir-deep-rl	RL algorithm implementation
"Evaluate architecture changes without mutation"	yzmir-deep-rl/counterfactual-reasoning	Counterfactual simulation
"Debug PyTorch gradient flow"	yzmir-pytorch-engineering	Low-level PyTorch debugging
"Optimize training loop performance"	yzmir-training-optimization	General training optimization
"Design transformer architecture"	yzmir-neural-architectures	Static architecture design
"Deploy morphogenetic model"	yzmir-ml-production	Production deployment

Intersection with deep-rl: If using RL to control architecture decisions (when to grow/prune), combine this pack's lifecycle orchestration with deep-rl's policy gradient or actor-critic methods.

Counterfactual evaluation: Before committing to a live mutation (grow/prune), use deep-rl's counterfactual-reasoning.md to simulate the change and evaluate outcomes without risk. This is critical for production morphogenetic systems.

Diagnostic Question Templates

Use these to route users:

Problem Classification

"Are you training on multiple tasks sequentially, or growing a single-task network?"
"Do you have an existing trained model you want to extend, or starting fresh?"
"Is the issue forgetting (old performance drops) or instability (training explodes)?"

Architectural Questions

"Where do new modules attach to the existing network?"
"How should new module outputs combine with existing outputs?"
"What triggers growth? Loss plateau, manual, or learned?"

Lifecycle Questions

"What states can a module be in? (training, integrating, permanent, removed)"
"What conditions must be met before integration?"
"What happens if a module fails to improve performance?"

Summary: Routing Decision Tree

START: Dynamic architecture problem

├─ Forgetting old tasks?
│  └─ → continual-learning-foundations

├─ New module destabilizes existing?
│  └─ → gradient-isolation-techniques

├─ Fine-tuning LLM efficiently?
│  └─ → peft-adapter-techniques

├─ When/where to add capacity?
│  └─ → dynamic-architecture-patterns

├─ How modules combine?
│  └─ → modular-neural-composition

├─ Managing grow/train/integrate cycle?
│  └─ → ml-lifecycle-orchestration

├─ Warmup/cooldown for new capacity?
│  └─ → progressive-training-strategies

└─ Building complete morphogenetic system?
   └─ → Start with dynamic-architecture-patterns
      → Then gradient-isolation-techniques
      → Then ml-lifecycle-orchestration

Reference Sheets

After routing, load the appropriate reference sheet:

continual-learning-foundations.md - EWC, PackNet, rehearsal, forgetting theory
gradient-isolation-techniques.md - Freezing, detach, alpha blending, hook surgery
peft-adapter-techniques.md - LoRA, QLoRA, DoRA, adapter merging
dynamic-architecture-patterns.md - Grow/prune patterns, triggers, scheduling
modular-neural-composition.md - MoE, gating, grafting, interface contracts
ml-lifecycle-orchestration.md - State machines, gates, controllers
progressive-training-strategies.md - Staged expansion, warmup/cooldown