Deco Site Memory Debugging

Debug memory issues on Deno sites running in Kubernetes pods using Chrome DevTools Protocol (CDP) over WebSocket.

When to Use This Skill

Pod memory usage is high or growing over time
Kubernetes OOMKill events on site pods
Need to identify what's consuming memory inside a Deno process
Investigating Response/Request body leaks
Need to determine if memory is a real leak vs lazy GC

Quick Start

1. IDENTIFY POD        → kubectl get pods -n sites-<sitename>
2. PORT-FORWARD        → kubectl port-forward <pod> 9229:9229
3. GET WS URL          → curl http://127.0.0.1:9229/json/list
4. CONNECT CDP         → WebSocket to the debuggerUrl
5. FORCE GC            → HeapProfiler.collectGarbage
6. ANALYZE MEMORY      → Deno.memoryUsage() + queryObjects + heap snapshot
7. DIAGNOSE            → Is it a leak or lazy GC?
8. RECOMMEND           → Tune --v8-flags=--max-old-space-size or fix leak

Files in This Skill

File	Purpose
`SKILL.md`	This overview and quick reference
`cdp-connection.md`	How to connect to pods and common pitfalls
`memory-analysis.md`	Step-by-step memory analysis procedures

Key Concept: GC is Lazy

V8's garbage collector is lazy by design. It won't collect garbage until memory pressure forces it to. A pod showing 1.8 GB RSS might drop to 700 MB after a forced GC — meaning there was no leak, just uncollected garbage.

Always force GC before concluding there's a leak:

HeapProfiler.collectGarbage  (via CDP)

Then check Deno.memoryUsage() again. The difference between before-GC and after-GC tells you how much was reclaimable garbage vs actual retained memory.

Recommendation: Reduce Max Heap Size

If post-GC memory is reasonable but pre-GC memory is causing OOMKills or high pod memory:

Decrease the V8 max old space size so GC runs more frequently:

--v8-flags=--max-old-space-size=512

This forces V8 to GC more aggressively instead of letting garbage accumulate. For most Deco sites this doesn't affect performance because the actual live heap is much smaller than the default limit. The GC runs are fast (milliseconds) and the trade-off is worth it to keep RSS predictable.

Example Deno flags:

deno run --v8-flags=--max-old-space-size=512 -A main.ts

Memory Breakdown Model

RSS is composed of multiple layers. You must understand what each layer represents:

RSS = V8 Heap + V8 External + Native (untracked)

V8 Heap     → JavaScript objects, closures, strings, compiled code
V8 External → ArrayBuffers (including a ~304MB static V8/ICU data buffer)
Native      → Deno Rust runtime, module graph, JIT code cache, mmap'd files, thread stacks

Use Deno.memoryUsage() to get the breakdown:

rss: Total resident set size
heapUsed: V8 JavaScript heap
heapTotal: V8 heap capacity
external: V8 external allocations (ArrayBuffers)
rss - heapUsed - external = native/untracked memory

The native gap is normal. For a large Deno/Fresh app with thousands of modules, 500MB-1GB of native memory is expected (JIT compiled code, Rust allocator, module cache). This is NOT a leak.

Common Memory Consumers

What	Typical Size	Is It a Problem?
Static V8/ICU ArrayBuffer	~304 MB	No — built into V8, constant
Deno module cache on disk	~100-200 MB	No — normal for large apps
V8 JIT compiled code	~200-500 MB	No — proportional to loaded modules
Response bodies not consumed	Variable, grows	YES — leak if bodyUsed=false
OpenTelemetry export buffers	~10-50 MB	Minor — accumulates slowly
Rendered HTML strings (SSR cache)	~20-100 MB	Monitor — should be bounded
LRU cache metadata	Small (booleans)	No — only stores `true` with size tracking

Diagnostic Decision Tree

Pod memory high?
├── Force GC → Memory drops significantly?
│   ├── YES → Not a leak. Recommend reducing --max-old-space-size
│   └── NO → Real retained memory. Continue investigation:
│       ├── Check Response objects (queryObjects Response.prototype)
│       │   └── bodyUsed=false count high? → Response body leak
│       ├── Check ArrayBuffers
│       │   └── Many large OTEL/JSON buffers? → Export/cache leak
│       ├── Check heap snapshot top consumers
│       │   └── Large HTML strings? → SSR cache unbounded
│       └── Large native gap (RSS - heap - external)?
│           └── Normal for large Deno apps (JIT + Rust runtime)

Related Skills

deco-incident-debugging — For general incident response and triage
deco-performance-audit — For deep performance analysis

deco-site-memory-debugging