Human Extractor Skill

Description

GPU-accelerated pipeline for detecting, tracking, and classifying humans in dashcam footage. Processes MP4 videos to extract human crops with optional CLIP-based head covering classification, saving all outputs to a unified directory with comprehensive indexing.

Purpose

Extract visual evidence of human presence from dashcam recordings for investigative analysis. Optimized for high throughput using NVDEC decoding, batched YOLOv8 detection, ByteTrack multi-object tracking, and optional CLIP classification.

Usage

Basic Invocation

Extract humans from Park_R videos on October 6, 2025

Advanced Invocation

Scan Park_R\20251006 and 20251007, keep only frames with people,
save all outputs in one folder, add one full-frame per timestamp with boxes,
use my GPU at max, filter for head-covered individuals at 80% confidence

Input Parameters

Required

roots (list[str]): One or more source directories containing MP4 files
- Example: ["G:\\My Drive\\PROJECTS\\INVESTIGATION\\DASHCAM\\Park_R\\20251006"]

Core Detection

confidence (float, default: 0.35): YOLOv8 detection confidence threshold (0.0-1.0)
iou (float, default: 0.50): IoU threshold for NMS (0.0-1.0)
yolo_batch (int, default: 64): YOLOv8 batch size (32-128 depending on VRAM)

CLIP Filtering (Optional)

clip_filter.enabled (bool, default: false): Enable head covering classification
clip_filter.threshold (float, default: 0.80): CLIP confidence threshold
clip_filter.batch (int, default: 384): CLIP batch size (256-512)

Hardware Acceleration

nvdec (bool, default: true): Use NVIDIA hardware video decoding
gpu_id (int, default: 0): CUDA device ID

Output Control

single_output_dir (str, default: "parsed\ALL_CROPS"): Unified output directory
save_full_frame (bool, default: false): Save one annotated full-frame per timestamp
full_frame_maxw (int, default: 1280): Max width for full-frame saves
draw_boxes (bool, default: true): Annotate boxes on full-frames

Filename Convention

filename_version (str, default: "v1"): Version tag for output filenames

Deduplication

dedup.enabled (bool, default: true): Enable similarity deduplication
dedup.ssim (float, default: 0.92): SSIM threshold (0.0-1.0)
dedup.rate_cap_per_track_per_min (int, default: 12): Max crops per track per minute

Parallel Processing

parallel.dates (list[str], optional): Process multiple dates concurrently
parallel.max_workers (int, default: 3): Max parallel date workers

Output Format

Success Response

{
  "status": "ok",
  "summary": {
    "videos_processed": 142,
    "crops_saved": 4414,
    "frames_saved": 728,
    "gpu_util_avg": 0.85,
    "processing_time_sec": 2847,
    "errors": 0
  },
  "artifacts": {
    "index_csv": "G:\\My Drive\\PROJECTS\\APPS\\Human_Detection\\parsed\\ALL_CROPS\\INDEX.csv",
    "output_dir": "G:\\My Drive\\PROJECTS\\APPS\\Human_Detection\\parsed\\ALL_CROPS",
    "log_file": "G:\\My Drive\\PROJECTS\\APPS\\Human_Detection\\parsed\\ALL_CROPS\\run_20251006_143022.log"
  },
  "performance": {
    "nvdec_active": true,
    "yolo_batch": 64,
    "clip_batch": 384,
    "avg_fps": 48.3,
    "vram_peak_gb": 9.2
  },
  "notes": [
    "NVDEC hardware decoding active",
    "Batched YOLO=64, CLIP=384",
    "GPU utilization: 85%"
  ]
}

Error Response

{
  "status": "error",
  "error": "CUDA out of memory",
  "suggestion": "Reduce batch sizes: yolo_batch=48, clip_batch=256",
  "partial_results": {
    "videos_processed": 67,
    "crops_saved": 2103
  }
}

Output Structure

Directory Layout

parsed\ALL_CROPS\
├── INDEX.csv                                    # Global master index
├── INDEX.20251006_pid1234.csv                   # Shard (pre-merge)
├── run_20251006_143022.log                      # Execution log
│
# Crop files (per person detection)
├── 20251006__20251006142644_070785B__t15234__f365__trk017__x1014y46w266h659__c85__v1.webp
├── 20251006__20251006143844_070787B__t8420__f202__trk003__x234y567w180h420__c92__v1.webp
│
# Full-frame files (optional, one per timestamp)
├── 20251006__20251006142644_070785B__t15234__FRAME__v1.webp
└── 20251006__20251006143844_070787B__t8420__FRAME__v1.webp

Filename Convention

Crop Format:

<date>__<video_stem>__t<ts_ms>__f<frame_idx>__trk<track_id>__x<x1>y<y1>w<w>h<h>__c<covered_0to100>__v<ver>.webp

Example:
20251006__20251006142644_070785B__t15234__f365__trk017__x1014y46w266h659__c85__v1.webp

Decoded:
- Date: 2025-10-06
- Video: 20251006142644_070785B.MP4
- Timestamp: 15234 ms
- Frame: 365
- Track: 17
- BBox: x=1014, y=46, w=266, h=659
- CLIP confidence: 85% (head covering)
- Version: v1

Full-Frame Format:

<date>__<video_stem>__t<ts_ms>__FRAME__v<ver>.webp

Example:
20251006__20251006142644_070785B__t15234__FRAME__v1.webp

INDEX.csv Schema

dataset,date,video_rel,video_stem,frame_idx,ts_ms,track_id,x1,y1,w,h,person_conf,covered_conf,file_type,crop_file,sha1,bboxes_json,annotated,pipeline_ver,yolo_batch,clip_batch,nvdec,created_utc

# Example rows:
Park_R,20251006,20251006\20251006142644_070785B.MP4,20251006142644_070785B,365,15234,17,1014,46,266,659,0.92,0.85,crop,20251006__20251006142644_070785B__t15234__f365__trk017__x1014y46w266h659__c85__v1.webp,a3f2c8b9...,,,v1,64,384,1,2025-10-06T14:30:22Z
Park_R,20251006,20251006\20251006142644_070785B.MP4,20251006142644_070785B,365,15234,,,,,,,frame,20251006__20251006142644_070785B__t15234__FRAME__v1.webp,d4e1a2c7...,"[{""x1"":1014,""y1"":46,""w"":266,""h"":659,""conf"":0.92,""track"":17}]",1,v1,64,384,1,2025-10-06T14:30:22Z

Column Definitions:

dataset: Source camera (Park_R, Park_F, Movie_F, Movie_R)
date: YYYYMMDD
video_rel: Relative path from dataset root
video_stem: Filename without .MP4 extension
frame_idx: Frame number in video
ts_ms: Timestamp in milliseconds
track_id: ByteTrack ID (empty for FRAME rows)
x1,y1,w,h: Bounding box (empty for FRAME rows)
person_conf: YOLOv8 detection confidence
covered_conf: CLIP head covering confidence (0-100 scale, empty if disabled)
file_type: "crop" or "frame"
crop_file: Relative filename
sha1: File hash for integrity
bboxes_json: All detections in frame (FRAME rows only)
annotated: 1 if boxes drawn on frame, 0 otherwise
pipeline_ver: Semantic version tag
yolo_batch: YOLO batch size used
clip_batch: CLIP batch size used (0 if disabled)
nvdec: 1 if NVDEC used, 0 otherwise
created_utc: ISO 8601 timestamp

Implementation Details

Processing Pipeline

[MP4 Videos]
     │
     ▼
[NVDEC Decoder (GPU)]
  RGB tensor → CUDA Stream A
     │
     ▼
[YOLOv8s Detection]
  Batched (64 frames)
  FP16, conf=0.35
     │
     ▼
[ByteTrack Tracking]
  IoU=0.5, max_age=10
     │
     ├──────────────────────► [Full-Frame Saver]
     │                        (optional, downscaled, annotated)
     ▼
[ROI Align (GPU)]
  Extract crops on GPU
     │
     ▼
[CLIP Classification]  ◄────── (optional)
  Batched (384 crops)
  FP16, threshold=0.80
     │
     ▼
[Deduplication Filter]
  SSIM ≥ 0.92
  Rate cap: 12/min/track
     │
     ▼
[Async I/O Thread Pool]
  WebP encode (q=85)
  Shard INDEX writes
     │
     ▼
[Final Merge]
  INDEX.csv

GPU Optimization Strategy

Dual CUDA Streams:

Stream A: YOLOv8 detection
Stream B: CLIP classification
Overlap compute + memory transfers

Dynamic Batching:

Accumulate frames until batch size reached
Process immediately on timeout (100ms)
Keep GPU pipeline full (80-90% utilization)

Memory Management:

Pinned memory for faster CPU↔GPU transfers
Pre-allocated tensor buffers
Stream-ordered operations

Decoder Priority:

NVDEC (GPU hardware decoder) - 5-10x faster
CPU fallback (OpenCV) if NVDEC unavailable
Multi-threaded DataLoader (8-12 workers)

Performance Targets (RTX 4080 16GB)

Metric	Target	Notes
GPU Utilization	80-90%	NVDEC + dual streams + large batches
Throughput	3-4 videos/min	Parking videos (2 FPS sampling)
VRAM Usage	6-10 GB	YOLO=64, CLIP=384
Latency	<30s per video	Including decode, detect, track, classify

Configuration Tuning

If GPU util < 70%:

Increase batch sizes: yolo_batch=80, clip_batch=448
Verify NVDEC active (check nvdec_active in response)
Increase parallel workers: max_workers=4

If CUDA OOM:

Reduce CLIP batch first: clip_batch=256
Then reduce YOLO batch: yolo_batch=48
Disable full-frame saves: save_full_frame=false

If disk I/O bottleneck:

Disable full-frame: save_full_frame=false
Reduce quality: full_frame_maxw=960, WebP q=75
Use faster storage (NVMe SSD)

CLI Equivalent

# Basic usage
python -m src.cli.run_multi_dates \
  --root "G:\My Drive\PROJECTS\INVESTIGATION\DASHCAM\Park_R" \
  --out parsed\ALL_CROPS \
  --dates 20251006 20251007 \
  --use-nvdec --conf 0.35 --iou 0.5

# Advanced usage with CLIP filtering
python -m src.cli.run_multi_dates \
  --root "G:\My Drive\PROJECTS\INVESTIGATION\DASHCAM\Park_R" \
  --out parsed\ALL_CROPS \
  --dates 20251006 20251007 20251008 \
  --use-nvdec \
  --yolo-batch 64 \
  --clip-batch 384 \
  --clip-threshold 0.80 \
  --conf 0.35 \
  --iou 0.5 \
  --save-full-frame \
  --draw-boxes \
  --parallel 3

Example Interactions

Example 1: Basic Detection

User: "Extract all humans from Park_R videos on October 6"

Skill invokes:

{
  "mode": "extract_humans",
  "roots": ["G:\\My Drive\\PROJECTS\\INVESTIGATION\\DASHCAM\\Park_R\\20251006"],
  "confidence": 0.35,
  "single_output_dir": "parsed\\ALL_CROPS",
  "nvdec": true
}

Example 2: Advanced with CLIP

User: "Scan Park_R for October 6-8, filter for people with head coverings at 80% confidence, save annotated frames, max GPU usage"

Skill invokes:

{
  "mode": "extract_humans",
  "roots": [
    "G:\\My Drive\\PROJECTS\\INVESTIGATION\\DASHCAM\\Park_R\\20251006",
    "G:\\My Drive\\PROJECTS\\INVESTIGATION\\DASHCAM\\Park_R\\20251007",
    "G:\\My Drive\\PROJECTS\\INVESTIGATION\\DASHCAM\\Park_R\\20251008"
  ],
  "confidence": 0.35,
  "iou": 0.50,
  "yolo_batch": 64,
  "clip_filter": {
    "enabled": true,
    "threshold": 0.80,
    "batch": 384
  },
  "nvdec": true,
  "save_full_frame": true,
  "draw_boxes": true,
  "single_output_dir": "parsed\\ALL_CROPS",
  "parallel": {
    "max_workers": 3
  }
}

Example 3: Low-Resource Mode

User: "Process Park_R October 6 with minimal GPU memory"

Skill invokes:

{
  "mode": "extract_humans",
  "roots": ["G:\\My Drive\\PROJECTS\\INVESTIGATION\\DASHCAM\\Park_R\\20251006"],
  "confidence": 0.35,
  "yolo_batch": 32,
  "clip_filter": {
    "enabled": false
  },
  "nvdec": false,
  "save_full_frame": false,
  "single_output_dir": "parsed\\ALL_CROPS"
}

Safety & Guardrails

Do Not

❌ Move or delete source MP4 files
❌ Infer gender unless explicitly enabled (sensitive, noisy)
❌ Process videos without user consent
❌ Share outputs containing identifiable persons

Do

✅ Verify GPU availability before processing
✅ Enforce longitude sign corrections for GPS overlays
✅ Maintain audit trail in INDEX.csv
✅ Log versions, batches, NVDEC usage
✅ Handle OOM gracefully with suggestions

Resume Safety

Idempotent: skip already-processed crops by filename
Shard-based: partial runs can resume
Index integrity: SHA1 hashes verify file correctness

Testing & Verification

Pre-Run Checks

# GPU availability
assert torch.cuda.is_available(), "CUDA required"
assert torch.cuda.device_count() > 0, "No GPU found"

# Model files
assert Path("models/yolov8s.pt").exists(), "YOLOv8 model missing"

# Output directory writable
output_dir = Path("parsed/ALL_CROPS")
output_dir.mkdir(parents=True, exist_ok=True)
assert os.access(output_dir, os.W_OK), "Output dir not writable"

Post-Run Verification

# Check outputs exist
assert Path("parsed/ALL_CROPS/INDEX.csv").exists()
assert len(list(Path("parsed/ALL_CROPS").glob("*.webp"))) > 0

# Validate INDEX.csv
df = pd.read_csv("parsed/ALL_CROPS/INDEX.csv")
assert df['crop_file'].notna().all()
assert df['person_conf'].between(0, 1).all()

# Sample roundtrip
sample = df.sample(1).iloc[0]
assert Path(f"parsed/ALL_CROPS/{sample['crop_file']}").exists()

# GPU utilization check
assert gpu_util_avg > 0.70, f"Low GPU util: {gpu_util_avg}"

Dependencies

Required

Python 3.10+
PyTorch 2.0+ with CUDA 11.8+
ultralytics (YOLOv8)
transformers (CLIP)
opencv-python
pillow
pandas
numpy

Optional (Performance)

NVIDIA Video Codec SDK (NVDEC)
TensorRT (future optimization)
nvJPEG (GPU JPEG encoding)

Installation

cd "G:\My Drive\PROJECTS\APPS\Human_Detection"
pip install -r requirements.txt

Troubleshooting

Common Issues

1. CUDA Out of Memory

Error: CUDA out of memory. Tried to allocate 2.50 GiB
Solution: Reduce batch sizes
  yolo_batch: 64 → 48 → 32
  clip_batch: 384 → 256 → 128

2. NVDEC Not Available

Warning: NVDEC unavailable, falling back to CPU decode
Solution: Check NVIDIA driver version (≥525.60)
  GPU must support Video Codec SDK
  Verify with: nvidia-smi --query-gpu=name --format=csv

3. Low GPU Utilization

Warning: GPU util only 45%
Solutions:
  1. Increase batch sizes (if VRAM allows)
  2. Enable NVDEC: nvdec=true
  3. Increase parallel workers: max_workers=4
  4. Check CPU bottleneck (use more DataLoader workers)

4. Slow Processing

Performance: 0.8 videos/min (expected 3-4)
Diagnostics:
  1. Check disk I/O (use SSD)
  2. Verify NVDEC active (5-10x faster than CPU)
  3. Profile with: python -m torch.utils.bottleneck script.py

Future Enhancements

Planned

TensorRT optimization (2-4x CLIP speedup)
Multi-GPU sharding (process different dates on different GPUs)
GPU JPEG/WebP encoding (nvJPEG)
Real-time streaming mode

Under Consideration

Face recognition integration
Gender classification (opt-in only, with warnings)
Action recognition (walking, standing, etc.)
Multi-camera fusion (correlate detections across cameras)

Version History

v1.0 (Current)

Initial release
YOLOv8s + ByteTrack + CLIP
NVDEC support
Unified output directory
Global INDEX.csv

References

Contact & Support

For issues or questions:

Check parsed/ALL_CROPS/run_*.log for error details
Review GPU diagnostics: nvidia-smi
Validate input paths exist and are readable
Verify CUDA/PyTorch installation: python -c "import torch; print(torch.cuda.is_available())"

human-extractor

Human Extractor Skill

Description

Purpose

Usage

Basic Invocation

Advanced Invocation

Input Parameters

Required

Core Detection

CLIP Filtering (Optional)

Hardware Acceleration

Output Control

Filename Convention

Deduplication

Parallel Processing

Output Format

Success Response

Error Response

Output Structure

Directory Layout

Filename Convention

INDEX.csv Schema

Implementation Details

Processing Pipeline

GPU Optimization Strategy

Performance Targets (RTX 4080 16GB)

Configuration Tuning

CLI Equivalent

Example Interactions

Example 1: Basic Detection

Example 2: Advanced with CLIP

Example 3: Low-Resource Mode

Safety & Guardrails

Do Not

Do

Resume Safety

Testing & Verification

Pre-Run Checks

Post-Run Verification

Dependencies

Required

Optional (Performance)

Installation

Troubleshooting

Common Issues

Future Enhancements

Planned

Under Consideration

Version History

References

Contact & Support