Extract Moves From Video

Overview

This skill provides a systematic approach for extracting text commands from video recordings. Common use cases include extracting gameplay commands from text adventure games (like Zork), capturing terminal commands from screen recordings, or transcribing any typed input visible in video content.

Workflow

Step 1: Analyze the Source Video

Before processing, understand the video characteristics:

1. Determine video properties: Resolution, duration, frame rate
2. Identify text regions: Where commands appear on screen (e.g., after a prompt character like >)
3. Assess text style: Font type, color, background contrast (terminal text on dark backgrounds requires specific handling)
4. Check for audio: Determine if audio transcription could supplement OCR (verify audio contains relevant content before installing large packages like Whisper)
5. Understand typing patterns: Estimate how frequently new commands appear to inform frame sampling rate

Step 2: Download and Prepare Video

1. Download video using appropriate tools (yt-dlp, youtube-dl, or direct download)
2. Verify download integrity before proceeding
3. Extract video metadata to confirm properties match expectations

Step 3: Extract Frames Strategically

Frame extraction requires balancing coverage against processing time:

1. Analyze command frequency first: Manually review a sample of the video to understand how often new commands appear
2. Choose appropriate sampling rate:
   • Fast typing: 0.5-1 second intervals
   • Slow typing: 2-3 second intervals
   • When uncertain, extract at higher frequency and subsample later (avoids re-extraction)
3. Use FFmpeg for extraction:

   ffmpeg -i video.mp4 -vf "fps=1" frames/frame_%04d.png
   

4. Focus on relevant screen regions: If commands appear in a specific area, crop frames to that region to improve OCR accuracy

Step 4: Optimize OCR Configuration

OCR accuracy depends heavily on proper configuration for the specific video type:

1. Test on sample frames first: Before processing all frames, tune OCR settings on 5-10 representative frames
2. Configure Tesseract page segmentation mode (--psm):
   • --psm 6: Assume uniform block of text
   • --psm 7: Single text line
   • --psm 13: Raw line (treat as single line, no analysis)
3. Preprocess images for better OCR:
   • Binarization: Convert to black/white with appropriate threshold
   • Invert colors if text is light on dark background
   • Increase contrast for low-contrast videos
   • Scale up small text (2x-3x enlargement often helps)
4. Test multiple threshold values: Common values (127, 150, 180) work differently depending on video; empirically test which produces best results

Example preprocessing with Python/OpenCV:

import cv2
img = cv2.imread('frame.png', cv2.IMREAD_GRAYSCALE)
# Invert if light text on dark background
img = cv2.bitwise_not(img)
# Binarize with tested threshold
_, img = cv2.threshold(img, 150, 255, cv2.THRESH_BINARY)
# Scale up for better OCR
img = cv2.resize(img, None, fx=2, fy=2, interpolation=cv2.INTER_CUBIC)

Step 5: Extract and Parse Commands

1. Run OCR on preprocessed frames: Use Python bindings (pytesseract) for efficiency over subprocess calls
2. Identify command patterns: Look for prompt markers (e.g., >, $, >>>) that precede commands
3. Handle OCR output carefully:
   • Do not assume commands start at line beginning (OCR introduces whitespace)
   • Account for partial prompt character recognition (e.g., > may become › or »)
4. Use flexible pattern matching:

   # More robust than grep "^>"
   import re
   command_pattern = re.compile(r'[>›»]\s*(.+)')
   

Step 6: Clean and Deduplicate Results

Critical: Understand the data domain before cleaning:

1. Preserve legitimate duplicates: In many contexts (games, shell sessions), the same command can appear multiple times intentionally
2. Use temporal deduplication: Only remove duplicates from consecutive frames showing the same command, not all duplicates globally
3. Handle partial commands: Commands being typed appear partially; only capture complete commands
4. Validate corrections: When fixing OCR errors, verify corrections are contextually appropriate

Temporal deduplication approach:

def temporal_dedupe(commands):
    """Remove only consecutive duplicates, preserving repeated commands."""
    result = []
    prev = None
    for cmd in commands:
        if cmd != prev:
            result.append(cmd)
            prev = cmd
    return result

Step 7: Verify Results

Verification is essential for accuracy:

1. Sample verification: Manually compare extracted commands against source frames for a random sample
2. Domain validation: If extracting game commands, verify they are valid commands for that game
3. Sequence logic check: Verify the command sequence makes logical sense (e.g., movement commands follow plausible paths)
4. Count verification: Compare total extracted commands against expected count based on video length and typing speed

Common Pitfalls

OCR Quality Issues

• Mistake: Using default OCR settings without optimization
• Solution: Always tune --psm mode and image preprocessing on sample frames first

Incorrect Deduplication

• Mistake: Using global deduplication (e.g., awk '!seen[$0]++') which removes all repeated commands
• Solution: Use temporal deduplication that only removes consecutive duplicates

Prompt Detection Failures

• Mistake: Using rigid patterns like grep "^>" that assume specific formatting
• Solution: Use flexible regex that accounts for OCR variations and whitespace

Wasted Tool Installation

• Mistake: Installing large packages (Whisper for audio) without verifying they're needed
• Solution: Check if audio contains useful content before installing audio processing tools

No Intermediate Checkpointing

• Mistake: Processing all frames without saving intermediate results, losing progress on timeouts
• Solution: Save results after each processing stage; implement progress checkpoints

Abandoned Verification

• Mistake: Not validating extracted commands against source material
• Solution: Always verify a sample of extractions and validate overall sequence logic

Verification Checklist

Before finalizing extracted commands:

• Sample of extracted commands verified against source frames
• Command count is reasonable for video duration
• No obvious OCR artifacts remain (random characters, split words)
• Legitimate repeated commands are preserved (not incorrectly deduplicated)
• Command sequence follows logical order
• Domain-specific validation performed (e.g., commands are valid for the game/application)

Tool Selection Guide

Task	Recommended Tool	Notes
Video download	yt-dlp	More maintained than youtube-dl
Frame extraction	ffmpeg	Industry standard, reliable
OCR	tesseract via pytesseract	Use Python bindings for efficiency
Image preprocessing	OpenCV (cv2)	Flexible, well-documented
Pattern matching	Python re module	More flexible than grep