Video Understanding

Multi-provider video understanding with automatic fallback and model selection.

Quick Start

# Check available providers
python3 scripts/check_providers.py

# Process a video (auto-selects best provider)
python3 scripts/process_video.py "https://youtube.com/watch?v=..."
python3 scripts/process_video.py /path/to/video.mp4

# Custom prompt
python3 scripts/process_video.py video.mp4 -p "List all products shown with timestamps"

# Use specific provider/model
python3 scripts/process_video.py video.mp4 --provider openrouter -m google/gemini-3-pro-preview

# List available models
python3 scripts/process_video.py --list-models

Provider Hierarchy

Automatically selects the best available provider:

Priority	Provider	Capability	Env Var	Default Model
1	Gemini	Full video	GEMINI_API_KEY	gemini-3-flash-preview
2	Vertex AI	Full video	GOOGLE_APPLICATION_CREDENTIALS	gemini-3-flash-preview
3	OpenRouter	Full video	OPENROUTER_API_KEY	google/gemini-3-flash-preview
4	FFMPEG	Frames + ASR	None (requires ffmpeg + whisper)	scene
5	OpenAI	ASR only	OPENAI_API_KEY	whisper-1
6	AssemblyAI	ASR + analysis	ASSEMBLYAI_API_KEY	best
7	Deepgram	ASR	DEEPGRAM_API_KEY	nova-2
8	Groq	ASR (fast)	GROQ_API_KEY	whisper-large-v3-turbo
9	Local Whisper	ASR (offline)	None	base

Full video = visual + audio analysis. Frames + ASR = extracted screenshots + audio transcription (free, offline). ASR = audio transcription only.

CLI Options

python3 scripts/process_video.py [OPTIONS] SOURCE

Arguments:
  SOURCE              YouTube URL, video URL, or local file path

Options:
  -p, --prompt TEXT   Custom prompt for video understanding
  --provider NAME     Force specific provider
  -m, --model NAME    Force specific model
  --asr-only          Force ASR-only mode (skip visual analysis)
  -o, --output FILE   Write JSON to file instead of stdout
  -q, --quiet         Suppress progress messages
  --list-models       Show available models per provider
  --list-providers    Show available providers as JSON

Model Selection

Each provider supports multiple models. Use --list-models to see options:

python3 scripts/process_video.py --list-models

OpenRouter models:

• google/gemini-3-flash-preview (default) - Fast, free tier
• google/gemini-3-pro-preview - Higher quality

Gemini models:

• gemini-3-flash-preview (default) - Latest, fast
• gemini-3-pro-preview - Highest quality
• gemini-2.5-flash - Stable production fallback

Local Whisper models:

• tiny, base (default), small, medium, large, large-v3

FFMPEG modes (frame extraction strategy):

• scene (default) - Extract frames when scene changes (smart, efficient)
• keyframe - Extract I-frames only (fastest)
• interval - Extract frames at regular intervals (predictable)

Quick Reference

Task	Reference
Setup & API keys	setup-guide.md
Use Gemini for video	gemini.md
Use OpenRouter	openrouter.md
FFMPEG frames (free)	ffmpeg-frames.md
ASR providers	asr-providers.md
Output JSON schema	output-format.md
Video sources & downloading	video-sources.md

Verify Setup

python3 scripts/setup.py  # Check dependencies and API keys

Output Format

All providers return consistent JSON:

{
  "source": {
    "type": "youtube|url|local",
    "path": "...",
    "duration_seconds": 120.5,
    "size_mb": 15.2
  },
  "provider": "openrouter",
  "model": "google/gemini-3-flash-preview",
  "capability": "full_video",
  "response": "...",
  "transcript": [{"start": 0.0, "end": 2.5, "text": "..."}],
  "text": "Full transcript..."
}

Features

• Automatic provider selection based on available API keys
• Model selection per provider with sensible defaults
• Robust path handling for macOS special characters and unicode
• Progress output (use -q for quiet mode)
• File size warnings for API limits
• Sane frame defaults for offline mode (downscaled/compressed images instead of huge 4k frames)
• Auto-conversion of video formats when needed
• YouTube URL support (direct or via download)

Requirements

For full video understanding:

pip install google-generativeai  # Gemini
pip install openai               # OpenRouter

For ASR fallback:

brew install yt-dlp ffmpeg       # Video tools
pip install openai               # OpenAI Whisper
pip install groq                 # Groq Whisper
pip install assemblyai           # AssemblyAI
pip install deepgram-sdk         # Deepgram
pip install openai-whisper       # Local Whisper