Financial Document Processor

Overview

This skill provides guidance for extracting structured data from financial documents (invoices, receipts, statements, etc.) using OCR and PDF text extraction. It emphasizes data safety practices that prevent catastrophic failures from destructive operations.

Critical Data Safety Principles

NEVER perform destructive operations on source data without verification or backup.

Before any file processing:

1. Create a backup of all source files before processing
2. Work on copies, not originals
3. Verify outputs match expectations before any cleanup
4. Use atomic operations (copy → verify → delete) instead of direct moves

Safe File Operation Pattern

# CORRECT: Copy first, verify, then clean up
cp -r /source/documents/ /backup/documents/
# ... process files ...
# ... verify outputs match expectations ...
# Only after verification: rm /backup/documents/

# WRONG: Delete before moving (data loss risk)
rm -f /source/*.pdf && mv /source/* /dest/  # Files deleted before move!

Workflow

Step 1: Assess the Environment and Requirements

Before writing any processing code:

1. List all source files and note their exact paths
2. Identify file types (PDF text-based, PDF scanned, JPG, PNG, etc.)
3. Check available tools: which tesseract, which pdftotext, python3 -c "import pypdf"
4. Understand the expected output format (CSV columns, required fields, etc.)

Step 2: Create Backup

Always backup source files before any processing:

# Create timestamped backup directory
BACKUP_DIR="/tmp/backup_$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"
cp -r /path/to/source/documents/* "$BACKUP_DIR/"
echo "Backup created at: $BACKUP_DIR"

Step 3: Test Extraction on Sample Files

Before processing all documents:

1. Select 2-3 representative files (different formats, edge cases)
2. Test OCR/extraction on these samples
3. Verify extracted values match visual inspection
4. Adjust extraction logic based on sample results

# Test extraction on a single file first
sample_file = "/path/to/sample_invoice.pdf"
extracted_data = extract_document(sample_file)
print(f"Extracted: {extracted_data}")
# Manually verify these values match the document

Step 4: Handle Format Variations

Financial documents often have format variations:

• Number formats: European (1.234,56) vs US (1,234.56)
• Date formats: DD/MM/YYYY vs MM/DD/YYYY vs YYYY-MM-DD
• Currency symbols: $, €, £, or spelled out
• Empty/missing fields: VAT may be blank, not zero

def parse_amount(text):
    """Handle multiple number format conventions."""
    # Remove currency symbols and whitespace
    cleaned = re.sub(r'[$€£\s]', '', text)

    # Detect European format (comma as decimal separator)
    if re.match(r'^\d{1,3}(\.\d{3})*,\d{2}$', cleaned):
        cleaned = cleaned.replace('.', '').replace(',', '.')
    # US format (comma as thousands separator)
    elif ',' in cleaned:
        cleaned = cleaned.replace(',', '')

    return float(cleaned) if cleaned else None

Step 5: Process All Documents

After successful sample testing:

1. Process documents one at a time with error handling
2. Log extraction results for each document
3. Collect all results before writing output file

results = []
errors = []

for doc_path in document_paths:
    try:
        data = extract_document(doc_path)
        results.append(data)
        print(f"✓ Processed: {doc_path}")
    except Exception as e:
        errors.append((doc_path, str(e)))
        print(f"✗ Failed: {doc_path} - {e}")

if errors:
    print(f"\nWarning: {len(errors)} documents failed to process")
    for path, error in errors:
        print(f"  - {path}: {error}")

Step 6: Verify Before File Operations

Before moving files or writing final outputs:

1. Compare extracted record count to source file count
2. Spot-check extracted values against source documents
3. Verify output format matches requirements

# Verification checklist
assert len(results) == len(document_paths), "Record count mismatch"

# Spot-check a few values
for sample in random.sample(results, min(3, len(results))):
    print(f"Please verify: {sample['filename']} -> Total: {sample['total']}")

Step 7: Move Files (Only After Verification)

Only after verification passes:

# Move files to destination (not delete!)
for file in /source/documents/*.pdf; do
    mv "$file" /processed/
done

# Only remove backup after confirming processed files exist
ls /processed/*.pdf && rm -rf "$BACKUP_DIR"

Common Pitfalls

1. Destructive Commands Without Backup

Problem: Using rm or overwriting files before verifying success. Prevention: Always create backups first; use copy-verify-delete pattern.

2. Command Order in Shell Pipelines

Problem: rm -f *.pdf && mv *.pdf /dest/ - files are deleted before move. Prevention: Test commands on sample data; understand execution order.

3. Incomplete Script Verification

Problem: Running truncated or incomplete scripts on production data. Prevention: Verify script content before execution; test on samples first.

4. Fabricating Missing Data

Problem: Writing guessed values when extraction fails. Prevention: Report failures explicitly; use null/empty for missing values.

5. Premature Optimization

Problem: Immediately reprocessing when values look wrong without investigation. Prevention: First analyze OCR output and extraction logic issues without moving files.

6. PDF vs Image Handling

Problem: Using OCR on text-based PDFs or text extraction on scanned PDFs. Prevention: Check if PDF has extractable text before choosing extraction method.

def is_text_based_pdf(pdf_path):
    """Check if PDF contains extractable text."""
    from pypdf import PdfReader
    reader = PdfReader(pdf_path)
    for page in reader.pages:
        if page.extract_text().strip():
            return True
    return False

Verification Strategies

Pre-Processing Verification

• Source files exist and are readable
• Backup created successfully
• Required tools installed (tesseract, pdftotext, pypdf)
• Sample extraction produces reasonable values

Post-Processing Verification

• Output record count matches input file count
• No extraction errors occurred (or errors are documented)
• Spot-checked values match source documents
• Output format matches requirements (correct columns, types)
• Files moved to correct destinations
• Original backup preserved until final verification

Recovery Plan

If something goes wrong:

1. Stop immediately - do not continue processing
2. Restore from backup: cp -r "$BACKUP_DIR"/* /source/
3. Investigate the failure before retrying
4. Fix extraction logic on samples before reprocessing all files

Tool Selection Guide

File Type	Primary Tool	Fallback
Text-based PDF	pypdf, pdftotext	-
Scanned PDF	tesseract (after pdf2image)	pypdf
JPG/PNG images	tesseract	-
Mixed PDF (text + scans)	pypdf first, tesseract for image pages	-

Install dependencies:

# System packages
apt-get install tesseract-ocr poppler-utils

# Python packages
pip install pypdf pytesseract pdf2image pillow