CSV Handler

Quick Start

Read a CSV file

Inspect structure and preview data using the bundled script:

python3 scripts/csv_read.py data.csv --info      # Structure analysis (columns, types, row count)
python3 scripts/csv_read.py data.csv --head 10    # Preview first 10 rows
python3 scripts/csv_read.py data.csv --search "keyword"  # Search rows

Write a CSV file

Create CSV from JSON data or transform existing files:

# From JSON array of objects
python3 scripts/csv_write.py output.csv --json '[{"name":"Alice","age":30},{"name":"Bob","age":25}]'

# Transform existing CSV (filter + sort + select columns)
python3 scripts/csv_write.py output.csv --from input.csv --filter "status==active" --sort name --select "name,email"

# Excel-compatible UTF-8 with BOM
python3 scripts/csv_write.py output.csv --from input.csv --bom

Reading CSV Files

Workflow

1. Inspect structure first - Run csv_read.py --info to understand columns, types, and encoding
2. Preview data - Run csv_read.py --head N to verify content
3. Search if needed - Use --search to find specific data
4. Force encoding/delimiter if auto-detection fails (see encoding-guide.md)

Direct Python (without bundled scripts)

For inline CSV processing within code, use Python's csv module:

import csv

# Read with encoding
with open("data.csv", "r", encoding="utf-8-sig") as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(row["column_name"])

# Read TSV
with open("data.tsv", "r", encoding="utf-8") as f:
    reader = csv.reader(f, delimiter="\t")
    headers = next(reader)
    for row in reader:
        print(row)

Writing CSV Files

Workflow

1. Determine target encoding - UTF-8 for modern systems, UTF-8 with BOM for Excel, Shift_JIS for legacy
2. Determine delimiter - comma for CSV, tab for TSV
3. Write using script or inline Python

Direct Python (without bundled scripts)

import csv

# Write CSV with BOM for Excel
with open("output.csv", "w", encoding="utf-8-sig", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["name", "age", "email"])
    writer.writerow(["Alice", 30, "alice@example.com"])

# Write from list of dicts
with open("output.csv", "w", encoding="utf-8-sig", newline="") as f:
    fieldnames = ["name", "age", "email"]
    writer = csv.DictWriter(f, fieldnames=fieldnames)
    writer.writeheader()
    writer.writerows(data)

Data Transformation

Available operations via csv_write.py

Operation	Flag	Example
Filter rows	--filter	--filter "status==active"
Regex filter	--filter	--filter "name~=^A"
Negative filter	--filter	--filter "status!=deleted"
Sort ascending	--sort	--sort "created_at"
Sort descending	--sort-desc	--sort-desc "score"
Select columns	--select	--select "name,email,phone"
Rename columns	--rename	--rename "old_name=new_name"
Deduplicate	--dedupe	--dedupe "email"
Change encoding	--encoding	--encoding cp932
Add BOM	--bom	For Excel compatibility
Output as TSV	--tsv	Tab-delimited output

Multiple --filter flags can be combined (AND logic).

Encoding Handling

For detailed encoding guidance including Japanese encoding scenarios, see encoding-guide.md.

Quick Reference

Scenario	Encoding	Flag
Modern systems	utf-8	(default)
Excel compatibility	utf-8-sig	--bom
Japanese legacy	cp932	--encoding cp932
Auto-detect input	(automatic)	(default in csv_read.py)

Critical Rules

• Always inspect before modifying - Run csv_read.py --info before any transformation
• Preserve original files - Write to a new file path, never overwrite source
• Verify encoding - If output shows garbled text, check encoding with --info and force correct encoding
• Use newline="" in open() - Required for Python csv module to handle line endings correctly
• Quote fields containing delimiters - Python csv module handles this automatically

Resources

scripts/

• csv_read.py - Read, inspect, and search CSV/TSV files with auto-detection
• csv_write.py - Write and transform CSV/TSV files with encoding and format control

references/

• encoding-guide.md - Detailed encoding detection, Japanese encoding handling, and troubleshooting