Data Ingest — Universal Text Source Handler

You are ingesting arbitrary text data into an Obsidian wiki. The source could be anything — conversation exports, log files, transcripts, data dumps. Your job is to figure out the format, extract knowledge, and distill it into wiki pages.

Before You Start

Read .env to get OBSIDIAN_VAULT_PATH
Read .manifest.json at the vault root — check if this source has been ingested before
Read index.md at the vault root to know what already exists

If the source path is already in .manifest.json and the file hasn't been modified since ingested_at, tell the user it's already been ingested. Ask if they want to re-ingest anyway.

Step 1: Identify the Source Format

Read the file(s) the user points you at. Common formats you'll encounter:

Format	How to identify	How to read
JSON / JSONL	`.json` / `.jsonl` extension, starts with `{` or `[`	Parse with Read tool, look for message/content fields
Markdown	`.md` extension	Read directly
Plain text	`.txt` extension or no extension	Read directly
CSV / TSV	`.csv` / `.tsv`, comma or tab separated	Parse rows, identify columns
HTML	`.html`, starts with `<`	Extract text content, ignore markup
Chat export	Varies — look for turn-taking patterns (user/assistant, human/ai, timestamps)	Extract the dialogue turns
Images	`.png` / `.jpg` / `.jpeg` / `.webp` / `.gif`	Requires a vision-capable model. Use the Read tool — it renders images into your context. Screenshots, whiteboards, diagrams all qualify. Models without vision support should skip and report which files were skipped.

Common Chat Export Formats

ChatGPT export (conversations.json):

[{"title": "...", "mapping": {"node-id": {"message": {"role": "user", "content": {"parts": ["text"]}}}}}]

Slack export (directory of JSON files per channel):

[{"user": "U123", "text": "message", "ts": "1234567890.123456"}]

Generic chat log (timestamped text):

[2024-03-15 10:30] User: message here
[2024-03-15 10:31] Bot: response here

Don't try to handle every format upfront — read the actual data, figure out the structure, and adapt.

Images and visual sources

When the user dumps a folder of screenshots, whiteboard photos, or diagram exports, treat each image as a source:

Use the Read tool on the image path — it will render the image into context.
Transcribe any visible text verbatim (this is the only extracted content from an image).
Describe structure: for diagrams, list nodes/edges; for screenshots, name the app and what's on screen.
Extract the concepts the image conveys — what's it about? Most of this is ^[inferred].
Flag anything you can't read, can't identify, or are guessing at with ^[ambiguous].

Image-derived pages will skew heavily inferred — that's expected and the provenance markers will reflect it. Set source_type: "image" in the manifest entry. Skip files with EXIF-only changes (re-saved with no visual diff) — compare via the standard delta logic.

For folders of mixed images (e.g. a screenshot timeline of a debugging session), cluster by visible topic rather than per-file. Twenty screenshots of the same UI bug should produce one wiki page, not twenty.

Step 2: Extract Knowledge

Regardless of format, extract the same things:

Topics discussed — what subjects come up?
Decisions made — what was concluded or decided?
Facts learned — what concrete information is stated?
Procedures described — how-to knowledge, workflows, steps
Entities mentioned — people, tools, projects, organizations
Connections — how do topics relate to each other and to existing wiki content?

For conversation data specifically:

Focus on the substance, not the dialogue. A 50-message debugging session might yield one skills page about the fix. A long brainstorming chat might yield three concept pages.

Skip:

Greetings, pleasantries, meta-conversation ("can you help me with...")
Repetitive back-and-forth that doesn't add new information
Raw code dumps (unless they illustrate a reusable pattern)

Step 3: Cluster and Deduplicate

Before creating pages:

Group extracted knowledge by topic (not by source file or conversation)
Check existing wiki pages — does this knowledge belong on an existing page?
Merge overlapping information from multiple sources
Note contradictions between sources

Step 4: Distill into Wiki Pages

Follow the wiki-ingest skill's process for creating/updating pages:

Use correct category directories (concepts/, entities/, skills/, etc.)
Add YAML frontmatter with title, category, tags, sources
Use [[wikilinks]] to connect to existing pages
Attribute claims to their source
Write a summary: frontmatter field on every new page (1–2 sentences, ≤200 characters) answering "what is this page about?" — this is what downstream skills read to avoid opening the page body.
Apply provenance markers per the convention in llm-wiki. Conversation, log, and chat data tend to be high-inference — you're often reading between the turns to extract a coherent claim. Be liberal with ^[inferred] for synthesized patterns and with ^[ambiguous] when speakers contradict each other or you're unsure who's right. Write a provenance: frontmatter block on each new/updated page.

Step 5: Update Manifest and Special Files

.manifest.json — Add an entry for each source file processed:

{
  "ingested_at": "TIMESTAMP",
  "size_bytes": FILE_SIZE,
  "modified_at": FILE_MTIME,
  "source_type": "data",  // or "image" for png/jpg/webp/gif sources
  "project": "project-name-or-null",
  "pages_created": ["list/of/pages.md"],
  "pages_updated": ["list/of/pages.md"]
}

index.md and log.md:

- [TIMESTAMP] DATA_INGEST source="path/to/data" format=FORMAT pages_updated=X pages_created=Y

Tips

When in doubt about format, just read it. The Read tool will show you what you're dealing with.
Large files: Read in chunks using offset/limit. Don't try to load a 10MB JSON in one go.
Multiple files: Process them in order, building up wiki pages incrementally.
Binary files: Skip them, except images — those are first-class sources via the Read tool's vision support.
Encoding issues: If you see garbled text, mention it to the user and move on.

data-ingest