Literature Engineer (evidence collector)

Goal: build a large, verifiable candidate pool for downstream dedupe/rank, mapping, notes, citations, and drafting.

This skill is intentionally evidence-first: if you can't reach the target size with verifiable IDs/provenance, the correct behavior is to block and ask for more exports / enable network, not to fabricate.

Load Order

Always read:

• references/domain_pack_overview.md — how domain packs drive topic-specific behavior

Domain packs (loaded by topic match):

• assets/domain_packs/llm_agents.json — pinned classic/survey arXiv IDs for LLM agent topics

Script Boundary

Use scripts/run.py only for:

• multi-route offline import, normalization, and provenance tagging
• online arXiv/Semantic Scholar API retrieval
• snowball expansion and deduplication
• retrieval report generation

Do not treat run.py as the place for:

• hardcoded pinned arXiv ID lists (use domain packs)
• hardcoded topic detection logic (use domain packs)

Inputs

• queries.md
  • keywords, exclude, max_results, time window
• Optional offline sources (any combination; all are merged):
  • papers/import.(csv|json|jsonl|bib)
  • papers/arxiv_export.(csv|json|jsonl|bib)
  • papers/imports/*.(csv|json|jsonl|bib)
• Optional snowball exports (offline):
  • papers/snowball/*.(csv|json|jsonl|bib)

Outputs

• papers/papers_raw.jsonl
  • 1 record per line; minimum fields:
    • title (str), authors (list[str]), year (int|""), url (str)
    • stable identifier(s): arxiv_id and/or doi
    • abstract (str; may be empty in offline mode)
    • source (str) + provenance (list[dict])
• papers/papers_raw.csv (human scan)
• papers/retrieval_report.md (route counts, missing-meta stats, next actions)

Workflow (multi-route)

1. Offline-first merge: ingest all available offline exports (and label provenance per file).
2. Online retrieval (optional): if enabled, run arXiv API retrieval for each keyword query.
3. Snowballing (optional): expand from seed papers via references/cited-by (online), or merge offline snowball exports.
4. Normalize + dedupe: canonicalize IDs/URLs, merge duplicates while unioning provenance.
5. Report: write a concise retrieval report with coverage buckets and missing-meta counts.

Quality checklist

• Candidate pool size target met (A150++: ≥1200) without fabrication.
• Each record has a stable identifier (arxiv_id or doi, plus url).
• Each record has provenance: which route/file/API produced it.

Script

Quick Start

• python .codex/skills/literature-engineer/scripts/run.py --help

All Options

• See python .codex/skills/literature-engineer/scripts/run.py --help.
• Reads retrieval config from queries.md.
• Offline inputs (merged if present): papers/import.(csv|json|jsonl|bib), papers/arxiv_export.(csv|json|jsonl|bib), papers/imports/*.(csv|json|jsonl|bib).
• Optional offline snowball inputs: papers/snowball/*.(csv|json|jsonl|bib).
• Online expansion requires network: use --online and/or --snowball.
• Online retrieval is best-effort: arXiv API can be flaky in some environments; the script will also attempt a Semantic Scholar route when needed.
• For LLM-agent topics, the script also performs a best-effort pinned arXiv id_list fetch (canonical classics like ReAct/Toolformer/Reflexion/Voyager/Tree-of-Thoughts + a small prior-survey seed set) so ref.bib can include must-cite anchors even when keyword search misses them.
• If HTTPS/TLS to external domains is unstable, the Semantic Scholar route is fetched via the r.jina.ai proxy so the pipeline can still self-boot without manual exports.
• When an online run returns 0 records due to transient network errors, a simple rerun is often sufficient (the pipeline should not fabricate).

Examples

• Offline imports only:

  • Put exports under papers/imports/ then run:
    • python .codex/skills/literature-engineer/scripts/run.py --workspace <ws>

• Explicit offline inputs (multi-route):

  • python .codex/skills/literature-engineer/scripts/run.py --workspace <ws> --input path/to/a.bib --input path/to/b.jsonl

• Online arXiv retrieval (needs network):

  • python .codex/skills/literature-engineer/scripts/run.py --workspace <ws> --online

• Snowballing (needs network unless you provide offline snowball exports):

  • python .codex/skills/literature-engineer/scripts/run.py --workspace <ws> --snowball

Troubleshooting

Issue: can't reach ≥1200 papers

Symptom:

• papers/papers_raw.jsonl size is far below target; later stages will fail mapping/bindings and citation density.

Causes:

• Only a small offline export was provided.
• Network is blocked so online retrieval/snowballing can't run.

Solutions:

• Provide additional exports under papers/imports/ (multiple routes/queries).
• Provide snowball exports under papers/snowball/.
• Enable network and rerun with --online --snowball.

Issue: many records missing stable IDs

Symptom:

• Report shows many entries with empty arxiv_id and doi.

Solutions:

• Prefer arXiv/OpenReview/ACL exports that include stable IDs.
• If you have network, rerun with --online to backfill arXiv IDs.
• Filter out ID-less entries before downstream citation generation.