Process FAQ - FAQ Knowledge Base Processor

Transform raw FAQ documents into RAG-optimized structured format with intelligent content expansion and analysis.

What This Skill Does

This skill helps you:

1. Convert FAQ documents to readable Markdown format (script)
2. Analyze FAQ content and identify expansion opportunities (Claude)
3. Expand content: split complex questions, rewrite answers (Claude)
4. Standardize format and generate keywords automatically (script)

Supported Input Formats

• Excel (.xlsx)
• Word (.docx)
• PDF (.pdf)
• Text (.txt)

Workflow Overview (3-Step Process)

Step 1: Convert → Markdown (script)
Step 2: Expand → Enhanced FAQ (Claude - THIS IS THE KEY STEP!)
Step 3: Standardize → Final RAG format (script)

New Workflow (Claude + Script Collaboration)

Step 1: Convert to Markdown (Script)

First, convert the input file to Markdown so Claude can read and analyze it:

python process-faq/scripts/convert_to_markdown.py <input_file>

This creates a *_for_analysis.md file with structured FAQ content.

Why Markdown?

• Claude can directly read and understand the content
• Better for analyzing content quality vs just checking format
• Allows for nuanced, intelligent analysis

Step 2: Claude Analyzes and Expands Content (CRITICAL!)

This is the most important step where YOU (Claude) create value!

Read the Markdown file and perform deep content analysis and expansion:

Phase A: Content Quality Analysis (质量检查)

IMPORTANT: Do this BEFORE expanding content!

Identify and document issues:

1. Logical Issues (逻辑问题)

   • Contradictions: Do different answers give conflicting information?
     • Example: Q1 says "支持退货" but Q2 says "不支持退货"
   • Inconsistencies: Do similar questions have different answers?
     • Example: "配送时间 3 天" vs "配送时间 5-7 天"
   • Outdated Information: References to old products, prices, or policies?

2. Duplicate/Redundant Content (重复内容)

   • Exact Duplicates: Same question appears multiple times
   • Semantic Duplicates: "如何登录?" vs "怎么登录?" (same meaning)
   • Overlapping Answers: Multiple questions share 80%+ identical content

3. Missing or Incomplete Information (缺失信息)

   • Incomplete Answers: Too brief, missing key steps or details
   • Missing Context: Assumes knowledge users may not have
   • Broken Logic: Answer doesn't actually address the question

4. Clarity Issues (表达问题)

   • Unclear Questions: Too vague ("支持什么?")
   • Ambiguous Terms: Undefined jargon or acronyms
   • Poor Structure: Wall of text without formatting

Action Required: Document all issues found and decide:

• ✅ Fix: Resolve contradictions, merge duplicates, clarify ambiguities
• ⚠️ Flag: Note issues that need user clarification
• ❌ Remove: Delete truly useless or incorrect content

Phase B: Content Expansion Strategy

Goal: Transform a small FAQ into a comprehensive, high-quality knowledge base

CRITICAL: Expansion must happen AFTER quality analysis!

Key expansion techniques:

1. Resolve Issues First (基于 Phase A 的发现)

   • Fix Contradictions: Choose the correct information, note uncertainty for user
   • Merge Duplicates: Combine semantically identical questions into one best version
   • Complete Incomplete: Fill in missing steps, add context
   • Clarify Ambiguities: Reword vague questions to be specific

2. Split Complex Questions

   • If one question like "如何使用产品?" contains multiple sub-topics
   • Break it into specific questions: "如何安装?", "如何配置?", "如何维护?"

3. Extract Knowledge Points from Long Answers

   • If one answer is 500+ characters and covers multiple topics
   • Identify each distinct knowledge point
   • Create a dedicated Q&A for each point
   • BUT: Ensure each extracted point is accurate and consistent

4. Identify Missing Common Questions

   • Based on the domain, what would users naturally ask?
   • Add questions that should exist but don't
   • Consider user journey: pre-purchase → purchase → usage → troubleshooting

5. Rewrite Answers for Clarity

   • Make each answer concise and focused
   • Remove sales language if not appropriate
   • Add structure (numbered lists, bullet points)
   • Ensure consistency with other related answers

Example: Quality Analysis + Expansion

Original (with issues):

Q1: 你们是怎么调理睡眠的?
A1: [500字，包含：产品介绍、使用流程、手环说明、"手环180元"、售后政策等]

Q2: 先用后付是什么意思?
A2: [与Q1相同的500字回答]

Q3: 华为手环多少钱？
A3: 手环200元左右

Phase A Analysis - Issues Found:

• ❌ Contradiction: Q1 说"手环 180 元"，Q3 说"手环 200 元左右"
• ❌ Duplicate: Q1 和 Q2 的回答完全相同
• ⚠️ Overlapping: 三个问题都提到手环，信息散乱

Phase B Expansion - After Fixes:

Q1: 你们是怎么调理睡眠的？
A1: 我们通过太赫兹能量睡垫来调理睡眠，能帮助疏通经络、改善气血循环...
[简洁，只讲核心调理原理，不再包含价格等无关信息]

Q2: 先用后付是什么意思？
A2: 先用后付就是您可以先把产品拿回家免费体验，有效果再付款...
[独立回答，不再重复Q1的内容]

Q3: 华为手环多少钱？
A3: 华为手环180元左右（已统一价格，解决矛盾）

Q4: 为什么要用华为手环测睡眠？
A4: 手环能精准测出入睡时间、深睡时长等数据...
[新增问题，补充手环相关信息]

Q5: 手环怎么使用？
A5: 充电后戴在手腕上，连接手机APP即可...
[新增问题，完善手环知识点]

Summary:

• Fixed 1 contradiction (价格统一)
• Merged 1 duplicate (Q1 和 Q2)
• Expanded 3 → 5 FAQs (提取知识点)
• Each answer is now focused and consistent

Phase C: Categorization

Design a clear category structure:

• Group related questions together
• Use domain-appropriate category names
• Aim for 5-10 main categories

Step 3: User Consultation (Optional)

Use the AskUserQuestion tool if you need clarification on:

• Domain-specific terminology
• Tone preferences (formal vs casual)
• Whether to keep sales language
• Priority topics to expand

In most cases, you can proceed directly to Step 4 based on your analysis.

Step 4: Generate Expanded FAQ (Excel)

CRITICAL: This is where you do the actual content expansion!

Create a new Excel file with the expanded FAQ content:

File naming: <original_name>_expanded.xlsx

Required columns:

• 分类 (Category)
• 问题 (Question)
• 回答 (Answer)

Optional column (script will generate if missing):

• 关键词 (Keywords) - you can leave this empty, script will auto-generate

How to create the file:

IMPORTANT (Cross-Platform Compatibility):

• DO NOT use python -c "..." to run inline Python code - this causes quote escaping issues on Windows
• ALWAYS use the Write tool to create a .py script file first, then run it with python script.py

Step-by-step approach:

1. First, use the Write tool to create a Python script (e.g., create_faq.py):

# create_faq.py - Use Write tool to create this file
import pandas as pd

data = [
    {
        "分类": "睡眠问题咨询",
        "问题": "你们是怎么调理睡眠的？",
        "回答": "我们通过太赫兹能量睡垫来调理睡眠..."
    },
    {
        "分类": "睡眠问题咨询",
        "问题": "我总是入睡困难怎么办？",
        "回答": "入睡困难通常和气血不畅有关..."
    },
    # ... 添加所有扩展后的FAQ
]

df = pd.DataFrame(data)
df.to_excel("filename_expanded.xlsx", index=False)
print("Successfully created filename_expanded.xlsx")

2. Then run the script using Bash:

python create_faq.py

3. Clean up the temporary script after use:

rm create_faq.py  # or 'del create_faq.py' on Windows CMD

Quality checklist before saving:

• Each question is specific and focused
• Each answer is concise (typically 50-200 characters)
• Questions are grouped by logical categories
• All important knowledge points are covered
• No redundant or duplicate questions

Step 5: Standardize Format with Script

Use the script to process the expanded file:

python process-faq/scripts/generate_rag_faq.py <expanded_file> <final_output_file>

What the script does:

• Auto-generates keywords using jieba TF-IDF
• Applies professional Excel formatting
• Sets proper column widths and styles
• Performs final duplicate check
• Creates the final RAG-optimized knowledge base

Example:

python process-faq/scripts/generate_rag_faq.py 申花太赫兹_expanded.xlsx 申花太赫兹_RAG_优化版.xlsx

Complete Example Workflow

User: "Please process 申花太赫兹知识库.xlsx and convert it to RAG format"

You (Claude):

Step 1: Convert to Markdown

python process-faq/scripts/convert_to_markdown.py 申花太赫兹知识库.xlsx

Step 2: Read and Analyze

• Read the generated 申花太赫兹知识库_for_analysis.md using Read tool
• Analyze: "I found that the original 7 FAQs have very long answers (500+ characters each)"
• Identify: "Each answer actually covers 3-5 different topics"

Step 3: Expand Content

• Extract knowledge points from long answers
• Create dedicated Q&A for each point
• Example: From 1 question about "如何调理睡眠", expand to:
  • 你们是怎么调理睡眠的？(调理原理)
  • 我总是入睡困难怎么办？(具体症状)
  • 先用后付是什么意思？(购买政策)
  • 为什么要用华为手环？(设备说明)
  • 手环怎么使用？(使用指南)
  • 等等...

Step 4: Generate Expanded Excel Use pandas to create 申花太赫兹知识库_expanded.xlsx with 31 focused FAQs (from original 7)

Step 5: Standardize Format

python process-faq/scripts/generate_rag_faq.py 申花太赫兹知识库_expanded.xlsx 申花太赫兹知识库_RAG_优化版.xlsx

Step 6: Report Results

• "Successfully expanded 7 FAQs into 31 focused entries"
• "Organized into 8 categories"
• "Auto-generated keywords for all entries"
• "Final file ready for RAG system"

Key Principles

1. Content Expansion is Key

The main value you provide is:

• Expanding small FAQs into comprehensive knowledge bases
• Extracting knowledge points from long answers
• Creating focused, specific Q&A pairs
• NOT just cleaning up format or removing duplicates

2. Quality Over Quantity (But More is Often Better)

• Each FAQ should be focused and specific
• Better to have 30 focused FAQs than 5 long ones
• Each answer should ideally be 50-200 characters
• Long answers (500+) should be split into multiple FAQs

3. Think Like a RAG System

• How would users search for this information?
• What specific questions would they ask?
• Would this answer be found by semantic search?
• Is the question specific enough to match user intent?

4. Division of Labor

Claude does (creative work):

• Content understanding
• Knowledge point extraction
• Question splitting and rewording
• Answer rewriting
• Category design

Script does (mechanical work):

• Keyword extraction (jieba TF-IDF)
• Format standardization
• Excel styling
• Final duplicate check

Quality Analysis and Expansion Checklist

Phase A: Quality Analysis (Must do FIRST!)

• Check for contradictions: Do different FAQs give conflicting information?
• Identify duplicates: Exact or semantic duplicates (same meaning, different wording)
• Find inconsistencies: Similar questions with different answers (e.g., different prices, timeframes)
• Spot incomplete info: Answers missing key steps or context
• Flag unclear content: Vague questions, ambiguous terms, undefined jargon
• Note outdated info: References to old products, policies, or prices

Action: Document all issues and plan how to resolve them

Phase B: Content Expansion (After quality fixes!)

• Contradictions resolved: Unified conflicting information
• Duplicates merged: Combined semantically identical questions
• Long answers split: Any answer >300 characters covering multiple topics
• Each Q&A focused: One question = one specific topic
• All knowledge points extracted: No information lost from original
• Questions are specific: Avoided vague questions like "如何使用?"
• Answers are concise: Typically 50-200 characters per answer
• Answers are consistent: Related FAQs give aligned information
• Categories are clear: 5-10 logical categories based on the domain
• Common questions added: Anticipated natural user questions
• Proper structure: Used lists, numbering, or bullet points where appropriate

Output Format

The final Excel file will have:

分类	问题	回答	关键词
Category	Question	Answer	Keywords

Example:

分类	问题	回答	关键词
账户管理	如何重置密码?	1. 点击"忘记密码"\n2. 输入邮箱\n3. 查收重置链接	密码,重置,账户
支付问题	支持哪些支付方式?	我们支持:\n- 支付宝\n- 微信支付\n- 银行卡	支付,方式,支付宝

Best Practices

1. Always Convert First: Don't try to analyze binary files directly
2. Read Thoroughly: Actually read the Markdown file, understand the domain
3. Quality BEFORE Expansion: Analyze issues first, then expand
   • Don't expand broken content - fix it first!
   • Resolve contradictions before creating more FAQs
   • Merge duplicates before splitting long answers
4. Look for Logic Issues:
   • Contradicting information across FAQs
   • Inconsistent answers to similar questions
   • Missing prerequisites or context
5. Extract Knowledge Points: Identify every distinct topic in long answers
6. Ensure Consistency: Related FAQs should give aligned, non-conflicting information
7. Create Focused FAQs: Each Q&A should cover one specific topic
8. Think Like Users: What would they search for? What questions would they ask?
9. Generate Intermediate File: Create *_expanded.xlsx before running the script
10. Let Script Handle Keywords: Don't manually generate keywords, let jieba do it

Common Expansion Scenarios

Scenario 1: Long Answer with Multiple Topics

Original:

Q: 你们的产品怎么样?
A: 我们的产品质量好、价格实惠、支持30天退货、全国包邮、还有24小时客服...

Expanded:

Q1: 你们的产品质量如何?
A1: 产品经过严格质检，质量可靠...

Q2: 产品价格贵吗?
A2: 价格实惠，性价比高...

Q3: 支持退货吗?
A3: 支持30天无理由退货...

Q4: 包邮吗?
A4: 全国包邮，无需额外运费...

Q5: 有客服支持吗?
A5: 提供24小时在线客服...

Scenario 2: Vague Question Needs Specificity

Original:

Q: 如何使用?
A: [Long explanation covering installation, configuration, daily use, troubleshooting...]

Expanded:

Q1: 如何安装产品?
A1: [Installation steps]

Q2: 如何进行初始配置?
A2: [Configuration guide]

Q3: 日常使用注意事项有哪些?
A3: [Daily usage tips]

Q4: 遇到问题怎么排查?
A4: [Troubleshooting steps]

Scenario 3: Contradictions and Inconsistencies

Original (with logic issues):

Q1: 配送需要多久?
A1: 一般3-5个工作日送达

Q2: 什么时候能收到货?
A2: 通常7天内送达

Q3: 支持退货吗?
A3: 支持7天无理由退货

Q4: 可以退款吗?
A4: 不支持退款，只能换货

Issues Found:

• ❌ Contradiction: Q1 说 3-5 天，Q2 说 7 天
• ❌ Contradiction: Q3 支持退货，Q4 说不支持退款
• ⚠️ Semantic duplicate: Q1 和 Q2 问的是同一件事

Fixed and Expanded:

Q1: 配送需要多久?
A1: 一般3-5个工作日送达（偏远地区可能需要7天）
[统一时间信息，说明例外情况]

Q2: 支持退货吗?
A2: 支持7天无理由退货退款
[解决矛盾：统一退货和退款政策]

Q3: 如何申请退货?
A3: 联系客服说明原因，获得退货地址后寄回即可
[新增：补充退货流程]

Q4: 退货运费谁承担?
A4: 质量问题我们承担，非质量问题需您承担
[新增：补充退货细节]

Summary:

• Resolved 2 contradictions
• Merged 1 semantic duplicate
• Expanded with 2 new related questions
• All information now consistent

Scenario 4: Missing Obvious Questions

Original: Only has "如何注册账户?"

Should also add:

• 注册需要提供哪些信息?
• 可以用手机号注册吗?
• 忘记密码怎么办?
• 如何修改个人信息?
• 如何注销账户?

Error Handling

If conversion fails:

• Check file format is supported
• Verify file is not corrupted
• Try opening the file manually first
• Check file encoding (should be UTF-8)

If content is unstructured:

• The Markdown will show raw text
• You'll need to manually identify Q&A pairs
• Suggest restructuring the source document

Dependencies

Required Python packages:

• pandas (data handling)
• openpyxl (Excel support)
• python-docx (Word support)
• PyPDF2 (PDF support)
• jieba (Chinese text processing)

Install with:

pip install -r process-faq/requirements.txt