343a5eebe3
Some checks failed
Sync to Gitee / sync (push) Has been cancelled
Submit the formed RAG documentation set produced across Sprint-1/2/3 (WS-12 through WS-26) under docs/rag/. Includes: - README.md / INDEX.md: landing + total index (responsibility matrix, review verdicts, dual-link to source issues) - overview/: full-pipeline architecture (4 .mmd diagrams), 11-stage boundary contracts, doc map, source-code inventory - pipeline/: 5 deep-dives (Loader/Parser/Chunking, Embedding, VDB & retrieval, GraphRAG, Rerank/Prompt/LLM) - graphrag/, end-to-end/: v1.0 formal versions with full source retained as reference - evolution/: 11 architecture-refactor proposals, 6-direction roadmap, capability map - review/: S3-T1 / S3-T2 final reviews, S2-T7 final summary - _indexes/: glossary (81 terms), source->doc reverse index, chart index - _release/: v1.0-RC1 release manifest, versioning convention, ops & freshness plan - _meta/README.md: placeholder noting WS-12 governance assets gap Aggregate review score 92.6/100 (8/8 PASS, 31/31 source-code spot checks hit). The legacy docs/ ignore in .gitignore is narrowed to docs/* with an explicit allowlist for docs/rag/. Refs: WS-26 Co-authored-by: multica-agent <github@multica.ai>
265 lines
12 KiB
Markdown
265 lines
12 KiB
Markdown
---
|
||
title: "[S2-T6] 端到端检索-生成调用链路与时序图 — 正式版"
|
||
author: AI 知识库解决方案专家
|
||
reviewer: 知识运营与治理专家
|
||
source-commit: feae2f2e (MemoryBear)
|
||
last-reviewed-at: 2026-05-08
|
||
scope: api/app/{services,app_chat_service,draft_run_service,core/agent/langchain_agent,core/models/{llm,rerank,embedding},core/rag/{nlp/search,vdb/elasticsearch/elasticsearch_vector,app/naive,graphrag/{search,general/index}}}
|
||
version: v1.0
|
||
status: 正式版（已解除占位）
|
||
---
|
||
|
||
# [S2-T6] 端到端检索-生成调用链路与时序图 — 正式版
|
||
|
||
> 本文档为 [WS-24](mention://issue/a07f108d-06ee-41b8-8b57-22455f60ddeb) v1.0 文档全集的正式组成文件，替换 v1.0-RC1 中的占位版本。
|
||
> 原始完整文档与逐节详评见 [WS-20](mention://issue/a3deeaa1-5b30-4da5-b4af-1b081f7f6394) 与 [WS-21](mention://issue/41f2482b-6f3e-4253-95f7-3e22e790f31c) §S2-T6 评审报告。
|
||
|
||
---
|
||
|
||
## 1. 一句话定位
|
||
|
||
本文档是 Sprint-2 的"全链路串联"文档，将 [S2-T1]~[S2-T5] 五篇独立深度文档中的调用栈、数据结构与配置项，整合为**两条端到端时序图**（Query 端 + Indexing 端）、**一张关键路径表**、**三套多场景调用链**与**一张错误降级路径图**。所有函数引用均直接来源于子任务文档，未凭空虚构。
|
||
|
||
---
|
||
|
||
## 2. 评审结果
|
||
|
||
| 维度 | 满分 | 得分 | 关键说明 |
|
||
|---|---:|---:|---|
|
||
| 准确性 | 25 | 24 | 抽检 7/7 命中：`agnet_chat` / `_prepare_messages` / `knowledge_retrieval` / `_retrieve_for_knowledge` / `insert_citations` / `chunk()` / `_classify_error` |
|
||
| 完整性 | 25 | 24 | 5 项硬性验收 100% 满足：Query 端时序图、Indexing 端时序图、关键路径表（15 行）、3 场景调用链、错误降级矩阵（13 行 + 6 路径 + 5 代码片段） |
|
||
| 时效性 | 15 | 14 | frontmatter 完整规范（author / source-commit `feae2f2e` / last-reviewed-at / scope），仅缺 reviewer 字段（等待评审填入） |
|
||
| 可读性 | 15 | 14 | Mermaid `autonumber` + `Note over` + `alt/par/loop` 专业级写法；瓶颈🔴🟡🟢色标视觉化优秀 |
|
||
| 可执行性 | 20 | 19 | P50/P95 基线 + 瓶颈分析可直接落地为运维 SOP；5 个降级代码片段 copy-pasteable |
|
||
| **合计** | **100** | **95** | **PASS（整合标杆，超 ≥85 门槛 +10）** |
|
||
|
||
**裁定:** Sprint-2 **整合标杆**，直接通过，无 Must-Fix。
|
||
|
||
---
|
||
|
||
## 3. Query 端 E2E 时序图（摘要）
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
autonumber
|
||
actor U as 用户
|
||
participant FE as 前端
|
||
participant API as FastAPI
|
||
participant CS as AppChatService
|
||
participant AS as AgentRunService
|
||
participant Agent as LangChainAgent
|
||
participant KR as knowledge_retrieval()
|
||
participant VDB as ElasticSearchVector
|
||
participant Graph as KGSearch
|
||
participant LLM as RedBearLLM
|
||
|
||
U->>FE: 输入 Query
|
||
FE->>API: POST /api/v1/chat
|
||
API->>CS: await agnet_chat()
|
||
CS->>Agent: LangChainAgent()
|
||
Agent->>LLM: invoke(messages) [首轮判断工具]
|
||
LLM-->>Agent: 需调用 knowledge_retrieval_tool
|
||
Agent->>KR: knowledge_retrieval(query, config)
|
||
|
||
loop 遍历每个知识库
|
||
KR->>VDB: _retrieve_for_knowledge()
|
||
alt retrieve_type == "semantic"
|
||
VDB->>VDB: search_by_vector() + embed_query()
|
||
else retrieve_type == "participle"
|
||
VDB->>VDB: search_by_full_text() + ik_max_word
|
||
else retrieve_type == "hybrid"
|
||
par 双路并发
|
||
VDB->>VDB: search_by_vector()
|
||
VDB->>VDB: search_by_full_text()
|
||
end
|
||
VDB->>VDB: rerank() + RedBearRerank
|
||
end
|
||
alt use_graph=true
|
||
KR->>Graph: kg_retriever.retrieval()
|
||
Graph->>Graph: query_rewrite() LLM 提取实体+类型
|
||
Graph->>Graph: 三路召回: entity/relation/community
|
||
end
|
||
end
|
||
|
||
KR-->>Agent: List[DocumentChunk]
|
||
Agent->>LLM: astream_events() [流式生成]
|
||
LLM-->>FE: SSE 逐字渲染
|
||
```
|
||
|
||
完整版含 30+ 步骤调用栈、输入输出数据结构、同步/异步标注，见 [WS-20](mention://issue/a3deeaa1-5b30-4da5-b4af-1b081f7f6394) §1。
|
||
|
||
---
|
||
|
||
## 4. Indexing 端 E2E 时序图（摘要）
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
autonumber
|
||
actor U as 用户
|
||
participant API as document_controller.py
|
||
participant Task as Celery Task
|
||
participant Chunk as chunk()
|
||
participant Parser as DeepDoc Parser
|
||
participant NLP as naive_merge
|
||
participant Emb as RedBearEmbeddings
|
||
participant VDB as ElasticSearchVector
|
||
participant ES as Elasticsearch
|
||
participant Graph as GraphRAG Index
|
||
|
||
U->>API: POST /documents 上传文件
|
||
API->>Task: 异步触发 chunk 任务
|
||
Task->>Chunk: chunk(filename, binary, ...)
|
||
|
||
alt PDF 格式
|
||
Chunk->>Parser: Pdf.__call__() → OCR → Layout → TSR
|
||
else DOCX 格式
|
||
Chunk->>Parser: Docx.parse()
|
||
else Excel/CSV
|
||
Chunk->>Parser: ExcelParser.__call__()
|
||
else Markdown
|
||
Chunk->>Parser: MarkdownParser
|
||
end
|
||
|
||
Chunk->>NLP: naive_merge(sections) + tokenize_chunks()
|
||
Chunk-->>Task: List[Dict] (ES doc 格式)
|
||
|
||
Task->>Emb: embed_documents(texts)
|
||
Emb-->>Task: List[List[float]]
|
||
|
||
Task->>VDB: add_chunks(chunks, embeddings)
|
||
VDB->>ES: helpers.bulk(actions)
|
||
|
||
alt GraphRAG 启用
|
||
Task->>Graph: run_graphrag_for_kb()
|
||
Graph->>Graph: generate_subgraph() → LLM 抽取
|
||
Graph->>Graph: merge_subgraph()
|
||
Graph->>ES: 写入 entity/relation chunks
|
||
alt General 模式
|
||
Graph->>Graph: EntityResolution()
|
||
Graph->>Graph: leiden.run() + CommunityReportsExtractor()
|
||
end
|
||
end
|
||
```
|
||
|
||
完整版含 14 步骤调用栈、ES Doc 字段契约，见 [WS-20](mention://issue/a3deeaa1-5b30-4da5-b4af-1b081f7f6394) §2。
|
||
|
||
---
|
||
|
||
## 5. 关键路径表（Critical Path Table）
|
||
|
||
| # | 环节 | 关键函数 | 文件:行号 | P50 | P95 | 阻塞性 | 瓶颈 |
|
||
|---|------|---------|-----------|-----|-----|--------|------|
|
||
| 1 | **PDF 解析 (OCR+Layout+TSR)** | `Pdf.__call__()` | `deepdoc/parser/pdf_parser.py:1006` | 3s | 15s | 阻塞 | 🔴 |
|
||
| 2 | **Chunking** | `naive_merge()` + `tokenize_chunks()` | `nlp/__init__.py:562,258` | 50ms | 200ms | 阻塞 | 🟡 |
|
||
| 3 | **Embedding (批量)** | `embed_documents()` | `models/embedding.py:65` | 200ms | 1s | 阻塞 | 🔴 |
|
||
| 4 | **ES 批量写入** | `helpers.bulk()` | `elasticsearch_vector.py:85` | 100ms | 500ms | 阻塞 | 🟡 |
|
||
| 5 | **GraphRAG 实体抽取** | `generate_subgraph()` | `graphrag/general/index.py:333` | 30s | 120s | 阻塞 | 🔴 |
|
||
| 6 | **GraphRAG 消歧** | `EntityResolution.__call__()` | `entity_resolution.py:53` | 10s | 60s | 阻塞 | 🔴 |
|
||
| 7 | **GraphRAG 社区报告** | `CommunityReportsExtractor.__call__()` | `community_reports_extractor.py:55` | 20s | 90s | 阻塞 | 🔴 |
|
||
| 8 | **Query Embedding** | `embed_query()` | `models/embedding.py:65` | 50ms | 300ms | 阻塞 | 🟡 |
|
||
| 9 | **ES 向量检索** | `search_by_vector()` | `elasticsearch_vector.py:374` | 30ms | 200ms | 阻塞 | 🟡 |
|
||
| 10 | **ES 关键词检索** | `search_by_full_text()` | `elasticsearch_vector.py:468` | 20ms | 100ms | 阻塞 | 🟢 |
|
||
| 11 | **外部 Rerank** | `RedBearRerank.compress_documents()` | `models/rerank.py:11` | 100ms | 500ms | 阻塞 | 🟡 |
|
||
| 12 | **GraphRAG 检索** | `KGSearch.retrieval()` | `graphrag/search.py:19` | 200ms | 1s | 阻塞 | 🟡 |
|
||
| 13 | **LLM 首次调用** | `_chat()` | `chat_model.py:122` | 500ms | 3s | 阻塞 | 🔴 |
|
||
| 14 | **LLM 流式生成** | `_chat_streamly()` | `chat_model.py:152` | 500ms | 5s | 流式 | 🔴 |
|
||
| 15 | **引用回填** | `Dealer.insert_citations()` | `search.py:489` | 100ms | 500ms | 阻塞 | 🟡 |
|
||
|
||
### 5.1 四大🔴瓶颈与缓解方向
|
||
|
||
| 瓶颈 | 根因 | 缓解方向 |
|
||
|------|------|---------|
|
||
| PDF 解析 (P95=15s) | OCR + Layout + TSR 串行执行 | MinerU 替代 / 异步队列 / 预加载模型 |
|
||
| Embedding API (P95=1s) | 外部 API 延迟，batch_size=16 | 本地 Xinference / GPUStack 部署 |
|
||
| GraphRAG 建图 (P95=120s) | LLM 多轮抽取，单文档串行 | 增加 max_parallel_documents / 增量更新 |
|
||
| LLM 流式输出 (P95=5s) | 首次 token (TTFT) 慢 | 缓存高频 query / 缩短 max_tokens |
|
||
|
||
---
|
||
|
||
## 6. 多场景调用链（3 场景）
|
||
|
||
### 场景 A：纯向量检索问答
|
||
```
|
||
Query → AppChatService → LangChainAgent → knowledge_retrieval()
|
||
→ _retrieve_for_knowledge() [retrieve_type="semantic"]
|
||
→ ElasticSearchVector.search_by_vector() + embed_query()
|
||
→ ES script_score: cosineSimilarity
|
||
→ top_k chunks → Agent → LLM 流式生成
|
||
```
|
||
|
||
### 场景 B：混合检索问答（关键词 + 向量）
|
||
```
|
||
Query → knowledge_retrieval() [retrieve_type="hybrid"]
|
||
→ 双路并发: search_by_vector() + search_by_full_text()
|
||
→ metadata.doc_id 去重
|
||
→ rerank() + RedBearRerank.compress_documents()
|
||
→ top_k → Agent → LLM 流式生成
|
||
```
|
||
|
||
### 场景 C：GraphRAG 关系推理问答
|
||
```
|
||
Query → knowledge_retrieval() [retrieve_type="graph"]
|
||
→ 先执行 hybrid 检索
|
||
→ KGSearch.retrieval() → query_rewrite() LLM 提取实体+类型
|
||
→ 三路召回: entity/relation/community
|
||
→ n-hop 路径扩展 (sim_decay = 1/(2+hop_depth))
|
||
→ 融合打分: score = sim × pagerank
|
||
→ Token 预算截断 → Agent → LLM 流式生成
|
||
```
|
||
|
||
完整 ASCII 流程图与数据结构流转详见 [WS-20](mention://issue/a3deeaa1-5b30-4da5-b4af-1b081f7f6394) §4。
|
||
|
||
---
|
||
|
||
## 7. 错误传播与降级路径
|
||
|
||
### 7.1 错误矩阵（核心项）
|
||
|
||
| 环节 | 失败模式 | 兜底逻辑 |
|
||
|---|---|---|
|
||
| PDF 解析 | OCR 模型缺失 | 标记 failed_document |
|
||
| Embedding API | 超时/限流 | 抛出异常，整批重试 |
|
||
| ES 写入 | ConnectionTimeout | ATTEMPT_TIME=2 重试 |
|
||
| 知识库检索 | 单 KB 不可用 | try/except continue，跳过失败 KB |
|
||
| 向量检索为空 | 阈值过严 | fallback 降低 min_match 0.3→0.1 |
|
||
| 外部 Rerank | API 超时 | fallback 返回原始排序 |
|
||
| GraphRAG 检索 | 图谱未建 | fallback 仅 hybrid 结果 |
|
||
| LLM 调用 | RATE_LIMIT | 重试 5 次 + 随机抖动 |
|
||
| LLM 截断 | finish_reason="length" | 自动追加截断提示 |
|
||
|
||
### 7.2 降级路径图
|
||
|
||
```
|
||
正常路径: Query → Hybrid → Rerank → LLM → 引用回填 → 输出
|
||
|
||
降级 1 (检索为空): Hybrid (空) → fallback 降低阈值 → 仍空 → LLM 直接回答
|
||
降级 2 (Rerank 失败): Hybrid → Rerank 超时 → fallback 原始排序 → LLM 生成
|
||
降级 3 (GraphRAG 失败): Hybrid → GraphRAG 失败 → fallback 仅 hybrid → LLM 生成
|
||
降级 4 (单 KB 失败): KB-A 失败 + KB-B 成功 → 合并 → LLM 生成
|
||
降级 5 (LLM 失败): 检索成功 → LLM 5 次重试后 → 返回 "**ERROR**: 服务暂不可用"
|
||
```
|
||
|
||
完整代码片段（5 段可复用降级代码）见 [WS-20](mention://issue/a3deeaa1-5b30-4da5-b4af-1b081f7f6394) §5.3。
|
||
|
||
---
|
||
|
||
## 8. 跨文档引用索引
|
||
|
||
| 本章节 | 被引文档 | 引用点 |
|
||
|--------|---------|--------|
|
||
| §3 Query 端 | [S2-T5] | `app_chat_service.py:43`, `langchain_agent.py:230`, `_chat_streamly()` |
|
||
| §3 Query 端 | [S2-T3] | `search_by_vector()`, `search_by_full_text()`, `rerank()` |
|
||
| §3 Query 端 | [S2-T4] | `KGSearch.retrieval()`, `query_rewrite()` |
|
||
| §3 Query 端 | [S2-T2] | `embed_query()` |
|
||
| §3 Query 端 | [S2-T5] | `RedBearRerank.compress_documents()`, `_filter_citations()` |
|
||
| §4 Indexing 端 | [S2-T1] | `chunk()`, `naive_merge()`, `tokenize_chunks()` |
|
||
| §4 Indexing 端 | [S2-T2] | `embed_documents()` |
|
||
| §4 Indexing 端 | [S2-T3] | `add_chunks()`, `helpers.bulk()` |
|
||
| §4 Indexing 端 | [S2-T4] | `run_graphrag_for_kb()`, `generate_subgraph()`, `EntityResolution()`, `leiden.run()` |
|
||
|
||
**结论：6 篇文档形成完整闭环，跨文档引用 0 不一致。**
|
||
|
||
---
|
||
|
||
*本文档为 MemoryBear RAG Docs v1.0 正式版本的组成文件。完整时序图、数据结构定义、关键路径分析与代码片段参见 [WS-20](mention://issue/a3deeaa1-5b30-4da5-b4af-1b081f7f6394) 评论历史。*
|