docs(rag): add MemoryBear RAG implementation docs v1.0
Some checks failed
Sync to Gitee / sync (push) Has been cancelled
Some checks failed
Sync to Gitee / sync (push) Has been cancelled
Submit the formed RAG documentation set produced across Sprint-1/2/3 (WS-12 through WS-26) under docs/rag/. Includes: - README.md / INDEX.md: landing + total index (responsibility matrix, review verdicts, dual-link to source issues) - overview/: full-pipeline architecture (4 .mmd diagrams), 11-stage boundary contracts, doc map, source-code inventory - pipeline/: 5 deep-dives (Loader/Parser/Chunking, Embedding, VDB & retrieval, GraphRAG, Rerank/Prompt/LLM) - graphrag/, end-to-end/: v1.0 formal versions with full source retained as reference - evolution/: 11 architecture-refactor proposals, 6-direction roadmap, capability map - review/: S3-T1 / S3-T2 final reviews, S2-T7 final summary - _indexes/: glossary (81 terms), source->doc reverse index, chart index - _release/: v1.0-RC1 release manifest, versioning convention, ops & freshness plan - _meta/README.md: placeholder noting WS-12 governance assets gap Aggregate review score 92.6/100 (8/8 PASS, 31/31 source-code spot checks hit). The legacy docs/ ignore in .gitignore is narrowed to docs/* with an explicit allowlist for docs/rag/. Refs: WS-26 Co-authored-by: multica-agent <github@multica.ai>
This commit is contained in:
102
docs/rag/overview/03-query-pipeline.mmd
Normal file
102
docs/rag/overview/03-query-pipeline.mmd
Normal file
@@ -0,0 +1,102 @@
|
||||
%% MemoryBear 在线检索时序图(Query Pipeline)
|
||||
%% 起点:用户 Query;终点:LLM 生成的回答
|
||||
|
||||
sequenceDiagram
|
||||
autonumber
|
||||
participant User as 用户/API
|
||||
participant WF as Workflow Engine<br/>(workflow/nodes/knowledge/node.py)
|
||||
participant Config as config.py<br/>KnowledgeRetrievalNodeConfig
|
||||
participant Retriever as nlp/search.py<br/>knowledge_retrieval()
|
||||
participant Dealer as nlp/search.py:349<br/>Dealer.search()
|
||||
participant Qryr as nlp/query.py<br/>Query理解
|
||||
participant ESVec as ESVector<br/>elasticsearch_vector.py
|
||||
participant Graph as graphrag/search.py<br/>KGSearch.retrieval()
|
||||
participant Rerank as models/rerank.py<br/>RedBearRerank
|
||||
participant Prompt as prompts/generator.py
|
||||
participant LLM as llm/chat_model.py<br/>Base.chat()
|
||||
participant Cache as utils/redis_conn.py
|
||||
|
||||
Note over User,Cache: === 阶段 1:Query 准备 ===
|
||||
User->>WF: 用户输入 Query
|
||||
WF->>WF: _render_template(query, variable_pool)
|
||||
WF->>Config: 读取 knowledge_bases[]<br/>reranker_id / retrieve_type
|
||||
|
||||
Note over Retriever,ESVec: === 阶段 2:多知识库检索 ===
|
||||
loop 每个 Knowledge Base
|
||||
WF->>Retriever: knowledge_retrieval(query, config)
|
||||
Retriever->>DB: 验证 KB 状态 (chunk_num>0, status=1)
|
||||
|
||||
alt RetrieveType == PARTICIPLE
|
||||
Retriever->>ESVec: search_by_full_text(query, top_k)
|
||||
ESVec->>ESVec: match on page_content (ik_max_word)
|
||||
ESVec-->>Retriever: List[DocumentChunk]
|
||||
|
||||
else RetrieveType == SEMANTIC
|
||||
Retriever->>ESVec: search_by_vector(query, top_k)
|
||||
ESVec->>ESVec: script_score cosineSimilarity
|
||||
ESVec-->>Retriever: List[DocumentChunk]
|
||||
|
||||
else RetrieveType == HYBRID
|
||||
par
|
||||
Retriever->>ESVec: search_by_vector()
|
||||
ESVec-->>Retriever: rs1
|
||||
and
|
||||
Retriever->>ESVec: search_by_full_text()
|
||||
ESVec-->>Retriever: rs2
|
||||
end
|
||||
Retriever->>Retriever: _deduplicate_docs(rs1, rs2)
|
||||
Retriever->>Rerank: rerank(query, docs, top_k)
|
||||
Rerank->>Rerank: similarity() 交叉编码评分
|
||||
Rerank-->>Retriever: sorted docs
|
||||
|
||||
else RetrieveType == GRAPH
|
||||
par
|
||||
Retriever->>ESVec: search_by_vector()
|
||||
ESVec-->>Retriever: rs1
|
||||
and
|
||||
Retriever->>ESVec: search_by_full_text()
|
||||
ESVec-->>Retriever: rs2
|
||||
end
|
||||
Retriever->>Retriever: dedup + rerank
|
||||
|
||||
Retriever->>Graph: kg_retriever.retrieval(question)
|
||||
Graph->>Graph: query_rewrite() → keywords + entities
|
||||
Graph->>ESVec: get_relevant_ents_by_keywords()
|
||||
Graph->>ESVec: get_relevant_relations_by_txt()
|
||||
Graph->>Graph: n_hop_with_weight 路径扩展
|
||||
Graph->>Graph: Score = pagerank * sim
|
||||
Graph->>Graph: _community_retrieval_()
|
||||
Graph-->>Retriever: Entity+Relation+CommunityReport chunk
|
||||
Retriever->>Retriever: insert(0, graph_result)
|
||||
end
|
||||
|
||||
Retriever-->>WF: List[DocumentChunk]
|
||||
end
|
||||
|
||||
WF->>WF: _deduplicate_docs(all_results)
|
||||
|
||||
alt reranker_id 配置
|
||||
WF->>Rerank: rerank(query, all_results, reranker_top_k)
|
||||
Rerank-->>WF: reranked chunks
|
||||
end
|
||||
|
||||
Note over Prompt,Cache: === 阶段 3:Prompt 组装 + LLM 生成 ===
|
||||
WF->>WF: 返回 {"chunks": [...], "citations": [...]}
|
||||
WF->>Prompt: citation_prompt(chunks)
|
||||
Prompt->>Prompt: 组装 System Prompt + 检索上下文
|
||||
|
||||
Prompt->>Cache: get_llm_cache(model, prompt)
|
||||
alt cache miss
|
||||
Prompt->>LLM: chat(system, history, gen_conf)
|
||||
LLM-->>Prompt: answer, tokens
|
||||
Prompt->>Cache: set_llm_cache(model, prompt, answer)
|
||||
else cache hit
|
||||
Cache-->>Prompt: cached answer
|
||||
end
|
||||
|
||||
Note over User,Cache: === 阶段 4:后处理 ===
|
||||
Prompt->>Dealer: insert_citations(answer, chunks, chunk_v)
|
||||
Dealer->>Dealer: pagerank*sim 定位引用位置
|
||||
Dealer-->>Prompt: answer_with_citations, cited_ids
|
||||
|
||||
Prompt-->>User: 最终回答(含引用标注)
|
||||
Reference in New Issue
Block a user