Commit Graph

55 Commits

Author SHA1 Message Date
lanceyq
cf389bb978 refactor(memory): remove expired_at field and add dialog_at timestamp
Remove the deprecated expired_at field from all graph models, Neo4j
Cypher queries, repositories, and pipeline code. Replace with dialog_at
on StatementNode to track the original dialog timestamp.

- Strip expired_at from DialogueNode, ChunkNode, StatementNode,
  ExtractedEntityNode, edges, and all Cypher queries
- Add dialog_at to MessageItem schema and propagate through extraction
  and graph build steps
- Extract emotion/metadata async submission from WritePipeline into
  a generic _submit_celery_task helper
- Add post_store_dedup_and_alias_merge Celery task for async alias
  merging and second-layer dedup after Neo4j write
- Switch pytest async backend from anyio to asyncio_mode=auto
2026-05-08 11:27:59 +08:00
lanceyq
d66d601e41 refactor(memory): redesign metadata extraction as async pipeline step
- Replace extract_user_metadata_task with entity-level extract_metadata_batch_task
- Add MetadataExtractionStep following ExtractionStep pattern with Jinja2 prompts
- Flatten MetadataExtractionResponse to 9-field schema (aliases, core_facts, etc.)
- Add Cypher queries for incremental metadata writeback and alias edge redirection
- Wire _extract_metadata into WritePipeline as Step 3.6 (fire-and-forget)
- Add pilot_write() to MemoryService; refactor pilot_run_service to use it
- Extract snapshot logic into WriteSnapshotRecorder
2026-05-08 11:27:51 +08:00
lanceyq
4af9b02815 feat(memory): propagate temporal validity fields through extraction pipeline
- Add valid_at/invalid_at passthrough in triplet extraction prompt (both zh/en)
- Propagate temporal_validity to EntityEntityEdge in ExtractionOrchestrator
- Use coalesce() for valid_at/invalid_at in Neo4j cypher queries to handle NULLs
- Fix workspace_id/config_id UUID parsing in read_memory config resolution
- Downgrade verbose extraction pipeline logs from info to debug
- Remove UUID and short API key patterns from sensitive filter to reduce false positives
- Standardize log message format (use = spacing, end_user_id label)
- Fix misindented TODO comment in write_pipeline.py
2026-05-08 11:26:24 +08:00
lanceyq
1f0c88a5f0 refactor(memory): consolidate write pipeline and rename statement extraction step
- Rename StatementExtractionStep → StatementTemporalExtractionStep and
  extract_statement.jinja2 → extract_statement_temporal.jinja2 to reflect
  merged temporal extraction logic
- Move extraction_pipeline_orchestrator.py out of steps/ to engine root
- Move dedup_step.py into steps/ directory
- Introduce WriteMemoryRequest schema to replace positional args in write_memory()
- Extract _resolve_and_load_config, _preprocess_files, _write_neo4j, and
  _invalidate_interest_cache as private helpers in MemoryAgentService
- Remove shadow pipeline and simplify NEW_PIPELINE_ENABLED branch
- Merge 类型归属/成员隶属/任职服务 relation types into single 归属身份关系 in triplet prompt
- Add alias merge logic (别名属于) in deduplication and MERGE_ALIAS_BELONGS_TO Cypher query
- Add StorageType, Language, MessageItem enums/models to memory_agent_schema
- Reduce AgentMemory_Long_Term.DEFAULT_SCOPE from 6 to 1
- Delete standalone extract_temporal.jinja2 (logic merged into statement step)
2026-05-08 11:26:24 +08:00
lanceyq
7747ed7ac1 refactor(memory): enhance extraction ontology and add assistant pruning graph support
- Expand entity type ontology with detailed definitions, examples, and notes
  (merged types: 地点设施, 物品设备, 产品服务, 软件平台, 角色职业, 知识能力, 偏好习惯目标, 称呼别名, 智能体)
- Add relation ontology taxonomy with 15 predicate categories and usage rules
- Strengthen reference resolution rules: resolve pronouns before extraction,
  skip unresolvable references entirely
- Add guidelines to avoid extracting abstract propositions, emotions, and
  low-value entities (effort/reward/success patterns)
- Add 7 new extraction examples covering edge cases
- Add AssistantOriginal/AssistantPruned node models and graph persistence
  (PRUNED_TO and BELONGS_TO_DIALOG edges, Neo4j indexes and constraints)
- Add graph_build_step.py for building graph nodes/edges from DialogData
- Update write_pipeline.py to pass assistant pruning nodes/edges to graph saver
- Update data_pruning.py with related preprocessing changes
2026-05-08 11:26:24 +08:00
lanceyq
2355536b44 refactor(memory): add PilotWritePipeline and enrich extraction schema
- Add dedicated PilotWritePipeline (statement → triplet → graph_build → layer-1 dedup, no Neo4j write)
- Add type_description/predicate_description fields across entity and triplet models, Cypher queries, and graph builders
- Refactor data_pruning with LRU cache and snapshot support; skip assistant chunks in extraction
- Remove strict Predicate enum whitelist; support statement_text alias in legacy extractor
- Wire PipelineSnapshot through preprocessing and emotion extraction for debug tracing
- Add PILOT_RUN_USE_REFACTORED_PIPELINE env toggle for pipeline selection
2026-05-08 11:26:04 +08:00
lanceyq
b0ddd12cc6 feat(memory): add emotion batch extraction task and improve extraction prompts
- Add extract_emotion_batch_task for async emotion extraction
- Refine Chinese entity types and relation types in extraction prompts
- Add STATEMENT_EMOTION_UPDATE Cypher query for Neo4j backfill
- Refactor statement_step and triplet_step implementations
2026-05-08 11:26:04 +08:00
Eternity
1191f0f54e fix(neo4j): correct community property name in search queries 2026-04-24 13:13:38 +08:00
Eternity
dc3207b1d3 Merge branch 'develop' into refactor/memory_search
# Conflicts:
#	api/app/core/memory/storage_services/search/__init__.py
2026-04-20 18:07:07 +08:00
Eternity
2716a55c7f feat(memory): implement quick search pipeline with Neo4j integration 2026-04-15 12:18:23 +08:00
lanceyq
811193dd75 fix(memory): make PgSQL the single source of truth for user entity aliases
- Skip alias merging for user entities during dedup (_merge_attribute and
  _merge_entities_with_aliases) to prevent dirty data from overwriting
  PgSQL authoritative aliases
- Add PgSQL→Neo4j alias sync after Neo4j write in write_tools to
  ensure Neo4j user entities always reflect the PgSQL source
- Remove deduped_aliases (Neo4j history) from alias sync in
  extraction_orchestrator, only append newly extracted aliases to PgSQL
- Guard Neo4j MERGE cypher to preserve existing aliases for user
  entities (name IN ['用户','我','User','I'])
- Fix emotion_analytics_service query to use ExtractedEntity label
  and entity_type property
2026-04-14 17:28:24 +08:00
Eternity
dca3173ed9 refactor(memory): restructure memory search architecture
- Replace storage_services/search with new read_services/memory_search structure
- Implement content_search and perceptual_search strategies
- Add query_preprocessor for search optimization
- Create memory_service as unified interface
- Update celery_app and graph_search for new architecture
- Add enums for memory operations
- Implement base_pipeline and memory_read pipeline patterns
2026-04-13 14:03:47 +08:00
lanceyq
f2d7479229 feat(memory): add async user metadata extraction pipeline
- Add MetadataExtractor to collect user-related statements post-dedup
  and extract profile/behavioral metadata via independent LLM call
- Add Celery task (extract_user_metadata) routed to memory_tasks queue
- Add metadata models (UserMetadata, UserMetadataProfile, etc.)
- Add metadata utility functions (clean, validate, merge with _op support)
- Add Jinja2 prompt template for metadata extraction (zh/en)
- Fix Lucene query parameter naming: rename `q` to `query` across all
  Cypher queries, graph_search functions, and callers
- Escape `/` in Lucene queries to prevent TokenMgrError
- Add `speaker` field to ChunkNode and persist it in Neo4j
- Remove unused imports (argparse, os, UUID) in search.py
- Fix unnecessary db context nesting in interest distribution task
2026-04-09 11:01:56 +08:00
Eternity
9cbe9d5edc feat(memory): add perceptual memory retrieval service with BM25+embedding fusion 2026-04-01 18:03:07 +08:00
lanceyq
3419bb137a [fix] Fix the alias query statement 2026-03-30 19:56:02 +08:00
lanceyq
2a12cb04bf [changes] Optimize the Cypher query statement 2026-03-25 18:47:30 +08:00
lanceyq
c4461c4917 【add】User alias extraction and retrieval 2026-03-25 18:47:29 +08:00
Eternity
89d188fbf3 Merge branch 'develop' into feature/multimodel_memory
# Conflicts:
#	api/app/core/memory/storage_services/extraction_engine/knowledge_extraction/embedding_generation.py
#	api/app/repositories/neo4j/add_nodes.py
#	api/app/repositories/neo4j/cypher_queries.py
#	api/app/repositories/neo4j/graph_saver.py
#	api/app/services/memory_agent_service.py
#	api/app/services/multimodal_service.py
2026-03-24 14:15:18 +08:00
Eternity
6bba574ca6 feat(memory, model): update multi-modal memory write and model list API
- Adjust multi-modal memory write behavior for text and visual data
- Mask API keys in model list response to prevent exposure
- Add capability-based filtering to the model list API
2026-03-24 13:54:15 +08:00
lanceyq
31b8a3764e 【change】 1.Standardize log specifications;2.Cluster settings trigger explicitly 2026-03-23 16:38:47 +08:00
Eternity
2ff81ba101 feat(memory): support perception-aware memory writing in workflow and Neo4j nodes 2026-03-23 16:33:25 +08:00
Ke Sun
37bc4beab4 Merge branch 'release/v0.2.8' into develop 2026-03-23 10:24:17 +08:00
Eternity
c17a2dad2d style(memory): Some code style optimizations 2026-03-20 21:05:22 +08:00
lanceyq
81f3b50200 Ensure stability 2026-03-20 20:45:29 +08:00
lanceyq
72a2f2a7e8 [add] Introduce examples and triples to enrich the community summaries 2026-03-20 20:19:44 +08:00
lanceyq
b4615bacdc [changes] Modify the execution conditions of the task 2026-03-19 20:17:43 +08:00
lanceyq
f644c84fbb [changes]Community node attribute check 2026-03-19 19:24:37 +08:00
lanceyq
5d6007aaff [add] Complete the missing cypher statement 2026-03-18 18:52:36 +08:00
lanceyq
56adca9f22 [changes] Batch mode for metadata creation and unified management of indexes 2026-03-16 23:06:41 +08:00
lanceyq
f32d92b9d0 [Changes] 2026-03-16 14:05:12 +08:00
lanceyq
f9fb480cc3 [changes] Community Clustering Retrieval Module 2026-03-16 13:38:38 +08:00
lanceyq
f033607c8b [changes] Queue uniformity, query statement uniformity 2026-03-13 20:07:18 +08:00
lanceyq
5141a91041 [changes] 2026-03-13 15:52:38 +08:00
lanceyq
6d8b1aede4 [add] Create the attribute values of the community nodes 2026-03-13 15:47:05 +08:00
lanceyq
744ba31ba6 [changes] Initial stage of community integration 2026-03-13 15:47:05 +08:00
lanceyq
db8257b67a [add] Create community nodes 2026-03-13 15:47:04 +08:00
lixinyue11
4e7ab3d7e3 Fix/release memory bug (#335)
* Write Missing None

* Write Missing None

* Write Missing None

* Apply suggestion from @sourcery-ai[bot]

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>

* Write Missing None

* redis update

* redis update

* redis update

* redis update

* writer_dup_bug/fix

* writer_graph_bug/fix

* writer_graph_bug/fix

---------

Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
2026-02-05 17:27:28 +08:00
乐力齐
47b25d7a26 Fix/fact summary (#333)
* [fix]Disable the contents related to fact_summary

* [fix]Disable the contents related to fact_summary

* [fix]Modify the code based on the AI review
2026-02-05 15:56:43 +08:00
Eternity
b471d56a86 fix(prompt): remove hard-coded import of prompt file paths (#279)
* Fix/develop memory bug (#274)

* 遗漏的历史映射

* 遗漏的历史映射

* fix_timeline_memories

* fix(web): update retrieve_type key

* Fix/develop memory bug (#276)

* 遗漏的历史映射

* 遗漏的历史映射

* fix_timeline_memories

* fix_timeline_memories

* write_gragp/bug_fix

* write_gragp/bug_fix

* write_gragp/bug_fix

* chore(celery): disable periodic task scheduling

* fix(prompt): remove hard-coded import of prompt file paths

---------

Co-authored-by: lixinyue11 <94037597+lixinyue11@users.noreply.github.com>
Co-authored-by: zhaoying <yzhao96@best-inc.com>
Co-authored-by: yingzhao <zhaoyingyz@126.com>
Co-authored-by: Ke Sun <kesun5@illinois.edu>
2026-02-03 10:29:51 +08:00
lixinyue11
3601737869 Fix/memory bug fix (#171) 2026-01-26 11:53:34 +08:00
lixinyue11
e518b57dea Fix/memory bug fix (#150)
* 图谱数据量限制数量去掉

* 图谱数据量限制数量去掉

* 图谱数据量限制数量去掉

* 用户详情优化

* 用户详情优化

* 用户详情优化

* 用户详情优化

* 用户详情优化

* 用户详情优化

* 读取的接口,去掉全局锁

* 输出数组

* 反思优化1.0(优化隐私输出、时间检索)

* 反思优化1.0(优化隐私输出、时间检索)

* 反思优化1.0(优化隐私输出、时间检索)
2026-01-20 10:39:12 +08:00
乐力齐
935f3d54b3 Feature/generate cache (#135)
* [feature]Generate emotions, implicit cache

* [feature]Generate emotions, implicit cache

* [changes]Improve the code based on AI review

* [changes]Improve the code based on AI review

* [changes]Improve the code

* [feature]Generate emotions, implicit cache

* [changes]Improve the code based on AI review

* [changes]Improve the code
2026-01-16 12:33:37 +08:00
乐力齐
4e1ff99cc5 [fix]Restore episodic memory and retrieve memory type (#78) 2026-01-12 19:02:25 +08:00
lixinyue11
f6ca6a547f Fix/develop memory deail (#69)
* 新增记忆空间详情

* 新增记忆空间详情

* 新增记忆关联的数量

* 修改记忆时间线

* 修改记忆时间线

* 修改记忆时间线

* Parameterize elementId in Cypher query

* 关系演化,互动频率优化

* 关系演化,互动频率优化

* 关系演化,互动频率优化

* 关系演化,互动频率优化

* 关系演化,互动频率优化

* 关系演化,互动频率优化

---------

Co-authored-by: Ke Sun <33739460+keeees@users.noreply.github.com>
2026-01-12 12:28:17 +08:00
乐力齐
9722601bae Feature/episodic memory (#70)
* [feature]episodic memory

* [feature]episodic memory

* [changes]AI review and modify code

* [feature]Explicit memory

* [feature]Explicit memory
2026-01-12 12:27:33 +08:00
乐力齐
539821454a Feature/episodic memory (#64)
* [feature]episodic memory

* [feature]episodic memory

* [changes]AI review and modify code
2026-01-10 16:35:32 +08:00
lixinyue11
7d28717030 Fix/develop memory deail (#63)
* 新增记忆空间详情

* 新增记忆空间详情

* 新增记忆关联的数量

* 修改记忆时间线

* 修改记忆时间线

* 修改记忆时间线

* Parameterize elementId in Cypher query

---------

Co-authored-by: Ke Sun <33739460+keeees@users.noreply.github.com>
2026-01-10 12:37:11 +08:00
lixinyue11
009ceefa30 新增记忆空间详情 (#58)
* 新增记忆空间详情

* 新增记忆空间详情
2026-01-08 17:51:49 +08:00
乐力齐
5fe8043ff8 Fix/actr config (#49)
* [fix]Remove the LLM

* [fix]Failed to restore access history record
2026-01-07 16:00:53 +08:00
乐力齐
e8a5cfe7e3 Merge #85 into develop from feature/actr-forget
[feature]actr-记忆遗忘需求开发

* feature/actr-forget: (12 commits squashed)

  - [feature]
    1.Extended fields of the date_config table;
    2.New activation value calculation has been added, and the ACTR parameter has been introduced in Neo4j.

  - [feature]1.Create a forgetting strategy executor;2.Create the forgetting scheduler

  - [feature]Introduce activation values for retrieval, and develop a two-stage retrieval reordering process

  - [feature]
    1.Extended fields of the date_config table;
    2.New activation value calculation has been added, and the ACTR parameter has been introduced in Neo4j.

  - [feature]1.Create a forgetting strategy executor;2.Create the forgetting scheduler

  - [feature]Introduce activation values for retrieval, and develop a two-stage retrieval reordering process

  - Merge branch &#39;feature/actr-forget&#39; of codeup.aliyun.com:redbearai/python/redbear-mem-open into feature/actr-forget

  - [fix]Eliminate the interference caused by redundant code

  - [feature]
    1.Extended fields of the date_config table;
    2.New activation value calculation has been added, and the ACTR parameter has been introduced in Neo4j.

  - [feature]1.Create a forgetting strategy executor;2.Create the forgetting scheduler

  - [feature]Introduce activation values for retrieval, and develop a two-stage retrieval reordering process

  - Merge branch &#39;feature/actr-forget&#39; of codeup.aliyun.com:redbearai/python/redbear-mem-open into feature/actr-forget

Signed-off-by: 乐力齐 <accounts_690c7b0af9007d7e338af636@mail.teambition.com>
Reviewed-by: aliyun6762716068 <accounts_68cb7c6b61f5dcc4200d6251@mail.teambition.com>
Merged-by: aliyun6762716068 <accounts_68cb7c6b61f5dcc4200d6251@mail.teambition.com>

CR-link: https://codeup.aliyun.com/redbearai/python/redbear-mem-open/change/85
2026-01-05 04:30:36 +00:00