Commit Graph

1297 Commits

Author SHA1 Message Date
lanceyq
15a863b41a feat(memory): unify alias extraction into metadata pipeline and deduplicate user entity nodes
- Merge alias add/remove into MetadataExtractionResponse and Celery metadata task,
  removing the separate sync step from extraction_orchestrator
- Replace first-person pronouns ("我") with "用户" in statement extraction to
  preserve identity semantics for downstream metadata/alias extraction
- Update extract_statement.jinja2 prompt to enforce "用户" as subject for user
  statements instead of resolving to real names
- Add alias change instructions (aliases_to_add/aliases_to_remove) to
  extract_user_metadata.jinja2 with incremental merge logic
- Deduplicate special entities ("用户", "AI助手") in graph_saver by reusing
  existing Neo4j node IDs per end_user_id
- Sync final aliases from PgSQL to Neo4j user entity nodes after metadata write
2026-04-09 21:55:59 +08:00
lanceyq
e0546e01ef refactor(memory): delegate metadata merging to LLM instead of code-based merge
- Remove merge_metadata and its helper functions from metadata_utils.py
- Pass existing_metadata to MetadataExtractor.extract_metadata() as LLM context
- Add merge instructions to extract_user_metadata.jinja2 prompt (zh/en)
- Update Celery task to read existing metadata before extraction and overwrite
- Simplify field descriptions in UserMetadataProfile model
- Add _update_timestamps helper to track changed fields
2026-04-09 15:10:29 +08:00
lanceyq
f2d7479229 feat(memory): add async user metadata extraction pipeline
- Add MetadataExtractor to collect user-related statements post-dedup
  and extract profile/behavioral metadata via independent LLM call
- Add Celery task (extract_user_metadata) routed to memory_tasks queue
- Add metadata models (UserMetadata, UserMetadataProfile, etc.)
- Add metadata utility functions (clean, validate, merge with _op support)
- Add Jinja2 prompt template for metadata extraction (zh/en)
- Fix Lucene query parameter naming: rename `q` to `query` across all
  Cypher queries, graph_search functions, and callers
- Escape `/` in Lucene queries to prevent TokenMgrError
- Add `speaker` field to ChunkNode and persist it in Neo4j
- Remove unused imports (argparse, os, UUID) in search.py
- Fix unnecessary db context nesting in interest distribution task
2026-04-09 11:01:56 +08:00
Ke Sun
cfbf83f71e Merge pull request #787 from SuanmoSuanyangTechnology/fix/atomic-update
fix(memory): improve optimistic lock resilience in access history man…
2026-04-07 10:57:20 +08:00
Timebomb2018
38f3455bab feat(workflow):
1. add list operator node for filtering, sorting, limiting, and extracting list items;
2. Increase the session variable to the "file" type
2026-04-03 18:57:28 +08:00
lanceyq
99862db7a0 refactor(forgetting-engine): replace optimistic locking with APOC atomic operations in access history manager
- Replace version-based optimistic locking and retry loop with apoc.atomic.add/insert for concurrent safety
- Merge duplicate accesses within a batch before updating (access_count_delta)
- Simplify _calculate_update to only compute on new timestamps instead of full history rebuild
- Remove max_retries instance variable (kept as param for backward compat)
- Trim verbose docstrings and inline comments
2026-04-03 18:40:03 +08:00
lanceyq
00a8099857 changes:(api) Change the "jitter" to "tremble". 2026-04-03 16:55:53 +08:00
lanceyq
117e29fbe3 fix(memory): improve optimistic lock resilience in access history manager
- Increase max_retries from 3 to 5 for concurrent conflict recovery
- Add randomized exponential backoff between retries to reduce contention
- Merge duplicate node accesses in batch operations to avoid self-conflicts
- Support access_times parameter for merged batch access counting
- Add Community node label support in atomic update content field map
2026-04-03 16:46:09 +08:00
Mark
32740e8159 Merge pull request #786 from wanxunyang/feat/app-log-wxy
fix(workflow): persist citations in conversation message meta_data
2026-04-03 16:26:02 +08:00
Ke Sun
bc5ea2d421 Merge pull request #784 from SuanmoSuanyangTechnology/fix/aliases-extract
feat(memory): prevent cross-role alias contamination between user and…
2026-04-03 15:26:31 +08:00
lanceyq
c4ff1a325b refactor(memory): harden alias extraction and sync PgSQL with Neo4j deduped aliases
- Strengthen anti-hallucination rules in extract_triplet prompt to
  enforce verbatim-only alias extraction, removing suggestive examples
- Add _extract_deduped_entity_aliases to sync historical aliases from
  Neo4j two-stage dedup into PgSQL end_user_info
- Remove unused _fetch_neo4j_user_aliases; reuse injected connector
  instead of instantiating new Neo4jConnector
- Simplify _would_merge_cross_role and reuse clean_cross_role_aliases
  in _normalize_special_entity_names
- Reuse _USER_PLACEHOLDER_NAMES from dedup module to avoid duplication
2026-04-03 14:38:55 +08:00
wxy
d1f0258065 fix(workflow): persist citations in conversation message meta_data 2026-04-03 14:36:43 +08:00
wxy
5db59bc9cf fix(workflow): persist citations in conversation message meta_data 2026-04-03 14:32:41 +08:00
Mark
a711635694 Merge pull request #785 from wanxunyang/feat/app-log-wxy
feat(workflow): add opening statement and citation support
2026-04-03 13:41:08 +08:00
lanceyq
15b3ce3dd5 refactor(memory): deduplicate assistant alias query and fix case-sensitive placeholder matching
- Extract fetch_neo4j_assistant_aliases() into deduped_and_disamb.py as
  single source of truth, replacing inline Cypher in write_tools and
  extraction_orchestrator
- Normalize USER_PLACEHOLDER_NAMES to lowercase and apply .lower() on
  all comparisons to prevent case-variant names leaking into aliases
2026-04-03 13:15:57 +08:00
lanceyq
9cc19047b4 fix(memory): prevent cross-role alias contamination in entity dedup
- Extract user aliases from raw dialog statements instead of post-dedup
  entities to bypass merge pollution
- Add alias cross-cleaning step in _normalize_special_entity_names to
  strip AI assistant aliases from user entities before dedup
- Call clean_cross_role_aliases after second-layer dedup to handle
  historical dirty data merged from Neo4j
- Fix syntax error in prompt_utils.py (ontology_types variable assignment)
2026-04-03 12:34:04 +08:00
wxy
2e8e63878e feat(workflow): add opening statement and citation support
- Trigger opening statement on new conversation in run/run_stream
- Fix /opening endpoint to support workflow app type
- Fix features field missing in workflow config release snapshot
- Knowledge node returns citations alongside chunks
- Aggregate citations from all knowledge nodes in result builder
- Filter citations based on features.citation.enabled switch
- Fix WorkflowConfigCreate circular import in app_schema
2026-04-03 11:44:07 +08:00
Ke Sun
38955d7d45 Merge pull request #783 from SuanmoSuanyangTechnology/fix/code-none-value
fix(code-node): prevent null errors by adding default value handling
2026-04-03 11:19:58 +08:00
Ke Sun
b6167d4e94 Merge pull request #761 from SuanmoSuanyangTechnology/feature/perceptual-read
feat(memory): add perceptual memory retrieval service with BM25+embedding fusionding fusion
2026-04-03 11:19:33 +08:00
lanceyq
7890970a39 feat(memory): prevent cross-role alias contamination between user and AI entities
- Add speaker context to triplet extraction prompt to distinguish alias ownership
- Add explicit examples and rules in extract_triplet.jinja2 for user vs AI alias attribution
- Introduce cross-role merge protection in dedup (accurate, fuzzy, and LLM stages)
- Normalize special entity names (用户/AI助手) before deduplication
- Add clean_cross_role_aliases() to sanitize aliases before Neo4j write
- Refactor _update_end_user_other_name to merge aliases from PgSQL instead of Neo4j
- Filter AI assistant aliases from user alias extraction in orchestrator
2026-04-03 10:57:30 +08:00
Eternity
203732de1d fix(code-node): prevent null errors by adding default value handling 2026-04-03 10:18:33 +08:00
Ke Sun
63235de42b Merge pull request #774 from SuanmoSuanyangTechnology/feat/data-transformation
Feat/data transformation
2026-04-02 15:37:50 +08:00
Mark
106a32bc3a Merge pull request #770 from wanxunyang/feat/app-log-wxy
fix: update app_shares.is_active and updated_at when deleting shared app
2026-04-02 15:25:17 +08:00
lanceyq
abbd92b74c Interface performance optimization, using only one function 2026-04-02 14:19:27 +08:00
lanceyq
960ee9f2df changes:(services) Modify the query method for user memory to batch processing. 2026-04-02 14:07:51 +08:00
Ke Sun
1c133d3d6c Merge pull request #775 from SuanmoSuanyangTechnology/pref/redis-connections
refactor(tasks, redis_lock): improve Redis connection and lock handling
2026-04-02 14:06:46 +08:00
Eternity
d270d25a99 refactor(tasks, redis_lock): improve Redis connection and lock handling
- Increased max_connections for Redis pool from 10 to 100.
- Extended socket_timeout from 5 to 10 seconds.
- Added retry mechanism with exponential backoff for Redis operations in `RedisFairLock`.
2026-04-02 13:57:22 +08:00
lanceyq
8abd59b26e changes:(controllers & services) The method for calculating general data is extracted and presented as a shared function. 2026-04-02 13:02:21 +08:00
lanceyq
bd48b4fdbe changes:(controllers) Modify the statistical method of the knowledge base 2026-04-02 12:26:20 +08:00
wxy
be38c9e385 fix: update app_shares.is_active and updated_at when deleting shared app 2026-04-02 11:37:38 +08:00
lanceyq
6eca5f6cdf feat:(controllers & services) Changes in data from yesterday to today 2026-04-02 10:44:14 +08:00
Mark
cf519738f4 Merge pull request #762 from wanxunyang/feat/app-log-wxy
fix: update app_shares.is_active to false when deleting shared app
2026-04-01 20:04:34 +08:00
Ke Sun
cdebe014cf fix(tasks): disable late acknowledgment for write_message task 2026-04-01 18:21:16 +08:00
Eternity
9cbe9d5edc feat(memory): add perceptual memory retrieval service with BM25+embedding fusion 2026-04-01 18:03:07 +08:00
Mark
7b5b2ab31a Merge pull request #757 from wanxunyang/feature/tenant-billing-user-management
feat: enhance homepage version management with database persistence
2026-04-01 17:50:38 +08:00
wxy
924d10ac5b fix: update app_shares.is_active to false when deleting shared app 2026-04-01 17:41:28 +08:00
wxy
0470a71d03 fix: update app_shares.is_active to false when deleting shared app 2026-04-01 17:35:27 +08:00
wxy
5f7db778b5 refactor: remove debug print statements from home page version endpoint 2026-04-01 17:22:49 +08:00
Mark
75bb96d4e7 Merge pull request #741 from SuanmoSuanyangTechnology/feature/agent-tool_xjn
fix(app)
2026-04-01 16:08:32 +08:00
Ke Sun
68fdf5d76f Merge pull request #758 from SuanmoSuanyangTechnology/refactor/redis-lock
fix(redis_lock): refactor RedisFairLock to use ZSET for queue management and fix loop shutdown
2026-04-01 16:04:10 +08:00
Timebomb2018
258c19f9e0 fix(app service)Sourcery mistook the log f-string for SQL.: 2026-04-01 16:02:27 +08:00
Timebomb2018
386ed2b914 feat(models): support reasoning_content streaming 2026-04-01 15:57:02 +08:00
Timebomb2018
264183cec2 feat(models): support reasoning_content streaming 2026-04-01 15:47:43 +08:00
Timebomb2018
9561578a2a Merge branch 'refs/heads/develop' into feature/agent-tool_xjn
# Conflicts:
#	api/app/core/agent/langchain_agent.py
#	api/app/core/tools/mcp/client.py
2026-04-01 15:27:34 +08:00
Ke Sun
99ff07ccac Merge pull request #760 from SuanmoSuanyangTechnology/feat/update-enduser-api
feat(end-user-api): add authenticated API endpoint for end user creation
2026-04-01 13:44:02 +08:00
wxy
70c3c7dd74 feat: enhance homepage version management with database persistence 2026-04-01 11:20:52 +08:00
Ke Sun
b482822629 Merge pull request #755 from SuanmoSuanyangTechnology/feature/enduser-page
Feature/enduser page
2026-04-01 11:15:33 +08:00
Eternity
8f609ba29c fix(redis_lock): refactor RedisFairLock to use ZSET for queue management and fix loop shutdown
- Replace list-based queue with sorted set for better dead client cleanup
- Add zombie cleanup buffer to handle expired queue entries
- Fix potential None loop reference in graceful shutdown
- Add task start time to write_message_task result
- Update lock acquisition script to use ZSET operations
- Remove unused queue cleanup scripts
- Ensure proper lock release and renewal failure handling
2026-04-01 11:15:06 +08:00
wxy
8b997b422a feat: enhance homepage version management with database persistence 2026-04-01 11:04:27 +08:00
lanceyq
6d6338eb06 [changes] Modify the data format and improve the query logic. 2026-04-01 10:36:29 +08:00