Commit Graph

107 Commits

Author SHA1 Message Date
Mark
140311048a [modify] rag qa chunk 2026-04-28 14:04:36 +08:00
Mark
c53fcf3981 [fix] old code file_path 2026-04-27 17:10:00 +08:00
Mark
2997558bc8 Merge branch 'release/v0.3.2' into feature/rag2
* release/v0.3.2: (245 commits)
  fix(conversation_schema): refine citations field type to Dict[str, Any]
  fix(tool_controller): re-raise HTTPException to preserve original status codes
  fix(workflow): add reasoning content, suggested questions, citations and audio status support
  feat(workflow): augment logging queries and ameliorate error handling
  fix(api_key): bypass publication check for SERVICE type API keys
  fix(multimodal_service): add '文档内容:' prefix to document text and simplify image placeholder text
  fix(api): convert config_id to string in write_router
  fix(api): convert end_user_id to string in write_router
  fix(multimodal_service): refactor image processing to use intermediate list before extending result
  fix(web): node status ui
  fix(api): correct import paths in memory_read and celery task command
  fix(api): correct import paths in memory_read and celery task command
  refactor(tool): flatten request body parameters for model exposure
  fix(api): correct import paths in memory_read and celery task command
  refactor(workflow): streamline node execution handling and log service logic
  feat(web): http request add process
  feat(web): workflow app logs
  fix(app_chat_service,draft_run_service): move system_prompt augmentation before LangChainAgent instantiation
  fix(app_chat_service,draft_run_service): move system_prompt augmentation before LangChainAgent instantiation
  refactor(http_request): simplify request handling and remove unused fields
  ...

# Conflicts:
#	api/app/controllers/file_controller.py
#	api/app/tasks.py
2026-04-27 16:13:57 +08:00
Mark
30cdf229de [modify] rag file system 2026-04-27 16:05:27 +08:00
Ke Sun
5f96a6380e Merge pull request #990 from SuanmoSuanyangTechnology/feature/celery-task-scheduler
Feature/celery task scheduler
2026-04-24 18:19:00 +08:00
Eternity
c5ae82c3c2 refactor(core): migrate memory write tasks to centralized scheduler 2026-04-22 16:50:06 +08:00
Mark
1ea0f308ba [fix] celery task 2026-04-22 11:47:32 +08:00
Mark
2b5bece9b6 [modify] nfs read error 2026-04-21 17:34:03 +08:00
Mark
ea0e65f1ec [modify] fix tasks 2026-04-21 17:29:35 +08:00
lanceyq
113ae59f84 refactor(memory): insert new metadata values at list head for recency ordering
Change list.append() to list.insert(0, ...) in extract_user_metadata_task so that newly extracted user metadata values appear at the front of each field list, maintaining a newest-first ordering.
2026-04-17 17:33:17 +08:00
lanceyq
643f69bb90 refactor(memory): tighten metadata field types and clean up descriptions
- Use Literal['set', 'remove'] for MetadataFieldChange.action instead of str
- Simplify field_path description to reflect current schema
- Remove redundant isinstance check in extract_user_metadata_task
2026-04-16 17:29:00 +08:00
lanceyq
73fbc19747 refactor(memory): switch metadata extraction from full-replace to incremental changes
- Replace UserMetadata full-object overwrite with incremental MetadataFieldChange
  operations (set/remove per field path)
- Convert profile.role and profile.domain from scalar strings to lists
- Remove UserMetadataBehavioralHints and knowledge_tags fields
- Update Jinja2 prompt to instruct LLM to output incremental changes
- Update extract_user_metadata_task to apply changes via deep-copy and
  per-field mutation for proper SQLAlchemy change detection
- Minor lint: remove unnecessary f-string prefixes in tasks.py
2026-04-16 17:14:30 +08:00
Mark
cba24e58db Merge branch 'feature/rag2' into develop
* feature/rag2:
  [modify] parse document workflow, add graph queue hand build graph
  [modify] mineru
  [modify] 优化tasks ,拆分graphirag 队列

# Conflicts:
#	api/app/tasks.py
2026-04-13 13:46:19 +08:00
Mark
a96f20ee05 [modify] parse document workflow, add graph queue hand build graph 2026-04-13 10:40:58 +08:00
lanceyq
cd018814fe fix(memory): improve metadata language detection and clean_metadata logic
- Make MetadataExtractor language param optional (default None) to
  support auto-detection fallback when no language is explicitly set
- Refactor clean_metadata from walrus-operator dict comprehension to
  explicit loop for correctness and readability
2026-04-10 00:42:11 +08:00
lanceyq
e0b7e95af6 refactor(memory): remove first-person pronoun replacement and inline metadata utils
- Remove _replace_first_person_with_user from StatementExtractor to preserve
  original user text for downstream metadata/alias extraction
- Delete metadata_utils.py module, inline clean_metadata into Celery task
- Remove unused imports and commented-out collect_user_raw_messages method
- Apply formatting cleanup across metadata models and extraction orchestrator
2026-04-10 00:29:18 +08:00
lanceyq
15a863b41a feat(memory): unify alias extraction into metadata pipeline and deduplicate user entity nodes
- Merge alias add/remove into MetadataExtractionResponse and Celery metadata task,
  removing the separate sync step from extraction_orchestrator
- Replace first-person pronouns ("我") with "用户" in statement extraction to
  preserve identity semantics for downstream metadata/alias extraction
- Update extract_statement.jinja2 prompt to enforce "用户" as subject for user
  statements instead of resolving to real names
- Add alias change instructions (aliases_to_add/aliases_to_remove) to
  extract_user_metadata.jinja2 with incremental merge logic
- Deduplicate special entities ("用户", "AI助手") in graph_saver by reusing
  existing Neo4j node IDs per end_user_id
- Sync final aliases from PgSQL to Neo4j user entity nodes after metadata write
2026-04-09 21:55:59 +08:00
lanceyq
e0546e01ef refactor(memory): delegate metadata merging to LLM instead of code-based merge
- Remove merge_metadata and its helper functions from metadata_utils.py
- Pass existing_metadata to MetadataExtractor.extract_metadata() as LLM context
- Add merge instructions to extract_user_metadata.jinja2 prompt (zh/en)
- Update Celery task to read existing metadata before extraction and overwrite
- Simplify field descriptions in UserMetadataProfile model
- Add _update_timestamps helper to track changed fields
2026-04-09 15:10:29 +08:00
Mark
0f50537d7d [modify] mineru 2026-04-09 14:11:01 +08:00
Mark
3ff44f0108 [modify] 优化tasks ,拆分graphirag 队列 2026-04-09 11:59:02 +08:00
lanceyq
f2d7479229 feat(memory): add async user metadata extraction pipeline
- Add MetadataExtractor to collect user-related statements post-dedup
  and extract profile/behavioral metadata via independent LLM call
- Add Celery task (extract_user_metadata) routed to memory_tasks queue
- Add metadata models (UserMetadata, UserMetadataProfile, etc.)
- Add metadata utility functions (clean, validate, merge with _op support)
- Add Jinja2 prompt template for metadata extraction (zh/en)
- Fix Lucene query parameter naming: rename `q` to `query` across all
  Cypher queries, graph_search functions, and callers
- Escape `/` in Lucene queries to prevent TokenMgrError
- Add `speaker` field to ChunkNode and persist it in Neo4j
- Remove unused imports (argparse, os, UUID) in search.py
- Fix unnecessary db context nesting in interest distribution task
2026-04-09 11:01:56 +08:00
Ke Sun
63235de42b Merge pull request #774 from SuanmoSuanyangTechnology/feat/data-transformation
Feat/data transformation
2026-04-02 15:37:50 +08:00
lanceyq
abbd92b74c Interface performance optimization, using only one function 2026-04-02 14:19:27 +08:00
Ke Sun
1c133d3d6c Merge pull request #775 from SuanmoSuanyangTechnology/pref/redis-connections
refactor(tasks, redis_lock): improve Redis connection and lock handling
2026-04-02 14:06:46 +08:00
Eternity
d270d25a99 refactor(tasks, redis_lock): improve Redis connection and lock handling
- Increased max_connections for Redis pool from 10 to 100.
- Extended socket_timeout from 5 to 10 seconds.
- Added retry mechanism with exponential backoff for Redis operations in `RedisFairLock`.
2026-04-02 13:57:22 +08:00
Ke Sun
cdebe014cf fix(tasks): disable late acknowledgment for write_message task 2026-04-01 18:21:16 +08:00
Eternity
8f609ba29c fix(redis_lock): refactor RedisFairLock to use ZSET for queue management and fix loop shutdown
- Replace list-based queue with sorted set for better dead client cleanup
- Add zombie cleanup buffer to handle expired queue entries
- Fix potential None loop reference in graceful shutdown
- Add task start time to write_message_task result
- Update lock acquisition script to use ZSET operations
- Remove unused queue cleanup scripts
- Ensure proper lock release and renewal failure handling
2026-04-01 11:15:06 +08:00
Ke Sun
dbe387f666 fix(tasks): increase redis lock timeout and expiration for write_message_task
- Increase lock expiration time from 120 to 600 seconds (5 minutes)
- Increase lock timeout from 300 to 3600 seconds (1 hour)
- Prevents premature lock release during long-running memory write operations
2026-03-30 20:53:17 +08:00
lanceyq
6e7c641fd4 [fix] Remove duplicate creations 2026-03-30 18:46:25 +08:00
lanceyq
0c677701c0 [fix] iron release 2026-03-30 18:29:17 +08:00
lanceyq
c90b58bbcd [fix] The "write_tools" module actively shuts down the client, and it closes before the task event loop is completed. 2026-03-30 18:19:50 +08:00
Eternity
8dd24533bf fix(memory,task): add Redis fair lock for ordered memory writes 2026-03-30 17:20:54 +08:00
lanceyq
2319432182 [changes] Set up Celery tasks to perform clustering 2026-03-26 18:18:23 +08:00
lanceyq
f2c9902a07 [fix] Fix the forgotten periodic tasks 2026-03-26 11:13:12 +08:00
Eternity
89d188fbf3 Merge branch 'develop' into feature/multimodel_memory
# Conflicts:
#	api/app/core/memory/storage_services/extraction_engine/knowledge_extraction/embedding_generation.py
#	api/app/repositories/neo4j/add_nodes.py
#	api/app/repositories/neo4j/cypher_queries.py
#	api/app/repositories/neo4j/graph_saver.py
#	api/app/services/memory_agent_service.py
#	api/app/services/multimodal_service.py
2026-03-24 14:15:18 +08:00
Eternity
6bba574ca6 feat(memory, model): update multi-modal memory write and model list API
- Adjust multi-modal memory write behavior for text and visual data
- Mask API keys in model list response to prevent exposure
- Add capability-based filtering to the model list API
2026-03-24 13:54:15 +08:00
lanceyq
31b8a3764e 【change】 1.Standardize log specifications;2.Cluster settings trigger explicitly 2026-03-23 16:38:47 +08:00
Eternity
2ff81ba101 feat(memory): support perception-aware memory writing in workflow and Neo4j nodes 2026-03-23 16:33:25 +08:00
Eternity
c17a2dad2d style(memory): Some code style optimizations 2026-03-20 21:05:22 +08:00
lanceyq
b4615bacdc [changes] Modify the execution conditions of the task 2026-03-19 20:17:43 +08:00
lanceyq
f644c84fbb [changes]Community node attribute check 2026-03-19 19:24:37 +08:00
Timebomb2018
ea391dc44e feat(app):
1. Add new functional features to the agent;
2. Enhance the voice output;
3. Modify the end_user binding;
4. Delete and modify the tools.
2026-03-16 18:00:09 +08:00
lanceyq
bdf0c256b3 [add] Change the "last_done" storage to UTC and remove the intermediate conversion. 2026-03-16 16:13:30 +08:00
lanceyq
382e4c5377 [changes] The user's personal configuration and the clustering trigger boundary are clearly defined 2026-03-13 18:02:23 +08:00
lanceyq
668539e737 [add] Selective merge submission 2026-03-13 15:47:53 +08:00
Eternity
b71bc1f875 feat(multimodel): support multimodal memory display and improve code style 2026-03-13 14:47:56 +08:00
lanceyq
5438d35f17 [add] Specify the error types and clearly define the downgrade conditions 2026-03-09 16:19:55 +08:00
lanceyq
21ae448ed7 [add] Throw out explicit error messages; Using the CST time zone 2026-03-09 14:58:03 +08:00
lanceyq
50466124c8 [add] Verification of the existence of interest distribution 2026-03-09 14:57:22 +08:00
Ke Sun
c8065b0c60 feat(implicit-emotions): add Redis resilience and connection pooling
- Replace single Redis client with connection pool for better concurrency and auto-reconnection
- Add graceful degradation when Redis is unavailable (None handling in get_users_needing_refresh)
- Add RedisError exception handling with fallback to process all users on mget failures
- Add type hints (Optional[redis.StrictRedis]) to Redis client parameters
- Add health check and socket timeout configuration to connection pool
- Add logging for Redis connection failures and degradation events
- Reorganize imports alphabetically for consistency across both files
- Update get_sync_redis_client to validate connection with ping() before returning
2026-03-09 14:12:53 +08:00