Release/v0.2.3 (#355)

* feat(web): add PageEmpty component * feat(web): add PageTabs component * feat(web): add PageEmpty component * feat(web): add PageTabs component * feat(prompt): add history tracking for prompt releases * feat(web): add prompt menu * refactor: The PageScrollList component supports two generic parameters * feat(web): BodyWrapper compoent update PageLoading * feat(web): add Ontology menu * feat(web): memory management add scene * feat(tasks): add celery task configuration for periodic jobs - Add ignore_result=True to prevent storing results for periodic tasks - Set max_retries=0 to skip failed periodic tasks without retry attempts - Configure acks_late=False for immediate acknowledgment in beat tasks - Add time_limit and soft_time_limit to regenerate_memory_cache task (3600s/3300s) - Add time_limit and soft_time_limit to workspace_reflection_task (300s/240s) - Add time_limit and soft_time_limit to run_forgetting_cycle_task (7200s/7000s) - Improve task reliability and resource management for scheduled jobs * feat(sandbox): add Node.js code execution support to sandbox * Release/v0.2.2 (#260) * [modify] migration script * [add] migration script * fix(web): change form message * fix(web): the memoryContent field is compatible with numbers and strings * feat(web): code node hidden * fix(model): 1. create a basic model to check if the name and provider are duplicated. 2. The result shows error models because the provider created API Keys for all matching models. --------- Co-authored-by: Mark <zhuwenhui5566@163.com> Co-authored-by: zhaoying <yzhao96@best-inc.com> Co-authored-by: yingzhao <zhaoyingyz@126.com> Co-authored-by: Timebomb2018 <18868801967@163.com> * Feature/ontology class clean (#249) * [add] Complete ontology engineering feature implementation * [add] Add ontology feature integration and validation utilities * [add] Add OWL validator and validation utilities * [fix] Add missing render_ontology_extraction_prompt function * [fix]Add dependencies, fix functionality * [add] migration script * feat(celery): add dedicated periodic tasks worker and queue (#261) * fix(web): conflict resolve * Fix/v022 bug (#263) * [fix]Fix the issue of inconsistent language in explicit and episodic memory. * [fix]Fix the issue of inconsistent language in explicit and episodic memory. * [add]Add scene_id * [fix]Based on the AI review to fix the code * Fix/develop memory reflex (#265) * 遗漏的历史映射 * 遗漏的历史映射 * 反思后台报错处理 * [add] migration script * fix: chat conversation_id add node_start * feat(web): show code node * fix(web): Restructure the CustomSelect component, repair the interface that is called multiple times when the form is updated * feat(web): RadioGroupCard support block mode * feat(web): create space add icon * feat(app and model): token consumption statistics * Add/develop memory (#264) * 遗漏的历史映射 * 遗漏的历史映射 * 遗漏的历史映射 * 遗漏的历史映射 * 遗漏的历史映射 * 遗漏的历史映射 * 遗漏的历史映射 * 遗漏的历史映射 * 遗漏的历史映射 * 新增长期记忆功能 * 新增长期记忆功能 * 新增长期记忆功能 * 知识库检索多余字段 * 长期 * feat(app and model): token consumption statistics of the cluster * memory_BUG_fix * fix(web): prompt history remove pageLoading * fix(prompt): remove hard-coded import of prompt file paths (#279) * Fix/develop memory bug (#274) * 遗漏的历史映射 * 遗漏的历史映射 * fix_timeline_memories * fix(web): update retrieve_type key * Fix/develop memory bug (#276) * 遗漏的历史映射 * 遗漏的历史映射 * fix_timeline_memories * fix_timeline_memories * write_gragp/bug_fix * write_gragp/bug_fix * write_gragp/bug_fix * chore(celery): disable periodic task scheduling * fix(prompt): remove hard-coded import of prompt file paths --------- Co-authored-by: lixinyue11 <94037597+lixinyue11@users.noreply.github.com> Co-authored-by: zhaoying <yzhao96@best-inc.com> Co-authored-by: yingzhao <zhaoyingyz@126.com> Co-authored-by: Ke Sun <kesun5@illinois.edu> * fix(web): remove delete confirm content * refactor(workflow): relocate template directory into workflow * feat(memory): add long-term storage task routing and batching * fix(web): PageScrollList loading update * fix(web): PageScrollList loading update * Ontology v1 bug (#291) * [changes]Add 'id' as the secondary sorting key, and 'scene_id' now returns a UUID object * [fix]Fix the "end_user" return to be sorted by update time. * [fix]Set the default values of the memory configuration model based on the spatial model. * [fix]Remove the entity extraction check combination model, read the configuration list, and add the return of scene_id * [fix]Fix the "end_user" return to be sorted by update time. * [fix] * fix(memory): add Redis session validation - Add macOS fork() safety configuration in celery_app.py to prevent initialization issues - Add null/False checks for Redis session queries in term_memory_save to handle missing sessions gracefully - Add null/False checks in memory_long_term_storage to prevent processing empty Redis results - Add null/False checks in aggregate_judgment before format_parsing to avoid errors on missing data - Initialize redis_messages variable in window_dialogue for consistency - Add debug logging when no existing session found in Redis for better troubleshooting - Add TODO comments for magic numbers (scope=6, time=5) to be extracted as constants - Improve error handling when Redis returns False or empty results instead of crashing * fix(web): PageScrollList style update * fix(workflow): fix argument passing in code execution nodes * fix(web): prompt add disabled * fix(web): space icon required * feat(app): modify the key of the token * fix(fix the key of the app's token): * fix(workflow): switch code input encoding to base64+URL encoding * [add]The main project adds multi-API Key load balancing. * [changes]Attribute security access, secure numerical conversion, unified use of local variables * fix(web): save add session update * fix(web): language editor support paste * [changes]Active status filtering logic, API Key selection strategy * memory_BUG * memory_BUG_long_term * [changes] * memory_BUG_long_term * memory_BUG_long_term * Fix/release memory bug (#306) * memory_BUG_fix * memory_BUG * memory_BUG_long_term * memory_BUG_long_term * memory_BUG_long_term * knowledge_retrieval/bug/fix * knowledge_retrieval/bug/fix * knowledge_retrieval/bug/fix * [fix]1.The "read_all_config" interface returns "scene_name";2.Memory configuration for lightweight query ontology scenarios * fix(web): replace code editor * [changes]Modify the description of the time for the recent event * [changes]Modify the code based on the AI review * feat(web): update memory config ontology api * fix(web): ui update * knowledge_retrieval/bug/fix * knowledge_retrieval/bug/fix * knowledge_retrieval/bug/fix * feat(workflow): add token usage statistics for question classifier and parameter extraction * feat(web): move prompt menu * Multiple independent transactions - single transaction * Multiple independent transactions - single transaction * Multiple independent transactions - single transaction * Multiple independent transactions - single transaction * Write Missing None (#321) * Write Missing None * Write Missing None * Write Missing None * Apply suggestion from @sourcery-ai[bot] Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Write Missing None --------- Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Fix/release memory bug (#324) * Write Missing None * Write Missing None * Write Missing None * Apply suggestion from @sourcery-ai[bot] Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Write Missing None * redis update * redis update * redis update * redis update --------- Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Fix/writer memory bug (#326) * [fix]Fix the bug * [fix]Fix the bug * [fix]Correct the direction indication. * fix(web): markdown table ui update * Fix/release memory bug (#332) * Write Missing None * Write Missing None * Write Missing None * Apply suggestion from @sourcery-ai[bot] Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Write Missing None * redis update * redis update * redis update * redis update * writer_dup_bug/fix --------- Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Fix/fact summary (#333) * [fix]Disable the contents related to fact_summary * [fix]Disable the contents related to fact_summary * [fix]Modify the code based on the AI review * Fix/release memory bug (#335) * Write Missing None * Write Missing None * Write Missing None * Apply suggestion from @sourcery-ai[bot] Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Write Missing None * redis update * redis update * redis update * redis update * writer_dup_bug/fix * writer_graph_bug/fix * writer_graph_bug/fix --------- Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Revert "feat(web): move prompt menu" This reverts commit 9e6e8f50f8. * fix(web): ui update * fix(web): update text * fix(web): ui update * fix(model): change the "vl" model type of dashscope to "chat" * fix(model): change the "vl" model type of dashscope to "chat" --------- Co-authored-by: zhaoying <yzhao96@best-inc.com> Co-authored-by: Eternity <1533512157@qq.com> Co-authored-by: Mark <zhuwenhui5566@163.com> Co-authored-by: yingzhao <zhaoyingyz@126.com> Co-authored-by: Timebomb2018 <18868801967@163.com> Co-authored-by: 乐力齐 <162269739+lanceyq@users.noreply.github.com> Co-authored-by: lixinyue11 <94037597+lixinyue11@users.noreply.github.com> Co-authored-by: lixinyue <2569494688@qq.com> Co-authored-by: Eternity <61316157+myhMARS@users.noreply.github.com> Co-authored-by: lanceyq <1982376970@qq.com> Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
2026-02-06 19:01:57 +08:00
parent eab7225d83
commit 79ab929fb0
187 changed files with 12252 additions and 1656 deletions
--- a/api/app/tasks.py
+++ b/api/app/tasks.py
@@ -774,7 +774,15 @@ def write_total_memory_task(workspace_id: str) -> Dict[str, Any]:
        }


-@celery_app.task(name="app.tasks.regenerate_memory_cache", bind=True)
+@celery_app.task(
+    name="app.tasks.regenerate_memory_cache",
+    bind=True,
+    ignore_result=True,      
+    max_retries=0,          
+    acks_late=False,     
+    time_limit=3600,         
+    soft_time_limit=3300,    
+)
 def regenerate_memory_cache(self) -> Dict[str, Any]:
    """定时任务：为所有用户重新生成记忆洞察和用户摘要缓存
    
@@ -966,7 +974,16 @@ def regenerate_memory_cache(self) -> Dict[str, Any]:
        }


-@celery_app.task(name="app.tasks.workspace_reflection_task", bind=True)
+
+@celery_app.task(
+    name="app.tasks.workspace_reflection_task",
+    bind=True,
+    ignore_result=True,      
+    max_retries=0,          
+    acks_late=False,       
+    time_limit=300,          
+    soft_time_limit=240,   
+)
 def workspace_reflection_task(self) -> Dict[str, Any]:
    """定时任务：每30秒运行工作空间反思功能

@@ -1049,6 +1066,7 @@ def workspace_reflection_task(self) -> Dict[str, Any]:
                            f"工作空间 {workspace_id} 反思处理完成，处理了 {len(workspace_reflection_results)} 个任务")

                    except Exception as e:
+                        db.rollback()  # Rollback failed transaction to allow next query
                        api_logger.error(f"处理工作空间 {workspace_id} 反思失败: {str(e)}")
                        all_reflection_results.append({
                            "workspace_id": str(workspace_id),
@@ -1111,7 +1129,16 @@ def workspace_reflection_task(self) -> Dict[str, Any]:



-@celery_app.task(name="app.tasks.run_forgetting_cycle_task", bind=True)
+
+@celery_app.task(
+    name="app.tasks.run_forgetting_cycle_task",
+    bind=True,
+    ignore_result=True,      
+    max_retries=0,           
+    acks_late=False,         
+    time_limit=7200,        
+    soft_time_limit=7000,    
+)
 def run_forgetting_cycle_task(self, config_id: Optional[uuid.UUID] = None) -> Dict[str, Any]:
    """定时任务：运行遗忘周期
    
@@ -1178,3 +1205,290 @@ def run_forgetting_cycle_task(self, config_id: Optional[uuid.UUID] = None) -> Di
        return result
    finally:
        loop.close()
+
+
+# =============================================================================
+# Long-term Memory Storage Tasks (Batched Write Strategies)
+# =============================================================================
+
+@celery_app.task(name="app.core.memory.agent.long_term_storage.window", bind=True)
+def long_term_storage_window_task(
+    self,
+    end_user_id: str,
+    langchain_messages: List[Dict[str, Any]],
+    config_id: str,
+    scope: int = 6
+) -> Dict[str, Any]:
+    """Celery task for window-based long-term memory storage.
+    
+    Accumulates messages in Redis buffer until window size (scope) is reached,
+    then writes batched messages to Neo4j.
+    
+    Args:
+        end_user_id: End user identifier
+        langchain_messages: List of messages [{"role": "user/assistant", "content": "..."}]
+        config_id: Memory configuration ID
+        scope: Window size (number of messages before triggering write)
+        
+    Returns:
+        Dict containing task status and metadata
+    """
+    from app.core.logging_config import get_logger
+    logger = get_logger(__name__)
+    
+    logger.info(f"[LONG_TERM_WINDOW] Starting task - end_user_id={end_user_id}, scope={scope}")
+    start_time = time.time()
+    
+    async def _run() -> Dict[str, Any]:
+        from app.core.memory.agent.langgraph_graph.routing.write_router import window_dialogue
+        from app.core.memory.agent.langgraph_graph.tools.write_tool import chat_data_format
+        from app.core.memory.agent.utils.redis_tool import write_store
+        from app.services.memory_config_service import MemoryConfigService
+        
+        db = next(get_db())
+        try:
+            # Save to Redis buffer first
+            write_store.save_session_write(end_user_id, await chat_data_format(langchain_messages))
+            
+            # Load memory config
+            config_service = MemoryConfigService(db)
+            memory_config = config_service.load_memory_config(
+                config_id=config_id,
+                service_name="LongTermStorageTask"
+            )
+            
+            # Execute window-based dialogue storage
+            await window_dialogue(end_user_id, langchain_messages, memory_config, scope)
+            
+            return {"status": "SUCCESS", "strategy": "window", "scope": scope}
+        finally:
+            db.close()
+    
+    try:
+        import nest_asyncio
+        nest_asyncio.apply()
+    except ImportError:
+        pass
+    
+    try:
+        loop = asyncio.get_event_loop()
+        if loop.is_closed():
+            loop = asyncio.new_event_loop()
+            asyncio.set_event_loop(loop)
+    except RuntimeError:
+        loop = asyncio.new_event_loop()
+        asyncio.set_event_loop(loop)
+    
+    try:
+        result = loop.run_until_complete(_run())
+        elapsed_time = time.time() - start_time
+        
+        logger.info(f"[LONG_TERM_WINDOW] Task completed - elapsed_time={elapsed_time:.2f}s")
+        
+        return {
+            **result,
+            "end_user_id": end_user_id,
+            "config_id": config_id,
+            "elapsed_time": elapsed_time,
+            "task_id": self.request.id
+        }
+    except Exception as e:
+        elapsed_time = time.time() - start_time
+        logger.error(f"[LONG_TERM_WINDOW] Task failed - error={str(e)}", exc_info=True)
+        
+        return {
+            "status": "FAILURE",
+            "strategy": "window",
+            "error": str(e),
+            "end_user_id": end_user_id,
+            "config_id": config_id,
+            "elapsed_time": elapsed_time,
+            "task_id": self.request.id
+        }
+
+
+# @celery_app.task(name="app.core.memory.agent.long_term_storage.time", bind=True)
+# def long_term_storage_time_task(
+#     self,
+#     end_user_id: str,
+#     config_id: str,
+#     time_window: int = 5
+# ) -> Dict[str, Any]:
+#     """Celery task for time-based long-term memory storage.
+    
+#     Retrieves recent sessions from Redis within time window and writes to Neo4j.
+    
+#     Args:
+#         end_user_id: End user identifier
+#         config_id: Memory configuration ID
+#         time_window: Time window in minutes for retrieving recent sessions
+        
+#     Returns:
+#         Dict containing task status and metadata
+#     """
+#     from app.core.logging_config import get_logger
+#     logger = get_logger(__name__)
+    
+#     logger.info(f"[LONG_TERM_TIME] Starting task - end_user_id={end_user_id}, time_window={time_window}")
+#     start_time = time.time()
+    
+#     async def _run() -> Dict[str, Any]:
+#         from app.core.memory.agent.langgraph_graph.routing.write_router import memory_long_term_storage
+#         from app.services.memory_config_service import MemoryConfigService
+        
+#         db = next(get_db())
+#         try:
+#             # Load memory config
+#             config_service = MemoryConfigService(db)
+#             memory_config = config_service.load_memory_config(
+#                 config_id=config_id,
+#                 service_name="LongTermStorageTask"
+#             )
+            
+#             # Execute time-based storage
+#             await memory_long_term_storage(end_user_id, memory_config, time_window)
+            
+#             return {"status": "SUCCESS", "strategy": "time", "time_window": time_window}
+#         finally:
+#             db.close()
+    
+#     try:
+#         import nest_asyncio
+#         nest_asyncio.apply()
+#     except ImportError:
+#         pass
+    
+#     try:
+#         loop = asyncio.get_event_loop()
+#         if loop.is_closed():
+#             loop = asyncio.new_event_loop()
+#             asyncio.set_event_loop(loop)
+#     except RuntimeError:
+#         loop = asyncio.new_event_loop()
+#         asyncio.set_event_loop(loop)
+    
+#     try:
+#         result = loop.run_until_complete(_run())
+#         elapsed_time = time.time() - start_time
+        
+#         logger.info(f"[LONG_TERM_TIME] Task completed - elapsed_time={elapsed_time:.2f}s")
+        
+#         return {
+#             **result,
+#             "end_user_id": end_user_id,
+#             "config_id": config_id,
+#             "elapsed_time": elapsed_time,
+#             "task_id": self.request.id
+#         }
+#     except Exception as e:
+#         elapsed_time = time.time() - start_time
+#         logger.error(f"[LONG_TERM_TIME] Task failed - error={str(e)}", exc_info=True)
+        
+#         return {
+#             "status": "FAILURE",
+#             "strategy": "time",
+#             "error": str(e),
+#             "end_user_id": end_user_id,
+#             "config_id": config_id,
+#             "elapsed_time": elapsed_time,
+#             "task_id": self.request.id
+#         }
+
+
+# @celery_app.task(name="app.core.memory.agent.long_term_storage.aggregate", bind=True)
+# def long_term_storage_aggregate_task(
+#     self,
+#     end_user_id: str,
+#     langchain_messages: List[Dict[str, Any]],
+#     config_id: str
+# ) -> Dict[str, Any]:
+#     """Celery task for aggregate-based long-term memory storage.
+    
+#     Uses LLM to determine if new messages describe the same event as history.
+#     Only writes to Neo4j if messages represent new information (not duplicates).
+    
+#     Args:
+#         end_user_id: End user identifier
+#         langchain_messages: List of messages [{"role": "user/assistant", "content": "..."}]
+#         config_id: Memory configuration ID
+        
+#     Returns:
+#         Dict containing task status, is_same_event flag, and metadata
+#     """
+#     from app.core.logging_config import get_logger
+#     logger = get_logger(__name__)
+    
+#     logger.info(f"[LONG_TERM_AGGREGATE] Starting task - end_user_id={end_user_id}")
+#     start_time = time.time()
+    
+#     async def _run() -> Dict[str, Any]:
+#         from app.core.memory.agent.langgraph_graph.routing.write_router import aggregate_judgment
+#         from app.core.memory.agent.langgraph_graph.tools.write_tool import chat_data_format
+#         from app.core.memory.agent.utils.redis_tool import write_store
+#         from app.services.memory_config_service import MemoryConfigService
+        
+#         db = next(get_db())
+#         try:
+#             # Save to Redis buffer first
+#             write_store.save_session_write(end_user_id, await chat_data_format(langchain_messages))
+            
+#             # Load memory config
+#             config_service = MemoryConfigService(db)
+#             memory_config = config_service.load_memory_config(
+#                 config_id=config_id,
+#                 service_name="LongTermStorageTask"
+#             )
+            
+#             # Execute aggregate judgment
+#             result = await aggregate_judgment(end_user_id, langchain_messages, memory_config)
+            
+#             return {
+#                 "status": "SUCCESS",
+#                 "strategy": "aggregate",
+#                 "is_same_event": result.get("is_same_event", False),
+#                 "wrote_to_neo4j": not result.get("is_same_event", False)
+#             }
+#         finally:
+#             db.close()
+    
+#     try:
+#         import nest_asyncio
+#         nest_asyncio.apply()
+#     except ImportError:
+#         pass
+    
+#     try:
+#         loop = asyncio.get_event_loop()
+#         if loop.is_closed():
+#             loop = asyncio.new_event_loop()
+#             asyncio.set_event_loop(loop)
+#     except RuntimeError:
+#         loop = asyncio.new_event_loop()
+#         asyncio.set_event_loop(loop)
+    
+#     try:
+#         result = loop.run_until_complete(_run())
+#         elapsed_time = time.time() - start_time
+        
+#         logger.info(f"[LONG_TERM_AGGREGATE] Task completed - is_same_event={result.get('is_same_event')}, elapsed_time={elapsed_time:.2f}s")
+        
+#         return {
+#             **result,
+#             "end_user_id": end_user_id,
+#             "config_id": config_id,
+#             "elapsed_time": elapsed_time,
+#             "task_id": self.request.id
+#         }
+#     except Exception as e:
+#         elapsed_time = time.time() - start_time
+#         logger.error(f"[LONG_TERM_AGGREGATE] Task failed - error={str(e)}", exc_info=True)
+        
+#         return {
+#             "status": "FAILURE",
+#             "strategy": "aggregate",
+#             "error": str(e),
+#             "end_user_id": end_user_id,
+#             "config_id": config_id,
+#             "elapsed_time": elapsed_time,
+#             "task_id": self.request.id
+#         }