Fix/memory celery fix (#168)
* refactor(celery): optimize task routing and worker configuration - Simplify Celery queue configuration with single default 'io_tasks' queue - Implement task routing strategy separating IO-bound and CPU-bound tasks - Add Flower monitoring support with task event tracking enabled - Add summary node search optimization to only retrieve summary nodes - Clean up unused imports and reorganize import statements for consistency - Update docker-compose configuration to support multi-queue worker setup * chore(celery): simplify flower configuration and add gevent dependency * chore(dependencies): add gevent dependency to requirements - Add gevent==24.11.1 to api/requirements.txt - Gevent is required for async worker support in Celery - Complements existing flower and celery configuration * refactor(celery): simplify async event loop handling and reorganize task queues - Replace complex nest_asyncio and manual event loop management with asyncio.run() in read_message_task, write_message_task, regenerate_memory_cache, and workspace_reflection_task - Rename task queues from io_tasks/cpu_tasks to memory_tasks/document_tasks for better semantic clarity - Update task routing configuration to reflect new queue names for memory agent tasks and document processing tasks - Remove redundant exception handling comments and simplify error handling logic - Update README with improved community support section including GitHub Issues, Pull Requests, Discussions, and WeChat community links - Simplifies event loop management by leveraging asyncio.run() which handles loop creation and cleanup automatically, reducing code complexity and potential race conditions
This commit is contained in:
@@ -7,10 +7,6 @@ services:
|
||||
- "8002:8000"
|
||||
env_file:
|
||||
- .env
|
||||
environment:
|
||||
- SERVER_IP=0.0.0.0
|
||||
# 如果代码里必须要 MCP_SERVER_URL,可以先注释或指向占位
|
||||
# - MCP_SERVER_URL=
|
||||
volumes:
|
||||
- ./files:/files
|
||||
- /etc/localtime:/etc/localtime:ro
|
||||
@@ -19,20 +15,53 @@ services:
|
||||
networks:
|
||||
- default
|
||||
- celery
|
||||
depends_on:
|
||||
- worker-memory
|
||||
- worker-document
|
||||
|
||||
# Celery worker
|
||||
worker:
|
||||
# Memory worker - Memory read/write tasks (threads pool for asyncio)
|
||||
worker-memory:
|
||||
image: redbear-mem-open:latest
|
||||
container_name: worker
|
||||
container_name: worker-memory
|
||||
env_file:
|
||||
- .env
|
||||
volumes:
|
||||
- ./files:/files
|
||||
- /etc/localtime:/etc/localtime:ro
|
||||
command: celery -A app.celery_worker.celery_app worker --loglevel=info
|
||||
command: celery -A app.celery_worker.celery_app worker -E --loglevel=info --pool=threads --concurrency=100 --queues=memory_tasks -n memory_worker@%h
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- celery
|
||||
|
||||
# Document worker - Document parsing tasks (prefork for CPU-bound)
|
||||
worker-document:
|
||||
image: redbear-mem-open:latest
|
||||
container_name: worker-document
|
||||
env_file:
|
||||
- .env
|
||||
volumes:
|
||||
- ./files:/files
|
||||
- /etc/localtime:/etc/localtime:ro
|
||||
command: celery -A app.celery_worker.celery_app worker -E --loglevel=info --pool=prefork --concurrency=4 --queues=document_tasks --max-tasks-per-child=100 -n document_worker@%h
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- celery
|
||||
|
||||
# Celery Beat - scheduler
|
||||
beat:
|
||||
image: redbear-mem-open:latest
|
||||
container_name: celery-beat
|
||||
env_file:
|
||||
- .env
|
||||
volumes:
|
||||
- ./files:/files
|
||||
- /etc/localtime:/etc/localtime:ro
|
||||
command: celery -A app.celery_worker.celery_app beat --loglevel=info
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- celery
|
||||
depends_on:
|
||||
- worker-memory
|
||||
|
||||
networks:
|
||||
celery:
|
||||
|
||||
Reference in New Issue
Block a user