Skip to content

feat: Core engines - CompactEngine, ParallelExecutor, MCPManager - #141

Open
raymondginger2018-sudo wants to merge 10 commits into
HKUDS:mainfrom
raymondginger2018-sudo:feat/upstream-core-engines
Open

feat: Core engines - CompactEngine, ParallelExecutor, MCPManager#141
raymondginger2018-sudo wants to merge 10 commits into
HKUDS:mainfrom
raymondginger2018-sudo:feat/upstream-core-engines

Conversation

@raymondginger2018-sudo

Copy link
Copy Markdown

新增 4 个核心引擎模块:\n\n1. compact_engine.py (1007行) — Token-aware 会话压缩引擎\n - 三级水位监控 (micro/compact/deep)\n - LLM 生成摘要 (forked sub-agent)\n - tiktoken 真实计数 + 启发式回退\n - 熔断保护 (3次失败上限)\n\n2. parallel_executor.py (443行) — DAG 并行执行引擎\n - 2D 工具矩阵编排\n - 依赖图自动解析\n - 并行 Worker 池\n\n3. mcp_manager.py (283行) — MCP 服务器生命周期管理\n - 进程启动/停止/健康检查\n - 工具注册/发现\n\n4. auto_compact.py (259行) — 自动压缩触发

@raymondginger2018-sudo
raymondginger2018-sudo force-pushed the feat/upstream-core-engines branch from dd72368 to 566dd08 Compare July 30, 2026 07:06
清除 core 引擎文件中的 Claude Code 等第三方品牌引用,替换为中性描述
@raymondginger2018-sudo

Copy link
Copy Markdown
Author

Hi @akaxlh @chaohuang-ai, this PR adds core engines (CompactEngine / ParallelExecutor / MCPManager) to the core/ directory. It's rebased on the latest main and npm run check passes with 0 errors. Could you please take a look when you have time? Thanks!

…rchitecture)

- v4.0: idempotency / event-sourcing / saga / data-port / hexagonal / contextual-bandit
- v4.1: transparency family (failure/replication/concurrency/scaling/mobility/security)
- v4.1: transparency family (failure/replication/concurrency/scaling/mobility/security)
- v4.2: AI patterns (MAPE-K self-healing / reflexion / blackboard / intent) + model cascade
@raymondginger2018-sudo

Copy link
Copy Markdown
Author

@Zongwei9888 The CI workflow (Linting and Formatting) for this PR is stuck in action_required -- GitHub requires a maintainer to approve running Actions on fork pull requests.

All code has been verified locally (npm run check / py_compile all pass, no conflicts). Could you please Approve and run workflows on this PR, or adjust repo Actions settings to allow fork PRs without approval?

Thanks!

1 similar comment
@raymondginger2018-sudo

Copy link
Copy Markdown
Author

@Zongwei9888 The CI workflow (Linting and Formatting) for this PR is stuck in action_required -- GitHub requires a maintainer to approve running Actions on fork pull requests.

All code has been verified locally (npm run check / py_compile all pass, no conflicts). Could you please Approve and run workflows on this PR, or adjust repo Actions settings to allow fork PRs without approval?

Thanks!

cmd.exe 不消费 POSIX 语法(单引号包 JSON),导致 hooks 的
block/context/decision 在 Windows 上全部失效。_default_shell()
改为优先用 git-bash 执行,无 bash 时才回退 cmd.exe。

配套测试修复:
- test_hooks: cwd 默认 /tmp 在 Windows 解析为盘根/tmp(可能不存在),
  改用 tempfile.gettempdir();hook 命令内插的 Windows 路径用 as_posix()
- test_exec_sandbox_wiring: 断言放宽为接受 which('bash') 的绝对路径
- harness_sandbox: seatbelt profile 中 Windows 路径被转义为 \,断言侧同步转义
- mcp_server: workspace 硬编码 /tmp/x(Windows 解析为盘根/tmp/x),改用 tmp_path
- shell_search_tools: glob 输出分隔符跨平台,断言改用 os.sep
command 为全局安装的 PATH 命令(如 mcp-server-fetch)时被误判为
不存在的文件路径,导致 pre-commit 健康检查误报。先 shutil.which
解析 PATH,解析不到再按文件路径检查。
- fix(auto_compact): deep/compact 级压缩真实写盘(原为桩实现)
- 新增 tests/test_auto_compact.py (5 用例) + pytest.ini
- 真实会话验证: 427,894 → 75,770 tokens (水位 314% → 59.2%)
@raymondginger2018-sudo

Copy link
Copy Markdown
Author

Update: auto_compact real compaction fix (e3d1723)

Added to this PR today — fixes a significant bug in the compression engine:

The bug

_conversation_compact() / _session_memory_compact() in deepcode-engine-mcp/mcp_servers/auto_compact.py were stubs: they only estimated saved_tokens via a fixed ratio (0.35/0.50) and never rewrote the session file. Result: watermark stayed at 100%, and sessions ballooned unbounded (one real session hit 402K tokens against a 128K window — 314%).

The fix

Real write-back implementation:

  • _compact_history() — groups old turns, keeps the last keep_turns, replaces older turns with a single summary message, and actually rewrites the JSONL
  • _extract_key_line() — heuristic extraction of conclusion/result/error lines (no LLM dependency)
  • _persist_memory() — deep compacts persist the summary to <session>.memory.json
  • Summary length capped (compact 40% / deep 25% of original) so it's always strictly shorter

Verification

Item Result
New tests tests/test_auto_compact.py 5 passed (real write-back, turn preservation, memory persistence, too-few-turns noop, micro non-regression)
Existing compact tests 26 passed
Real-session deep compact 427,894 → 75,770 tokens (watermark 314% → 59.2%), JSONL integrity: 0 bad lines

Also added pytest.ini in the submodule to fix a rootdir collision (the parent repo's __init__.py was being imported as a package by pytest).

@HKUDS maintainers — this PR is ready for review; happy to split it into smaller pieces if preferred. 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants