Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions .github/workflows/benchmark-regression.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
name: Benchmark Regression

on:
schedule:
- cron: '0 2 * * *'
workflow_dispatch:

permissions:
contents: read

env:
TURBO_TELEMETRY_DISABLED: '1'
BENCHMARK_ENFORCE_AFTER: '2026-03-24T00:00:00Z'

jobs:
benchmark:
name: benchmark-regression
runs-on: ubuntu-latest

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 9.15.0

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20.x
cache: pnpm

- name: Install dependencies
run: pnpm install --frozen-lockfile

- name: Build web
run: pnpm --filter web build

- name: Start web server
run: |
pnpm --filter web start -- --port 3000 > /tmp/confession-web.log 2>&1 &
echo $! > /tmp/confession-web.pid

- name: Wait for /api/health
run: |
for i in {1..60}; do
if curl -fsS "http://127.0.0.1:3000/api/health" >/dev/null; then
echo "web server ready"
exit 0
fi
sleep 2
done
echo "web server did not become ready"
exit 1

- name: Run benchmark
run: |
pnpm --filter web benchmark:scan -- \
--api http://127.0.0.1:3000 \
--sizes 1000,3000 \
--runs 2 \
--warmup-runs 1 \
--status-clients 4 \
--depth quick \
--engine-mode baseline \
--output /tmp/confession-benchmark-current.json

- name: Check regression threshold
run: |
node web/scripts/check-benchmark-regression.mjs \
--baseline web/benchmarks/scan-baseline.json \
--current /tmp/confession-benchmark-current.json \
--latency-threshold 0.15 \
--rps-threshold 0.2 \
--enforce-after "$BENCHMARK_ENFORCE_AFTER"

- name: Upload benchmark artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: benchmark-regression
path: |
/tmp/confession-benchmark-current.json
/tmp/confession-web.log
web/benchmarks/scan-baseline.json

- name: Stop web server
if: always()
run: |
if [[ -f /tmp/confession-web.pid ]]; then
kill "$(cat /tmp/confession-web.pid)" || true
fi
84 changes: 84 additions & 0 deletions .github/workflows/code-scanning.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
name: Code Scanning

on:
pull_request:
branches: [main]
paths:
- '.github/workflows/code-scanning.yml'
- 'web/src/server/**'
- 'web/scripts/**'
- 'web/package.json'
- 'pnpm-lock.yaml'
- 'AGENTS.md'
- '.kiro/steering/**'
push:
branches: [main]
paths:
- '.github/workflows/code-scanning.yml'
- 'web/src/server/**'
- 'web/scripts/**'
- 'web/package.json'
- 'pnpm-lock.yaml'
- 'AGENTS.md'
- '.kiro/steering/**'
workflow_dispatch:

permissions:
contents: read
security-events: write

env:
TURBO_TELEMETRY_DISABLED: '1'

jobs:
sarif-upload:
name: sarif-${{ matrix.engine_mode }}-${{ matrix.depth }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- engine_mode: baseline
depth: standard
- engine_mode: agentic_beta
depth: standard

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup pnpm
uses: pnpm/action-setup@v4
with:
version: 9.15.0

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 20.x
cache: pnpm

- name: Install dependencies
run: pnpm install --frozen-lockfile

- name: Generate SARIF from fixture
run: |
node web/scripts/generate-sarif-ci.mjs \
--fixture web/scripts/code-scanning-fixture.json \
--output "/tmp/confession-${{ matrix.engine_mode }}-${{ matrix.depth }}.sarif.json" \
--engine-mode "${{ matrix.engine_mode }}" \
--depth "${{ matrix.depth }}" \
--max-results 5000 \
--max-bytes 9000000

- name: Upload SARIF artifact
uses: actions/upload-artifact@v4
with:
name: sarif-${{ matrix.engine_mode }}-${{ matrix.depth }}
path: /tmp/confession-${{ matrix.engine_mode }}-${{ matrix.depth }}.sarif.json

- name: Upload to GitHub Code Scanning
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: /tmp/confession-${{ matrix.engine_mode }}-${{ matrix.depth }}.sarif.json
category: confession-${{ matrix.engine_mode }}-${{ matrix.depth }}
9 changes: 7 additions & 2 deletions .kiro/steering/api-standards.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,19 +93,22 @@ fileMatchPattern: '**/src/server/**/*'
- LLM 回應 `confidence` 需以 0..1 儲存;若模型回傳 0..100 百分制,後端需正規化後再驗證
- 漏洞事件規範:
- `scan_detected`:新漏洞建立時寫入
- `scan_relocated`:同 `stableFingerprint` 命中舊漏洞但檔案/行號位移時寫入
- `review_saved`:`humanStatus/humanComment/owaspCategory` 任一變更時寫入
- `status_changed`:`status` 變更時寫入
- `scan_relocated` 事件需帶 `fromFilePath/fromLine/toFilePath/toLine`
- 狀態更新與事件寫入需在同一 transaction,確保一致性
- 相容舊 DB:`vulnerability_events` 尚未存在時,`/trend` 回退舊邏輯,`/:id/events` 回空陣列
- 漏洞語義去重規範:
- `upsertVulnerabilities` 寫入前需做語義去重(同一行同一敏感資料主題僅保留一筆)
- `hardcoded_secret` 與 `keyword_*` 重疊時,優先保留 `hardcoded_secret`
- 去重聚合鍵需採 `stableFingerprint` 優先;缺值時才回退舊語義鍵
- `GET /api/vulnerabilities` 與 `GET /api/vulnerabilities/stats` 需基於語義去重後資料回傳,避免同源重複告警膨脹
- `GET /api/vulnerabilities/stats` 需回傳 `bySeverityOpen`(僅 `status=open` 的嚴重度分佈),供 Dashboard 做資源分配建議
- 工作區快照收斂規範:
- 僅 `scanScope=workspace` 且 `workspaceSnapshotComplete !== false` 時啟用
- 必須以 `workspaceRoots` 限定收斂範圍;缺少 roots 時跳過收斂
- 若開放漏洞的 `filePath` 不在本次快照清單,後端需自動將其 `status` 由 `open` 轉為 `fixed`
- 收斂判斷需 fingerprint-aware:僅在 `filePath` 不在快照且其 `stableFingerprint` 未於本輪掃描觀測到時,才可 auto-fix
- 收斂時需寫入 `status_changed` 事件,訊息需說明來源檔案不在本次工作區快照(可能刪除或改名)
- 收斂失敗不得中斷整體掃描任務完成;需輸出結構化 log 供追查
- `POST /api/export` 規範:
Expand All @@ -115,9 +118,11 @@ fileMatchPattern: '**/src/server/**/*'
- `markdown`:`text/markdown; charset=utf-8`
- `pdf`:`text/html; charset=utf-8`(列印版 HTML,由前端觸發瀏覽器列印另存 PDF)
- `sarif`:`application/sarif+json; charset=utf-8`,版本固定 `2.1.0`,需包含 `partialFingerprints.stableFingerprint`
- SARIF 需套用 `maxResults`/`maxBytes` guard,截斷時以 `X-Confession-Sarif-Warning` 回傳警告摘要
- `Content-Disposition` 檔名格式統一:`confession-vulnerabilities-YYYYMMDD-HHmmss.<ext>`
- 掃描完成需輸出結構化 LLM 用量 log(`[Confession][LLMUsage]`),至少含 requestCount、token 用量、cacheHits、skippedByPolicy、successfulFiles、requestFailures、parseFailures、failureKinds
- 掃描完成需輸出引擎結構化 log(`[Confession][EngineMetrics]`),至少含 `agentic_attempt_count`、`agentic_failure_count`、`baseline_fallback_count`、`fallback_success_rate`
- 掃描完成需輸出引擎結構化 log(`[Confession][EngineMetrics]`),至少含 `agentic_attempt_count`、`agentic_failure_count`、`baseline_fallback_count`、`fallback_success_rate`、`fs_write_ops_per_scan`、`db_lock_wait_ms_p95`、`db_lock_timeout_count`
- 漏洞寫入需輸出儲存層結構化 log(`[Confession][StorageWriteMetrics]`),至少含 `taskId`、`vuln_count`、`write_ops`、`lock_wait_ms_p95`、`lock_timeout_count`、`relocation_count`

## Agent 系統

Expand Down
2 changes: 1 addition & 1 deletion .kiro/steering/code-conventions.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ export default function MyComponent({ children }: MyComponentProps) {

- **InteractionPoint**:AST 輸出(type, language, location, codeSnippet, patternName, confidence)
- **Vulnerability**:完整記錄,含位置、分類(type, cweId, severity)、修復建議、歸因、狀態、`stableFingerprint`、`source(sast|dast)`
- **VulnerabilityEvent**:漏洞事件流(scan_detected / review_saved / status_changed)
- **VulnerabilityEvent**:漏洞事件流(scan_detected / scan_relocated / review_saved / status_changed),relocation 事件需帶 `fromFilePath/fromLine/toFilePath/toLine`
- **ScanRequest**:files + depth + includeLlmScan
- **PluginConfig**:llm、analysis、ignore、api 設定
- **ExtToWebMsg / WebToExtMsg**:擴充套件與 webview 間的 postMessage 協議
Expand Down
2 changes: 1 addition & 1 deletion .kiro/steering/product.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ inclusion: always
- 當 `agentic_beta` 失敗時,後端需在同一 task 內自動回退 `baseline`
- 掃描前端輪詢若逾時,需主動送出取消請求中止任務,避免殘留 `running` 任務造成狀態誤導
- 工作區掃描需以「快照一致性」收斂舊漏洞:
- 來源檔案不在最新工作區快照時,自動將該漏洞由 `open` 收斂為 `fixed`
- 僅在來源檔案不在最新工作區快照,且同 `stableFingerprint` 未於本輪掃描再次出現時,才自動將漏洞由 `open` 收斂為 `fixed`
- 快照不完整(例如觸及檔案上限)時必須跳過收斂,避免誤判

## 專家審核與修復流程
Expand Down
11 changes: 10 additions & 1 deletion .kiro/steering/structure.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@ inclusion: always
confession/
├── .github/
│ └── workflows/
│ └── ci.yml # GitHub Actions:lint/build/test + quality + commit-check
│ ├── ci.yml # GitHub Actions:lint/build/test + quality + commit-check
│ ├── code-scanning.yml # 產生 SARIF 並上傳 GitHub Code Scanning
│ └── benchmark-regression.yml # 夜間/手動掃描效能回歸守門
├── .husky/
│ └── commit-msg # 本機 commit 訊息檢查 hook
├── confession-cli/ # npm 全域 CLI(bin: confession)
Expand All @@ -26,6 +28,12 @@ confession/
│ ├── src/status-bar.ts
│ └── src/types.ts
├── web/ # Next.js (App Router) + Hono 後端
│ ├── benchmarks/
│ │ └── scan-baseline.json # benchmark:scan 基線
│ ├── scripts/
│ │ ├── code-scanning-fixture.json
│ │ ├── generate-sarif-ci.mjs # CI SARIF 產生器(category/限制)
│ │ └── check-benchmark-regression.mjs
│ ├── src/app/
│ │ ├── layout.tsx
│ │ ├── globals.css
Expand All @@ -50,6 +58,7 @@ confession/
│ ├── mcp/
│ ├── db.ts # FileStore(.confession)+ SQLite 一次性遷移
│ ├── file-analysis-cache-store.ts
│ ├── sarif-generator.js
│ ├── advice-gate.ts
│ ├── health-score.ts
│ └── monitoring.ts
Expand Down
14 changes: 13 additions & 1 deletion .kiro/steering/tech.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,18 +21,28 @@ inclusion: always
| Agentic Engine | Planner/Skill/Analyst/Critic/Judge 多代理管線(正式預設,失敗自動回退 baseline) |
| MCP | 內建 broker + policy(白名單 server、僅允許安全能力) |
| 測試 | Vitest + fast-check(web/extension)+ Node.js `node:test`(CLI) |
| CI/CD | GitHub Actions(`lint`/`build`/`test` 並行 + `quality` 聚合 + `commit-check`) |
| CI/CD | GitHub Actions(`ci.yml` + `code-scanning.yml` + `benchmark-regression.yml`) |
| Commit 檢查 | commitlint + husky(`commit-msg` hook) |

## CI 快取與觸發

- `ci.yml` 需使用 `paths`/`paths-ignore` 做精準觸發,避免無關變更浪費 CI。
- `code-scanning.yml` 需限制為安全相關路徑觸發,並上傳 `category=confession-{engineMode}-{depth}` 的 SARIF。
- `benchmark-regression.yml` 以夜間排程 + 手動觸發執行;前期可 warning-only,達 `BENCHMARK_ENFORCE_AFTER` 後改為阻擋。
- Turborepo Remote Cache 以環境變數注入:
- `TURBO_TOKEN`
- `TURBO_TEAM`
- `TURBO_REMOTE_CACHE_SIGNATURE_KEY`
- CI 環境需設定 `TURBO_TELEMETRY_DISABLED=1`。

## SARIF 與回歸守門

- SARIF 產生邏輯需集中於 `web/src/server/sarif-generator.js`,供 route 與 CI 腳本共用。
- SARIF 需支援 `maxResults`/`maxBytes` guard,超限時截斷並附 warning。
- Benchmark regression 預設門檻:
- `scan_workspace_p95_ms` 惡化 > 15%
- `status_api_rps_p95` 惡化 > 20%

## 部署備註(Vercel)

- API 入口 `web/src/app/api/[...route]/route.ts` 需固定:
Expand Down Expand Up @@ -67,6 +77,8 @@ inclusion: always
- 測試(extension):`pnpm --filter confession-extension test`
- 測試(CLI):`pnpm --filter confession-cli test`
- 掃描基準(1000/3000 檔):`pnpm --filter web benchmark:scan`
- CI SARIF(本地模擬):`pnpm --filter web sarif:ci -- --output /tmp/confession.sarif.json`
- 效能回歸比對:`node web/scripts/check-benchmark-regression.mjs --baseline <baseline.json> --current <current.json>`
- 程式碼格式化:`pnpm format`
- 格式檢查:`pnpm format:check`
- CI 檢查彙總:`pnpm check:ci`
Expand Down
4 changes: 4 additions & 0 deletions .kiro/steering/testing-standards.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,13 +31,17 @@ fileMatchPattern: '**/*.test.*'
- `.confession/*.json` 讀寫與原子寫入測試
- `prisma` 外觀相容層(find/update/upsert/transaction)測試
- `scanTask` 快速寫路徑測試(僅寫 `scan-tasks.json` + `meta.json`)
- vulnerability 單次掃描單鎖/單次寫回測試(避免 chunk 交易放大)
- relocation match 測試(同 `stableFingerprint` rename/移行不重建、需產生 `scan_relocated`)
- `analysis-cache.json` 版本不相容清空與持久化測試
- 匯出能力需補齊:
- SARIF 2.1.0 schema 相容測試(含 `partialFingerprints.stableFingerprint`)
- SARIF `maxResults`/`maxBytes` guard 與 warning header 測試
- CSV 新欄位(`stableFingerprint`、`source`)與 BOM 驗證
- 效能基準需可重複執行:
- `pnpm --filter web benchmark:scan` 量測 `scan_workspace_p95_ms` 與 `status_api_rps`(預設 `engineMode=baseline`)
- 至少覆蓋 1000 與 3000 檔工作區場景
- benchmark regression script 需驗證門檻:`scan_workspace_p95_ms > +15%`、`status_api_rps_p95 < -20%`
- CLI 需覆蓋:
- `init` 建檔與重跑冪等
- `scan` 成功 / 失敗 / 逾時 cancel / SIGINT cancel
Expand Down
Loading
Loading