Compare commits

...

9 Commits

Author SHA1 Message Date
Tak Hoffman
80ca2bf451 Add embed transport hardening and sandbox controls 2026-04-10 12:05:11 -05:00
Tak Hoffman
999cb095c4 Restore offloaded chat attachment persistence 2026-04-10 08:38:25 -05:00
Tak Hoffman
e0668ee22d Fix embed follow-up review regressions 2026-04-10 08:27:58 -05:00
Tak Hoffman
3288d97428 Harden embed iframe URL handling 2026-04-10 08:21:59 -05:00
Tak Hoffman
49319ca986 Fix chat media and history regressions 2026-04-10 08:13:08 -05:00
Tak Hoffman
3b591566c6 Secure assistant media route and preserve UI avatar override 2026-04-10 01:28:27 -05:00
Tak Hoffman
eab40d89a1 Harden canvas path resolution and stage isolation 2026-04-10 00:47:34 -05:00
Tak Hoffman
8ef1415e94 Add changelog entry for embed rendering 2026-04-10 00:36:56 -05:00
Tak Hoffman
c0a2798a03 Add embed rendering for Control UI assistant output 2026-04-10 00:35:04 -05:00
64 changed files with 7105 additions and 2157 deletions

View File

@@ -11,6 +11,7 @@ Docs: https://docs.openclaw.ai
- Docs i18n: chunk raw doc translation, reject truncated tagged outputs, avoid ambiguous body-only wrapper unwrapping, and recover from terminated Pi translation sessions without changing the default `openai/gpt-5.4` path. (#62969, #63808) Thanks @hxy91819.
- QA/testing: add a `--runner multipass` lane for `openclaw qa suite` so repo-backed QA scenarios can run inside a disposable Linux VM and write back the usual report, summary, and VM logs. (#63426) Thanks @shakkernerd.
- Gateway: split startup and runtime seams so gateway lifecycle sequencing, reload state, and shutdown behavior stay easier to maintain without changing observed behavior. (#63975) Thanks @gumadeiras.
- Control UI/webchat: normalize assistant `MEDIA:`/reply/voice directives into structured bubble rendering, rename the unreleased rich web shortcode to `[embed ...]`, and surface session runtime roots so hosted web content is written to the correct document path instead of guessed local files.
### Fixes

View File

@@ -2402,6 +2402,7 @@ OpenClaw uses the built-in model catalog. Add custom providers via `models.provi
- `request.auth`: auth strategy override. Modes: `"provider-default"` (use provider's built-in auth), `"authorization-bearer"` (with `token`), `"header"` (with `headerName`, `value`, optional `prefix`).
- `request.proxy`: HTTP proxy override. Modes: `"env-proxy"` (use `HTTP_PROXY`/`HTTPS_PROXY` env vars), `"explicit-proxy"` (with `url`). Both modes accept an optional `tls` sub-object.
- `request.tls`: TLS override for direct connections. Fields: `ca`, `cert`, `key`, `passphrase` (all accept SecretRef), `serverName`, `insecureSkipVerify`.
- `request.allowPrivateNetwork`: when `true`, allow HTTPS to `baseUrl` when DNS resolves to private, CGNAT, or similar ranges, via the provider HTTP fetch guard (operator opt-in for trusted self-hosted OpenAI-compatible endpoints). WebSocket uses the same `request` for headers/TLS but not that fetch SSRF gate. Default `false`.
- `models.providers.*.models`: explicit provider model catalog entries.
- `models.providers.*.models.*.contextWindow`: native model context window metadata.
- `models.providers.*.models.*.contextTokens`: optional runtime context cap. Use this when you want a smaller effective context budget than the model's native `contextWindow`.
@@ -2858,6 +2859,7 @@ See [Plugins](/tools/plugin).
enabled: true,
basePath: "/openclaw",
// root: "dist/control-ui",
// embedSandbox: "powerful", // powerful | isolated
// allowedOrigins: ["https://control.example.com"], // required for non-loopback Control UI
// dangerouslyAllowHostHeaderOriginFallback: false, // dangerous Host-header origin fallback mode
// allowInsecureAuth: false,

View File

@@ -0,0 +1,50 @@
# Rich Output Protocol
Assistant output can carry a small set of delivery/render directives:
- `MEDIA:` for attachment delivery
- `[[audio_as_voice]]` for audio presentation hints
- `[[reply_to_current]]` / `[[reply_to:<id>]]` for reply metadata
- `[canvas ...]` for Control UI rich rendering
These directives are separate. `MEDIA:` and reply/voice tags remain delivery metadata; `[canvas ...]` is the web-only rich render path.
## `[canvas ...]`
`[canvas ...]` is the only agent-facing rich render syntax for the Control UI.
Self-closing example:
```text
[canvas ref="cv_123" title="Status" /]
```
Rules:
- `[view ...]` is no longer valid for new output.
- Canvas shortcodes render in the assistant message surface only.
- Only URL-backed canvases are rendered. Use `ref="..."` or `url="..."`.
- Block-form inline HTML canvas shortcodes are not rendered.
- The web UI strips the shortcode from visible text and renders the canvas inline.
- `MEDIA:` is not a canvas alias and should not be used for rich canvas rendering.
## Stored Rendering Shape
The normalized/stored assistant content block is a structured `canvas` item:
```json
{
"type": "canvas",
"preview": {
"kind": "canvas",
"surface": "assistant_message",
"render": "url",
"viewId": "cv_123",
"url": "/__openclaw__/canvas/documents/cv_123/index.html",
"title": "Status",
"preferredHeight": 320
}
}
```
Stored/rendered rich blocks use this `canvas` shape directly. `present_view` is not recognized.

View File

@@ -42,6 +42,7 @@ export function buildEmbeddedSystemPrompt(params: {
channel?: string;
/** Supported message actions for the current channel (e.g., react, edit, unsend) */
channelActions?: string[];
canvasRootDir?: string;
};
messageToolHints?: string[];
sandboxInfo?: EmbeddedSandboxInfo;

View File

@@ -3,6 +3,7 @@ import os from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
import type { OpenClawConfig } from "../config/config.js";
import { resolveStateDir } from "../config/paths.js";
import { buildSystemPromptParams } from "./system-prompt-params.js";
async function makeTempDir(label: string): Promise<string> {
@@ -101,4 +102,12 @@ describe("buildSystemPromptParams repo root", () => {
expect(runtimeInfo.repoRoot).toBeUndefined();
});
it("includes the default profile canvas root in runtimeInfo", async () => {
const workspaceDir = await makeTempDir("canvas-root");
const { runtimeInfo } = buildParams({ workspaceDir });
expect(runtimeInfo.canvasRootDir).toBe(path.resolve(path.join(resolveStateDir(), "canvas")));
});
});

View File

@@ -1,7 +1,9 @@
import fs from "node:fs";
import path from "node:path";
import type { OpenClawConfig } from "../config/config.js";
import { resolveStateDir } from "../config/paths.js";
import { findGitRoot } from "../infra/git-root.js";
import { resolveHomeRelativePath } from "../infra/home-dir.js";
import {
formatUserTime,
resolveUserTimeFormat,
@@ -23,6 +25,7 @@ export type RuntimeInfoInput = {
/** Supported message actions for the current channel (e.g., react, edit, unsend) */
channelActions?: string[];
repoRoot?: string;
canvasRootDir?: string;
};
export type SystemPromptRuntimeParams = {
@@ -47,11 +50,17 @@ export function buildSystemPromptParams(params: {
const userTimezone = resolveUserTimezone(params.config?.agents?.defaults?.userTimezone);
const userTimeFormat = resolveUserTimeFormat(params.config?.agents?.defaults?.timeFormat);
const userTime = formatUserTime(new Date(), userTimezone, userTimeFormat);
const stateDir = resolveStateDir(process.env);
const canvasRootDir = resolveCanvasRootDir({
config: params.config,
stateDir,
});
return {
runtimeInfo: {
agentId: params.agentId,
...params.runtime,
repoRoot,
canvasRootDir,
},
userTimezone,
userTime,
@@ -59,6 +68,18 @@ export function buildSystemPromptParams(params: {
};
}
function resolveCanvasRootDir(params: { config?: OpenClawConfig; stateDir: string }): string {
const configured = params.config?.canvasHost?.root?.trim();
if (configured) {
return path.resolve(
resolveHomeRelativePath(configured, {
env: process.env,
}),
);
}
return path.resolve(path.join(params.stateDir, "canvas"));
}
function resolveRepoRoot(params: {
config?: OpenClawConfig;
workspaceDir?: string;

View File

@@ -2,7 +2,6 @@ import { describe, expect, it } from "vitest";
import { SILENT_REPLY_TOKEN } from "../auto-reply/tokens.js";
import { typedCases } from "../test-utils/typed-cases.js";
import { buildSubagentSystemPrompt } from "./subagent-announce.js";
import { SYSTEM_PROMPT_CACHE_BOUNDARY } from "./system-prompt-cache-boundary.js";
import { buildAgentSystemPrompt, buildRuntimeLine } from "./system-prompt.js";
describe("buildAgentSystemPrompt", () => {
@@ -102,7 +101,7 @@ describe("buildAgentSystemPrompt", () => {
skillsPrompt:
"<available_skills>\n <skill>\n <name>demo</name>\n </skill>\n</available_skills>",
heartbeatPrompt: "ping",
toolNames: ["message", "memory_search", "cron"],
toolNames: ["message", "memory_search"],
docsPath: "/tmp/openclaw/docs",
extraSystemPrompt: "Subagent details",
ttsHint: "Voice (TTS) is enabled.",
@@ -120,16 +119,7 @@ describe("buildAgentSystemPrompt", () => {
expect(prompt).not.toContain("## Heartbeats");
expect(prompt).toContain("## Safety");
expect(prompt).toContain(
'For follow-up at a future time (for example "check back in 10 minutes", reminders, run-later work, or recurring tasks), use cron instead of exec sleep, yieldMs delays, or process polling.',
);
expect(prompt).toContain(
"Use exec/process only for commands that start now and continue running in the background.",
);
expect(prompt).toContain(
"For long-running work that starts now, start it once and rely on automatic completion wake when it is enabled and the command emits output or fails; otherwise use process to confirm completion, and use it for logs, status, input, or intervention.",
);
expect(prompt).toContain(
"Do not emulate scheduling with sleep loops, timeout loops, or repeated polling.",
"For long waits, avoid rapid poll loops: use exec with enough yieldMs or process(action=poll, timeout=<ms>).",
);
expect(prompt).toContain("You have no independent goals");
expect(prompt).toContain("Prioritize safety and human oversight");
@@ -159,89 +149,6 @@ describe("buildAgentSystemPrompt", () => {
);
});
it("tells the agent not to execute /approve through exec", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
});
expect(prompt).toContain(
"Never execute /approve through exec or any other shell/tool path; /approve is a user-facing approval command, not a shell command.",
);
});
it("adds stronger execution-bias guidance for actionable turns", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
});
expect(prompt).toContain("## Execution Bias");
expect(prompt).toContain(
"If the user asks you to do the work, start doing it in the same turn.",
);
expect(prompt).toContain(
"Commentary-only turns are incomplete when tools are available and the next action is clear.",
);
});
it("narrows silent reply guidance to true no-delivery cases", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
});
expect(prompt).toContain(
`Use ${SILENT_REPLY_TOKEN} ONLY when no user-visible reply is required.`,
);
expect(prompt).toContain(
"Never use it to avoid doing requested work or to end an actionable turn early.",
);
});
it("keeps manual /approve instructions for non-native approval channels", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
runtimeInfo: { channel: "signal" },
});
expect(prompt).toContain(
"When exec returns approval-pending, include the concrete /approve command from tool output",
);
expect(prompt).not.toContain("allow-once|allow-always|deny");
});
it("tells native approval channels not to duplicate plain chat /approve instructions", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
runtimeInfo: { channel: "telegram", capabilities: ["inlineButtons"] },
});
expect(prompt).toContain(
"When exec returns approval-pending on this channel, rely on native approval card/buttons when they appear and do not also send plain chat /approve instructions. Only include the concrete /approve command if the tool result says chat approvals are unavailable or only manual approval is possible.",
);
expect(prompt).toContain(
"Only include the concrete /approve command if the tool result says chat approvals are unavailable or only manual approval is possible.",
);
expect(prompt).not.toContain(
"When exec returns approval-pending, include the concrete /approve command from tool output",
);
});
it("treats webchat as a native approval surface", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
runtimeInfo: { channel: "webchat" },
});
expect(prompt).toContain(
"When exec returns approval-pending on this channel, rely on native approval card/buttons when they appear",
);
expect(prompt).toContain(
"Only include the concrete /approve command if the tool result says chat approvals are unavailable or only manual approval is possible.",
);
expect(prompt).not.toContain(
"When exec returns approval-pending, include the concrete /approve command from tool output",
);
});
it("omits skills in minimal prompt mode when skillsPrompt is absent", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
@@ -318,19 +225,46 @@ describe("buildAgentSystemPrompt", () => {
expect(prompt).toContain("do not forward raw internal metadata");
});
it("does not include canvas guidance in the default global prompt", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
});
expect(prompt).not.toContain("## Control UI Canvas");
expect(prompt).not.toContain("Use `[canvas ...]` only in Control UI/webchat sessions");
});
it("includes canvas guidance only for webchat sessions", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
runtimeInfo: {
channel: "webchat",
canvasRootDir: "/Users/example/.openclaw-dev/canvas",
},
});
expect(prompt).toContain("## Control UI Canvas");
expect(prompt).toContain("Use `[canvas ...]` only in Control UI/webchat sessions");
expect(prompt).toContain('[canvas ref="cv_123" title="Status" height="320" /]');
expect(prompt).toContain(
'[canvas url="/__openclaw__/canvas/documents/cv_123/index.html" title="Status" height="320" /]',
);
expect(prompt).toContain(
"Never use local filesystem paths or `file://...` URLs in `[canvas ...]`.",
);
expect(prompt).toContain(
"The active hosted canvas root for this session is: `/Users/example/.openclaw-dev/canvas`.",
);
expect(prompt).not.toContain('[canvas content_type="html" title="Status"]...[/canvas]');
});
it("guides subagent workflows to avoid polling loops", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
});
expect(prompt).toContain(
'For follow-up at a future time (for example "check back in 10 minutes", reminders, run-later work, or recurring tasks), use cron instead of exec sleep, yieldMs delays, or process polling.',
);
expect(prompt).toContain(
"Use exec/process only for commands that start now and continue running in the background.",
);
expect(prompt).toContain(
"For long-running work that starts now, start it once and rely on automatic completion wake when it is enabled and the command emits output or fails; otherwise use process to confirm completion, and use it for logs, status, input, or intervention.",
"For long waits, avoid rapid poll loops: use exec with enough yieldMs or process(action=poll, timeout=<ms>).",
);
expect(prompt).toContain("Completion is push-based: it will auto-announce when done.");
expect(prompt).toContain("Do not poll `subagents list` / `sessions_list` in a loop");
@@ -339,25 +273,16 @@ describe("buildAgentSystemPrompt", () => {
);
});
it("uses structured tool definitions as the source of truth", () => {
it("lists available tools when provided", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
toolNames: ["exec", "sessions_list", "sessions_history", "sessions_send"],
});
expect(prompt).toContain(
"Structured tool definitions are the source of truth for tool names, descriptions, and parameters.",
);
expect(prompt).toContain(
"Tool names are case-sensitive. Call tools exactly as listed in the structured tool definitions.",
);
expect(prompt).toContain(
"TOOLS.md does not control tool availability; it is user guidance for how to use external tools.",
);
expect(prompt).not.toContain("Tool availability (filtered by policy):");
expect(prompt).not.toContain("- sessions_list:");
expect(prompt).not.toContain("- sessions_history:");
expect(prompt).not.toContain("- sessions_send:");
expect(prompt).toContain("Tool availability (filtered by policy):");
expect(prompt).toContain("sessions_list");
expect(prompt).toContain("sessions_history");
expect(prompt).toContain("sessions_send");
});
it("documents ACP sessions_spawn agent targeting requirements", () => {
@@ -367,8 +292,10 @@ describe("buildAgentSystemPrompt", () => {
});
expect(prompt).toContain("sessions_spawn");
expect(prompt).toContain("Set `agentId` explicitly unless `acp.defaultAgent` is configured");
expect(prompt).toContain("`subagents`/`agents_list`");
expect(prompt).toContain(
'runtime="acp" requires `agentId` unless `acp.defaultAgent` is configured',
);
expect(prompt).toContain("not agents_list");
});
it("guides harness requests to ACP thread-bound spawns", () => {
@@ -403,9 +330,8 @@ describe("buildAgentSystemPrompt", () => {
);
expect(prompt).not.toContain('runtime="acp" requires `agentId`');
expect(prompt).not.toContain("not ACP harness ids");
expect(prompt).toContain(
"If a task is more complex or takes longer, spawn a sub-agent. Completion is push-based: it will auto-announce when done.",
);
expect(prompt).toContain("- sessions_spawn: Spawn an isolated sub-agent session");
expect(prompt).toContain("- agents_list: List OpenClaw agent ids allowed for sessions_spawn");
});
it("omits ACP harness spawn guidance for sandboxed sessions and shows ACP block note", () => {
@@ -439,12 +365,8 @@ describe("buildAgentSystemPrompt", () => {
docsPath: "/tmp/openclaw/docs",
});
expect(prompt).toContain(
"Tool names are case-sensitive. Call tools exactly as listed in the structured tool definitions.",
);
expect(prompt).toContain(
"For long waits, avoid rapid poll loops: use Exec with enough yieldMs or process(action=poll, timeout=<ms>).",
);
expect(prompt).toContain("- Read: Read file contents");
expect(prompt).toContain("- Exec: Run shell commands");
expect(prompt).toContain(
"- If exactly one skill clearly applies: read its SKILL.md at <location> with `Read`, then follow it.",
);
@@ -454,25 +376,6 @@ describe("buildAgentSystemPrompt", () => {
);
});
it("adds update_plan guidance only when the tool is available", () => {
const promptWithPlan = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
toolNames: ["exec", "update_plan"],
});
const promptWithoutPlan = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
toolNames: ["exec"],
});
expect(promptWithPlan).toContain(
"For non-trivial multi-step work, keep a short plan updated with `update_plan`.",
);
expect(promptWithPlan).toContain(
"When you use `update_plan`, keep exactly one step `in_progress` until the work is done.",
);
expect(promptWithoutPlan).not.toContain("keep a short plan updated with `update_plan`");
});
it("includes docs guidance when docsPath is provided", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
@@ -502,7 +405,7 @@ describe("buildAgentSystemPrompt", () => {
params: {
workspaceDir: "/tmp/openclaw",
userTimezone: "America/Chicago",
userTime: "Monday, January 5th, 2026 - 3:26 PM",
userTime: "Monday, January 5th, 2026 3:26 PM",
userTimeFormat: "12" as const,
},
},
@@ -511,7 +414,7 @@ describe("buildAgentSystemPrompt", () => {
params: {
workspaceDir: "/tmp/openclaw",
userTimezone: "America/Chicago",
userTime: "Monday, January 5th, 2026 - 15:26",
userTime: "Monday, January 5th, 2026 15:26",
userTimeFormat: "24" as const,
},
},
@@ -544,19 +447,23 @@ describe("buildAgentSystemPrompt", () => {
// The system prompt intentionally does NOT include the current date/time.
// Only the timezone is included, to keep the prompt stable for caching.
// See: https://github.com/moltbot/moltbot/commit/66eec295b894bce8333886cfbca3b960c57c4946
// Agents should use session_status or message timestamps to determine the date/time.
// Related: https://github.com/moltbot/moltbot/issues/1897
// https://github.com/moltbot/moltbot/issues/3658
it("does NOT include a date or time in the system prompt (cache stability)", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/clawd",
userTimezone: "America/Chicago",
userTime: "Monday, January 5th, 2026 - 3:26 PM",
userTime: "Monday, January 5th, 2026 3:26 PM",
userTimeFormat: "12",
});
// The prompt should contain the timezone but NOT the formatted date/time string.
// This is intentional for prompt cache stability. If you want to add date/time
// awareness, do it through gateway-level timestamp injection into messages, not
// the system prompt.
// This is intentional for prompt cache stability — the date/time was removed in
// commit 66eec295b. If you're here because you want to add it back, please see
// https://github.com/moltbot/moltbot/issues/3658 for the preferred approach:
// gateway-level timestamp injection into messages, not the system prompt.
expect(prompt).toContain("Time zone: America/Chicago");
expect(prompt).not.toContain("Monday, January 5th, 2026");
expect(prompt).not.toContain("3:26 PM");
@@ -567,14 +474,14 @@ describe("buildAgentSystemPrompt", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
modelAliasLines: [
"- Opus: anthropic/claude-opus-4-6",
"- Sonnet: anthropic/claude-sonnet-4-6",
"- Opus: anthropic/claude-opus-4-5",
"- Sonnet: anthropic/claude-sonnet-4-5",
],
});
expect(prompt).toContain("## Model Aliases");
expect(prompt).toContain("Prefer aliases when specifying model overrides");
expect(prompt).toContain("- Opus: anthropic/claude-opus-4-6");
expect(prompt).toContain("- Opus: anthropic/claude-opus-4-5");
});
it("adds ClaudeBot self-update guidance when gateway tool is available", () => {
@@ -681,112 +588,14 @@ describe("buildAgentSystemPrompt", () => {
expect(prompt).not.toContain("# Project Context");
});
it("orders stable project context before the cache boundary and moves HEARTBEAT below it", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
contextFiles: [
{ path: "HEARTBEAT.md", content: "Check inbox." },
{ path: "MEMORY.md", content: "Long-term notes." },
{ path: "AGENTS.md", content: "Follow repo rules." },
{ path: "SOUL.md", content: "Warm but direct." },
{ path: "TOOLS.md", content: "Prefer rg." },
],
});
const agentsIndex = prompt.indexOf("## AGENTS.md");
const soulIndex = prompt.indexOf("## SOUL.md");
const toolsIndex = prompt.indexOf("## TOOLS.md");
const memoryIndex = prompt.indexOf("## MEMORY.md");
const boundaryIndex = prompt.indexOf(SYSTEM_PROMPT_CACHE_BOUNDARY);
const heartbeatHeadingIndex = prompt.indexOf("# Dynamic Project Context");
const heartbeatFileIndex = prompt.indexOf("## HEARTBEAT.md");
expect(agentsIndex).toBeGreaterThan(-1);
expect(soulIndex).toBeGreaterThan(agentsIndex);
expect(toolsIndex).toBeGreaterThan(soulIndex);
expect(memoryIndex).toBeGreaterThan(toolsIndex);
expect(boundaryIndex).toBeGreaterThan(memoryIndex);
expect(heartbeatHeadingIndex).toBeGreaterThan(boundaryIndex);
expect(heartbeatFileIndex).toBeGreaterThan(heartbeatHeadingIndex);
expect(prompt).toContain(
"The following frequently-changing project context files are kept below the cache boundary when possible:",
);
});
it("keeps heartbeat-only project context below the cache boundary", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
contextFiles: [{ path: "HEARTBEAT.md", content: "Check inbox." }],
});
const boundaryIndex = prompt.indexOf(SYSTEM_PROMPT_CACHE_BOUNDARY);
const projectContextIndex = prompt.indexOf("# Project Context");
const heartbeatFileIndex = prompt.indexOf("## HEARTBEAT.md");
expect(boundaryIndex).toBeGreaterThan(-1);
expect(projectContextIndex).toBeGreaterThan(boundaryIndex);
expect(heartbeatFileIndex).toBeGreaterThan(projectContextIndex);
expect(prompt).not.toContain("# Dynamic Project Context");
});
it("replaces provider-owned prompt sections without disturbing core ordering", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
promptContribution: {
sectionOverrides: {
interaction_style: "## Interaction Style\n\nCustom interaction guidance.",
execution_bias: "## Execution Bias\n\nCustom execution guidance.",
},
},
});
expect(prompt).toContain("## Interaction Style\n\nCustom interaction guidance.");
expect(prompt).toContain("## Execution Bias\n\nCustom execution guidance.");
expect(prompt).not.toContain("Bias toward action and momentum.");
});
it("places provider stable prefixes above the cache boundary", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
promptContribution: {
stablePrefix: "## Provider Stable Block\n\nStable provider guidance.",
},
});
const boundaryIndex = prompt.indexOf(SYSTEM_PROMPT_CACHE_BOUNDARY);
const stableIndex = prompt.indexOf("## Provider Stable Block");
const safetyIndex = prompt.indexOf("## Safety");
expect(stableIndex).toBeGreaterThan(-1);
expect(boundaryIndex).toBeGreaterThan(stableIndex);
expect(safetyIndex).toBeGreaterThan(stableIndex);
});
it("places provider dynamic suffixes below the cache boundary", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
promptContribution: {
dynamicSuffix: "## Provider Dynamic Block\n\nPer-turn provider guidance.",
},
});
const boundaryIndex = prompt.indexOf(SYSTEM_PROMPT_CACHE_BOUNDARY);
const dynamicIndex = prompt.indexOf("## Provider Dynamic Block");
const heartbeatIndex = prompt.indexOf("## Heartbeats");
expect(boundaryIndex).toBeGreaterThan(-1);
expect(dynamicIndex).toBeGreaterThan(boundaryIndex);
expect(heartbeatIndex).toBe(-1);
});
it("summarizes the message tool when available", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
toolNames: ["message"],
});
expect(prompt).toContain("message: Send messages and channel actions");
expect(prompt).toContain("### message tool");
expect(prompt).toContain("Use `message` for proactive sends + channel actions");
expect(prompt).toContain(`respond with ONLY: ${SILENT_REPLY_TOKEN}`);
});
@@ -814,7 +623,7 @@ describe("buildAgentSystemPrompt", () => {
});
expect(prompt).toContain("channel=telegram");
expect(prompt).toContain("capabilities=inlinebuttons");
expect(prompt).toContain("capabilities=inlineButtons");
});
it("includes agent id in runtime when provided", () => {
@@ -854,7 +663,7 @@ describe("buildAgentSystemPrompt", () => {
arch: "arm64",
node: "v20",
model: "anthropic/claude",
defaultModel: "anthropic/claude-opus-4-6",
defaultModel: "anthropic/claude-opus-4-5",
},
"telegram",
["inlineButtons"],
@@ -867,58 +676,12 @@ describe("buildAgentSystemPrompt", () => {
expect(line).toContain("os=macOS (arm64)");
expect(line).toContain("node=v20");
expect(line).toContain("model=anthropic/claude");
expect(line).toContain("default_model=anthropic/claude-opus-4-6");
expect(line).toContain("default_model=anthropic/claude-opus-4-5");
expect(line).toContain("channel=telegram");
expect(line).toContain("capabilities=inlinebuttons");
expect(line).toContain("capabilities=inlineButtons");
expect(line).toContain("thinking=low");
});
it("normalizes runtime capability ordering and casing for cache stability", () => {
const line = buildRuntimeLine(
{
agentId: "work",
},
"telegram",
[" React ", "inlineButtons", "react"],
"low",
);
expect(line).toContain("capabilities=inlinebuttons,react");
});
it("keeps semantically equivalent structured prompt inputs byte-stable", () => {
const clean = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
runtimeInfo: {
channel: "telegram",
capabilities: ["inlinebuttons", "react"],
},
skillsPrompt:
"<available_skills>\n <skill>\n <name>demo</name>\n </skill>\n</available_skills>",
heartbeatPrompt: "ping",
extraSystemPrompt: "Group chat context\nSecond line",
workspaceNotes: ["Reminder: keep commits scoped."],
modelAliasLines: ["- Sonnet: anthropic/claude-sonnet-4-5"],
});
const noisy = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",
runtimeInfo: {
channel: "telegram",
capabilities: [" react ", "inlineButtons", "react"],
},
skillsPrompt:
"<available_skills>\r\n <skill> \r\n <name>demo</name>\t\r\n </skill>\r\n</available_skills>\r\n",
heartbeatPrompt: " ping \r\n",
extraSystemPrompt: " Group chat context \r\nSecond line \t\r\n",
workspaceNotes: [" Reminder: keep commits scoped. \t\r\n"],
modelAliasLines: [" - Sonnet: anthropic/claude-sonnet-4-5 \t\r\n"],
});
expect(noisy).toBe(clean);
expect(noisy).not.toContain("\r");
expect(noisy).not.toMatch(/[ \t]+$/m);
});
it("describes sandboxed runtime and elevated when allowed", () => {
const prompt = buildAgentSystemPrompt({
workspaceDir: "/tmp/openclaw",

View File

@@ -1,28 +1,13 @@
import { createHmac, createHash } from "node:crypto";
import type { ReasoningLevel, ThinkLevel } from "../auto-reply/thinking.js";
import { SILENT_REPLY_TOKEN } from "../auto-reply/tokens.js";
import { resolveChannelApprovalCapability } from "../channels/plugins/approvals.js";
import { getChannelPlugin } from "../channels/plugins/index.js";
import type { MemoryCitationsMode } from "../config/types.memory.js";
import { buildMemoryPromptSection } from "../plugins/memory-state.js";
import {
normalizeLowercaseStringOrEmpty,
normalizeOptionalLowercaseString,
} from "../shared/string-coerce.js";
import { listDeliverableMessageChannels } from "../utils/message-channel.js";
import type { ResolvedTimeFormat } from "./date-time.js";
import type { EmbeddedContextFile } from "./pi-embedded-helpers.js";
import type { EmbeddedSandboxInfo } from "./pi-embedded-runner/types.js";
import {
normalizePromptCapabilityIds,
normalizeStructuredPromptSection,
} from "./prompt-cache-stability.js";
import { sanitizeForPromptLiteral } from "./sanitize-for-prompt.js";
import { SYSTEM_PROMPT_CACHE_BOUNDARY } from "./system-prompt-cache-boundary.js";
import type {
ProviderSystemPromptContribution,
ProviderSystemPromptSectionId,
} from "./system-prompt-contribution.js";
/**
* Controls which hardcoded sections are included in the system prompt.
@@ -33,81 +18,6 @@ import type {
export type PromptMode = "full" | "minimal" | "none";
type OwnerIdDisplay = "raw" | "hash";
const CONTEXT_FILE_ORDER = new Map<string, number>([
["agents.md", 10],
["soul.md", 20],
["identity.md", 30],
["user.md", 40],
["tools.md", 50],
["bootstrap.md", 60],
["memory.md", 70],
]);
const DYNAMIC_CONTEXT_FILE_BASENAMES = new Set(["heartbeat.md"]);
function normalizeContextFilePath(pathValue: string): string {
return pathValue.trim().replace(/\\/g, "/");
}
function getContextFileBasename(pathValue: string): string {
const normalizedPath = normalizeContextFilePath(pathValue);
return normalizeLowercaseStringOrEmpty(normalizedPath.split("/").pop() ?? normalizedPath);
}
function isDynamicContextFile(pathValue: string): boolean {
return DYNAMIC_CONTEXT_FILE_BASENAMES.has(getContextFileBasename(pathValue));
}
function sortContextFilesForPrompt(contextFiles: EmbeddedContextFile[]): EmbeddedContextFile[] {
return contextFiles.toSorted((a, b) => {
const aPath = normalizeContextFilePath(a.path);
const bPath = normalizeContextFilePath(b.path);
const aBase = getContextFileBasename(a.path);
const bBase = getContextFileBasename(b.path);
const aOrder = CONTEXT_FILE_ORDER.get(aBase) ?? Number.MAX_SAFE_INTEGER;
const bOrder = CONTEXT_FILE_ORDER.get(bBase) ?? Number.MAX_SAFE_INTEGER;
if (aOrder !== bOrder) {
return aOrder - bOrder;
}
if (aBase !== bBase) {
return aBase.localeCompare(bBase);
}
return aPath.localeCompare(bPath);
});
}
function buildProjectContextSection(params: {
files: EmbeddedContextFile[];
heading: string;
dynamic: boolean;
}) {
if (params.files.length === 0) {
return [];
}
const lines = [params.heading, ""];
if (params.dynamic) {
lines.push(
"The following frequently-changing project context files are kept below the cache boundary when possible:",
"",
);
} else {
const hasSoulFile = params.files.some(
(file) => getContextFileBasename(file.path) === "soul.md",
);
lines.push("The following project context files have been loaded:");
if (hasSoulFile) {
lines.push(
"If SOUL.md is present, embody its persona and tone. Avoid stiff, generic replies; follow its guidance unless higher-priority instructions override it.",
);
}
lines.push("");
}
for (const file of params.files) {
lines.push(`## ${file.path}`, "", file.content, "");
}
return lines;
}
function buildSkillsSection(params: { skillsPrompt?: string; readToolName: string }) {
const trimmed = params.skillsPrompt?.trim();
if (!trimmed) {
@@ -128,11 +38,10 @@ function buildSkillsSection(params: { skillsPrompt?: string; readToolName: strin
function buildMemorySection(params: {
isMinimal: boolean;
includeMemorySection?: boolean;
availableTools: Set<string>;
citationsMode?: MemoryCitationsMode;
}) {
if (params.isMinimal || params.includeMemorySection === false) {
if (params.isMinimal) {
return [];
}
return buildMemoryPromptSection({
@@ -179,18 +88,46 @@ function buildTimeSection(params: { userTimezone?: string }) {
return ["## Current Date & Time", `Time zone: ${params.userTimezone}`, ""];
}
function buildReplyTagsSection(isMinimal: boolean) {
function buildAssistantOutputDirectivesSection(isMinimal: boolean) {
if (isMinimal) {
return [];
}
return [
"## Reply Tags",
"To request a native reply/quote on supported surfaces, include one tag in your reply:",
"## Assistant Output Directives",
"Use these when you need delivery metadata in an assistant message:",
"- `MEDIA:<path-or-url>` on its own line requests attachment delivery. The web UI strips supported MEDIA lines and renders them inline; channels still decide actual delivery behavior.",
"- `[[audio_as_voice]]` marks attached audio as a voice-note style delivery hint. The web UI may show a voice-note badge when audio is present; channels still own delivery semantics.",
"- To request a native reply/quote on supported surfaces, include one reply tag in your reply:",
"- Reply tags must be the very first token in the message (no leading text/newlines): [[reply_to_current]] your reply.",
"- [[reply_to_current]] replies to the triggering message.",
"- Prefer [[reply_to_current]]. Use [[reply_to:<id>]] only when an id was explicitly provided (e.g. by the user or a tool).",
"Whitespace inside the tag is allowed (e.g. [[ reply_to_current ]] / [[ reply_to: 123 ]]).",
"Tags are stripped before sending; support depends on the current channel config.",
"- Channel-specific interactive directives are separate and should not be mixed into this web render guidance.",
"Supported tags are stripped before user-visible rendering; support still depends on the current channel config.",
"",
];
}
function buildWebchatCanvasSection(params: {
isMinimal: boolean;
runtimeChannel?: string;
canvasRootDir?: string;
}) {
if (params.isMinimal || params.runtimeChannel !== "webchat") {
return [];
}
return [
"## Control UI Canvas",
"Use `[canvas ...]` only in Control UI/webchat sessions for inline rich rendering inside the assistant bubble.",
"- Do not use `[canvas ...]` for non-web channels.",
"- `[canvas ...]` is separate from `MEDIA:`. Use `MEDIA:` for attachments; use `[canvas ...]` for web-only rich rendering.",
'- Use self-closing form for hosted canvas documents: `[canvas ref="cv_123" title="Status" height="320" /]`.',
'- You may also use an explicit hosted URL: `[canvas url="/__openclaw__/canvas/documents/cv_123/index.html" title="Status" height="320" /]`.',
'- Never use local filesystem paths or `file://...` URLs in `[canvas ...]`. Web canvases must point at hosted `/__openclaw__/canvas/...` URLs or use `ref="..."`.',
params.canvasRootDir
? `- The active hosted canvas root for this session is: \`${sanitizeForPromptLiteral(params.canvasRootDir)}\`. If you manually stage a hosted canvas file, write it there, not in the workspace.`
: "- The active hosted canvas root is profile-scoped, not workspace-scoped. If you manually stage a hosted canvas file, write it under the active profile canvas root, not in the workspace.",
"- Quote all attribute values. Prefer `ref` for hosted canvas documents unless you already have the full `/__openclaw__/canvas/documents/<id>/index.html` URL.",
"",
];
}
@@ -264,56 +201,6 @@ function buildDocsSection(params: { docsPath?: string; isMinimal: boolean; readT
];
}
function buildExecutionBiasSection(params: { isMinimal: boolean }) {
if (params.isMinimal) {
return [];
}
return [
"## Execution Bias",
"If the user asks you to do the work, start doing it in the same turn.",
"Use a real tool call or concrete action first when the task is actionable; do not stop at a plan or promise-to-act reply.",
"Commentary-only turns are incomplete when tools are available and the next action is clear.",
"If the work will take multiple steps or a while to finish, send one short progress update before or while acting.",
"",
];
}
function normalizeProviderPromptBlock(value?: string): string | undefined {
if (typeof value !== "string") {
return undefined;
}
const normalized = normalizeStructuredPromptSection(value);
return normalized || undefined;
}
function buildOverridablePromptSection(params: {
override?: string;
fallback: string[];
}): string[] {
const override = normalizeProviderPromptBlock(params.override);
if (override) {
return [override, ""];
}
return params.fallback;
}
function buildExecApprovalPromptGuidance(params: {
runtimeChannel?: string;
inlineButtonsEnabled?: boolean;
}) {
const runtimeChannel = normalizeOptionalLowercaseString(params.runtimeChannel);
const usesNativeApprovalUi =
runtimeChannel === "webchat" ||
params.inlineButtonsEnabled === true ||
(runtimeChannel
? Boolean(resolveChannelApprovalCapability(getChannelPlugin(runtimeChannel))?.native)
: false);
if (usesNativeApprovalUi) {
return "When exec returns approval-pending on this channel, rely on native approval card/buttons when they appear and do not also send plain chat /approve instructions. Only include the concrete /approve command if the tool result says chat approvals are unavailable or only manual approval is possible.";
}
return "When exec returns approval-pending, include the concrete /approve command from tool output as plain chat text for the user, and do not ask for a different or rotated code.";
}
export function buildAgentSystemPrompt(params: {
workspaceDir: string;
defaultThinkLevel?: ThinkLevel;
@@ -324,6 +211,7 @@ export function buildAgentSystemPrompt(params: {
ownerDisplaySecret?: string;
reasoningTagHint?: boolean;
toolNames?: string[];
toolSummaries?: Record<string, string>;
modelAliasLines?: string[];
userTimezone?: string;
userTime?: string;
@@ -344,13 +232,13 @@ export function buildAgentSystemPrompt(params: {
os?: string;
arch?: string;
node?: string;
provider?: string;
model?: string;
defaultModel?: string;
shell?: string;
channel?: string;
capabilities?: string[];
repoRoot?: string;
canvasRootDir?: string;
};
messageToolHints?: string[];
sandboxInfo?: EmbeddedSandboxInfo;
@@ -359,20 +247,80 @@ export function buildAgentSystemPrompt(params: {
level: "minimal" | "extensive";
channel: string;
};
/** Whether to include the active memory plugin prompt guidance in the base system prompt. Defaults to true. */
includeMemorySection?: boolean;
memoryCitationsMode?: MemoryCitationsMode;
promptContribution?: ProviderSystemPromptContribution;
}) {
const acpEnabled = params.acpEnabled !== false;
const sandboxedRuntime = params.sandboxInfo?.enabled === true;
const acpSpawnRuntimeEnabled = acpEnabled && !sandboxedRuntime;
const coreToolSummaries: Record<string, string> = {
read: "Read file contents",
write: "Create or overwrite files",
edit: "Make precise edits to files",
apply_patch: "Apply multi-file patches",
grep: "Search file contents for patterns",
find: "Find files by glob pattern",
ls: "List directory contents",
exec: "Run shell commands (pty available for TTY-required CLIs)",
process: "Manage background exec sessions",
web_search: "Search the web (Brave API)",
web_fetch: "Fetch and extract readable content from a URL",
// Channel docking: add login tools here when a channel needs interactive linking.
browser: "Control web browser",
canvas: "Present/eval/snapshot the Canvas",
nodes: "List/describe/notify/camera/screen on paired nodes",
cron: "Manage cron jobs and wake events (use for reminders; when scheduling a reminder, write the systemEvent text as something that will read like a reminder when it fires, and mention that it is a reminder depending on the time gap between setting and firing; include recent context in reminder text if appropriate)",
message: "Send messages and channel actions",
gateway: "Restart, apply config, or run updates on the running OpenClaw process",
agents_list: acpSpawnRuntimeEnabled
? 'List OpenClaw agent ids allowed for sessions_spawn when runtime="subagent" (not ACP harness ids)'
: "List OpenClaw agent ids allowed for sessions_spawn",
sessions_list: "List other sessions (incl. sub-agents) with filters/last",
sessions_history: "Fetch history for another session/sub-agent",
sessions_send: "Send a message to another session/sub-agent",
sessions_spawn: acpSpawnRuntimeEnabled
? 'Spawn an isolated sub-agent or ACP coding session (runtime="acp" requires `agentId` unless `acp.defaultAgent` is configured; ACP harness ids follow acp.allowedAgents, not agents_list)'
: "Spawn an isolated sub-agent session",
subagents: "List, steer, or kill sub-agent runs for this requester session",
session_status:
"Show a /status-equivalent status card (usage + time + Reasoning/Verbose/Elevated); use for model-use questions (📊 session_status); optional per-session model override",
image: "Analyze an image with the configured image model",
image_generate: "Generate images with the configured image-generation model",
};
const toolOrder = [
"read",
"write",
"edit",
"apply_patch",
"grep",
"find",
"ls",
"exec",
"process",
"web_search",
"web_fetch",
"browser",
"canvas",
"nodes",
"cron",
"message",
"gateway",
"agents_list",
"sessions_list",
"sessions_history",
"sessions_send",
"subagents",
"session_status",
"image",
"image_generate",
];
const rawToolNames = (params.toolNames ?? []).map((tool) => tool.trim());
const canonicalToolNames = rawToolNames.filter(Boolean);
// Preserve caller casing while deduping tool names by lowercase.
const canonicalByNormalized = new Map<string, string>();
for (const name of canonicalToolNames) {
const normalized = normalizeLowercaseStringOrEmpty(name);
const normalized = name.toLowerCase();
if (!canonicalByNormalized.has(normalized)) {
canonicalByNormalized.set(normalized, name);
}
@@ -380,31 +328,38 @@ export function buildAgentSystemPrompt(params: {
const resolveToolName = (normalized: string) =>
canonicalByNormalized.get(normalized) ?? normalized;
const normalizedTools = canonicalToolNames.map((tool) => normalizeLowercaseStringOrEmpty(tool));
const normalizedTools = canonicalToolNames.map((tool) => tool.toLowerCase());
const availableTools = new Set(normalizedTools);
const hasSessionsSpawn = availableTools.has("sessions_spawn");
const hasUpdatePlanTool = availableTools.has("update_plan");
const acpHarnessSpawnAllowed = hasSessionsSpawn && acpSpawnRuntimeEnabled;
const externalToolSummaries = new Map<string, string>();
for (const [key, value] of Object.entries(params.toolSummaries ?? {})) {
const normalized = key.trim().toLowerCase();
if (!normalized || !value?.trim()) {
continue;
}
externalToolSummaries.set(normalized, value.trim());
}
const extraTools = Array.from(
new Set(normalizedTools.filter((tool) => !toolOrder.includes(tool))),
);
const enabledTools = toolOrder.filter((tool) => availableTools.has(tool));
const toolLines = enabledTools.map((tool) => {
const summary = coreToolSummaries[tool] ?? externalToolSummaries.get(tool);
const name = resolveToolName(tool);
return summary ? `- ${name}: ${summary}` : `- ${name}`;
});
for (const tool of extraTools.toSorted()) {
const summary = coreToolSummaries[tool] ?? externalToolSummaries.get(tool);
const name = resolveToolName(tool);
toolLines.push(summary ? `- ${name}: ${summary}` : `- ${name}`);
}
const hasGateway = availableTools.has("gateway");
const hasCronTool = availableTools.has("cron") || canonicalToolNames.length === 0;
const readToolName = resolveToolName("read");
const execToolName = resolveToolName("exec");
const processToolName = resolveToolName("process");
const extraSystemPrompt =
typeof params.extraSystemPrompt === "string"
? normalizeStructuredPromptSection(params.extraSystemPrompt)
: undefined;
const promptContribution = params.promptContribution;
const providerStablePrefix = normalizeProviderPromptBlock(promptContribution?.stablePrefix);
const providerDynamicSuffix = normalizeProviderPromptBlock(promptContribution?.dynamicSuffix);
const providerSectionOverrides = Object.fromEntries(
Object.entries(promptContribution?.sectionOverrides ?? {})
.map(([key, value]) => [
key,
normalizeProviderPromptBlock(typeof value === "string" ? value : undefined),
])
.filter(([, value]) => Boolean(value)),
) as Partial<Record<ProviderSystemPromptSectionId, string>>;
const extraSystemPrompt = params.extraSystemPrompt?.trim();
const ownerDisplay = params.ownerDisplay === "hash" ? "hash" : "raw";
const ownerLine = buildOwnerIdentityLine(
params.ownerNumbers ?? [],
@@ -425,20 +380,14 @@ export function buildAgentSystemPrompt(params: {
: undefined;
const reasoningLevel = params.reasoningLevel ?? "off";
const userTimezone = params.userTimezone?.trim();
const skillsPrompt =
typeof params.skillsPrompt === "string"
? normalizeStructuredPromptSection(params.skillsPrompt)
: undefined;
const heartbeatPrompt =
typeof params.heartbeatPrompt === "string"
? normalizeStructuredPromptSection(params.heartbeatPrompt)
: undefined;
const skillsPrompt = params.skillsPrompt?.trim();
const heartbeatPrompt = params.heartbeatPrompt?.trim();
const runtimeInfo = params.runtimeInfo;
const runtimeChannel = normalizeOptionalLowercaseString(runtimeInfo?.channel);
const runtimeCapabilities = runtimeInfo?.capabilities ?? [];
const runtimeCapabilitiesLower = new Set(
runtimeCapabilities.map((cap) => normalizeLowercaseStringOrEmpty(String(cap))).filter(Boolean),
);
const runtimeChannel = runtimeInfo?.channel?.trim().toLowerCase();
const runtimeCapabilities = (runtimeInfo?.capabilities ?? [])
.map((cap) => String(cap).trim())
.filter(Boolean);
const runtimeCapabilitiesLower = new Set(runtimeCapabilities.map((cap) => cap.toLowerCase()));
const inlineButtonsEnabled = runtimeCapabilitiesLower.has("inlinebuttons");
const messageChannelOptions = listDeliverableMessageChannels().join("|");
const promptMode = params.promptMode ?? "full";
@@ -469,7 +418,6 @@ export function buildAgentSystemPrompt(params: {
});
const memorySection = buildMemorySection({
isMinimal,
includeMemorySection: params.includeMemorySection,
availableTools,
citationsMode: params.memoryCitationsMode,
});
@@ -478,45 +426,41 @@ export function buildAgentSystemPrompt(params: {
isMinimal,
readToolName,
});
const workspaceNotes = (params.workspaceNotes ?? [])
.map((note) => normalizeStructuredPromptSection(note))
.filter(Boolean);
const modelAliasLines = (params.modelAliasLines ?? [])
.map((line) => normalizeStructuredPromptSection(line))
.filter(Boolean);
const workspaceNotes = (params.workspaceNotes ?? []).map((note) => note.trim()).filter(Boolean);
// For "none" mode, return just the basic identity line
if (promptMode === "none") {
return "You are a personal assistant operating inside OpenClaw.";
return "You are a personal assistant running inside OpenClaw.";
}
const lines = [
"You are a personal assistant operating inside OpenClaw.",
"You are a personal assistant running inside OpenClaw.",
"",
"## Tooling",
"Structured tool definitions are the source of truth for tool names, descriptions, and parameters.",
"Tool names are case-sensitive. Call tools exactly as listed in the structured tool definitions.",
"If a tool is present in the structured tool definitions, it is available unless a later tool call reports a policy/runtime restriction.",
"TOOLS.md does not control tool availability; it is user guidance for how to use external tools.",
...(hasCronTool
? [
`For follow-up at a future time (for example "check back in 10 minutes", reminders, run-later work, or recurring tasks), use cron instead of ${execToolName} sleep, yieldMs delays, or ${processToolName} polling.`,
`Use ${execToolName}/${processToolName} only for commands that start now and continue running in the background.`,
`For long-running work that starts now, start it once and rely on automatic completion wake when it is enabled and the command emits output or fails; otherwise use ${processToolName} to confirm completion, and use it for logs, status, input, or intervention.`,
"Do not emulate scheduling with sleep loops, timeout loops, or repeated polling.",
]
"Tool availability (filtered by policy):",
"Tool names are case-sensitive. Call tools exactly as listed.",
toolLines.length > 0
? toolLines.join("\n")
: [
`For long waits, avoid rapid poll loops: use ${execToolName} with enough yieldMs or ${processToolName}(action=poll, timeout=<ms>).`,
`For long-running work that starts now, start it once and rely on automatic completion wake when it is enabled and the command emits output or fails; otherwise use ${processToolName} to confirm completion, and use it for logs, status, input, or intervention.`,
]),
...(hasUpdatePlanTool
? [
"For non-trivial multi-step work, keep a short plan updated with `update_plan`.",
"Skip `update_plan` for simple tasks, obvious one-step fixes, or work you can finish in a few direct actions.",
"When you use `update_plan`, keep exactly one step `in_progress` until the work is done.",
"After calling `update_plan`, continue the work and do not repeat the full plan unless the user asks.",
]
: []),
"Pi lists the standard tools above. This runtime enables:",
"- grep: search file contents for patterns",
"- find: find files by glob pattern",
"- ls: list directory contents",
"- apply_patch: apply multi-file patches",
`- ${execToolName}: run shell commands (supports background via yieldMs/background)`,
`- ${processToolName}: manage background exec sessions`,
"- browser: control OpenClaw's dedicated browser",
"- canvas: present/eval/snapshot the Canvas",
"- nodes: list/describe/notify/camera/screen on paired nodes",
"- cron: manage cron jobs and wake events (use for reminders; when scheduling a reminder, write the systemEvent text as something that will read like a reminder when it fires, and mention that it is a reminder depending on the time gap between setting and firing; include recent context in reminder text if appropriate)",
"- sessions_list: list sessions",
"- sessions_history: fetch session history",
"- sessions_send: send to another session",
"- subagents: list/steer/kill sub-agent runs",
'- session_status: show usage/time/model state and answer "what model are we using?"',
].join("\n"),
"TOOLS.md does not control tool availability; it is user guidance for how to use external tools.",
`For long waits, avoid rapid poll loops: use ${execToolName} with enough yieldMs or ${processToolName}(action=poll, timeout=<ms>).`,
"If a task is more complex or takes longer, spawn a sub-agent. Completion is push-based: it will auto-announce when done.",
...(acpHarnessSpawnAllowed
? [
@@ -528,39 +472,16 @@ export function buildAgentSystemPrompt(params: {
: []),
"Do not poll `subagents list` / `sessions_list` in a loop; only check status on-demand (for intervention, debugging, or when explicitly asked).",
"",
...buildOverridablePromptSection({
override: providerSectionOverrides.interaction_style,
fallback: [],
}),
...buildOverridablePromptSection({
override: providerSectionOverrides.tool_call_style,
fallback: [
"## Tool Call Style",
"Default: do not narrate routine, low-risk tool calls (just call the tool).",
"Narrate only when it helps: multi-step work, complex/challenging problems, sensitive actions (e.g., deletions), or when the user explicitly asks.",
"Keep narration brief and value-dense; avoid repeating obvious steps.",
"Use plain human language for narration unless in a technical context.",
"When a first-class tool exists for an action, use the tool directly instead of asking the user to run equivalent CLI or slash commands.",
buildExecApprovalPromptGuidance({
runtimeChannel: params.runtimeInfo?.channel,
inlineButtonsEnabled,
}),
"Never execute /approve through exec or any other shell/tool path; /approve is a user-facing approval command, not a shell command.",
"Treat allow-once as single-command only: if another elevated command needs approval, request a fresh /approve and do not claim prior approval covered it.",
"When approvals are required, preserve and show the full command/script exactly as provided (including chained operators like &&, ||, |, ;, or multiline shells) so the user can approve what will actually run.",
"",
],
}),
...buildOverridablePromptSection({
override: providerSectionOverrides.execution_bias,
fallback: buildExecutionBiasSection({
isMinimal,
}),
}),
...buildOverridablePromptSection({
override: providerStablePrefix,
fallback: [],
}),
"## Tool Call Style",
"Default: do not narrate routine, low-risk tool calls (just call the tool).",
"Narrate only when it helps: multi-step work, complex/challenging problems, sensitive actions (e.g., deletions), or when the user explicitly asks.",
"Keep narration brief and value-dense; avoid repeating obvious steps.",
"Use plain human language for narration unless in a technical context.",
"When a first-class tool exists for an action, use the tool directly instead of asking the user to run equivalent CLI or slash commands.",
"When exec returns approval-pending, include the concrete /approve command from tool output (with allow-once|allow-always|deny) and do not ask for a different or rotated code.",
"Treat allow-once as single-command only: if another elevated command needs approval, request a fresh /approve and do not claim prior approval covered it.",
"When approvals are required, preserve and show the full command/script exactly as provided (including chained operators like &&, ||, |, ;, or multiline shells) so the user can approve what will actually run.",
"",
...safetySection,
"## OpenClaw CLI Quick Reference",
"OpenClaw is controlled via subcommands. Do not invent commands.",
@@ -580,19 +501,23 @@ export function buildAgentSystemPrompt(params: {
"Get Updates (self-update) is ONLY allowed when the user explicitly asks for it.",
"Do not run config.apply or update.run unless the user explicitly requests an update or config change; if it's not explicit, ask first.",
"Use config.schema.lookup with a specific dot path to inspect only the relevant config subtree before making config changes or answering config-field questions; avoid guessing field names/types.",
"Actions: config.schema.lookup, config.get, config.apply (validate + write full config, then hot-reload or restart as needed), config.patch (partial update, merges with existing), update.run (update deps or git, then restart).",
"Actions: config.schema.lookup, config.get, config.apply (validate + write full config, then restart), config.patch (partial update, merges with existing), update.run (update deps or git, then restart).",
"After restart, OpenClaw pings the last active session automatically.",
].join("\n")
: "",
hasGateway && !isMinimal ? "" : "",
"",
// Skip model aliases for subagent/none modes
modelAliasLines.length > 0 && !isMinimal ? "## Model Aliases" : "",
modelAliasLines.length > 0 && !isMinimal
params.modelAliasLines && params.modelAliasLines.length > 0 && !isMinimal
? "## Model Aliases"
: "",
params.modelAliasLines && params.modelAliasLines.length > 0 && !isMinimal
? "Prefer aliases when specifying model overrides; full provider/model is also accepted."
: "",
modelAliasLines.length > 0 && !isMinimal ? modelAliasLines.join("\n") : "",
modelAliasLines.length > 0 && !isMinimal ? "" : "",
params.modelAliasLines && params.modelAliasLines.length > 0 && !isMinimal
? params.modelAliasLines.join("\n")
: "",
params.modelAliasLines && params.modelAliasLines.length > 0 && !isMinimal ? "" : "",
userTimezone
? "If you need the current date, time, or day of week, run session_status (📊 session_status)."
: "",
@@ -657,7 +582,12 @@ export function buildAgentSystemPrompt(params: {
"## Workspace Files (injected)",
"These user-editable files are loaded by OpenClaw and included below in Project Context.",
"",
...buildReplyTagsSection(isMinimal),
...buildAssistantOutputDirectivesSection(isMinimal),
...buildWebchatCanvasSection({
isMinimal,
runtimeChannel,
canvasRootDir: params.runtimeInfo?.canvasRootDir,
}),
...buildMessagingSection({
isMinimal,
availableTools,
@@ -669,6 +599,12 @@ export function buildAgentSystemPrompt(params: {
...buildVoiceSection({ isMinimal, ttsHint: params.ttsHint }),
];
if (extraSystemPrompt) {
// Use "Subagent Context" header for minimal mode (subagents), otherwise "Group Chat Context"
const contextHeader =
promptMode === "minimal" ? "## Subagent Context" : "## Group Chat Context";
lines.push(contextHeader, extraSystemPrompt, "");
}
if (params.reactionGuidance) {
const { level, channel } = params.reactionGuidance;
const guidanceText =
@@ -700,27 +636,35 @@ export function buildAgentSystemPrompt(params: {
const validContextFiles = contextFiles.filter(
(file) => typeof file.path === "string" && file.path.trim().length > 0,
);
const orderedContextFiles = sortContextFilesForPrompt(validContextFiles);
const stableContextFiles = orderedContextFiles.filter((file) => !isDynamicContextFile(file.path));
const dynamicContextFiles = orderedContextFiles.filter((file) => isDynamicContextFile(file.path));
lines.push(
...buildProjectContextSection({
files: stableContextFiles,
heading: "# Project Context",
dynamic: false,
}),
);
if (validContextFiles.length > 0) {
lines.push("# Project Context", "");
if (validContextFiles.length > 0) {
const hasSoulFile = validContextFiles.some((file) => {
const normalizedPath = file.path.trim().replace(/\\/g, "/");
const baseName = normalizedPath.split("/").pop() ?? normalizedPath;
return baseName.toLowerCase() === "soul.md";
});
lines.push("The following project context files have been loaded:");
if (hasSoulFile) {
lines.push(
"If SOUL.md is present, embody its persona and tone. Avoid stiff, generic replies; follow its guidance unless higher-priority instructions override it.",
);
}
lines.push("");
}
for (const file of validContextFiles) {
lines.push(`## ${file.path}`, "", file.content, "");
}
}
// Skip silent replies for subagent/none modes
if (!isMinimal) {
lines.push(
"## Silent Replies",
`Use ${SILENT_REPLY_TOKEN} ONLY when no user-visible reply is required.`,
`When you have nothing to say, respond with ONLY: ${SILENT_REPLY_TOKEN}`,
"",
"⚠️ Rules:",
"- Valid cases: silent housekeeping, deliberate no-op ambient wakeups, or after a messaging tool already delivered the user-visible reply.",
"- Never use it to avoid doing requested work or to end an actionable turn early.",
"- It must be your ENTIRE message - nothing else",
"- It must be your ENTIRE message — nothing else",
`- Never append it to an actual response (never include "${SILENT_REPLY_TOKEN}" in real replies)`,
"- Never wrap it in markdown or code blocks",
"",
@@ -731,29 +675,6 @@ export function buildAgentSystemPrompt(params: {
);
}
// Keep large stable prompt context above this seam so Anthropic-family
// transports can reuse it across labs and turns. Dynamic group/session
// additions and volatile project context below it are the primary cache invalidators.
lines.push(SYSTEM_PROMPT_CACHE_BOUNDARY);
lines.push(
...buildProjectContextSection({
files: dynamicContextFiles,
heading: stableContextFiles.length > 0 ? "# Dynamic Project Context" : "# Project Context",
dynamic: true,
}),
);
if (extraSystemPrompt) {
// Use "Subagent Context" header for minimal mode (subagents), otherwise "Group Chat Context"
const contextHeader =
promptMode === "minimal" ? "## Subagent Context" : "## Group Chat Context";
lines.push(contextHeader, extraSystemPrompt, "");
}
if (providerDynamicSuffix) {
lines.push(providerDynamicSuffix, "");
}
// Skip heartbeats for subagent/none modes
if (!isMinimal && heartbeatPrompt) {
lines.push(
@@ -792,7 +713,6 @@ export function buildRuntimeLine(
runtimeCapabilities: string[] = [],
defaultThinkLevel?: ThinkLevel,
): string {
const normalizedRuntimeCapabilities = normalizePromptCapabilityIds(runtimeCapabilities);
return `Runtime: ${[
runtimeInfo?.agentId ? `agent=${runtimeInfo.agentId}` : "",
runtimeInfo?.host ? `host=${runtimeInfo.host}` : "",
@@ -808,11 +728,7 @@ export function buildRuntimeLine(
runtimeInfo?.shell ? `shell=${runtimeInfo.shell}` : "",
runtimeChannel ? `channel=${runtimeChannel}` : "",
runtimeChannel
? `capabilities=${
normalizedRuntimeCapabilities.length > 0
? normalizedRuntimeCapabilities.join(",")
: "none"
}`
? `capabilities=${runtimeCapabilities.length > 0 ? runtimeCapabilities.join(",") : "none"}`
: "",
`thinking=${defaultThinkLevel ?? "off"}`,
]

245
src/chat/canvas-render.ts Normal file
View File

@@ -0,0 +1,245 @@
import { parseFenceSpans } from "../markdown/fences.js";
export type CanvasSurface = "assistant_message";
export type CanvasPreview = {
kind: "canvas";
surface: CanvasSurface;
render: "url";
title?: string;
preferredHeight?: number;
url?: string;
viewId?: string;
className?: string;
style?: string;
};
function tryParseJsonRecord(value: string | undefined): Record<string, unknown> | undefined {
if (typeof value !== "string") {
return undefined;
}
try {
const parsed = JSON.parse(value);
return parsed && typeof parsed === "object" && !Array.isArray(parsed)
? (parsed as Record<string, unknown>)
: undefined;
} catch {
return undefined;
}
}
function getRecordStringField(
record: Record<string, unknown> | undefined,
key: string,
): string | undefined {
const value = record?.[key];
return typeof value === "string" && value.trim() ? value : undefined;
}
function getRecordNumberField(
record: Record<string, unknown> | undefined,
key: string,
): number | undefined {
const value = record?.[key];
return typeof value === "number" && Number.isFinite(value) ? value : undefined;
}
function getNestedRecord(
record: Record<string, unknown> | undefined,
key: string,
): Record<string, unknown> | undefined {
const value = record?.[key];
return value && typeof value === "object" && !Array.isArray(value)
? (value as Record<string, unknown>)
: undefined;
}
function normalizeSurface(value: string | undefined): CanvasSurface | undefined {
return value === "assistant_message" ? value : undefined;
}
function normalizePreferredHeight(value: number | undefined): number | undefined {
return typeof value === "number" && Number.isFinite(value) && value >= 160
? Math.min(Math.trunc(value), 1200)
: undefined;
}
function coerceCanvasPreview(
record: Record<string, unknown> | undefined,
): CanvasPreview | undefined {
if (!record) {
return undefined;
}
const kind = getRecordStringField(record, "kind")?.trim().toLowerCase();
if (kind !== "canvas") {
return undefined;
}
const presentation = getNestedRecord(record, "presentation");
const view = getNestedRecord(record, "view");
const source = getNestedRecord(record, "source");
const requestedSurface =
getRecordStringField(presentation, "target") ?? getRecordStringField(record, "target");
const surface = requestedSurface ? normalizeSurface(requestedSurface) : "assistant_message";
if (!surface) {
return undefined;
}
const title = getRecordStringField(presentation, "title") ?? getRecordStringField(view, "title");
const preferredHeight = normalizePreferredHeight(
getRecordNumberField(presentation, "preferred_height") ??
getRecordNumberField(presentation, "preferredHeight") ??
getRecordNumberField(view, "preferred_height") ??
getRecordNumberField(view, "preferredHeight"),
);
const className =
getRecordStringField(presentation, "class_name") ??
getRecordStringField(presentation, "className");
const style = getRecordStringField(presentation, "style");
const viewUrl = getRecordStringField(view, "url") ?? getRecordStringField(view, "entryUrl");
const viewId = getRecordStringField(view, "id") ?? getRecordStringField(view, "docId");
if (viewUrl) {
return {
kind: "canvas",
surface,
render: "url",
url: viewUrl,
...(viewId ? { viewId } : {}),
...(title ? { title } : {}),
...(preferredHeight ? { preferredHeight } : {}),
...(className ? { className } : {}),
...(style ? { style } : {}),
};
}
const sourceType = getRecordStringField(source, "type")?.trim().toLowerCase();
if (sourceType === "url") {
const url = getRecordStringField(source, "url");
if (!url) {
return undefined;
}
return {
kind: "canvas",
surface,
render: "url",
url,
...(title ? { title } : {}),
...(preferredHeight ? { preferredHeight } : {}),
...(className ? { className } : {}),
...(style ? { style } : {}),
};
}
return undefined;
}
function parseCanvasAttributes(raw: string): Record<string, string> {
const attrs: Record<string, string> = {};
const re = /([A-Za-z_][A-Za-z0-9_-]*)\s*=\s*(?:"([^"]*)"|'([^']*)')/g;
let match: RegExpExecArray | null;
while ((match = re.exec(raw))) {
const key = match[1]?.trim().toLowerCase();
const value = (match[2] ?? match[3] ?? "").trim();
if (key && value) {
attrs[key] = value;
}
}
return attrs;
}
function defaultCanvasEntryUrl(ref: string): string {
const encoded = encodeURIComponent(ref.trim());
return `/__openclaw__/canvas/documents/${encoded}/index.html`;
}
function previewFromShortcode(attrs: Record<string, string>): CanvasPreview | undefined {
if (attrs.target && normalizeSurface(attrs.target) !== "assistant_message") {
return undefined;
}
const surface = "assistant_message";
const title = attrs.title?.trim() || undefined;
const preferredHeight =
attrs.height && Number.isFinite(Number(attrs.height))
? normalizePreferredHeight(Number(attrs.height))
: undefined;
const className = attrs.class?.trim() || attrs.class_name?.trim() || undefined;
const style = attrs.style?.trim() || undefined;
const ref = attrs.ref?.trim();
const url = attrs.url?.trim();
if (url || ref) {
return {
kind: "canvas",
surface,
render: "url",
url: url ?? defaultCanvasEntryUrl(ref),
...(ref ? { viewId: ref } : {}),
...(title ? { title } : {}),
...(preferredHeight ? { preferredHeight } : {}),
...(className ? { className } : {}),
...(style ? { style } : {}),
};
}
return undefined;
}
export function extractCanvasFromText(
outputText: string | undefined,
_toolName?: string,
): CanvasPreview | undefined {
const parsed = tryParseJsonRecord(outputText);
return coerceCanvasPreview(parsed);
}
export function extractCanvasShortcodes(text: string | undefined): {
text: string;
previews: CanvasPreview[];
} {
if (!text?.trim() || !text.includes("[canvas")) {
return { text: text ?? "", previews: [] };
}
const fenceSpans = parseFenceSpans(text);
const matches: Array<{
start: number;
end: number;
attrs: Record<string, string>;
body?: string;
}> = [];
const blockRe = /\[canvas\s+([^\]]*?)\]([\s\S]*?)\[\/canvas\]/gi;
const selfClosingRe = /\[canvas\s+([^\]]*?)\/\]/gi;
for (const re of [blockRe, selfClosingRe]) {
let match: RegExpExecArray | null;
while ((match = re.exec(text))) {
const start = match.index ?? 0;
if (fenceSpans.some((span) => start >= span.start && start < span.end)) {
continue;
}
matches.push({
start,
end: start + match[0].length,
attrs: parseCanvasAttributes(match[1] ?? ""),
...(match[2] !== undefined ? { body: match[2] } : {}),
});
}
}
if (matches.length === 0) {
return { text, previews: [] };
}
matches.sort((a, b) => a.start - b.start);
const previews: CanvasPreview[] = [];
let cursor = 0;
let stripped = "";
for (const match of matches) {
if (match.start < cursor) {
continue;
}
stripped += text.slice(cursor, match.start);
const preview = previewFromShortcode(match.attrs);
if (!preview) {
stripped += text.slice(match.start, match.end);
} else {
previews.push(preview);
}
cursor = match.end;
}
stripped += text.slice(cursor);
return {
text: stripped.replace(/\n{3,}/g, "\n\n").trim(),
previews,
};
}

View File

@@ -1,9 +1,7 @@
import { normalizeLowercaseStringOrEmpty } from "../shared/string-coerce.js";
export type ToolContentBlock = Record<string, unknown>;
export function normalizeToolContentType(value: unknown): string {
return normalizeLowercaseStringOrEmpty(value);
return typeof value === "string" ? value.toLowerCase() : "";
}
export function isToolCallContentType(value: unknown): boolean {

View File

@@ -376,6 +376,8 @@ export const FIELD_HELP: Record<string, string> = {
"Optional URL prefix where the Control UI is served (e.g. /openclaw).",
"gateway.controlUi.root":
"Optional filesystem root for Control UI assets (defaults to dist/control-ui).",
"gateway.controlUi.embedSandbox":
'Iframe sandbox policy for hosted Control UI embeds. "powerful" keeps `allow-scripts allow-same-origin` for interactive same-origin embeds; "isolated" uses `allow-scripts` only to reduce iframe privileges at the cost of stricter runtime isolation.',
"gateway.controlUi.allowedOrigins":
'Allowed browser origins for Control UI/WebChat websocket connections (full origins only, e.g. https://control.example.com). Required for non-loopback Control UI deployments unless dangerous Host-header fallback is explicitly enabled. Setting ["*"] means allow any browser origin and should be avoided outside tightly controlled local testing.',
"gateway.controlUi.dangerouslyAllowHostHeaderOriginFallback":
@@ -745,7 +747,7 @@ export const FIELD_HELP: Record<string, string> = {
"models.providers.*.authHeader":
"When true, credentials are sent via the HTTP Authorization header even if alternate auth is possible. Use this only when your provider or proxy explicitly requires Authorization forwarding.",
"models.providers.*.request":
"Optional request overrides for model-provider requests, including extra headers, auth overrides, proxy routing, and TLS client settings. Use these only when your upstream or enterprise network path requires transport customization.",
"Optional request overrides for model-provider requests, including extra headers, auth overrides, proxy routing, TLS client settings, and optional allowPrivateNetwork for trusted self-hosted endpoints. Use these only when your upstream or enterprise network path requires transport customization.",
"models.providers.*.request.headers":
"Extra headers merged into provider requests after default attribution and auth resolution.",
"models.providers.*.request.auth":
@@ -794,6 +796,8 @@ export const FIELD_HELP: Record<string, string> = {
"Optional SNI/server-name override used when establishing upstream TLS.",
"models.providers.*.request.tls.insecureSkipVerify":
"Skips upstream TLS certificate verification. Use only for controlled development environments.",
"models.providers.*.request.allowPrivateNetwork":
"When true, allow HTTPS to the model base URL when DNS resolves to private, CGNAT, or similar ranges, via the provider HTTP fetch guard (fetchWithSsrFGuard). OpenAI Responses WebSocket reuses request for headers/TLS but does not use that fetch SSRF path. Use only for operator-controlled self-hosted OpenAI-compatible endpoints (LAN, overlay, split DNS). Default is false.",
"models.providers.*.models":
"Declared model list for a provider including identifiers, metadata, and optional compatibility/cost hints. Keep IDs exact to provider catalog values so selection and fallback resolve correctly.",
auth: "Authentication profile root used for multi-profile provider credentials and cooldown-based failover ordering. Keep profiles minimal and explicit so automatic failover behavior stays auditable.",

View File

@@ -259,6 +259,7 @@ export const FIELD_LABELS: Record<string, string> = {
"Web Fetch Allow RFC 2544 Benchmark Range",
"gateway.controlUi.basePath": "Control UI Base Path",
"gateway.controlUi.root": "Control UI Assets Root",
"gateway.controlUi.embedSandbox": "Control UI Embed Sandbox Mode",
"gateway.controlUi.allowedOrigins": "Control UI Allowed Origins",
"gateway.controlUi.dangerouslyAllowHostHeaderOriginFallback":
"Dangerously Allow Host-Header Origin Fallback",
@@ -482,6 +483,7 @@ export const FIELD_LABELS: Record<string, string> = {
"models.providers.*.request.tls.passphrase": "Model Provider Request TLS Passphrase",
"models.providers.*.request.tls.serverName": "Model Provider Request TLS Server Name",
"models.providers.*.request.tls.insecureSkipVerify": "Model Provider Request TLS Skip Verify",
"models.providers.*.request.allowPrivateNetwork": "Model Provider Request Allow Private Network",
"models.providers.*.models": "Model Provider Model List",
"auth.cooldowns.billingBackoffHours": "Billing Backoff (hours)",
"auth.cooldowns.billingBackoffHoursByProvider": "Billing Backoff Overrides",

View File

@@ -43,6 +43,7 @@ const TAG_OVERRIDES: Record<string, ConfigTag[]> = {
"gateway.auth.token": ["security", "auth", "access", "network"],
"gateway.auth.password": ["security", "auth", "access", "network"],
"gateway.push.apns.relay.baseUrl": ["network", "advanced"],
"gateway.controlUi.embedSandbox": ["security", "access", "network", "advanced"],
"gateway.controlUi.dangerouslyAllowHostHeaderOriginFallback": [
"security",
"access",

View File

@@ -85,6 +85,11 @@ export type GatewayControlUiConfig = {
basePath?: string;
/** Optional filesystem root for Control UI assets (defaults to dist/control-ui). */
root?: string;
/**
* Embed sandbox mode for hosted Control UI previews.
* "powerful" keeps same-origin iframe behavior; "isolated" removes same-origin.
*/
embedSandbox?: "powerful" | "isolated";
/** Allowed browser origins for Control UI/WebChat websocket connections. */
allowedOrigins?: string[];
/**

View File

@@ -680,6 +680,7 @@ export const OpenClawSchema = z
enabled: z.boolean().optional(),
basePath: z.string().optional(),
root: z.string().optional(),
embedSandbox: z.union([z.literal("powerful"), z.literal("isolated")]).optional(),
allowedOrigins: z.array(z.string()).optional(),
dangerouslyAllowHostHeaderOriginFallback: z.boolean().optional(),
allowInsecureAuth: z.boolean().optional(),

View File

@@ -0,0 +1,235 @@
import { mkdtemp, mkdir, writeFile, readFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import {
buildCanvasDocumentEntryUrl,
createCanvasDocument,
resolveCanvasDocumentAssets,
resolveCanvasDocumentDir,
resolveCanvasHttpPathToLocalPath,
} from "./canvas-documents.js";
const tempDirs: string[] = [];
afterEach(async () => {
await Promise.all(
tempDirs.splice(0).map(async (dir) => {
await import("node:fs/promises").then((fs) => fs.rm(dir, { recursive: true, force: true }));
}),
);
});
describe("canvas documents", () => {
it("builds entry urls for materialized path documents under managed storage", async () => {
const stateDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-"));
tempDirs.push(stateDir);
const workspaceDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-workspace-"));
tempDirs.push(workspaceDir);
await mkdir(path.join(workspaceDir, "player"), { recursive: true });
await writeFile(path.join(workspaceDir, "player/index.html"), "<div>ok</div>", "utf8");
const document = await createCanvasDocument(
{
kind: "html_bundle",
entrypoint: {
type: "path",
value: "player/index.html",
},
},
{ stateDir, workspaceDir },
);
expect(document.entryUrl).toContain("/__openclaw__/canvas/documents/");
expect(document.localEntrypoint).toBe("index.html");
expect(resolveCanvasDocumentDir(document.id, { stateDir })).toContain(stateDir);
});
it("normalizes nested local entrypoint urls", () => {
const url = buildCanvasDocumentEntryUrl("cv_example", "collection.media/index.html");
expect(url).toBe("/__openclaw__/canvas/documents/cv_example/collection.media/index.html");
});
it("materializes inline html bundles as index documents", async () => {
const stateDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-"));
tempDirs.push(stateDir);
const document = await createCanvasDocument(
{
kind: "html_bundle",
title: "Preview",
entrypoint: {
type: "html",
value:
"<!doctype html><html><head><style>.demo{color:red}</style></head><body><div class='demo'>Front</div></body></html>",
},
},
{ stateDir },
);
const indexHtml = await import("node:fs/promises").then((fs) =>
fs.readFile(
path.join(resolveCanvasDocumentDir(document.id, { stateDir }), "index.html"),
"utf8",
),
);
expect(indexHtml).toContain("<div class='demo'>Front</div>");
expect(indexHtml).toContain("<style>.demo{color:red}</style>");
expect(document.title).toBe("Preview");
expect(document.entryUrl).toBe(`/__openclaw__/canvas/documents/${document.id}/index.html`);
});
it("reuses a supplied stable document id by replacing the prior materialized view", async () => {
const stateDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-"));
tempDirs.push(stateDir);
const first = await createCanvasDocument(
{
id: "status-card",
kind: "html_bundle",
entrypoint: { type: "html", value: "<div>first</div>" },
},
{ stateDir },
);
const second = await createCanvasDocument(
{
id: "status-card",
kind: "html_bundle",
entrypoint: { type: "html", value: "<div>second</div>" },
},
{ stateDir },
);
expect(first.id).toBe("status-card");
expect(second.id).toBe("status-card");
const indexHtml = await import("node:fs/promises").then((fs) =>
fs.readFile(
path.join(resolveCanvasDocumentDir(second.id, { stateDir }), "index.html"),
"utf8",
),
);
expect(indexHtml).toContain("second");
expect(indexHtml).not.toContain("first");
});
it("exposes stable managed asset urls for copied canvas assets", async () => {
const stateDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-"));
tempDirs.push(stateDir);
const workspaceDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-workspace-"));
tempDirs.push(workspaceDir);
await mkdir(path.join(workspaceDir, "collection.media"), { recursive: true });
await writeFile(path.join(workspaceDir, "collection.media/audio.mp3"), "audio", "utf8");
const document = await createCanvasDocument(
{
kind: "html_bundle",
entrypoint: {
type: "html",
value:
'<audio controls><source src="collection.media/audio.mp3" type="audio/mpeg" /></audio>',
},
assets: [
{
logicalPath: "collection.media/audio.mp3",
sourcePath: "collection.media/audio.mp3",
contentType: "audio/mpeg",
},
],
},
{ stateDir, workspaceDir },
);
expect(resolveCanvasDocumentAssets(document, { stateDir })).toEqual([
{
logicalPath: "collection.media/audio.mp3",
contentType: "audio/mpeg",
localPath: path.join(
resolveCanvasDocumentDir(document.id, { stateDir }),
"collection.media/audio.mp3",
),
url: `/__openclaw__/canvas/documents/${document.id}/collection.media/audio.mp3`,
},
]);
expect(
resolveCanvasDocumentAssets(document, {
baseUrl: "http://127.0.0.1:19003",
stateDir,
}),
).toEqual([
{
logicalPath: "collection.media/audio.mp3",
contentType: "audio/mpeg",
localPath: path.join(
resolveCanvasDocumentDir(document.id, { stateDir }),
"collection.media/audio.mp3",
),
url: `http://127.0.0.1:19003/__openclaw__/canvas/documents/${document.id}/collection.media/audio.mp3`,
},
]);
});
it("wraps local pdf documents in an index viewer page", async () => {
const stateDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-"));
tempDirs.push(stateDir);
const workspaceDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-workspace-"));
tempDirs.push(workspaceDir);
await writeFile(path.join(workspaceDir, "demo.pdf"), "%PDF-1.4", "utf8");
const document = await createCanvasDocument(
{
kind: "document",
entrypoint: {
type: "path",
value: "demo.pdf",
},
},
{ stateDir, workspaceDir },
);
expect(document.entryUrl).toBe(`/__openclaw__/canvas/documents/${document.id}/index.html`);
const indexHtml = await readFile(
path.join(resolveCanvasDocumentDir(document.id, { stateDir }), "index.html"),
"utf8",
);
expect(indexHtml).toContain('type="application/pdf"');
expect(indexHtml).toContain('data="demo.pdf"');
});
it("wraps remote pdf urls in an index viewer page", async () => {
const stateDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-"));
tempDirs.push(stateDir);
const document = await createCanvasDocument(
{
kind: "document",
entrypoint: {
type: "url",
value: "https://example.com/demo.pdf",
},
},
{ stateDir },
);
expect(document.entryUrl).toBe(`/__openclaw__/canvas/documents/${document.id}/index.html`);
const indexHtml = await readFile(
path.join(resolveCanvasDocumentDir(document.id, { stateDir }), "index.html"),
"utf8",
);
expect(indexHtml).toContain('type="application/pdf"');
expect(indexHtml).toContain('data="https://example.com/demo.pdf"');
});
it("rejects traversal-style document ids in hosted canvas paths", async () => {
const stateDir = await mkdtemp(path.join(tmpdir(), "openclaw-canvas-documents-"));
tempDirs.push(stateDir);
expect(
resolveCanvasHttpPathToLocalPath(
"/__openclaw__/canvas/documents/../collection.media/index.html",
{ stateDir },
),
).toBeNull();
});
});

View File

@@ -0,0 +1,343 @@
import { randomUUID } from "node:crypto";
import fs from "node:fs/promises";
import path from "node:path";
import { CANVAS_HOST_PATH } from "../canvas-host/a2ui.js";
import { resolveStateDir } from "../config/paths.js";
import { resolveUserPath } from "../utils.js";
export type CanvasDocumentKind = "html_bundle" | "url_embed" | "document" | "image" | "video_asset";
export type CanvasDocumentAsset = {
logicalPath: string;
sourcePath: string;
contentType?: string;
};
export type CanvasDocumentEntrypoint =
| { type: "html"; value: string }
| { type: "path"; value: string }
| { type: "url"; value: string };
export type CanvasDocumentCreateInput = {
id?: string;
kind: CanvasDocumentKind;
title?: string;
preferredHeight?: number;
entrypoint?: CanvasDocumentEntrypoint;
assets?: CanvasDocumentAsset[];
surface?: "assistant_message" | "tool_card" | "sidebar";
};
export type CanvasDocumentManifest = {
id: string;
kind: CanvasDocumentKind;
title?: string;
preferredHeight?: number;
createdAt: string;
entryUrl: string;
localEntrypoint?: string;
externalUrl?: string;
surface?: "assistant_message" | "tool_card" | "sidebar";
assets: Array<{
logicalPath: string;
contentType?: string;
}>;
};
export type CanvasDocumentResolvedAsset = {
logicalPath: string;
contentType?: string;
url: string;
localPath: string;
};
const CANVAS_DOCUMENTS_DIR_NAME = "documents";
function isPdfPathLike(value: string): boolean {
return /\.pdf(?:[?#].*)?$/i.test(value.trim());
}
function buildPdfWrapper(url: string): string {
const escaped = escapeHtml(url);
return `<!doctype html><html><body style="margin:0;background:#e5e7eb;"><object data="${escaped}" type="application/pdf" style="width:100%;height:100vh;border:0;"><iframe src="${escaped}" style="width:100%;height:100vh;border:0;"></iframe><p style="padding:16px;font:14px system-ui,sans-serif;">Unable to render PDF preview. <a href="${escaped}" target="_blank" rel="noopener noreferrer">Open PDF</a>.</p></object></body></html>`;
}
function escapeHtml(value: string): string {
return value
.replaceAll("&", "&amp;")
.replaceAll("<", "&lt;")
.replaceAll(">", "&gt;")
.replaceAll('"', "&quot;")
.replaceAll("'", "&#39;");
}
function normalizeLogicalPath(value: string): string {
const normalized = value.replaceAll("\\", "/").replace(/^\/+/, "");
const parts = normalized.split("/").filter(Boolean);
if (parts.length === 0 || parts.some((part) => part === "." || part === "..")) {
throw new Error("canvas document logicalPath invalid");
}
return parts.join("/");
}
function canvasDocumentId(): string {
return `cv_${randomUUID().replaceAll("-", "")}`;
}
function normalizeCanvasDocumentId(value: string): string {
const normalized = value.trim();
if (
!normalized ||
normalized === "." ||
normalized === ".." ||
!/^[A-Za-z0-9._-]+$/.test(normalized)
) {
throw new Error("canvas document id invalid");
}
return normalized;
}
export function resolveCanvasRootDir(rootDir?: string, stateDir = resolveStateDir()): string {
const resolved = rootDir?.trim() ? resolveUserPath(rootDir) : path.join(stateDir, "canvas");
return path.resolve(resolved);
}
export function resolveCanvasDocumentsDir(rootDir?: string, stateDir = resolveStateDir()): string {
return path.join(resolveCanvasRootDir(rootDir, stateDir), CANVAS_DOCUMENTS_DIR_NAME);
}
export function resolveCanvasDocumentDir(
documentId: string,
options?: { rootDir?: string; stateDir?: string },
): string {
return path.join(resolveCanvasDocumentsDir(options?.rootDir, options?.stateDir), documentId);
}
export function buildCanvasDocumentEntryUrl(documentId: string, entrypoint: string): string {
const normalizedEntrypoint = normalizeLogicalPath(entrypoint);
return `${CANVAS_HOST_PATH}/${CANVAS_DOCUMENTS_DIR_NAME}/${encodeURIComponent(documentId)}/${normalizedEntrypoint}`;
}
export function buildCanvasDocumentAssetUrl(documentId: string, logicalPath: string): string {
return buildCanvasDocumentEntryUrl(documentId, logicalPath);
}
export function resolveCanvasHttpPathToLocalPath(
requestPath: string,
options?: { rootDir?: string; stateDir?: string },
): string | null {
const trimmed = requestPath.trim();
const prefix = `${CANVAS_HOST_PATH}/${CANVAS_DOCUMENTS_DIR_NAME}/`;
if (!trimmed.startsWith(prefix)) {
return null;
}
const pathWithoutQuery = trimmed.replace(/[?#].*$/, "");
const relative = pathWithoutQuery.slice(prefix.length);
const segments = relative
.split("/")
.map((segment) => {
try {
return decodeURIComponent(segment);
} catch {
return segment;
}
})
.filter(Boolean);
if (segments.length < 2) {
return null;
}
const [rawDocumentId, ...entrySegments] = segments;
try {
const documentId = normalizeCanvasDocumentId(rawDocumentId);
const normalizedEntrypoint = normalizeLogicalPath(entrySegments.join("/"));
const documentsDir = path.resolve(
resolveCanvasDocumentsDir(options?.rootDir, options?.stateDir),
);
const candidatePath = path.resolve(
resolveCanvasDocumentDir(documentId, options),
normalizedEntrypoint,
);
if (
!(candidatePath === documentsDir || candidatePath.startsWith(`${documentsDir}${path.sep}`))
) {
return null;
}
return candidatePath;
} catch {
return null;
}
}
async function writeManifest(rootDir: string, manifest: CanvasDocumentManifest): Promise<void> {
await fs.writeFile(
path.join(rootDir, "manifest.json"),
`${JSON.stringify(manifest, null, 2)}\n`,
"utf8",
);
}
async function copyAssets(
rootDir: string,
assets: CanvasDocumentAsset[] | undefined,
workspaceDir: string,
): Promise<CanvasDocumentManifest["assets"]> {
const copied: CanvasDocumentManifest["assets"] = [];
for (const asset of assets ?? []) {
const logicalPath = normalizeLogicalPath(asset.logicalPath);
const sourcePath = asset.sourcePath.startsWith("~")
? resolveUserPath(asset.sourcePath)
: path.isAbsolute(asset.sourcePath)
? path.resolve(asset.sourcePath)
: path.resolve(workspaceDir, asset.sourcePath);
const destination = path.join(rootDir, logicalPath);
await fs.mkdir(path.dirname(destination), { recursive: true });
await fs.copyFile(sourcePath, destination);
copied.push({
logicalPath,
...(asset.contentType ? { contentType: asset.contentType } : {}),
});
}
return copied;
}
async function materializeEntrypoint(
rootDir: string,
input: CanvasDocumentCreateInput,
workspaceDir: string,
): Promise<Pick<CanvasDocumentManifest, "entryUrl" | "localEntrypoint" | "externalUrl">> {
const entrypoint = input.entrypoint;
if (!entrypoint) {
throw new Error("canvas document entrypoint required");
}
if (entrypoint.type === "html") {
const fileName = "index.html";
await fs.writeFile(path.join(rootDir, fileName), entrypoint.value, "utf8");
return {
localEntrypoint: fileName,
entryUrl: buildCanvasDocumentEntryUrl(path.basename(rootDir), fileName),
};
}
if (entrypoint.type === "url") {
if (input.kind === "document" && isPdfPathLike(entrypoint.value)) {
const fileName = "index.html";
await fs.writeFile(path.join(rootDir, fileName), buildPdfWrapper(entrypoint.value), "utf8");
return {
localEntrypoint: fileName,
externalUrl: entrypoint.value,
entryUrl: buildCanvasDocumentEntryUrl(path.basename(rootDir), fileName),
};
}
return {
externalUrl: entrypoint.value,
entryUrl: entrypoint.value,
};
}
const resolvedPath = entrypoint.value.startsWith("~")
? resolveUserPath(entrypoint.value)
: path.isAbsolute(entrypoint.value)
? path.resolve(entrypoint.value)
: path.resolve(workspaceDir, entrypoint.value);
if (input.kind === "image" || input.kind === "video_asset") {
const copiedName = path.basename(resolvedPath);
await fs.copyFile(resolvedPath, path.join(rootDir, copiedName));
const wrapper =
input.kind === "image"
? `<!doctype html><html><body style="margin:0;background:#0f172a;display:flex;align-items:center;justify-content:center;"><img src="${escapeHtml(copiedName)}" style="max-width:100%;max-height:100vh;object-fit:contain;" /></body></html>`
: `<!doctype html><html><body style="margin:0;background:#0f172a;"><video src="${escapeHtml(copiedName)}" controls autoplay style="width:100%;height:100vh;object-fit:contain;background:#000;"></video></body></html>`;
await fs.writeFile(path.join(rootDir, "index.html"), wrapper, "utf8");
return {
localEntrypoint: "index.html",
entryUrl: buildCanvasDocumentEntryUrl(path.basename(rootDir), "index.html"),
};
}
const fileName = path.basename(resolvedPath);
await fs.copyFile(resolvedPath, path.join(rootDir, fileName));
if (input.kind === "document" && isPdfPathLike(fileName)) {
await fs.writeFile(path.join(rootDir, "index.html"), buildPdfWrapper(fileName), "utf8");
return {
localEntrypoint: "index.html",
entryUrl: buildCanvasDocumentEntryUrl(path.basename(rootDir), "index.html"),
};
}
return {
localEntrypoint: fileName,
entryUrl: buildCanvasDocumentEntryUrl(path.basename(rootDir), fileName),
};
}
export async function createCanvasDocument(
input: CanvasDocumentCreateInput,
options?: { stateDir?: string; workspaceDir?: string; canvasRootDir?: string },
): Promise<CanvasDocumentManifest> {
const workspaceDir = options?.workspaceDir ?? process.cwd();
const id = input.id?.trim() ? normalizeCanvasDocumentId(input.id) : canvasDocumentId();
const rootDir = resolveCanvasDocumentDir(id, {
stateDir: options?.stateDir,
rootDir: options?.canvasRootDir,
});
await fs.rm(rootDir, { recursive: true, force: true }).catch(() => undefined);
await fs.mkdir(rootDir, { recursive: true });
const assets = await copyAssets(rootDir, input.assets, workspaceDir);
const entry = await materializeEntrypoint(rootDir, input, workspaceDir);
const manifest: CanvasDocumentManifest = {
id,
kind: input.kind,
...(input.title?.trim() ? { title: input.title.trim() } : {}),
...(typeof input.preferredHeight === "number"
? { preferredHeight: input.preferredHeight }
: {}),
...(input.surface ? { surface: input.surface } : {}),
createdAt: new Date().toISOString(),
entryUrl: entry.entryUrl,
...(entry.localEntrypoint ? { localEntrypoint: entry.localEntrypoint } : {}),
...(entry.externalUrl ? { externalUrl: entry.externalUrl } : {}),
assets,
};
await writeManifest(rootDir, manifest);
return manifest;
}
export async function loadCanvasDocumentManifest(
documentId: string,
options?: { stateDir?: string; canvasRootDir?: string },
): Promise<CanvasDocumentManifest | null> {
const id = normalizeCanvasDocumentId(documentId);
const manifestPath = path.join(
resolveCanvasDocumentDir(id, {
stateDir: options?.stateDir,
rootDir: options?.canvasRootDir,
}),
"manifest.json",
);
try {
const raw = await fs.readFile(manifestPath, "utf8");
const parsed = JSON.parse(raw);
return parsed && typeof parsed === "object" && !Array.isArray(parsed)
? (parsed as CanvasDocumentManifest)
: null;
} catch {
return null;
}
}
export function resolveCanvasDocumentAssets(
manifest: CanvasDocumentManifest,
options?: { baseUrl?: string; stateDir?: string; canvasRootDir?: string },
): CanvasDocumentResolvedAsset[] {
const baseUrl = options?.baseUrl?.trim().replace(/\/+$/, "");
const documentDir = resolveCanvasDocumentDir(manifest.id, {
stateDir: options?.stateDir,
rootDir: options?.canvasRootDir,
});
return manifest.assets.map((asset) => ({
logicalPath: asset.logicalPath,
...(asset.contentType ? { contentType: asset.contentType } : {}),
localPath: path.join(documentDir, asset.logicalPath),
url: baseUrl
? `${baseUrl}${buildCanvasDocumentAssetUrl(manifest.id, asset.logicalPath)}`
: buildCanvasDocumentAssetUrl(manifest.id, asset.logicalPath),
}));
}

View File

@@ -1,7 +1,13 @@
export const CONTROL_UI_BOOTSTRAP_CONFIG_PATH = "/__openclaw/control-ui-config.json";
export type ControlUiEmbedSandboxMode = "powerful" | "isolated";
export type ControlUiBootstrapConfig = {
basePath: string;
assistantName: string;
assistantAvatar: string;
assistantAgentId: string;
serverVersion?: string;
localMediaPreviewRoots?: string[];
embedSandbox?: ControlUiEmbedSandboxMode;
};

View File

@@ -4,8 +4,13 @@ import type { IncomingMessage } from "node:http";
import os from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
import type { ResolvedGatewayAuth } from "./auth.js";
import { CONTROL_UI_BOOTSTRAP_CONFIG_PATH } from "./control-ui-contract.js";
import { handleControlUiAvatarRequest, handleControlUiHttpRequest } from "./control-ui.js";
import {
handleControlUiAssistantMediaRequest,
handleControlUiAvatarRequest,
handleControlUiHttpRequest,
} from "./control-ui.js";
import { makeMockHttpResponse } from "./test-http-response.js";
describe("handleControlUiHttpRequest", () => {
@@ -27,6 +32,8 @@ describe("handleControlUiHttpRequest", () => {
basePath: string;
assistantName: string;
assistantAvatar: string;
assistantAgentId: string;
localMediaPreviewRoots?: string[];
};
}
@@ -77,6 +84,33 @@ describe("handleControlUiHttpRequest", () => {
return { res, end, handled };
}
async function runAssistantMediaRequest(params: {
url: string;
method: "GET" | "HEAD";
basePath?: string;
auth?: ResolvedGatewayAuth;
headers?: IncomingMessage["headers"];
trustedProxies?: string[];
remoteAddress?: string;
}) {
const { res, end } = makeMockHttpResponse();
const handled = await handleControlUiAssistantMediaRequest(
{
url: params.url,
method: params.method,
headers: params.headers ?? {},
socket: { remoteAddress: params.remoteAddress ?? "127.0.0.1" },
} as IncomingMessage,
res,
{
...(params.basePath ? { basePath: params.basePath } : {}),
...(params.auth ? { auth: params.auth } : {}),
...(params.trustedProxies ? { trustedProxies: params.trustedProxies } : {}),
},
);
return { res, end, handled };
}
async function writeAssetFile(rootPath: string, filename: string, contents: string) {
const assetsDir = path.join(rootPath, "assets");
await fs.mkdir(assetsDir, { recursive: true });
@@ -131,6 +165,127 @@ describe("handleControlUiHttpRequest", () => {
});
});
it("serves assistant local media through the control ui media route", async () => {
const tmpRoot = path.join("/tmp/openclaw", `ui-media-${Date.now()}`);
try {
await fs.mkdir(tmpRoot, { recursive: true });
const filePath = path.join(tmpRoot, "photo.png");
await fs.writeFile(filePath, Buffer.from("not-a-real-png"));
const { res, handled } = await runAssistantMediaRequest({
url: `/__openclaw__/assistant-media?source=${encodeURIComponent(filePath)}&token=test-token`,
method: "GET",
auth: { mode: "token", token: "test-token", allowTailscale: false },
});
expect(handled).toBe(true);
expect(res.statusCode).toBe(200);
} finally {
await fs.rm(tmpRoot, { recursive: true, force: true });
}
});
it("rejects assistant local media outside allowed preview roots", async () => {
const tmp = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-ui-media-blocked-"));
try {
const filePath = path.join(tmp, "photo.png");
await fs.writeFile(filePath, Buffer.from("not-a-real-png"));
const { res, handled, end } = await runAssistantMediaRequest({
url: `/__openclaw__/assistant-media?source=${encodeURIComponent(filePath)}&token=test-token`,
method: "GET",
auth: { mode: "token", token: "test-token", allowTailscale: false },
});
expectNotFoundResponse({ handled, res, end });
} finally {
await fs.rm(tmp, { recursive: true, force: true });
}
});
it("reports assistant local media availability metadata", async () => {
const tmpRoot = path.join("/tmp/openclaw", `ui-media-meta-${Date.now()}`);
try {
await fs.mkdir(tmpRoot, { recursive: true });
const filePath = path.join(tmpRoot, "photo.png");
await fs.writeFile(filePath, Buffer.from("not-a-real-png"));
const { res, handled, end } = await runAssistantMediaRequest({
url: `/__openclaw__/assistant-media?meta=1&source=${encodeURIComponent(filePath)}&token=test-token`,
method: "GET",
auth: { mode: "token", token: "test-token", allowTailscale: false },
});
expect(handled).toBe(true);
expect(res.statusCode).toBe(200);
expect(JSON.parse(String(end.mock.calls[0]?.[0] ?? ""))).toEqual({ available: true });
} finally {
await fs.rm(tmpRoot, { recursive: true, force: true });
}
});
it("reports assistant local media availability failures with a reason", async () => {
const { res, handled, end } = await runAssistantMediaRequest({
url: `/__openclaw__/assistant-media?meta=1&source=${encodeURIComponent("/Users/test/Documents/private.pdf")}&token=test-token`,
method: "GET",
auth: { mode: "token", token: "test-token", allowTailscale: false },
});
expect(handled).toBe(true);
expect(res.statusCode).toBe(200);
expect(JSON.parse(String(end.mock.calls[0]?.[0] ?? ""))).toEqual({
available: false,
code: "outside-allowed-folders",
reason: "Outside allowed folders",
});
});
it("rejects assistant local media without a valid auth token when auth is enabled", async () => {
const tmpRoot = path.join("/tmp/openclaw", `ui-media-auth-${Date.now()}`);
try {
await fs.mkdir(tmpRoot, { recursive: true });
const filePath = path.join(tmpRoot, "photo.png");
await fs.writeFile(filePath, Buffer.from("not-a-real-png"));
const { res, handled, end } = await runAssistantMediaRequest({
url: `/__openclaw__/assistant-media?source=${encodeURIComponent(filePath)}`,
method: "GET",
auth: { mode: "token", token: "test-token", allowTailscale: false },
});
expect(handled).toBe(true);
expect(res.statusCode).toBe(401);
expect(String(end.mock.calls[0]?.[0] ?? "")).toContain("Unauthorized");
} finally {
await fs.rm(tmpRoot, { recursive: true, force: true });
}
});
it("rejects trusted-proxy assistant media requests from disallowed browser origins", async () => {
const tmpRoot = path.join("/tmp/openclaw", `ui-media-proxy-${Date.now()}`);
try {
await fs.mkdir(tmpRoot, { recursive: true });
const filePath = path.join(tmpRoot, "photo.png");
await fs.writeFile(filePath, Buffer.from("not-a-real-png"));
const { res, handled, end } = await runAssistantMediaRequest({
url: `/__openclaw__/assistant-media?source=${encodeURIComponent(filePath)}`,
method: "GET",
auth: {
mode: "trusted-proxy",
allowTailscale: false,
trustedProxy: {
userHeaders: ["x-forwarded-user"],
allowedOrigins: ["https://control.example.com"],
},
},
trustedProxies: ["10.0.0.1"],
remoteAddress: "10.0.0.1",
headers: {
host: "gateway.example.com",
origin: "https://evil.example",
"x-forwarded-user": "nick@example.com",
"x-forwarded-proto": "https",
},
});
expect(handled).toBe(true);
expect(res.statusCode).toBe(401);
expect(String(end.mock.calls[0]?.[0] ?? "")).toContain("Unauthorized");
} finally {
await fs.rm(tmpRoot, { recursive: true, force: true });
}
});
it("includes CSP hash for inline scripts in index.html", async () => {
const scriptContent = "(function(){ var x = 1; })();";
const html = `<html><head><script>${scriptContent}</script></head><body></body></html>\n`;
@@ -195,8 +350,8 @@ describe("handleControlUiHttpRequest", () => {
expect(parsed.basePath).toBe("");
expect(parsed.assistantName).toBe("</script><script>alert(1)//");
expect(parsed.assistantAvatar).toBe("/avatar/main");
expect(parsed).not.toHaveProperty("assistantAgentId");
expect(parsed).not.toHaveProperty("serverVersion");
expect(parsed.assistantAgentId).toBe("main");
expect(Array.isArray(parsed.localMediaPreviewRoots)).toBe(true);
},
});
});
@@ -222,8 +377,8 @@ describe("handleControlUiHttpRequest", () => {
expect(parsed.basePath).toBe("/openclaw");
expect(parsed.assistantName).toBe("Ops");
expect(parsed.assistantAvatar).toBe("/openclaw/avatar/main");
expect(parsed).not.toHaveProperty("assistantAgentId");
expect(parsed).not.toHaveProperty("serverVersion");
expect(parsed.assistantAgentId).toBe("main");
expect(Array.isArray(parsed.localMediaPreviewRoots)).toBe(true);
},
});
});

View File

@@ -7,11 +7,18 @@ import {
isPackageProvenControlUiRootSync,
resolveControlUiRootSync,
} from "../infra/control-ui-assets.js";
import { openLocalFileSafely, SafeOpenError } from "../infra/fs-safe.js";
import { safeFileURLToPath } from "../infra/local-file-access.js";
import { isWithinDir } from "../infra/path-safety.js";
import { openVerifiedFileSync } from "../infra/safe-open-sync.js";
import { assertLocalMediaAllowed, getDefaultLocalRoots } from "../media/local-media-access.js";
import { detectMime } from "../media/mime.js";
import { AVATAR_MAX_BYTES } from "../shared/avatar-policy.js";
import { normalizeLowercaseStringOrEmpty } from "../shared/string-coerce.js";
import { resolveUserPath } from "../utils.js";
import { resolveRuntimeServiceVersion } from "../version.js";
import { DEFAULT_ASSISTANT_IDENTITY, resolveAssistantIdentity } from "./assistant-identity.js";
import type { AuthRateLimiter } from "./auth-rate-limit.js";
import { authorizeHttpGatewayConnect, type ResolvedGatewayAuth } from "./auth.js";
import {
CONTROL_UI_BOOTSTRAP_CONFIG_PATH,
type ControlUiBootstrapConfig,
@@ -29,8 +36,11 @@ import {
normalizeControlUiBasePath,
resolveAssistantAvatarUrl,
} from "./control-ui-shared.js";
import { sendGatewayAuthFailure } from "./http-common.js";
import { getBearerToken, resolveHttpBrowserOriginPolicy } from "./http-utils.js";
const ROOT_PREFIX = "/";
const CONTROL_UI_ASSISTANT_MEDIA_PREFIX = "/__openclaw__/assistant-media";
const CONTROL_UI_ASSETS_MISSING_MESSAGE =
"Control UI assets not found. Build them with `pnpm ui:build` (auto-installs UI deps), or run `pnpm ui:dev` during development.";
@@ -153,6 +163,210 @@ function isValidAgentId(agentId: string): boolean {
return /^[a-z0-9][a-z0-9_-]{0,63}$/i.test(agentId);
}
function normalizeAssistantMediaSource(source: string): string | null {
const trimmed = source.trim();
if (!trimmed) {
return null;
}
if (trimmed.startsWith("file://")) {
try {
return safeFileURLToPath(trimmed);
} catch {
return null;
}
}
if (trimmed.startsWith("~")) {
return resolveUserPath(trimmed);
}
return trimmed;
}
function resolveAssistantMediaRoutePath(basePath?: string): string {
const normalizedBasePath =
basePath && basePath !== "/" ? (basePath.endsWith("/") ? basePath.slice(0, -1) : basePath) : "";
return `${normalizedBasePath}${CONTROL_UI_ASSISTANT_MEDIA_PREFIX}`;
}
function resolveAssistantMediaAuthToken(req: IncomingMessage): string | undefined {
const bearer = getBearerToken(req);
if (bearer) {
return bearer;
}
const urlRaw = req.url;
if (!urlRaw) {
return undefined;
}
try {
const url = new URL(urlRaw, "http://localhost");
const token = url.searchParams.get("token")?.trim();
return token || undefined;
} catch {
return undefined;
}
}
type AssistantMediaAvailability =
| { available: true }
| { available: false; reason: string; code: string };
function classifyAssistantMediaError(err: unknown): AssistantMediaAvailability {
if (err instanceof SafeOpenError) {
switch (err.code) {
case "not-found":
return { available: false, code: "file-not-found", reason: "File not found" };
case "not-file":
return { available: false, code: "not-a-file", reason: "Not a file" };
case "invalid-path":
case "path-mismatch":
case "symlink":
return { available: false, code: "invalid-file", reason: "Invalid file" };
default:
return {
available: false,
code: "attachment-unavailable",
reason: "Attachment unavailable",
};
}
}
if (err instanceof Error && "code" in err) {
const errorCode = (err as { code?: unknown }).code;
switch (typeof errorCode === "string" ? errorCode : "") {
case "path-not-allowed":
return {
available: false,
code: "outside-allowed-folders",
reason: "Outside allowed folders",
};
case "invalid-file-url":
case "invalid-path":
case "unsafe-bypass":
case "network-path-not-allowed":
case "invalid-root":
return { available: false, code: "blocked-local-file", reason: "Blocked local file" };
case "not-found":
return { available: false, code: "file-not-found", reason: "File not found" };
case "not-file":
return { available: false, code: "not-a-file", reason: "Not a file" };
default:
break;
}
}
return { available: false, code: "attachment-unavailable", reason: "Attachment unavailable" };
}
async function resolveAssistantMediaAvailability(
source: string,
): Promise<AssistantMediaAvailability> {
try {
await assertLocalMediaAllowed(source, getDefaultLocalRoots());
const opened = await openLocalFileSafely({ filePath: source });
await opened.handle.close();
return { available: true };
} catch (err) {
return classifyAssistantMediaError(err);
}
}
export async function handleControlUiAssistantMediaRequest(
req: IncomingMessage,
res: ServerResponse,
opts?: {
basePath?: string;
auth?: ResolvedGatewayAuth;
trustedProxies?: string[];
allowRealIpFallback?: boolean;
rateLimiter?: AuthRateLimiter;
},
): Promise<boolean> {
const urlRaw = req.url;
if (!urlRaw || !isReadHttpMethod(req.method)) {
return false;
}
const url = new URL(urlRaw, "http://localhost");
if (url.pathname !== resolveAssistantMediaRoutePath(opts?.basePath)) {
return false;
}
applyControlUiSecurityHeaders(res);
if (opts?.auth) {
const token = resolveAssistantMediaAuthToken(req);
const authResult = await authorizeHttpGatewayConnect({
auth: opts.auth,
connectAuth: token ? { token, password: token } : null,
req,
browserOriginPolicy: resolveHttpBrowserOriginPolicy(req),
trustedProxies: opts.trustedProxies,
allowRealIpFallback: opts.allowRealIpFallback,
rateLimiter: opts.rateLimiter,
});
if (!authResult.ok) {
sendGatewayAuthFailure(res, authResult);
return true;
}
}
const source = normalizeAssistantMediaSource(url.searchParams.get("source") ?? "");
if (!source) {
respondControlUiNotFound(res);
return true;
}
if (url.searchParams.get("meta") === "1") {
const availability = await resolveAssistantMediaAvailability(source);
sendJson(res, 200, availability);
return true;
}
try {
await assertLocalMediaAllowed(source, getDefaultLocalRoots());
const opened = await openLocalFileSafely({ filePath: source });
let handleClosed = false;
const closeHandle = async () => {
if (handleClosed) {
return;
}
handleClosed = true;
await opened.handle.close().catch(() => {});
};
const sniffLength = Math.min(opened.stat.size, 8192);
const sniffBuffer = sniffLength > 0 ? Buffer.allocUnsafe(sniffLength) : undefined;
const bytesRead =
sniffBuffer && sniffLength > 0
? (await opened.handle.read(sniffBuffer, 0, sniffLength, 0)).bytesRead
: 0;
const mime = await detectMime({
buffer: sniffBuffer?.subarray(0, bytesRead),
filePath: source,
});
if (mime) {
res.setHeader("Content-Type", mime);
} else {
res.setHeader("Content-Type", "application/octet-stream");
}
res.setHeader("Cache-Control", "no-cache");
res.setHeader("Content-Length", String(opened.stat.size));
const stream = opened.handle.createReadStream({ start: 0, autoClose: false });
const finishClose = () => {
void closeHandle();
};
stream.once("end", finishClose);
stream.once("close", finishClose);
stream.once("error", () => {
void closeHandle();
if (!res.headersSent) {
respondControlUiNotFound(res);
} else {
res.destroy();
}
});
res.once("close", finishClose);
stream.pipe(res);
return true;
} catch {
respondControlUiNotFound(res);
return true;
}
}
export function handleControlUiAvatarRequest(
req: IncomingMessage,
res: ServerResponse,
@@ -221,7 +435,7 @@ export function handleControlUiAvatarRequest(
}
function setStaticFileHeaders(res: ServerResponse, filePath: string) {
const ext = normalizeLowercaseStringOrEmpty(path.extname(filePath));
const ext = path.extname(filePath).toLowerCase();
res.setHeader("Content-Type", contentTypeForExt(ext));
// Static UI should never be cached aggressively while iterating; allow the
// browser to revalidate.
@@ -365,6 +579,11 @@ export function handleControlUiHttpRequest(
basePath,
assistantName: identity.name,
assistantAvatar: avatarValue ?? identity.avatar,
assistantAgentId: identity.agentId,
serverVersion: resolveRuntimeServiceVersion(process.env),
localMediaPreviewRoots: [...getDefaultLocalRoots()],
embedSandbox:
config?.gateway?.controlUi?.embedSandbox === "isolated" ? "isolated" : "powerful",
} satisfies ControlUiBootstrapConfig);
return true;
}
@@ -463,7 +682,7 @@ export function handleControlUiHttpRequest(
// against the same set of extensions that contentTypeForExt() recognises so
// that dotted SPA routes (e.g. /user/jane.doe, /v2.0) still get the
// client-side router fallback.
if (STATIC_ASSET_EXTENSIONS.has(normalizeLowercaseStringOrEmpty(path.extname(fileRel)))) {
if (STATIC_ASSET_EXTENSIONS.has(path.extname(fileRel).toLowerCase())) {
respondControlUiNotFound(res);
return true;
}

View File

@@ -89,6 +89,52 @@ describe("gateway probe endpoints", () => {
});
});
it("hides readiness details when trusted-proxy auth violates browser origin policy", async () => {
const getReadiness: ReadinessChecker = () => ({
ready: false,
failing: ["discord", "telegram"],
uptimeMs: 8_000,
});
await withGatewayServer({
prefix: "probe-remote-origin-rejected",
resolvedAuth: {
mode: "trusted-proxy",
allowTailscale: false,
trustedProxy: { userHeader: "x-forwarded-user" },
},
overrides: {
getReadiness,
configPatch: {
gateway: {
trustedProxies: ["10.0.0.1"],
controlUi: {
allowedOrigins: ["https://control.example"],
},
},
},
},
run: async (server) => {
const req = createRequest({
path: "/ready",
remoteAddress: "10.0.0.1",
host: "gateway.test",
headers: {
origin: "https://evil.example",
forwarded: "for=203.0.113.10;proto=https;host=gateway.test",
"x-forwarded-user": "user@example.com",
"x-forwarded-proto": "https",
},
});
const { res, getBody } = createResponse();
await dispatchRequest(server, req, res);
expect(res.statusCode).toBe(503);
expect(JSON.parse(getBody())).toEqual({ ready: false });
},
});
});
it("returns typed internal error payload when readiness evaluation throws", async () => {
const getReadiness: ReadinessChecker = () => {
throw new Error("boom");

View File

@@ -16,7 +16,6 @@ import { loadConfig } from "../config/config.js";
import type { createSubsystemLogger } from "../logging/subsystem.js";
import { resolveHookExternalContentSource as resolveHookExternalContentSourceFromSession } from "../security/external-content.js";
import { safeEqualSecret } from "../security/secret-equal.js";
import { normalizeLowercaseStringOrEmpty } from "../shared/string-coerce.js";
import {
AUTH_RATE_LIMIT_SCOPE_HOOK_AUTH,
createAuthRateLimiter,
@@ -31,6 +30,7 @@ import {
} from "./auth.js";
import { normalizeCanvasScopedUrl } from "./canvas-capability.js";
import {
handleControlUiAssistantMediaRequest,
handleControlUiAvatarRequest,
handleControlUiHttpRequest,
type ControlUiRootState,
@@ -59,6 +59,7 @@ import {
} from "./hooks.js";
import { sendGatewayAuthFailure, setDefaultSecurityHeaders } from "./http-common.js";
import {
type AuthorizedGatewayHttpRequest,
authorizeGatewayHttpRequestOrReply,
getBearerToken,
resolveHttpBrowserOriginPolicy,
@@ -98,7 +99,8 @@ function resolveMappedHookExternalContentSource(params: {
payload: Record<string, unknown>;
sessionKey: string;
}) {
const payloadSource = normalizeLowercaseStringOrEmpty(params.payload.source);
const payloadSource =
typeof params.payload.source === "string" ? params.payload.source.trim().toLowerCase() : "";
if (params.subPath === "gmail" || payloadSource === "gmail") {
return "gmail" as const;
}
@@ -134,6 +136,7 @@ const GATEWAY_PROBE_STATUS_BY_PATH = new Map<string, "live" | "ready">([
["/ready", "ready"],
["/readyz", "ready"],
]);
function resolvePluginGatewayAuthBypassPaths(
configSnapshot: ReturnType<typeof loadConfig>,
): Set<string> {
@@ -265,20 +268,6 @@ function writeUpgradeAuthFailure(
socket.write("HTTP/1.1 401 Unauthorized\r\nConnection: close\r\n\r\n");
}
function writeUpgradeServiceUnavailable(
socket: { write: (chunk: string) => void },
responseBody: string,
) {
socket.write(
"HTTP/1.1 503 Service Unavailable\r\n" +
"Connection: close\r\n" +
"Content-Type: text/plain; charset=utf-8\r\n" +
`Content-Length: ${Buffer.byteLength(responseBody, "utf8")}\r\n` +
"\r\n" +
responseBody,
);
}
export type HooksRequestHandler = (req: IncomingMessage, res: ServerResponse) => Promise<boolean>;
type GatewayHttpRequestStage = {
@@ -296,9 +285,8 @@ export async function runGatewayHttpRequestStages(
}
} catch (err) {
// Log and skip the failing stage so subsequent stages (control-ui,
// gateway-probes, etc.) remain reachable. A common trigger is a
// plugin-owned route/runtime code can still fail to load when an
// optional dependency is missing. Keep later stages reachable.
// gateway-probes, etc.) remain reachable. A common trigger is a
// plugin-owned route/runtime code still failing to load an optional dependency.
console.error(`[gateway-http] stage "${stage.name}" threw — skipping:`, err);
}
}
@@ -322,6 +310,7 @@ function buildPluginRequestStages(params: {
return [];
}
let pluginGatewayAuthSatisfied = false;
let pluginGatewayRequestAuth: AuthorizedGatewayHttpRequest | undefined;
let pluginRequestOperatorScopes: string[] | undefined;
return [
{
@@ -346,11 +335,13 @@ function buildPluginRequestStages(params: {
trustedProxies: params.trustedProxies,
allowRealIpFallback: params.allowRealIpFallback,
rateLimiter: params.rateLimiter,
browserOriginPolicy: resolveHttpBrowserOriginPolicy(params.req),
});
if (!requestAuth) {
return true;
}
pluginGatewayAuthSatisfied = true;
pluginGatewayRequestAuth = requestAuth;
pluginRequestOperatorScopes = resolvePluginRouteRuntimeOperatorScopes(
params.req,
requestAuth,
@@ -366,6 +357,7 @@ function buildPluginRequestStages(params: {
return (
params.handlePluginRequest?.(params.req, params.res, pathContext, {
gatewayAuthSatisfied: pluginGatewayAuthSatisfied,
gatewayRequestAuth: pluginGatewayRequestAuth,
gatewayRequestOperatorScopes: pluginRequestOperatorScopes,
}) ?? false
);
@@ -795,7 +787,7 @@ export function createGatewayHttpServer(opts: {
});
// Don't interfere with WebSocket upgrades; ws handles the 'upgrade' event.
if (normalizeLowercaseStringOrEmpty(req.headers.upgrade) === "websocket") {
if (String(req.headers.upgrade ?? "").toLowerCase() === "websocket") {
return;
}
@@ -955,6 +947,17 @@ export function createGatewayHttpServer(opts: {
);
if (controlUiEnabled) {
requestStages.push({
name: "control-ui-assistant-media",
run: () =>
handleControlUiAssistantMediaRequest(req, res, {
basePath: controlUiBasePath,
auth: resolvedAuth,
trustedProxies,
allowRealIpFallback,
rateLimiter,
}),
});
requestStages.push({
name: "control-ui-avatar",
run: () =>
@@ -1064,15 +1067,29 @@ export function attachGatewayUpgradeHandler(opts: {
}
}
const preauthBudgetKey = resolveRequestClientIp(req, trustedProxies, allowRealIpFallback);
// Keep startup upgrades inside the pre-auth budget until WS handlers attach.
if (!preauthConnectionBudget.acquire(preauthBudgetKey)) {
writeUpgradeServiceUnavailable(socket, "Too many unauthenticated sockets");
if (wss.listenerCount("connection") === 0) {
const responseBody = "Gateway websocket handlers unavailable";
socket.write(
"HTTP/1.1 503 Service Unavailable\r\n" +
"Connection: close\r\n" +
"Content-Type: text/plain; charset=utf-8\r\n" +
`Content-Length: ${Buffer.byteLength(responseBody, "utf8")}\r\n` +
"\r\n" +
responseBody,
);
socket.destroy();
return;
}
if (wss.listenerCount("connection") === 0) {
preauthConnectionBudget.release(preauthBudgetKey);
writeUpgradeServiceUnavailable(socket, "Gateway websocket handlers unavailable");
if (!preauthConnectionBudget.acquire(preauthBudgetKey)) {
const responseBody = "Too many unauthenticated sockets";
socket.write(
"HTTP/1.1 503 Service Unavailable\r\n" +
"Connection: close\r\n" +
"Content-Type: text/plain; charset=utf-8\r\n" +
`Content-Length: ${Buffer.byteLength(responseBody, "utf8")}\r\n` +
"\r\n" +
responseBody,
);
socket.destroy();
return;
}

View File

@@ -18,12 +18,17 @@ const mockState = vi.hoisted(() => ({
sessionId: "sess-1",
mainSessionKey: "main",
finalText: "[[reply_to_current]]",
finalPayload: null as { text?: string; mediaUrl?: string } | null,
dispatchError: null as Error | null,
triggerAgentRunStart: false,
agentRunId: "run-agent-1",
sessionEntry: {} as Record<string, unknown>,
lastDispatchCtx: undefined as MsgContext | undefined,
lastDispatchImages: undefined as Array<{ mimeType: string; data: string }> | undefined,
lastDispatchImageOrder: undefined as string[] | undefined,
modelCatalog: null as
| Array<{ provider: string; id: string; name?: string; input?: string[] }>
| null,
emittedTranscriptUpdates: [] as Array<{
sessionFile: string;
sessionKey?: string;
@@ -31,6 +36,7 @@ const mockState = vi.hoisted(() => ({
messageId?: string;
}>,
savedMediaResults: [] as Array<{ path: string; contentType?: string }>,
saveMediaError: null as Error | null,
savedMediaCalls: [] as Array<{ contentType?: string; subdir?: string; size: number }>,
saveMediaWait: null as Promise<void> | null,
activeSaveMediaCalls: 0,
@@ -86,17 +92,19 @@ vi.mock("../../auto-reply/dispatch.js", () => ({
replyOptions?: {
onAgentRunStart?: (runId: string) => void;
images?: Array<{ mimeType: string; data: string }>;
imageOrder?: string[];
};
}) => {
mockState.lastDispatchCtx = params.ctx;
mockState.lastDispatchImages = params.replyOptions?.images;
mockState.lastDispatchImageOrder = params.replyOptions?.imageOrder;
if (mockState.dispatchError) {
throw mockState.dispatchError;
}
if (mockState.triggerAgentRunStart) {
params.replyOptions?.onAgentRunStart?.(mockState.agentRunId);
}
params.dispatcher.sendFinalReply({ text: mockState.finalText });
params.dispatcher.sendFinalReply(mockState.finalPayload ?? { text: mockState.finalText });
params.dispatcher.markComplete();
await params.dispatcher.waitForIdle();
return { ok: true };
@@ -131,6 +139,10 @@ vi.mock("../../media/store.js", async () => {
if (mockState.saveMediaWait) {
await mockState.saveMediaWait;
}
if (mockState.saveMediaError) {
mockState.activeSaveMediaCalls -= 1;
throw mockState.saveMediaError;
}
mockState.savedMediaCalls.push({ contentType, subdir, size: buffer.byteLength });
const next = mockState.savedMediaResults.shift();
try {
@@ -254,20 +266,21 @@ function createChatContext(): Pick<
chatAbortedRuns: new Map(),
removeChatRun: vi.fn(),
dedupe: new Map(),
loadGatewayModelCatalog: async () => [
{
provider: "openai",
id: "gpt-5.4",
name: "GPT-5.4",
input: ["text", "image"],
},
{
provider: "anthropic",
id: "claude-opus-4-6",
name: "Claude Opus 4.6",
input: ["text", "image"],
},
],
loadGatewayModelCatalog: async () =>
mockState.modelCatalog ?? [
{
provider: "openai",
id: "gpt-5.4",
name: "GPT-5.4",
input: ["text", "image"],
},
{
provider: "anthropic",
id: "claude-opus-4-6",
name: "Claude Opus 4.6",
input: ["text", "image"],
},
],
registerToolEventRecipient: vi.fn(),
logGateway: {
warn: vi.fn(),
@@ -351,6 +364,7 @@ async function runNonStreamingChatSend(params: {
describe("chat directive tag stripping for non-streaming final payloads", () => {
afterEach(() => {
mockState.finalText = "[[reply_to_current]]";
mockState.finalPayload = null;
mockState.dispatchError = null;
mockState.mainSessionKey = "main";
mockState.triggerAgentRunStart = false;
@@ -358,8 +372,11 @@ describe("chat directive tag stripping for non-streaming final payloads", () =>
mockState.sessionEntry = {};
mockState.lastDispatchCtx = undefined;
mockState.lastDispatchImages = undefined;
mockState.lastDispatchImageOrder = undefined;
mockState.modelCatalog = null;
mockState.emittedTranscriptUpdates = [];
mockState.savedMediaResults = [];
mockState.saveMediaError = null;
mockState.savedMediaCalls = [];
mockState.saveMediaWait = null;
mockState.activeSaveMediaCalls = 0;
@@ -1584,6 +1601,78 @@ describe("chat directive tag stripping for non-streaming final payloads", () =>
});
});
it("preserves offloaded attachment media paths in transcript order", async () => {
createTranscriptFixture("openclaw-chat-send-user-transcript-offloaded-");
mockState.finalText = "ok";
mockState.triggerAgentRunStart = true;
mockState.sessionEntry = {
modelProvider: "test-provider",
model: "vision-model",
};
mockState.modelCatalog = [
{
provider: "test-provider",
id: "vision-model",
name: "Vision model",
input: ["text", "image"],
},
];
mockState.savedMediaResults = [
{ path: "/tmp/offloaded-big.png", contentType: "image/png" },
{ path: "/tmp/chat-send-inline.png", contentType: "image/png" },
];
const respond = vi.fn();
const context = createChatContext();
const bigPng = Buffer.alloc(2_100_000);
bigPng.set([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a], 0);
await runNonStreamingChatSend({
context,
respond,
idempotencyKey: "idem-user-transcript-offloaded",
message: "edit both",
requestParams: {
attachments: [
{
mimeType: "image/png",
content:
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+aYoYAAAAASUVORK5CYII=",
},
{
mimeType: "image/png",
content: bigPng.toString("base64"),
},
],
},
expectBroadcast: false,
waitForCompletion: false,
});
await waitForAssertion(() => {
const userUpdate = mockState.emittedTranscriptUpdates.find(
(update) =>
typeof update.message === "object" &&
update.message !== null &&
(update.message as { role?: unknown }).role === "user",
);
const message = userUpdate?.message as
| {
MediaPath?: string;
MediaPaths?: string[];
MediaType?: string;
MediaTypes?: string[];
}
| undefined;
expect(message?.MediaPath).toBe("/tmp/chat-send-inline.png");
expect(message?.MediaPaths).toEqual([
"/tmp/chat-send-inline.png",
"/tmp/offloaded-big.png",
]);
expect(message?.MediaType).toBe("image/png");
expect(message?.MediaTypes).toEqual(["image/png", "image/png"]);
});
});
it("skips transcript media notes for ACP bridge clients", async () => {
createTranscriptFixture("openclaw-chat-send-user-transcript-acp-images-");
mockState.finalText = "ok";
@@ -1680,6 +1769,149 @@ describe("chat directive tag stripping for non-streaming final payloads", () =>
});
});
it("preserves media-only final replies in the final broadcast message", async () => {
createTranscriptFixture("openclaw-chat-send-media-only-final-");
mockState.finalPayload = { mediaUrl: "https://example.com/final.png" };
const respond = vi.fn();
const context = createChatContext();
const payload = await runNonStreamingChatSend({
context,
respond,
idempotencyKey: "idem-media-only-final",
});
expect(extractFirstTextBlock(payload)).toBe("MEDIA:https://example.com/final.png");
});
it("drops image attachments for text-only session models", async () => {
createTranscriptFixture("openclaw-chat-send-text-only-attachments-");
mockState.finalText = "ok";
mockState.sessionEntry = {
modelProvider: "test-provider",
model: "text-only",
};
mockState.modelCatalog = [
{
provider: "test-provider",
id: "text-only",
name: "Text only",
input: ["text"],
},
];
const respond = vi.fn();
const context = createChatContext();
await runNonStreamingChatSend({
context,
respond,
idempotencyKey: "idem-text-only-attachments",
message: "describe image",
requestParams: {
attachments: [
{
mimeType: "image/png",
content:
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=",
},
],
},
expectBroadcast: false,
});
expect(mockState.lastDispatchImages).toBeUndefined();
expect(mockState.lastDispatchImageOrder).toBeUndefined();
});
it("passes imageOrder for mixed inline and offloaded chat.send attachments", async () => {
createTranscriptFixture("openclaw-chat-send-image-order-");
mockState.finalText = "ok";
mockState.sessionEntry = {
modelProvider: "test-provider",
model: "vision-model",
};
mockState.modelCatalog = [
{
provider: "test-provider",
id: "vision-model",
name: "Vision model",
input: ["text", "image"],
},
];
mockState.savedMediaResults = [{ path: "/tmp/offloaded-big.png", contentType: "image/png" }];
const respond = vi.fn();
const context = createChatContext();
const bigPng = Buffer.alloc(2_100_000);
bigPng.set([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a], 0);
await runNonStreamingChatSend({
context,
respond,
idempotencyKey: "idem-image-order",
message: "describe both",
requestParams: {
attachments: [
{
mimeType: "image/png",
content:
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=",
},
{
mimeType: "image/png",
content: bigPng.toString("base64"),
},
],
},
expectBroadcast: false,
});
expect(mockState.lastDispatchImages).toHaveLength(1);
expect(mockState.lastDispatchImageOrder).toEqual(["inline", "offloaded"]);
});
it("maps media offload failures to UNAVAILABLE in chat.send", async () => {
createTranscriptFixture("openclaw-chat-send-media-offload-error-");
mockState.sessionEntry = {
modelProvider: "test-provider",
model: "vision-model",
};
mockState.modelCatalog = [
{
provider: "test-provider",
id: "vision-model",
name: "Vision model",
input: ["text", "image"],
},
];
mockState.saveMediaError = new Error("disk full");
const respond = vi.fn();
const context = createChatContext();
const bigPng = Buffer.alloc(2_100_000);
bigPng.set([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a], 0);
await runNonStreamingChatSend({
context,
respond,
idempotencyKey: "idem-media-offload-error",
message: "describe image",
requestParams: {
attachments: [
{
mimeType: "image/png",
content: bigPng.toString("base64"),
},
],
},
waitFor: "none",
});
expect(respond).toHaveBeenCalledWith(
false,
undefined,
expect.objectContaining({ code: ErrorCodes.UNAVAILABLE }),
);
});
it("persists chat.send attachments one at a time", async () => {
createTranscriptFixture("openclaw-chat-send-image-serial-save-");
mockState.finalText = "ok";

View File

@@ -1,21 +1,20 @@
import fs from "node:fs";
import path from "node:path";
import { CURRENT_SESSION_VERSION, SessionManager } from "@mariozechner/pi-coding-agent";
import { resolveSendableOutboundReplyParts } from "openclaw/plugin-sdk/reply-payload";
import { resolveSessionAgentId } from "../../agents/agent-scope.js";
import { resolveThinkingDefault } from "../../agents/model-selection.js";
import { rewriteTranscriptEntriesInSessionFile } from "../../agents/pi-embedded-runner/transcript-rewrite.js";
import { resolveAgentTimeoutMs } from "../../agents/timeout.js";
import {
extractAssistantText as extractAssistantHistoryText,
hasAssistantPhaseMetadata,
} from "../../agents/tools/chat-history-text.js";
import { dispatchInboundMessage } from "../../auto-reply/dispatch.js";
import { createReplyDispatcher } from "../../auto-reply/reply/reply-dispatcher.js";
import type { MsgContext } from "../../auto-reply/templating.js";
import { isSilentReplyText, SILENT_REPLY_TOKEN } from "../../auto-reply/tokens.js";
import type { ReplyPayload } from "../../auto-reply/types.js";
import { extractCanvasFromText } from "../../chat/canvas-render.js";
import { resolveSessionFilePath } from "../../config/sessions.js";
import { formatErrorMessage } from "../../infra/errors.js";
import { jsonUtf8Bytes } from "../../infra/json-utf8-bytes.js";
import { isAudioFileName } from "../../media/mime.js";
import type { PromptImageOrderEntry } from "../../media/prompt-image-order.js";
import { type SavedMedia, saveMediaBuffer } from "../../media/store.js";
import { createChannelReplyPipeline } from "../../plugin-sdk/channel-reply-pipeline.js";
@@ -23,11 +22,7 @@ import { normalizeInputProvenance, type InputProvenance } from "../../sessions/i
import { resolveSendPolicy } from "../../sessions/send-policy.js";
import { parseAgentSessionKey } from "../../sessions/session-key-utils.js";
import { emitSessionTranscriptUpdate } from "../../sessions/transcript-events.js";
import { resolveAssistantMessagePhase } from "../../shared/chat-message-content.js";
import {
normalizeLowercaseStringOrEmpty,
normalizeOptionalString,
} from "../../shared/string-coerce.js";
import { isSuppressedControlReplyText } from "../control-reply-text.js";
import {
stripInlineDirectiveTagsForDisplay,
stripInlineDirectiveTagsFromMessageForDisplay,
@@ -47,13 +42,11 @@ import {
} from "../chat-abort.js";
import {
type ChatImageContent,
MediaOffloadError,
type OffloadedRef,
parseMessageWithAttachments,
} from "../chat-attachments.js";
import { stripEnvelopeFromMessage, stripEnvelopeFromMessages } from "../chat-sanitize.js";
import { augmentChatHistoryWithCliSessionImports } from "../cli-session-history.js";
import { isSuppressedControlReplyText } from "../control-reply-text.js";
import { ADMIN_SCOPE } from "../method-scopes.js";
import {
GATEWAY_CLIENT_CAPS,
@@ -84,12 +77,12 @@ import { injectTimestamp, timestampOptsFromConfig } from "./agent-timestamp.js";
import { setGatewayDedupeEntry } from "./agent-wait-dedupe.js";
import { normalizeRpcAttachmentsToChatAttachments } from "./attachment-normalize.js";
import { appendInjectedAssistantMessageToTranscript } from "./chat-transcript-inject.js";
import { buildWebchatAudioContentBlocksFromReplyPayloads } from "./chat-webchat-media.js";
import type {
GatewayRequestContext,
GatewayRequestHandlerOptions,
GatewayRequestHandlers,
} from "./types.js";
import { MediaOffloadError } from "../chat-attachments.js";
type TranscriptAppendResult = {
ok: boolean;
@@ -175,6 +168,35 @@ type SideResultPayload = {
ts: number;
};
function buildTranscriptReplyText(payloads: ReplyPayload[]): string {
const chunks = payloads
.map((payload) => {
const parts = resolveSendableOutboundReplyParts(payload);
const lines: string[] = [];
if (typeof payload.replyToId === "string" && payload.replyToId.trim()) {
lines.push(`[[reply_to:${payload.replyToId.trim()}]]`);
} else if (payload.replyToCurrent) {
lines.push("[[reply_to_current]]");
}
const text = payload.text?.trim();
if (text) {
lines.push(text);
}
for (const mediaUrl of parts.mediaUrls) {
const trimmed = mediaUrl.trim();
if (trimmed) {
lines.push(`MEDIA:${trimmed}`);
}
}
if (payload.audioAsVoice && parts.mediaUrls.some((mediaUrl) => isAudioFileName(mediaUrl))) {
lines.push("[[audio_as_voice]]");
}
return lines.join("\n").trim();
})
.filter(Boolean);
return chunks.join("\n\n").trim();
}
function resolveChatSendOriginatingRoute(params: {
client?: { mode?: string | null; id?: string | null } | null;
deliver?: boolean;
@@ -231,9 +253,9 @@ function resolveChatSendOriginatingRoute(params: {
.filter(Boolean);
const sessionScopeHead = sessionScopeParts[0];
const sessionChannelHint = normalizeMessageChannel(sessionScopeHead);
const normalizedSessionScopeHead = normalizeLowercaseStringOrEmpty(sessionScopeHead);
const normalizedSessionScopeHead = (sessionScopeHead ?? "").trim().toLowerCase();
const sessionPeerShapeCandidates = [sessionScopeParts[1], sessionScopeParts[2]]
.map((part) => normalizeLowercaseStringOrEmpty(part))
.map((part) => (part ?? "").trim().toLowerCase())
.filter(Boolean);
const isChannelAgnosticSessionScope = CHANNEL_AGNOSTIC_SESSION_SCOPES.has(
normalizedSessionScopeHead,
@@ -250,7 +272,7 @@ function resolveChatSendOriginatingRoute(params: {
const hasClientMetadata =
(typeof params.client?.mode === "string" && params.client.mode.trim().length > 0) ||
(typeof params.client?.id === "string" && params.client.id.trim().length > 0);
const configuredMainKey = normalizeLowercaseStringOrEmpty(params.mainKey ?? "main");
const configuredMainKey = (params.mainKey ?? "main").trim().toLowerCase();
const isConfiguredMainSessionScope =
normalizedSessionScopeHead.length > 0 && normalizedSessionScopeHead === configuredMainKey;
const canInheritConfiguredMainRoute =
@@ -344,17 +366,6 @@ function canInjectSystemProvenance(client: GatewayRequestHandlerOptions["client"
return scopes.includes(ADMIN_SCOPE);
}
/**
* Persist inline images and offloaded-ref media to the transcript media store.
*
* Inline images are re-saved from their base64 payload so that a stable
* filesystem path can be stored in the transcript. Offloaded refs are already
* on disk (saved by parseMessageWithAttachments); their SavedMedia metadata is
* synthesised directly from the OffloadedRef, avoiding a redundant write.
*
* Both sets are combined so that transcript media fields remain complete
* regardless of whether attachments were inlined or offloaded.
*/
async function persistChatSendImages(params: {
images: ChatImageContent[];
imageOrder: PromptImageOrderEntry[];
@@ -362,41 +373,56 @@ async function persistChatSendImages(params: {
client: GatewayRequestHandlerOptions["client"];
logGateway: GatewayRequestContext["logGateway"];
}): Promise<SavedMedia[]> {
if (isAcpBridgeClient(params.client)) {
if ((params.images.length === 0 && params.offloadedRefs.length === 0) || isAcpBridgeClient(params.client)) {
return [];
}
const saved: SavedMedia[] = [];
let inlineIndex = 0;
let offloadedIndex = 0;
for (const entry of params.imageOrder) {
if (entry === "offloaded") {
const ref = params.offloadedRefs[offloadedIndex++];
if (!ref) {
continue;
}
saved.push({
id: ref.id,
path: ref.path,
size: 0,
contentType: ref.mimeType,
});
continue;
}
const img = params.images[inlineIndex++];
if (!img) {
continue;
}
const inlineSaved: SavedMedia[] = [];
for (const img of params.images) {
try {
saved.push(await saveMediaBuffer(Buffer.from(img.data, "base64"), img.mimeType, "inbound"));
inlineSaved.push(await saveMediaBuffer(Buffer.from(img.data, "base64"), img.mimeType, "inbound"));
} catch (err) {
params.logGateway.warn(
`chat.send: failed to persist inbound image (${img.mimeType}): ${formatForLog(err)}`,
);
}
}
const offloadedSaved = params.offloadedRefs.map((ref) => ({
id: ref.id,
path: ref.path,
size: 0,
contentType: ref.mimeType,
}));
if (params.imageOrder.length === 0) {
return [...inlineSaved, ...offloadedSaved];
}
const saved: SavedMedia[] = [];
let inlineIndex = 0;
let offloadedIndex = 0;
for (const entry of params.imageOrder) {
if (entry === "inline") {
const inline = inlineSaved[inlineIndex++];
if (inline) {
saved.push(inline);
}
continue;
}
const offloaded = offloadedSaved[offloadedIndex++];
if (offloaded) {
saved.push(offloaded);
}
}
for (; inlineIndex < inlineSaved.length; inlineIndex++) {
const inline = inlineSaved[inlineIndex];
if (inline) {
saved.push(inline);
}
}
for (; offloadedIndex < offloadedSaved.length; offloadedIndex++) {
const offloaded = offloadedSaved[offloadedIndex];
if (offloaded) {
saved.push(offloaded);
}
}
return saved;
}
@@ -496,7 +522,7 @@ async function rewriteChatSendUserTurnMediaPaths(params: {
function truncateChatHistoryText(
text: string,
maxChars: number,
maxChars: number = DEFAULT_CHAT_HISTORY_TEXT_MAX_CHARS,
): { text: string; truncated: boolean } {
if (text.length <= maxChars) {
return { text, truncated: false };
@@ -507,30 +533,94 @@ function truncateChatHistoryText(
};
}
function isToolHistoryBlockType(type: unknown): boolean {
if (typeof type !== "string") {
return false;
}
const normalized = type.trim().toLowerCase();
return (
normalized === "toolcall" ||
normalized === "tool_call" ||
normalized === "tooluse" ||
normalized === "tool_use" ||
normalized === "toolresult" ||
normalized === "tool_result"
);
}
function extractChatHistoryBlockText(message: unknown): string | undefined {
if (!message || typeof message !== "object") {
return undefined;
}
const entry = message as Record<string, unknown>;
if (typeof entry.content === "string") {
return entry.content;
}
if (typeof entry.text === "string") {
return entry.text;
}
if (!Array.isArray(entry.content)) {
return undefined;
}
const textParts = entry.content
.map((block) => {
if (!block || typeof block !== "object") {
return undefined;
}
const typed = block as { text?: unknown; type?: unknown };
return typeof typed.text === "string" ? typed.text : undefined;
})
.filter((value): value is string => typeof value === "string");
return textParts.length > 0 ? textParts.join("\n") : undefined;
}
function sanitizeChatHistoryContentBlock(
block: unknown,
maxChars: number,
opts?: { preserveExactToolPayload?: boolean; maxChars?: number },
): { block: unknown; changed: boolean } {
if (!block || typeof block !== "object") {
return { block, changed: false };
}
const entry = { ...(block as Record<string, unknown>) };
let changed = false;
const preserveExactToolPayload =
opts?.preserveExactToolPayload === true || isToolHistoryBlockType(entry.type);
const maxChars = opts?.maxChars ?? DEFAULT_CHAT_HISTORY_TEXT_MAX_CHARS;
if (typeof entry.text === "string") {
const stripped = stripInlineDirectiveTagsForDisplay(entry.text);
const res = truncateChatHistoryText(stripped.text, maxChars);
entry.text = res.text;
changed ||= stripped.changed || res.truncated;
if (preserveExactToolPayload) {
entry.text = stripped.text;
changed ||= stripped.changed;
} else {
const res = truncateChatHistoryText(stripped.text, maxChars);
entry.text = res.text;
changed ||= stripped.changed || res.truncated;
}
}
if (typeof entry.content === "string") {
const stripped = stripInlineDirectiveTagsForDisplay(entry.content);
if (preserveExactToolPayload) {
entry.content = stripped.text;
changed ||= stripped.changed;
} else {
const res = truncateChatHistoryText(stripped.text, maxChars);
entry.content = res.text;
changed ||= stripped.changed || res.truncated;
}
}
if (typeof entry.partialJson === "string") {
const res = truncateChatHistoryText(entry.partialJson, maxChars);
entry.partialJson = res.text;
changed ||= res.truncated;
if (!preserveExactToolPayload) {
const res = truncateChatHistoryText(entry.partialJson, maxChars);
entry.partialJson = res.text;
changed ||= res.truncated;
}
}
if (typeof entry.arguments === "string") {
const res = truncateChatHistoryText(entry.arguments, maxChars);
entry.arguments = res.text;
changed ||= res.truncated;
if (!preserveExactToolPayload) {
const res = truncateChatHistoryText(entry.arguments, maxChars);
entry.arguments = res.text;
changed ||= res.truncated;
}
}
if (typeof entry.thinking === "string") {
const res = truncateChatHistoryText(entry.thinking, maxChars);
@@ -616,13 +706,23 @@ function sanitizeCost(raw: unknown): { total?: number } | undefined {
function sanitizeChatHistoryMessage(
message: unknown,
maxChars: number,
maxChars: number = DEFAULT_CHAT_HISTORY_TEXT_MAX_CHARS,
): { message: unknown; changed: boolean } {
if (!message || typeof message !== "object") {
return { message, changed: false };
}
const entry = { ...(message as Record<string, unknown>) };
let changed = false;
const role = typeof entry.role === "string" ? entry.role.toLowerCase() : "";
const preserveExactToolPayload =
role === "toolresult" ||
role === "tool_result" ||
role === "tool" ||
role === "function" ||
typeof entry.toolName === "string" ||
typeof entry.tool_name === "string" ||
typeof entry.toolCallId === "string" ||
typeof entry.tool_call_id === "string";
if ("details" in entry) {
delete entry.details;
@@ -664,35 +764,34 @@ function sanitizeChatHistoryMessage(
if (typeof entry.content === "string") {
const stripped = stripInlineDirectiveTagsForDisplay(entry.content);
const res = truncateChatHistoryText(stripped.text, maxChars);
entry.content = res.text;
changed ||= stripped.changed || res.truncated;
} else if (Array.isArray(entry.content)) {
const updated = entry.content.map((block) => sanitizeChatHistoryContentBlock(block, maxChars));
const sanitizedBlocks = updated.map((item) => item.block);
const hasPhaseMetadata = hasAssistantPhaseMetadata(entry);
if (hasPhaseMetadata) {
const stripped = stripInlineDirectiveTagsForDisplay(extractAssistantHistoryText(entry) ?? "");
if (preserveExactToolPayload) {
entry.content = stripped.text;
changed ||= stripped.changed;
} else {
const res = truncateChatHistoryText(stripped.text, maxChars);
const nonTextBlocks = sanitizedBlocks.filter(
(block) =>
!block || typeof block !== "object" || (block as { type?: unknown }).type !== "text",
);
entry.content = res.text
? [{ type: "text", text: res.text }, ...nonTextBlocks]
: nonTextBlocks;
changed = true;
} else if (updated.some((item) => item.changed)) {
entry.content = sanitizedBlocks;
entry.content = res.text;
changed ||= stripped.changed || res.truncated;
}
} else if (Array.isArray(entry.content)) {
const updated = entry.content.map((block) =>
sanitizeChatHistoryContentBlock(block, { preserveExactToolPayload, maxChars }),
);
if (updated.some((item) => item.changed)) {
entry.content = updated.map((item) => item.block);
changed = true;
}
}
if (typeof entry.text === "string") {
const stripped = stripInlineDirectiveTagsForDisplay(entry.text);
const res = truncateChatHistoryText(stripped.text, maxChars);
entry.text = res.text;
changed ||= stripped.changed || res.truncated;
if (preserveExactToolPayload) {
entry.text = stripped.text;
changed ||= stripped.changed;
} else {
const res = truncateChatHistoryText(stripped.text, maxChars);
entry.text = res.text;
changed ||= stripped.changed || res.truncated;
}
}
return { message: changed ? entry : message, changed };
@@ -705,7 +804,35 @@ function sanitizeChatHistoryMessage(
* dropping messages that carry real text alongside a stale `content: "NO_REPLY"`.
*/
function extractAssistantTextForSilentCheck(message: unknown): string | undefined {
return extractAssistantHistoryText(message);
if (!message || typeof message !== "object") {
return undefined;
}
const entry = message as Record<string, unknown>;
if (entry.role !== "assistant") {
return undefined;
}
if (typeof entry.text === "string") {
return entry.text;
}
if (typeof entry.content === "string") {
return entry.content;
}
if (!Array.isArray(entry.content) || entry.content.length === 0) {
return undefined;
}
const texts: string[] = [];
for (const block of entry.content) {
if (!block || typeof block !== "object") {
return undefined;
}
const typed = block as { type?: unknown; text?: unknown };
if (typed.type !== "text" || typeof typed.text !== "string") {
return undefined;
}
texts.push(typed.text);
}
return texts.length > 0 ? texts.join("\n") : undefined;
}
function hasAssistantNonTextContent(message: unknown): boolean {
@@ -725,10 +852,11 @@ function shouldDropAssistantHistoryMessage(message: unknown): boolean {
if (!message || typeof message !== "object") {
return false;
}
if ((message as { role?: unknown }).role !== "assistant") {
const entry = message as { role?: unknown; phase?: unknown };
if (entry.role !== "assistant") {
return false;
}
if (resolveAssistantMessagePhase(message) === "commentary") {
if (entry.phase === "commentary") {
return true;
}
const text = extractAssistantTextForSilentCheck(message);
@@ -738,24 +866,22 @@ function shouldDropAssistantHistoryMessage(message: unknown): boolean {
return !hasAssistantNonTextContent(message);
}
export function sanitizeChatHistoryMessages(messages: unknown[], maxChars: number): unknown[] {
export function sanitizeChatHistoryMessages(
messages: unknown[],
maxChars: number = DEFAULT_CHAT_HISTORY_TEXT_MAX_CHARS,
): unknown[] {
if (messages.length === 0) {
return messages;
}
let changed = false;
const next: unknown[] = [];
for (const message of messages) {
// Drop raw control-token replies before any maxChars truncation can make
// an exact token look like partial user-visible text.
if (shouldDropAssistantHistoryMessage(message)) {
changed = true;
continue;
}
const res = sanitizeChatHistoryMessage(message, maxChars);
changed ||= res.changed;
// Drop assistant commentary-only entries and exact control replies, but
// keep mixed assistant entries that still carry non-text content. Run this
// again after sanitizing so display-only cleanup can still suppress stale tokens.
if (shouldDropAssistantHistoryMessage(res.message)) {
changed = true;
continue;
@@ -765,6 +891,149 @@ export function sanitizeChatHistoryMessages(messages: unknown[], maxChars: numbe
return changed ? next : messages;
}
function appendCanvasBlockToAssistantHistoryMessage(params: {
message: unknown;
preview: ReturnType<typeof extractCanvasFromText>;
rawText: string | null;
}): unknown {
const preview = params.preview;
if (!preview || !params.message || typeof params.message !== "object") {
return params.message;
}
const entry = params.message as Record<string, unknown>;
const baseContent = Array.isArray(entry.content)
? [...entry.content]
: typeof entry.content === "string"
? [{ type: "text", text: entry.content }]
: typeof entry.text === "string"
? [{ type: "text", text: entry.text }]
: [];
const alreadyPresent = baseContent.some((block) => {
if (!block || typeof block !== "object") {
return false;
}
const typed = block as { type?: unknown; preview?: unknown };
return (
typed.type === "canvas" &&
typed.preview &&
typeof typed.preview === "object" &&
(((typed.preview as { viewId?: unknown }).viewId &&
(typed.preview as { viewId?: unknown }).viewId === preview.viewId) ||
((typed.preview as { url?: unknown }).url &&
(typed.preview as { url?: unknown }).url === preview.url))
);
});
if (!alreadyPresent) {
baseContent.push({
type: "canvas",
preview,
rawText: params.rawText,
});
}
return {
...entry,
content: baseContent,
};
}
function messageContainsToolHistoryContent(message: unknown): boolean {
if (!message || typeof message !== "object") {
return false;
}
const entry = message as Record<string, unknown>;
if (
typeof entry.toolCallId === "string" ||
typeof entry.tool_call_id === "string" ||
typeof entry.toolName === "string" ||
typeof entry.tool_name === "string"
) {
return true;
}
if (!Array.isArray(entry.content)) {
return false;
}
return entry.content.some((block) => {
if (!block || typeof block !== "object") {
return false;
}
return isToolHistoryBlockType((block as { type?: unknown }).type);
});
}
function augmentChatHistoryWithCanvasBlocks(messages: unknown[]): unknown[] {
if (messages.length === 0) {
return messages;
}
const next = [...messages];
let changed = false;
let lastAssistantIndex = -1;
let lastRenderableAssistantIndex = -1;
const pending: Array<{
preview: NonNullable<ReturnType<typeof extractCanvasFromText>>;
rawText: string | null;
}> = [];
for (let index = 0; index < next.length; index++) {
const message = next[index];
if (!message || typeof message !== "object") {
continue;
}
const entry = message as Record<string, unknown>;
const role = typeof entry.role === "string" ? entry.role.toLowerCase() : "";
if (role === "assistant") {
lastAssistantIndex = index;
if (!messageContainsToolHistoryContent(entry)) {
lastRenderableAssistantIndex = index;
if (pending.length > 0) {
let target = next[index];
for (const item of pending) {
target = appendCanvasBlockToAssistantHistoryMessage({
message: target,
preview: item.preview,
rawText: item.rawText,
});
}
next[index] = target;
pending.length = 0;
changed = true;
}
}
continue;
}
const toolName =
typeof entry.toolName === "string"
? entry.toolName
: typeof entry.tool_name === "string"
? entry.tool_name
: undefined;
const text = extractChatHistoryBlockText(entry);
const preview = extractCanvasFromText(text, toolName);
if (!preview) {
continue;
}
pending.push({
preview,
rawText: text ?? null,
});
}
if (pending.length > 0) {
const targetIndex =
lastRenderableAssistantIndex >= 0 ? lastRenderableAssistantIndex : lastAssistantIndex;
if (targetIndex >= 0) {
let target = next[targetIndex];
for (const item of pending) {
target = appendCanvasBlockToAssistantHistoryMessage({
message: target,
preview: item.preview,
rawText: item.rawText,
});
}
next[targetIndex] = target;
changed = true;
}
}
return changed ? next : messages;
}
function buildOversizedHistoryPlaceholder(message?: unknown): Record<string, unknown> {
const role =
message &&
@@ -871,7 +1140,7 @@ function ensureTranscriptFile(params: { transcriptPath: string; sessionId: strin
});
return { ok: true };
} catch (err) {
return { ok: false, error: formatErrorMessage(err) };
return { ok: false, error: err instanceof Error ? err.message : String(err) };
}
}
@@ -895,8 +1164,6 @@ function transcriptHasIdempotencyKey(transcriptPath: string, idempotencyKey: str
function appendAssistantTranscriptMessage(params: {
message: string;
/** Rich Pi message blocks (text, embedded audio, etc.). Overrides plain `message` when set. */
content?: Array<Record<string, unknown>>;
label?: string;
sessionId: string;
storePath: string | undefined;
@@ -941,7 +1208,6 @@ function appendAssistantTranscriptMessage(params: {
transcriptPath,
message: params.message,
label: params.label,
content: params.content,
idempotencyKey: params.idempotencyKey,
abortMeta: params.abortMeta,
});
@@ -1018,13 +1284,18 @@ function createChatAbortOps(context: GatewayRequestContext): ChatAbortOps {
};
}
function normalizeOptionalText(value?: string | null): string | undefined {
const trimmed = value?.trim();
return trimmed || undefined;
}
function normalizeExplicitChatSendOrigin(
params: ChatSendExplicitOrigin,
): { ok: true; value?: ChatSendExplicitOrigin } | { ok: false; error: string } {
const originatingChannel = normalizeOptionalString(params.originatingChannel);
const originatingTo = normalizeOptionalString(params.originatingTo);
const accountId = normalizeOptionalString(params.accountId);
const messageThreadId = normalizeOptionalString(params.messageThreadId);
const originatingChannel = normalizeOptionalText(params.originatingChannel);
const originatingTo = normalizeOptionalText(params.originatingTo);
const accountId = normalizeOptionalText(params.accountId);
const messageThreadId = normalizeOptionalText(params.messageThreadId);
const hasAnyExplicitOriginField = Boolean(
originatingChannel || originatingTo || accountId || messageThreadId,
);
@@ -1060,8 +1331,8 @@ function resolveChatAbortRequester(
): ChatAbortRequester {
const scopes = Array.isArray(client?.connect?.scopes) ? client.connect.scopes : [];
return {
connId: normalizeOptionalString(client?.connId),
deviceId: normalizeOptionalString(client?.connect?.device?.id),
connId: normalizeOptionalText(client?.connId),
deviceId: normalizeOptionalText(client?.connect?.device?.id),
isAdmin: scopes.includes(ADMIN_SCOPE),
};
}
@@ -1073,8 +1344,8 @@ function canRequesterAbortChatRun(
if (requester.isAdmin) {
return true;
}
const ownerDeviceId = normalizeOptionalString(entry.ownerDeviceId);
const ownerConnId = normalizeOptionalString(entry.ownerConnId);
const ownerDeviceId = normalizeOptionalText(entry.ownerDeviceId);
const ownerConnId = normalizeOptionalText(entry.ownerConnId);
if (!ownerDeviceId && !ownerConnId) {
return true;
}
@@ -1245,25 +1516,13 @@ export const chatHandlers: GatewayRequestHandlers = {
);
return;
}
const {
sessionKey,
limit,
maxChars: rpcMaxChars,
} = params as {
const { sessionKey, limit } = params as {
sessionKey: string;
limit?: number;
maxChars?: number;
};
const { cfg, storePath, entry } = loadSessionEntry(sessionKey);
const configMaxChars = cfg.gateway?.webchat?.chatHistoryMaxChars;
const effectiveMaxChars =
typeof rpcMaxChars === "number"
? rpcMaxChars
: typeof configMaxChars === "number"
? configMaxChars
: DEFAULT_CHAT_HISTORY_TEXT_MAX_CHARS;
const sessionAgentId = resolveSessionAgentId({ sessionKey, config: cfg });
const sessionId = entry?.sessionId;
const sessionAgentId = resolveSessionAgentId({ sessionKey, config: cfg });
const resolvedSessionModel = resolveSessionModelRef(cfg, entry, sessionAgentId);
const localMessages =
sessionId && storePath ? readSessionMessages(sessionId, storePath, entry?.sessionFile) : [];
@@ -1278,7 +1537,7 @@ export const chatHandlers: GatewayRequestHandlers = {
const max = Math.min(hardMax, requested);
const sliced = rawMessages.length > max ? rawMessages.slice(-max) : rawMessages;
const sanitized = stripEnvelopeFromMessages(sliced);
const normalized = sanitizeChatHistoryMessages(sanitized, effectiveMaxChars);
const normalized = augmentChatHistoryWithCanvasBlocks(sanitizeChatHistoryMessages(sanitized));
const maxHistoryBytes = getMaxChatHistoryMessagesBytes();
const perMessageHardCap = Math.min(CHAT_HISTORY_MAX_SINGLE_MESSAGE_BYTES, maxHistoryBytes);
const replaced = replaceOversizedChatHistoryMessages({
@@ -1296,16 +1555,11 @@ export const chatHandlers: GatewayRequestHandlers = {
}
let thinkingLevel = entry?.thinkingLevel;
if (!thinkingLevel) {
const sessionAgentId = resolveSessionAgentId({
sessionKey,
config: cfg,
});
const resolvedModel = resolveSessionModelRef(cfg, entry, sessionAgentId);
const catalog = await context.loadGatewayModelCatalog();
thinkingLevel = resolveThinkingDefault({
cfg,
provider: resolvedModel.provider,
model: resolvedModel.model,
provider: resolvedSessionModel.provider,
model: resolvedSessionModel.model,
catalog,
});
}
@@ -1486,18 +1740,41 @@ export const chatHandlers: GatewayRequestHandlers = {
);
return;
}
// Load session entry before attachment parsing so we can gate media-URI
// marker injection on the model's image capability. This prevents opaque
// media:// markers from leaking into prompts for text-only model runs.
const rawSessionKey = p.sessionKey;
const { cfg, entry, canonicalKey: sessionKey } = loadSessionEntry(rawSessionKey);
let parsedMessage = inboundMessage;
let parsedImages: ChatImageContent[] = [];
let parsedImageOrder: PromptImageOrderEntry[] = [];
let parsedOffloadedRefs: OffloadedRef[] = [];
let imageOrder: PromptImageOrderEntry[] = [];
let offloadedRefs: OffloadedRef[] = [];
if (normalizedAttachments.length > 0) {
const modelRef = resolveSessionModelRef(cfg, entry, undefined);
const supportsImages = await resolveGatewayModelSupportsImages({
loadGatewayModelCatalog: context.loadGatewayModelCatalog,
provider: modelRef.provider,
model: modelRef.model,
});
try {
const parsed = await parseMessageWithAttachments(inboundMessage, normalizedAttachments, {
maxBytes: 5_000_000,
log: context.logGateway,
supportsImages,
});
parsedMessage = parsed.message;
parsedImages = parsed.images;
imageOrder = parsed.imageOrder;
offloadedRefs = parsed.offloadedRefs;
} catch (err) {
respond(
false,
undefined,
errorShape(
err instanceof MediaOffloadError ? ErrorCodes.UNAVAILABLE : ErrorCodes.INVALID_REQUEST,
String(err),
),
);
return;
}
}
const timeoutMs = resolveAgentTimeoutMs({
cfg,
overrideMs: p.timeoutMs,
@@ -1555,43 +1832,6 @@ export const chatHandlers: GatewayRequestHandlers = {
return;
}
if (normalizedAttachments.length > 0) {
const sessionAgentId = resolveSessionAgentId({ sessionKey, config: cfg });
const modelRef = resolveSessionModelRef(cfg, entry, sessionAgentId);
const supportsImages = await resolveGatewayModelSupportsImages({
loadGatewayModelCatalog: context.loadGatewayModelCatalog,
provider: modelRef.provider,
model: modelRef.model,
});
try {
const parsed = await parseMessageWithAttachments(inboundMessage, normalizedAttachments, {
maxBytes: 5_000_000,
log: context.logGateway,
supportsImages,
});
parsedMessage = parsed.message;
parsedImages = parsed.images;
parsedImageOrder = parsed.imageOrder;
parsedOffloadedRefs = parsed.offloadedRefs;
} catch (err) {
// MediaOffloadError indicates a server-side storage fault (ENOSPC, EPERM,
// etc.). All other errors are client-side input validation failures.
// Map them to different HTTP status codes so callers can retry server
// faults without treating them as bad requests.
const isServerFault = err instanceof MediaOffloadError;
respond(
false,
undefined,
errorShape(
isServerFault ? ErrorCodes.UNAVAILABLE : ErrorCodes.INVALID_REQUEST,
String(err),
),
);
return;
}
}
try {
const abortController = new AbortController();
context.chatAbortControllers.set(clientRunId, {
@@ -1600,23 +1840,18 @@ export const chatHandlers: GatewayRequestHandlers = {
sessionKey: rawSessionKey,
startedAtMs: now,
expiresAtMs: resolveChatRunExpiresAtMs({ now, timeoutMs }),
ownerConnId: normalizeOptionalString(client?.connId),
ownerDeviceId: normalizeOptionalString(client?.connect?.device?.id),
ownerConnId: normalizeOptionalText(client?.connId),
ownerDeviceId: normalizeOptionalText(client?.connect?.device?.id),
});
const ackPayload = {
runId: clientRunId,
status: "started" as const,
};
respond(true, ackPayload, undefined, { runId: clientRunId });
// Persist both inline images and already-offloaded refs to the media
// store so that transcript media fields remain complete for all attachment
// sizes. Offloaded refs are already on disk; persistChatSendImages converts
// their metadata without re-writing the files.
const persistedImagesPromise = persistChatSendImages({
images: parsedImages,
imageOrder: parsedImageOrder,
offloadedRefs: parsedOffloadedRefs,
imageOrder,
offloadedRefs,
client,
logGateway: context.logGateway,
});
@@ -1647,7 +1882,7 @@ export const chatHandlers: GatewayRequestHandlers = {
});
// Inject timestamp so agents know the current date/time.
// Only BodyForAgent gets the timestamp — Body stays raw for UI display.
// See: https://github.com/openclaw/openclaw/issues/3658
// See: https://github.com/moltbot/moltbot/issues/3658
const stampedMessage = injectTimestamp(messageForAgent, timestampOptsFromConfig(cfg));
const ctx: MsgContext = {
@@ -1776,7 +2011,7 @@ export const chatHandlers: GatewayRequestHandlers = {
runId: clientRunId,
abortSignal: abortController.signal,
images: parsedImages.length > 0 ? parsedImages : undefined,
imageOrder: parsedImageOrder.length > 0 ? parsedImageOrder : undefined,
imageOrder: imageOrder.length > 0 ? imageOrder : undefined,
onAgentRunStart: (runId) => {
agentRunStarted = true;
void emitUserTranscriptUpdate();
@@ -1831,33 +2066,18 @@ export const chatHandlers: GatewayRequestHandlers = {
sessionKey,
});
} else {
const finalPayloads = deliveredReplies
const combinedReply = buildTranscriptReplyText(
deliveredReplies
.filter((entry) => entry.kind === "final")
.map((entry) => entry.payload);
const combinedReply = finalPayloads
.map((part) => part.text?.trim() ?? "")
.filter(Boolean)
.join("\n\n")
.trim();
const audioBlocks = buildWebchatAudioContentBlocksFromReplyPayloads(finalPayloads);
const assistantContent: Array<Record<string, unknown>> = [];
if (combinedReply) {
assistantContent.push({ type: "text", text: combinedReply });
} else if (audioBlocks.length > 0) {
assistantContent.push({ type: "text", text: "Audio reply" });
}
assistantContent.push(...audioBlocks);
.map((entry) => entry.payload)
);
let message: Record<string, unknown> | undefined;
if (assistantContent.length > 0) {
if (combinedReply) {
const { storePath: latestStorePath, entry: latestEntry } =
loadSessionEntry(sessionKey);
const sessionId = latestEntry?.sessionId ?? entry?.sessionId ?? clientRunId;
const transcriptFallbackText =
combinedReply || (audioBlocks.length > 0 ? "Audio reply" : "");
const appended = appendAssistantTranscriptMessage({
message: transcriptFallbackText,
content: assistantContent,
message: combinedReply,
sessionId,
storePath: latestStorePath,
sessionFile: latestEntry?.sessionFile,
@@ -1873,7 +2093,7 @@ export const chatHandlers: GatewayRequestHandlers = {
const now = Date.now();
message = {
role: "assistant",
content: assistantContent,
content: [{ type: "text", text: combinedReply }],
timestamp: now,
// Keep this compatible with Pi stopReason enums even though this message isn't
// persisted to the transcript due to the append failure.

View File

@@ -91,7 +91,7 @@ function makeWsClient(params: {
connId: string;
clientIp: string;
role: "node" | "operator";
mode: "node" | "backend";
mode: "node" | "backend" | "webchat";
canvasCapability?: string;
canvasCapabilityExpiresAtMs?: number;
}): GatewayWsClient {
@@ -219,7 +219,7 @@ describe("gateway canvas host auth", () => {
handleHttpRequest: allowCanvasHostHttp,
run: async ({ listener, clients }) => {
const host = "127.0.0.1";
const operatorOnlyCapability = "operator-only";
const webchatCapability = "webchat-cap";
const expiredNodeCapability = "expired-node";
const activeNodeCapability = "active-node";
const activeCanvasPath = scopedCanvasPath(activeNodeCapability, `${CANVAS_HOST_PATH}/`);
@@ -235,19 +235,19 @@ describe("gateway canvas host auth", () => {
clients.add(
makeWsClient({
connId: "c-operator",
connId: "c-webchat",
clientIp: "192.168.1.10",
role: "operator",
mode: "backend",
canvasCapability: operatorOnlyCapability,
mode: "webchat",
canvasCapability: webchatCapability,
canvasCapabilityExpiresAtMs: Date.now() + 60_000,
}),
);
const operatorCapabilityBlocked = await fetch(
`http://${host}:${listener.port}${scopedCanvasPath(operatorOnlyCapability, `${CANVAS_HOST_PATH}/`)}`,
const webchatCapabilityAllowed = await fetch(
`http://${host}:${listener.port}${scopedCanvasPath(webchatCapability, `${CANVAS_HOST_PATH}/`)}`,
);
expect(operatorCapabilityBlocked.status).toBe(401);
expect(webchatCapabilityAllowed.status).toBe(200);
clients.add(
makeWsClient({

View File

@@ -450,6 +450,32 @@ describe("gateway server hooks", () => {
});
});
test("rejects mapped hook session rebinding into a disallowed target-agent prefix", async () => {
testState.hooksConfig = {
enabled: true,
token: HOOK_TOKEN,
allowRequestSessionKey: true,
allowedSessionKeyPrefixes: ["hook:", "agent:main:"],
mappings: [
{
match: { path: "mapped-rebind-denied" },
action: "agent",
agentId: "hooks",
messageTemplate: "Mapped: {{payload.subject}}",
sessionKey: "agent:main:slack:channel:c123",
},
],
};
setMainAndHooksAgents();
await withGatewayServer(async ({ port }) => {
const denied = await postHook(port, "/hooks/mapped-rebind-denied", { subject: "hello" });
expect(denied.status).toBe(400);
const body = (await denied.json()) as { error?: string };
expect(body.error).toContain("sessionKey must start with one of");
expect(cronIsolatedRun).not.toHaveBeenCalled();
});
});
test("dedupes repeated /hooks/agent deliveries by idempotency key", async () => {
testState.hooksConfig = { enabled: true, token: HOOK_TOKEN };
await withGatewayServer(async ({ port }) => {

View File

@@ -9,7 +9,6 @@ import {
} from "../auth.js";
import { CANVAS_CAPABILITY_TTL_MS } from "../canvas-capability.js";
import { getBearerToken, resolveHttpBrowserOriginPolicy } from "../http-utils.js";
import { GATEWAY_CLIENT_MODES, normalizeGatewayClientMode } from "../protocol/client-info.js";
import type { GatewayWsClient } from "./ws-types.js";
export function isCanvasPath(pathname: string): boolean {
@@ -22,22 +21,12 @@ export function isCanvasPath(pathname: string): boolean {
);
}
function isNodeWsClient(client: GatewayWsClient): boolean {
if (client.connect.role === "node") {
return true;
}
return normalizeGatewayClientMode(client.connect.client.mode) === GATEWAY_CLIENT_MODES.NODE;
}
function hasAuthorizedNodeWsClientForCanvasCapability(
function hasAuthorizedWsClientForCanvasCapability(
clients: Set<GatewayWsClient>,
capability: string,
): boolean {
const nowMs = Date.now();
for (const client of clients) {
if (!isNodeWsClient(client)) {
continue;
}
if (!client.canvasCapability || !client.canvasCapabilityExpiresAtMs) {
continue;
}
@@ -95,7 +84,7 @@ export async function authorizeCanvasRequest(params: {
lastAuthFailure = authResult;
}
if (canvasCapability && hasAuthorizedNodeWsClientForCanvasCapability(clients, canvasCapability)) {
if (canvasCapability && hasAuthorizedWsClientForCanvasCapability(clients, canvasCapability)) {
return { ok: true };
}
return lastAuthFailure ?? { ok: false, reason: "unauthorized" };

View File

@@ -21,7 +21,6 @@ import {
attachGatewayWsMessageHandler,
type WsOriginCheckMetrics,
} from "./ws-connection/message-handler.js";
import { resolveSharedGatewaySessionGeneration } from "./ws-shared-generation.js";
import type { GatewayWsClient } from "./ws-types.js";
type SubsystemLogger = ReturnType<typeof createSubsystemLogger>;
@@ -107,8 +106,6 @@ export function attachGatewayWsConnectionHandler(params: AttachGatewayWsConnecti
canvasHostServerPort,
resolvedAuth,
getResolvedAuth = () => resolvedAuth,
getRequiredSharedGatewaySessionGeneration = () =>
resolveSharedGatewaySessionGeneration(getResolvedAuth()),
rateLimiter,
browserRateLimiter,
gatewayMethods,
@@ -321,8 +318,7 @@ export function attachGatewayWsConnectionHandler(params: AttachGatewayWsConnecti
requestUserAgent,
canvasHostUrl,
connectNonce,
getResolvedAuth,
getRequiredSharedGatewaySessionGeneration,
resolvedAuth: getResolvedAuth(),
rateLimiter,
browserRateLimiter,
gatewayMethods,

View File

@@ -40,7 +40,6 @@ import {
type DeviceBootstrapProfile,
} from "../../../shared/device-bootstrap-profile.js";
import { roleScopesAllow } from "../../../shared/operator-scope-compat.js";
import { normalizeOptionalString } from "../../../shared/string-coerce.js";
import {
isBrowserOperatorUiClient,
isGatewayCliClient,
@@ -161,6 +160,10 @@ export function attachGatewayWsMessageHandler(params: {
upgradeReq: IncomingMessage;
connId: string;
remoteAddr?: string;
remotePort?: number;
localAddr?: string;
localPort?: number;
endpoint?: string;
forwardedFor?: string;
realIp?: string;
requestHost?: string;
@@ -168,8 +171,8 @@ export function attachGatewayWsMessageHandler(params: {
requestUserAgent?: string;
canvasHostUrl?: string;
connectNonce: string;
getResolvedAuth: () => ResolvedGatewayAuth;
getRequiredSharedGatewaySessionGeneration: () => string | undefined;
resolvedAuth: ResolvedGatewayAuth;
getRequiredSharedGatewaySessionGeneration?: () => string | undefined;
/** Optional rate limiter for auth brute-force protection. */
rateLimiter?: AuthRateLimiter;
/** Browser-origin fallback limiter (loopback is never exempt). */
@@ -197,6 +200,10 @@ export function attachGatewayWsMessageHandler(params: {
upgradeReq,
connId,
remoteAddr,
remotePort,
localAddr,
localPort,
endpoint,
forwardedFor,
realIp,
requestHost,
@@ -204,7 +211,7 @@ export function attachGatewayWsMessageHandler(params: {
requestUserAgent,
canvasHostUrl,
connectNonce,
getResolvedAuth,
resolvedAuth,
getRequiredSharedGatewaySessionGeneration,
rateLimiter,
browserRateLimiter,
@@ -248,6 +255,7 @@ export function attachGatewayWsMessageHandler(params: {
trustedProxies,
allowRealIpFallback,
});
const peerLabel = endpoint ?? remoteAddr ?? "n/a";
// If proxy headers are present but the remote address isn't trusted, don't treat
// the connection as local. This prevents auth bypass when running behind a reverse
@@ -258,7 +266,6 @@ export function attachGatewayWsMessageHandler(params: {
const hasUntrustedProxyHeaders = hasProxyHeaders && !remoteIsTrustedProxy;
const hostIsLocalish = isLocalishHost(requestHost);
const isLocalClient = isLocalDirectRequest(upgradeReq, trustedProxies, allowRealIpFallback);
const reportedClientIp =
isLocalClient || hasUntrustedProxyHeaders
? undefined
@@ -369,7 +376,7 @@ export function attachGatewayWsMessageHandler(params: {
});
} else {
logWsControl.warn(
`invalid handshake conn=${connId} remote=${remoteAddr ?? "?"} fwd=${forwardedFor ?? "n/a"} origin=${requestOrigin ?? "n/a"} host=${requestHost ?? "n/a"} ua=${requestUserAgent ?? "n/a"}`,
`invalid handshake conn=${connId} peer=${formatForLog(peerLabel)} remote=${remoteAddr ?? "?"} fwd=${formatForLog(forwardedFor ?? "n/a")} origin=${formatForLog(requestOrigin ?? "n/a")} host=${formatForLog(requestHost ?? "n/a")} ua=${formatForLog(requestUserAgent ?? "n/a")}`,
);
}
const closeReason = truncateCloseReason(handshakeError || "invalid handshake");
@@ -389,6 +396,10 @@ export function attachGatewayWsMessageHandler(params: {
clientDisplayName: connectParams.client.displayName,
mode: connectParams.client.mode,
version: connectParams.client.version,
platform: connectParams.client.platform,
deviceFamily: connectParams.client.deviceFamily,
modelIdentifier: connectParams.client.modelIdentifier,
instanceId: connectParams.client.instanceId,
};
const markHandshakeFailure = (cause: string, meta?: Record<string, unknown>) => {
setHandshakeState("failed");
@@ -445,7 +456,6 @@ export function attachGatewayWsMessageHandler(params: {
const isControlUi = isOperatorUiClient(connectParams.client);
const isBrowserOperatorUi = isBrowserOperatorUiClient(connectParams.client);
const isWebchat = isWebchatConnect(connectParams);
const resolvedAuth = getResolvedAuth();
if (enforceOriginCheckForAnyClient || isBrowserOperatorUi || isWebchat) {
const hostHeaderOriginFallbackEnabled =
configSnapshot.gateway?.controlUi?.dangerouslyAllowHostHeaderOriginFallback === true;
@@ -529,9 +539,17 @@ export function attachGatewayWsMessageHandler(params: {
authProvided,
authReason: failedAuth.reason,
allowTailscale: resolvedAuth.allowTailscale,
peer: peerLabel,
remoteAddr,
remotePort,
localAddr,
localPort,
role,
scopeCount: scopes.length,
hasDeviceIdentity: Boolean(device),
});
logWsControl.warn(
`unauthorized conn=${connId} remote=${remoteAddr ?? "?"} client=${clientLabel} ${connectParams.client.mode} v${connectParams.client.version} reason=${failedAuth.reason ?? "unknown"}`,
`unauthorized conn=${connId} peer=${formatForLog(peerLabel)} remote=${remoteAddr ?? "?"} client=${formatForLog(clientLabel)} ${connectParams.client.mode} v${formatForLog(connectParams.client.version)} role=${role} scopes=${scopes.length} auth=${authProvided} device=${device ? "yes" : "no"} platform=${formatForLog(connectParams.client.platform)} instance=${formatForLog(connectParams.client.instanceId ?? "n/a")} host=${formatForLog(requestHost ?? "n/a")} origin=${formatForLog(requestOrigin ?? "n/a")} ua=${formatForLog(requestUserAgent ?? "n/a")} reason=${failedAuth.reason ?? "unknown"}`,
);
const authMessage = formatGatewayAuthFailureMessage({
authMode: resolvedAuth.mode,
@@ -660,7 +678,7 @@ export function attachGatewayWsMessageHandler(params: {
rejectDeviceAuthInvalid("device-signature-stale", "device signature expired");
return;
}
const providedNonce = normalizeOptionalString(device.nonce) ?? "";
const providedNonce = typeof device.nonce === "string" ? device.nonce.trim() : "";
if (!providedNonce) {
rejectDeviceAuthInvalid("device-nonce-missing", "device nonce required");
return;
@@ -723,14 +741,15 @@ export function attachGatewayWsMessageHandler(params: {
rejectUnauthorized(authResult);
return;
}
const sharedGatewaySessionGeneration =
authMethod === "token" || authMethod === "password"
? resolveSharedGatewaySessionGeneration(resolvedAuth)
: undefined;
if (authMethod === "token" || authMethod === "password") {
const sharedGatewaySessionGeneration =
resolveSharedGatewaySessionGeneration(resolvedAuth);
const requiredSharedGatewaySessionGeneration =
getRequiredSharedGatewaySessionGeneration();
if (sharedGatewaySessionGeneration !== requiredSharedGatewaySessionGeneration) {
getRequiredSharedGatewaySessionGeneration?.();
if (
requiredSharedGatewaySessionGeneration !== undefined &&
sharedGatewaySessionGeneration !== requiredSharedGatewaySessionGeneration
) {
setCloseCause("gateway-auth-rotated", {
authGenerationStale: true,
});
@@ -816,8 +835,6 @@ export function attachGatewayWsMessageHandler(params: {
displayName: connectParams.client.displayName,
clientId: connectParams.client.id,
clientMode: connectParams.client.mode,
role,
scopes,
remoteIp: reportedClientIp,
};
const requirePairing = async (
@@ -872,9 +889,6 @@ export function attachGatewayWsMessageHandler(params: {
isWebchat,
reason,
});
// QR bootstrap onboarding stays single-use, but the first node bootstrap handshake
// should seed bounded device tokens and only consume the bootstrap token once the
// hello-ok path succeeds so reconnects can recover from pre-hello failures.
const allowSilentBootstrapPairing =
authMethod === "bootstrap-token" &&
reason === "not-paired" &&
@@ -917,14 +931,21 @@ export function attachGatewayWsMessageHandler(params: {
return replacementPending?.requestId;
};
if (pairing.request.silent === true) {
approved = bootstrapProfileForSilentApproval
? await approveBootstrapDevicePairing(
pairing.request.requestId,
bootstrapProfileForSilentApproval,
)
: await approveDevicePairing(pairing.request.requestId, {
callerScopes: scopes,
});
const requestedOperatorScopes = scopes.filter((scope) =>
scope.startsWith("operator."),
);
approved =
bootstrapProfileForSilentApproval
? await approveBootstrapDevicePairing(
pairing.request.requestId,
bootstrapProfileForSilentApproval,
)
: requestedOperatorScopes.length > 0
? {
status: "forbidden" as const,
missingScope: requestedOperatorScopes[0] ?? "callerScopes-required",
}
: await approveDevicePairing(pairing.request.requestId);
if (approved?.status === "approved") {
if (bootstrapProfileForSilentApproval) {
handoffBootstrapProfile = bootstrapProfileForSilentApproval;
@@ -994,10 +1015,6 @@ export function attachGatewayWsMessageHandler(params: {
const isPaired = paired?.publicKey === devicePublicKey;
if (!isPaired) {
if (!(skipLocalBackendSelfPairing || skipControlUiPairingForDevice)) {
// Initial local backend/control-ui self-pairing can bypass the
// pairing prompt, but only while the device is still unpaired.
// Once a device is paired, reconnects must stay inside the
// approved role/scope baseline below.
const ok = await requirePairing("not-paired", paired);
if (!ok) {
return;
@@ -1193,11 +1210,14 @@ export function attachGatewayWsMessageHandler(params: {
snapshot.health = cachedHealth;
snapshot.stateVersion.health = getHealthVersion();
}
const canvasCapability =
role === "node" && canvasHostUrl ? mintCanvasCapabilityToken() : undefined;
const canvasCapability = canvasHostUrl ? mintCanvasCapabilityToken() : undefined;
const canvasCapabilityExpiresAtMs = canvasCapability
? Date.now() + CANVAS_CAPABILITY_TTL_MS
: undefined;
const usesSharedGatewayAuth = authMethod === "token" || authMethod === "password";
const sharedGatewaySessionGeneration = usesSharedGatewayAuth
? resolveSharedGatewaySessionGeneration(resolvedAuth)
: undefined;
const scopedCanvasHostUrl =
canvasHostUrl && canvasCapability
? (buildCanvasScopedHostUrl(canvasHostUrl, canvasCapability) ?? canvasHostUrl)
@@ -1235,7 +1255,7 @@ export function attachGatewayWsMessageHandler(params: {
socket,
connect: connectParams,
connId,
usesSharedGatewayAuth: authMethod === "token" || authMethod === "password",
usesSharedGatewayAuth,
sharedGatewaySessionGeneration,
presenceKey,
clientIp: reportedClientIp,
@@ -1252,7 +1272,7 @@ export function attachGatewayWsMessageHandler(params: {
remoteIp: reportedClientIp,
});
const instanceIdRaw = connectParams.client.instanceId;
const instanceId = normalizeOptionalString(instanceIdRaw) ?? "";
const instanceId = typeof instanceIdRaw === "string" ? instanceIdRaw.trim() : "";
const nodeIdsForPairing = new Set<string>([nodeSession.nodeId]);
if (instanceId) {
nodeIdsForPairing.add(instanceId);
@@ -1350,17 +1370,6 @@ export function attachGatewayWsMessageHandler(params: {
return;
}
if (client.usesSharedGatewayAuth) {
const requiredSharedGatewaySessionGeneration = getRequiredSharedGatewaySessionGeneration();
if (client.sharedGatewaySessionGeneration !== requiredSharedGatewaySessionGeneration) {
setCloseCause("gateway-auth-rotated", {
authGenerationStale: true,
});
close(4001, "gateway auth changed");
return;
}
}
// After handshake, accept only req frames
if (!validateRequestFrame(parsed)) {
send({

View File

@@ -1,3 +1,4 @@
import { EventEmitter } from "node:events";
import type { ServerResponse } from "node:http";
import { vi } from "vitest";
@@ -7,12 +8,34 @@ export function makeMockHttpResponse(): {
end: ReturnType<typeof vi.fn>;
} {
const setHeader = vi.fn();
const end = vi.fn();
const emitter = new EventEmitter();
const res = {
headersSent: false,
statusCode: 200,
writable: true,
writableEnded: false,
setHeader,
end,
write: vi.fn(() => true),
once: emitter.once.bind(emitter),
on: emitter.on.bind(emitter),
emit: emitter.emit.bind(emitter),
removeListener: emitter.removeListener.bind(emitter),
destroy: vi.fn(() => {
res.writableEnded = true;
emitter.emit("close");
return res;
}),
} as unknown as ServerResponse;
const end = vi.fn((chunk?: unknown) => {
if (chunk !== undefined) {
(res.write as unknown as ReturnType<typeof vi.fn>)(chunk);
}
res.headersSent = true;
res.writableEnded = true;
emitter.emit("finish");
emitter.emit("close");
return res;
});
(res as ServerResponse & { end: typeof end }).end = end;
return { res, setHeader, end };
}

View File

@@ -307,7 +307,7 @@ export async function readLocalFileSafely(params: {
filePath: string;
maxBytes?: number;
}): Promise<SafeLocalReadResult> {
const opened = await openVerifiedLocalFile(params.filePath);
const opened = await openLocalFileSafely({ filePath: params.filePath });
try {
return await readOpenedFileSafely({ opened, maxBytes: params.maxBytes });
} finally {
@@ -315,6 +315,10 @@ export async function readLocalFileSafely(params: {
}
}
export async function openLocalFileSafely(params: { filePath: string }): Promise<SafeOpenResult> {
return await openVerifiedLocalFile(params.filePath);
}
async function readOpenedFileSafely(params: {
opened: SafeOpenResult;
maxBytes?: number;

View File

@@ -1,9 +1,8 @@
import os from "node:os";
import path from "node:path";
import { pathToFileURL } from "node:url";
import { afterEach, describe, expect, it, vi } from "vitest";
import {
buildMediaLocalRoots,
appendLocalMediaParentRoots,
getAgentScopedMediaLocalRoots,
getAgentScopedMediaLocalRootsForSources,
getDefaultMediaLocalRoots,
@@ -53,14 +52,6 @@ describe("local media roots", () => {
expect(normalizedRoots).not.toContain(picturesRoot);
}
function expectPicturesRootAbsent(roots: readonly string[], picturesRoot?: string) {
expectPicturesRootPresence({
roots,
shouldContainPictures: false,
picturesRoot,
});
}
function expectAgentMediaRootsCase(params: {
stateDir: string;
getRoots: () => readonly string[];
@@ -86,12 +77,12 @@ describe("local media roots", () => {
it.each([
{
name: "keeps temp, media cache, and workspace roots by default",
name: "keeps temp, media cache, canvas, and workspace roots by default",
stateDir: path.join("/tmp", "openclaw-media-roots-state"),
getRoots: () => getDefaultMediaLocalRoots(),
expectedContained: ["media", "workspace", "sandboxes"],
expectedContained: ["media", "canvas", "workspace", "sandboxes"],
expectedExcluded: ["agents"],
minLength: 3,
minLength: 4,
},
{
name: "adds the active agent workspace without re-opening broad agent state roots",
@@ -110,12 +101,38 @@ describe("local media roots", () => {
});
});
it("adds concrete parent roots for local media sources without widening to filesystem root", () => {
const picturesDir =
process.platform === "win32" ? "C:\\Users\\peter\\Pictures" : "/Users/peter/Pictures";
const moviesDir =
process.platform === "win32" ? "C:\\Users\\peter\\Movies" : "/Users/peter/Movies";
const roots = appendLocalMediaParentRoots(
["/tmp/base"],
[
path.join(picturesDir, "photo.png"),
pathToFileURL(path.join(moviesDir, "clip.mp4")).href,
"https://example.com/remote.png",
"/top-level-file.png",
],
);
expect(roots.map(normalizeHostPath)).toEqual(
expect.arrayContaining([
normalizeHostPath("/tmp/base"),
normalizeHostPath(picturesDir),
normalizeHostPath(moviesDir),
]),
);
expect(roots.map(normalizeHostPath)).not.toContain(normalizeHostPath("/"));
});
it.each([
{
name: "does not widen agent media roots for concrete local sources when workspaceOnly is disabled",
name: "widens agent media roots for concrete local sources when workspaceOnly is disabled",
stateDir: path.join("/tmp", "openclaw-flexible-media-roots-state"),
cfg: {},
shouldContainPictures: false,
shouldContainPictures: true,
},
{
name: "does not widen agent media roots when workspaceOnly is enabled",
@@ -130,7 +147,7 @@ describe("local media roots", () => {
shouldContainPictures: false,
},
{
name: "does not widen media roots even when messaging-profile agents explicitly enable filesystem tools",
name: "widens media roots again when messaging-profile agents explicitly enable filesystem tools",
stateDir: path.join("/tmp", "openclaw-messaging-fs-media-roots-state"),
cfg: {
tools: {
@@ -138,7 +155,7 @@ describe("local media roots", () => {
fs: { workspaceOnly: false },
},
},
shouldContainPictures: false,
shouldContainPictures: true,
},
] as const)("$name", ({ stateDir, cfg, shouldContainPictures }) => {
const roots = withStateDir(stateDir, () =>
@@ -150,48 +167,4 @@ describe("local media roots", () => {
);
expectPicturesRootPresence({ roots, shouldContainPictures });
});
it("keeps agent-scoped defaults even when mediaSources include file URLs and top-level paths", () => {
const stateDir = path.join("/tmp", "openclaw-file-url-media-roots-state");
const picturesDir =
process.platform === "win32" ? "C:\\Users\\peter\\Pictures" : "/Users/peter/Pictures";
const moviesDir =
process.platform === "win32" ? "C:\\Users\\peter\\Movies" : "/Users/peter/Movies";
const roots = withStateDir(stateDir, () =>
getAgentScopedMediaLocalRootsForSources({
cfg: {},
agentId: "ops",
mediaSources: [
path.join(picturesDir, "photo.png"),
pathToFileURL(path.join(moviesDir, "clip.mp4")).href,
"/top-level-file.png",
],
}),
);
expectNormalizedRootsContain(roots, [
path.join(stateDir, "media"),
path.join(stateDir, "workspace"),
path.join(stateDir, "workspace-ops"),
]);
expectPicturesRootAbsent(roots, picturesDir);
expectPicturesRootAbsent(roots, moviesDir);
expect(roots.map(normalizeHostPath)).not.toContain(normalizeHostPath("/"));
});
it("includes the config media root when legacy state and config dirs diverge", () => {
const homeRoot = path.join(os.tmpdir(), "openclaw-legacy-home-test");
const roots = buildMediaLocalRoots(
path.join(homeRoot, ".clawdbot"),
path.join(homeRoot, ".openclaw"),
);
expectNormalizedRootsContain(roots, [
path.join(homeRoot, ".clawdbot", "media"),
path.join(homeRoot, ".clawdbot", "workspace"),
path.join(homeRoot, ".clawdbot", "sandboxes"),
path.join(homeRoot, ".openclaw", "media"),
]);
});
});

View File

@@ -1,16 +1,23 @@
import path from "node:path";
import { resolveAgentWorkspaceDir } from "../agents/agent-scope.js";
import {
resolveEffectiveToolFsRootExpansionAllowed,
resolveEffectiveToolFsWorkspaceOnly,
} from "../agents/tool-fs-policy.js";
import type { OpenClawConfig } from "../config/config.js";
import { resolveStateDir } from "../config/paths.js";
import { safeFileURLToPath } from "../infra/local-file-access.js";
import { resolvePreferredOpenClawTmpDir } from "../infra/tmp-openclaw-dir.js";
import { normalizeOptionalString } from "../shared/string-coerce.js";
import { resolveConfigDir } from "../utils.js";
import { resolveUserPath } from "../utils.js";
type BuildMediaLocalRootsOptions = {
preferredTmpDir?: string;
};
let cachedPreferredTmpDir: string | undefined;
const HTTP_URL_RE = /^https?:\/\//i;
const DATA_URL_RE = /^data:/i;
const WINDOWS_DRIVE_RE = /^[A-Za-z]:[\\/]/;
function resolveCachedPreferredTmpDir(): string {
if (!cachedPreferredTmpDir) {
@@ -19,43 +26,34 @@ function resolveCachedPreferredTmpDir(): string {
return cachedPreferredTmpDir;
}
export function buildMediaLocalRoots(
function buildMediaLocalRoots(
stateDir: string,
configDir: string,
options: BuildMediaLocalRootsOptions = {},
): string[] {
const resolvedStateDir = path.resolve(stateDir);
const resolvedConfigDir = path.resolve(configDir);
const preferredTmpDir = options.preferredTmpDir ?? resolveCachedPreferredTmpDir();
return Array.from(
new Set([
preferredTmpDir,
path.join(resolvedStateDir, "media"),
path.join(resolvedStateDir, "workspace"),
path.join(resolvedStateDir, "sandboxes"),
// Upgraded installs can still resolve the active state dir to the legacy
// ~/.clawdbot tree while new media writes already go under ~/.openclaw/media.
// Keep inbound media readable across that split without widening roots beyond
// the managed media cache.
path.join(resolvedConfigDir, "media"),
]),
);
return [
preferredTmpDir,
path.join(resolvedStateDir, "media"),
path.join(resolvedStateDir, "canvas"),
path.join(resolvedStateDir, "workspace"),
path.join(resolvedStateDir, "sandboxes"),
];
}
export function getDefaultMediaLocalRoots(): readonly string[] {
return buildMediaLocalRoots(resolveStateDir(), resolveConfigDir());
return buildMediaLocalRoots(resolveStateDir());
}
export function getAgentScopedMediaLocalRoots(
cfg: OpenClawConfig,
agentId?: string,
): readonly string[] {
const roots = buildMediaLocalRoots(resolveStateDir(), resolveConfigDir());
const normalizedAgentId = normalizeOptionalString(agentId);
if (!normalizedAgentId) {
const roots = buildMediaLocalRoots(resolveStateDir());
if (!agentId?.trim()) {
return roots;
}
const workspaceDir = resolveAgentWorkspaceDir(cfg, normalizedAgentId);
const workspaceDir = resolveAgentWorkspaceDir(cfg, agentId);
if (!workspaceDir) {
return roots;
}
@@ -66,24 +64,60 @@ export function getAgentScopedMediaLocalRoots(
return roots;
}
/**
* @deprecated Kept for plugin-sdk compatibility. Media sources no longer widen allowed roots.
*/
export function appendLocalMediaParentRoots(
roots: readonly string[],
_mediaSources?: readonly string[],
): string[] {
return Array.from(new Set(roots.map((root) => path.resolve(root))));
function resolveLocalMediaPath(source: string): string | undefined {
const trimmed = source.trim();
if (!trimmed || HTTP_URL_RE.test(trimmed) || DATA_URL_RE.test(trimmed)) {
return undefined;
}
if (trimmed.startsWith("file://")) {
try {
return safeFileURLToPath(trimmed);
} catch {
return undefined;
}
}
if (trimmed.startsWith("~")) {
return resolveUserPath(trimmed);
}
if (path.isAbsolute(trimmed) || WINDOWS_DRIVE_RE.test(trimmed)) {
return path.resolve(trimmed);
}
return undefined;
}
export function getAgentScopedMediaLocalRootsForSources({
cfg,
agentId,
mediaSources: _mediaSources,
}: {
export function appendLocalMediaParentRoots(
roots: readonly string[],
mediaSources?: readonly string[],
): string[] {
const appended = Array.from(new Set(roots.map((root) => path.resolve(root))));
for (const source of mediaSources ?? []) {
const localPath = resolveLocalMediaPath(source);
if (!localPath) {
continue;
}
const parentDir = path.dirname(localPath);
if (parentDir === path.parse(parentDir).root) {
continue;
}
const normalizedParent = path.resolve(parentDir);
if (!appended.includes(normalizedParent)) {
appended.push(normalizedParent);
}
}
return appended;
}
export function getAgentScopedMediaLocalRootsForSources(params: {
cfg: OpenClawConfig;
agentId?: string;
mediaSources?: readonly string[];
}): readonly string[] {
return getAgentScopedMediaLocalRoots(cfg, agentId);
const roots = getAgentScopedMediaLocalRoots(params.cfg, params.agentId);
if (resolveEffectiveToolFsWorkspaceOnly({ cfg: params.cfg, agentId: params.agentId })) {
return roots;
}
if (!resolveEffectiveToolFsRootExpansionAllowed({ cfg: params.cfg, agentId: params.agentId })) {
return roots;
}
return appendLocalMediaParentRoots(roots, params.mediaSources);
}

View File

@@ -1,17 +1,7 @@
import path from "node:path";
import {
normalizeLowercaseStringOrEmpty,
normalizeOptionalLowercaseString,
} from "../shared/string-coerce.js";
import { fileTypeFromBuffer } from "file-type";
import { type MediaKind, mediaKindFromMime } from "./constants.js";
let fileTypeModulePromise: Promise<typeof import("file-type")> | undefined;
function loadFileTypeModule(): Promise<typeof import("file-type")> {
fileTypeModulePromise ??= import("file-type");
return fileTypeModulePromise;
}
// Map common mimes to preferred file extensions.
const EXT_BY_MIME: Record<string, string> = {
"image/heic": ".heic",
@@ -46,10 +36,6 @@ const EXT_BY_MIME: Record<string, string> = {
"text/csv": ".csv",
"text/plain": ".txt",
"text/markdown": ".md",
"text/html": ".html",
"text/xml": ".xml",
"text/css": ".css",
"application/xml": ".xml",
};
const MIME_BY_EXT: Record<string, string> = {
@@ -57,8 +43,6 @@ const MIME_BY_EXT: Record<string, string> = {
// Additional extension aliases
".jpeg": "image/jpeg",
".js": "text/javascript",
".htm": "text/html",
".xml": "text/xml", // pin text/xml as canonical (application/xml also maps to .xml in EXT_BY_MIME)
};
const AUDIO_FILE_EXTENSIONS = new Set([
@@ -74,7 +58,11 @@ const AUDIO_FILE_EXTENSIONS = new Set([
]);
export function normalizeMimeType(mime?: string | null): string | undefined {
return normalizeOptionalLowercaseString(mime?.split(";")[0]);
if (!mime) {
return undefined;
}
const cleaned = mime.split(";")[0]?.trim().toLowerCase();
return cleaned || undefined;
}
async function sniffMime(buffer?: Buffer): Promise<string | undefined> {
@@ -82,7 +70,6 @@ async function sniffMime(buffer?: Buffer): Promise<string | undefined> {
return undefined;
}
try {
const { fileTypeFromBuffer } = await loadFileTypeModule();
const type = await fileTypeFromBuffer(buffer);
return type?.mime ?? undefined;
} catch {
@@ -97,15 +84,23 @@ export function getFileExtension(filePath?: string | null): string | undefined {
try {
if (/^https?:\/\//i.test(filePath)) {
const url = new URL(filePath);
return normalizeLowercaseStringOrEmpty(path.extname(url.pathname)) || undefined;
return path.extname(url.pathname).toLowerCase() || undefined;
}
} catch {
// fall back to plain path parsing
}
const ext = normalizeLowercaseStringOrEmpty(path.extname(filePath));
const ext = path.extname(filePath).toLowerCase();
return ext || undefined;
}
export function mimeTypeFromFilePath(filePath?: string | null): string | undefined {
const ext = getFileExtension(filePath);
if (!ext) {
return undefined;
}
return MIME_BY_EXT[ext];
}
export function isAudioFileName(fileName?: string | null): boolean {
const ext = getFileExtension(fileName);
if (!ext) {
@@ -126,7 +121,7 @@ function isGenericMime(mime?: string): boolean {
if (!mime) {
return true;
}
const m = normalizeLowercaseStringOrEmpty(mime);
const m = mime.toLowerCase();
return m === "application/octet-stream" || m === "application/zip";
}
@@ -174,7 +169,7 @@ export function isGifMedia(opts: {
contentType?: string | null;
fileName?: string | null;
}): boolean {
if (normalizeOptionalLowercaseString(opts.contentType) === "image/gif") {
if (opts.contentType?.toLowerCase() === "image/gif") {
return true;
}
const ext = getFileExtension(opts.fileName);
@@ -185,7 +180,7 @@ export function imageMimeFromFormat(format?: string | null): string | undefined
if (!format) {
return undefined;
}
switch (normalizeLowercaseStringOrEmpty(format)) {
switch (format.toLowerCase()) {
case "jpg":
case "jpeg":
return "image/jpeg";

View File

@@ -91,4 +91,16 @@ describe("splitMediaFromOutput", () => {
expectStableAudioAsVoiceDetectionCase(input);
}
});
it("returns ordered text and media segments while ignoring fenced MEDIA lines", () => {
const result = splitMediaFromOutput(
"Before\nMEDIA:https://example.com/a.png\n```text\nMEDIA:https://example.com/ignored.png\n```\nAfter",
);
expect(result.segments).toEqual([
{ type: "text", text: "Before" },
{ type: "media", url: "https://example.com/a.png" },
{ type: "text", text: "```text\nMEDIA:https://example.com/ignored.png\n```\nAfter" },
]);
});
});

View File

@@ -6,6 +6,16 @@ import { parseAudioTag } from "./audio-tags.js";
// Allow optional wrapping backticks and punctuation after the token; capture the core token.
export const MEDIA_TOKEN_RE = /\bMEDIA:\s*`?([^\n]+)`?/gi;
export type ParsedMediaOutputSegment =
| {
type: "text";
text: string;
}
| {
type: "media";
url: string;
};
export function normalizeMediaSource(src: string) {
return src.startsWith("file://") ? src.replace("file://", "") : src;
}
@@ -125,6 +135,7 @@ export function splitMediaFromOutput(raw: string): {
mediaUrls?: string[];
mediaUrl?: string; // legacy first item for backward compatibility
audioAsVoice?: boolean; // true if [[audio_as_voice]] tag was found
segments?: ParsedMediaOutputSegment[];
} {
// KNOWN: Leading whitespace is semantically meaningful in Markdown (lists, indented fences).
// We only trim the end; token cleanup below handles removing `MEDIA:` lines.
@@ -140,6 +151,19 @@ export function splitMediaFromOutput(raw: string): {
const media: string[] = [];
let foundMediaToken = false;
const segments: ParsedMediaOutputSegment[] = [];
const pushTextSegment = (text: string) => {
if (!text) {
return;
}
const last = segments[segments.length - 1];
if (last?.type === "text") {
last.text = `${last.text}\n${text}`;
return;
}
segments.push({ type: "text", text });
};
// Parse fenced code blocks to avoid extracting MEDIA tokens from inside them
const hasFenceMarkers = mayContainFenceMarkers(trimmedRaw);
@@ -154,6 +178,7 @@ export function splitMediaFromOutput(raw: string): {
// Skip MEDIA extraction if this line is inside a fenced code block
if (hasFenceMarkers && isInsideFence(fenceSpans, lineOffset)) {
keptLines.push(line);
pushTextSegment(line);
lineOffset += line.length + 1; // +1 for newline
continue;
}
@@ -161,6 +186,7 @@ export function splitMediaFromOutput(raw: string): {
const trimmedStart = line.trimStart();
if (!trimmedStart.startsWith("MEDIA:")) {
keptLines.push(line);
pushTextSegment(line);
lineOffset += line.length + 1; // +1 for newline
continue;
}
@@ -168,11 +194,13 @@ export function splitMediaFromOutput(raw: string): {
const matches = Array.from(line.matchAll(MEDIA_TOKEN_RE));
if (matches.length === 0) {
keptLines.push(line);
pushTextSegment(line);
lineOffset += line.length + 1; // +1 for newline
continue;
}
const pieces: string[] = [];
const lineSegments: ParsedMediaOutputSegment[] = [];
let cursor = 0;
for (const match of matches) {
@@ -219,6 +247,17 @@ export function splitMediaFromOutput(raw: string): {
}
}
if (!hasValidMedia && !unwrapped && /\s/.test(payloadValue)) {
const spacedFallback = normalizeMediaSource(cleanCandidate(payloadValue));
if (isValidMedia(spacedFallback, { allowSpaces: true, allowBareFilename: true })) {
media.splice(mediaStartIndex, media.length - mediaStartIndex, spacedFallback);
hasValidMedia = true;
foundMediaToken = true;
validCount = 1;
invalidParts.length = 0;
}
}
if (!hasValidMedia) {
const fallback = normalizeMediaSource(cleanCandidate(payloadValue));
if (isValidMedia(fallback, { allowSpaces: true, allowBareFilename: true })) {
@@ -230,6 +269,17 @@ export function splitMediaFromOutput(raw: string): {
}
if (hasValidMedia) {
const beforeText = pieces
.join("")
.replace(/[ \t]{2,}/g, " ")
.trim();
if (beforeText) {
lineSegments.push({ type: "text", text: beforeText });
}
pieces.length = 0;
for (const url of media.slice(mediaStartIndex, mediaStartIndex + validCount)) {
lineSegments.push({ type: "media", url });
}
if (invalidParts.length > 0) {
pieces.push(invalidParts.join(" "));
}
@@ -255,6 +305,14 @@ export function splitMediaFromOutput(raw: string): {
// If the line becomes empty, drop it.
if (cleanedLine) {
keptLines.push(cleanedLine);
lineSegments.push({ type: "text", text: cleanedLine });
}
for (const segment of lineSegments) {
if (segment.type === "text") {
pushTextSegment(segment.text);
continue;
}
segments.push(segment);
}
lineOffset += line.length + 1; // +1 for newline
}
@@ -274,9 +332,10 @@ export function splitMediaFromOutput(raw: string): {
}
if (media.length === 0) {
const parsedText = foundMediaToken || hasAudioAsVoice ? cleanedText : trimmedRaw;
const result: ReturnType<typeof splitMediaFromOutput> = {
// Return cleaned text if we found a media token OR audio tag, otherwise original
text: foundMediaToken || hasAudioAsVoice ? cleanedText : trimmedRaw,
text: parsedText,
segments: parsedText ? [{ type: "text", text: parsedText }] : [],
};
if (hasAudioAsVoice) {
result.audioAsVoice = true;
@@ -288,6 +347,7 @@ export function splitMediaFromOutput(raw: string): {
text: cleanedText,
mediaUrls: media,
mediaUrl: media[0],
segments: segments.length > 0 ? segments : [{ type: "text", text: cleanedText }],
...(hasAudioAsVoice ? { audioAsVoice: true } : {}),
};
}

View File

@@ -2,47 +2,48 @@ import fs from "node:fs/promises";
import path from "node:path";
import { pathToFileURL } from "node:url";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { CANVAS_HOST_PATH } from "../canvas-host/a2ui.js";
import { resolveStateDir } from "../config/paths.js";
import { resolvePreferredOpenClawTmpDir } from "../infra/tmp-openclaw-dir.js";
import { createSuiteTempRootTracker } from "../test-helpers/temp-dir.js";
import { createJpegBufferWithDimensions, createPngBufferWithDimensions } from "./test-helpers.js";
let loadWebMedia: typeof import("./web-media.js").loadWebMedia;
const mediaRootTracker = createSuiteTempRootTracker({
prefix: "web-media-core-",
parentDir: resolvePreferredOpenClawTmpDir(),
});
const TINY_PNG_BASE64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=";
let fixtureRoot = "";
let fakePdfFile = "";
let oversizedJpegFile = "";
let realPdfFile = "";
let tinyPngFile = "";
let stateDir = "";
let canvasPngFile = "";
beforeAll(async () => {
({ loadWebMedia } = await import("./web-media.js"));
await mediaRootTracker.setup();
fixtureRoot = await mediaRootTracker.make("case");
fakePdfFile = path.join(fixtureRoot, "fake.pdf");
realPdfFile = path.join(fixtureRoot, "real.pdf");
fixtureRoot = await fs.mkdtemp(path.join(resolvePreferredOpenClawTmpDir(), "web-media-core-"));
tinyPngFile = path.join(fixtureRoot, "tiny.png");
oversizedJpegFile = path.join(fixtureRoot, "oversized.jpg");
await fs.writeFile(fakePdfFile, "TOP_SECRET_TEXT", "utf8");
await fs.writeFile(
realPdfFile,
Buffer.from("%PDF-1.4\n1 0 obj\n<<>>\nendobj\ntrailer\n<<>>\n%%EOF"),
);
await fs.writeFile(tinyPngFile, Buffer.from(TINY_PNG_BASE64, "base64"));
await fs.writeFile(
oversizedJpegFile,
createJpegBufferWithDimensions({ width: 6_000, height: 5_000 }),
stateDir = resolveStateDir();
canvasPngFile = path.join(
stateDir,
"canvas",
"documents",
"cv_test",
"collection.media",
"tiny.png",
);
await fs.mkdir(path.dirname(canvasPngFile), { recursive: true });
await fs.writeFile(canvasPngFile, Buffer.from(TINY_PNG_BASE64, "base64"));
});
afterAll(async () => {
await mediaRootTracker.cleanup();
if (fixtureRoot) {
await fs.rm(fixtureRoot, { recursive: true, force: true });
}
if (stateDir) {
await fs.rm(path.join(stateDir, "canvas", "documents", "cv_test"), {
recursive: true,
force: true,
});
}
});
describe("loadWebMedia", () => {
@@ -108,24 +109,6 @@ describe("loadWebMedia", () => {
await expectLoadedWebMediaCase(createUrl());
});
it("rejects oversized pixel-count images before decode/resize backends run", async () => {
const oversizedPngFile = path.join(fixtureRoot, "oversized.png");
await fs.writeFile(
oversizedPngFile,
createPngBufferWithDimensions({ width: 8_000, height: 4_000 }),
);
await expect(loadWebMedia(oversizedPngFile, createLocalWebMediaOptions())).rejects.toThrow(
/pixel input limit/i,
);
});
it("preserves pixel-limit errors for oversized JPEG optimization", async () => {
await expect(loadWebMedia(oversizedJpegFile, createLocalWebMediaOptions())).rejects.toThrow(
/pixel input limit/i,
);
});
it.each([
{
name: "rejects remote-host file URLs before filesystem checks",
@@ -147,67 +130,35 @@ describe("loadWebMedia", () => {
await expectRejectedWebMediaWithoutFilesystemAccess(testCase);
});
describe("workspaceDir relative path resolution", () => {
it("resolves a bare filename against workspaceDir", async () => {
const result = await loadWebMedia("tiny.png", {
...createLocalWebMediaOptions(),
workspaceDir: fixtureRoot,
});
expect(result.kind).toBe("image");
expect(result.buffer.length).toBeGreaterThan(0);
});
it("loads browser-style canvas media paths as managed local files", async () => {
const result = await loadWebMedia(
`${CANVAS_HOST_PATH}/documents/cv_test/collection.media/tiny.png`,
{ maxBytes: 1024 * 1024 },
);
expect(result.kind).toBe("image");
expect(result.buffer.length).toBeGreaterThan(0);
});
it("resolves a dot-relative path against workspaceDir", async () => {
const result = await loadWebMedia("./tiny.png", {
...createLocalWebMediaOptions(),
workspaceDir: fixtureRoot,
});
expect(result.kind).toBe("image");
expect(result.buffer.length).toBeGreaterThan(0);
});
it("resolves a MEDIA:-prefixed relative path against workspaceDir", async () => {
const result = await loadWebMedia("MEDIA:tiny.png", {
...createLocalWebMediaOptions(),
workspaceDir: fixtureRoot,
});
expect(result.kind).toBe("image");
expect(result.buffer.length).toBeGreaterThan(0);
});
it("leaves absolute paths unchanged when workspaceDir is set", async () => {
const result = await loadWebMedia(tinyPngFile, {
...createLocalWebMediaOptions(),
workspaceDir: "/some/other/dir",
});
expect(result.kind).toBe("image");
expect(result.buffer.length).toBeGreaterThan(0);
it("rejects host-read text files outside local roots", async () => {
const secretFile = path.join(fixtureRoot, "secret.txt");
await fs.writeFile(secretFile, "secret", "utf8");
await expect(
loadWebMedia(secretFile, {
maxBytes: 1024 * 1024,
localRoots: "any",
readFile: async (filePath) => await fs.readFile(filePath),
hostReadCapability: true,
}),
).rejects.toMatchObject({
code: "path-not-allowed",
});
});
describe("host read capability", () => {
it("rejects document uploads that only match by file extension", async () => {
await expect(
loadWebMedia(fakePdfFile, {
maxBytes: 1024 * 1024,
localRoots: [fixtureRoot],
hostReadCapability: true,
}),
).rejects.toMatchObject({
code: "path-not-allowed",
});
});
it("still allows real PDF uploads detected from file content", async () => {
const result = await loadWebMedia(realPdfFile, {
maxBytes: 1024 * 1024,
localRoots: [fixtureRoot],
hostReadCapability: true,
});
expect(result.kind).toBe("document");
expect(result.contentType).toBe("application/pdf");
expect(result.fileName).toBe("real.pdf");
it("rejects traversal-style canvas media paths before filesystem access", async () => {
await expect(
loadWebMedia(`${CANVAS_HOST_PATH}/documents/../collection.media/tiny.png`),
).rejects.toMatchObject({
code: "path-not-allowed",
});
});
});

View File

@@ -1,19 +1,15 @@
import path from "node:path";
import { resolveCanvasHttpPathToLocalPath } from "../gateway/canvas-documents.js";
import { logVerbose, shouldLogVerbose } from "../globals.js";
import { SafeOpenError, readLocalFileSafely } from "../infra/fs-safe.js";
import { assertNoWindowsNetworkPath, safeFileURLToPath } from "../infra/local-file-access.js";
import type { SsrFPolicy } from "../infra/net/ssrf.js";
import {
normalizeLowercaseStringOrEmpty,
normalizeOptionalString,
} from "../shared/string-coerce.js";
import { resolveUserPath } from "../utils.js";
import { maxBytesForKind, type MediaKind } from "./constants.js";
import { fetchRemoteMedia } from "./fetch.js";
import {
convertHeicToJpeg,
hasAlphaChannel,
MAX_IMAGE_INPUT_PIXELS,
optimizeImageToPng,
resizeToJpeg,
} from "./image-ops.js";
@@ -46,8 +42,6 @@ type WebMediaOptions = {
readFile?: (filePath: string) => Promise<Buffer>;
/** Host-local fs-policy read piggyback; rejects plaintext-like document sends. */
hostReadCapability?: boolean;
/** Agent workspace directory for resolving relative MEDIA: paths. */
workspaceDir?: string;
};
function resolveWebMediaOptions(params: {
@@ -96,18 +90,11 @@ function formatCapReduce(label: string, cap: number, size: number): string {
return `${label} could not be reduced below ${formatMb(cap, 0)}MB (got ${formatMb(size)}MB)`;
}
function isPixelLimitError(error: unknown): boolean {
return (
error instanceof Error &&
error.message.includes(`${MAX_IMAGE_INPUT_PIXELS.toLocaleString("en-US")} pixel input limit`)
);
}
function isHeicSource(opts: { contentType?: string; fileName?: string }): boolean {
if (HEIC_MIME_RE.test(normalizeOptionalString(opts.contentType) ?? "")) {
if (opts.contentType && HEIC_MIME_RE.test(opts.contentType.trim())) {
return true;
}
if (HEIC_EXT_RE.test(normalizeOptionalString(opts.fileName) ?? "")) {
if (opts.fileName && HEIC_EXT_RE.test(opts.fileName.trim())) {
return true;
}
return false;
@@ -120,15 +107,8 @@ function assertHostReadMediaAllowed(params: {
if (params.kind === "image" || params.kind === "audio" || params.kind === "video") {
return;
}
if (params.kind !== "document") {
const contentType = normalizeMimeType(params.contentType);
throw new LocalMediaAccessError(
"path-not-allowed",
`Host-local media sends only allow images, audio, video, PDF, and Office documents (got ${contentType ?? "unknown"}).`,
);
}
const normalizedMime = normalizeMimeType(params.contentType);
if (normalizedMime && HOST_READ_ALLOWED_DOCUMENT_MIMES.has(normalizedMime)) {
if (params.kind === "document" && normalizedMime && HOST_READ_ALLOWED_DOCUMENT_MIMES.has(normalizedMime)) {
return;
}
throw new LocalMediaAccessError(
@@ -185,9 +165,7 @@ async function optimizeImageWithFallback(params: {
meta?: { contentType?: string; fileName?: string };
}): Promise<OptimizedImage> {
const { buffer, cap, meta } = params;
const isPng =
meta?.contentType === "image/png" ||
normalizeLowercaseStringOrEmpty(meta?.fileName).endsWith(".png");
const isPng = meta?.contentType === "image/png" || meta?.fileName?.toLowerCase().endsWith(".png");
const hasAlpha = isPng && (await hasAlphaChannel(buffer));
if (hasAlpha) {
@@ -218,7 +196,6 @@ async function loadWebMediaInternal(
sandboxValidated = false,
readFile: readFileOverride,
hostReadCapability = false,
workspaceDir,
} = options;
// Strip MEDIA: prefix used by agent tools (e.g. TTS) to tag media paths.
// Be lenient: LLM output may add extra whitespace (e.g. " MEDIA : /tmp/x.png").
@@ -231,6 +208,7 @@ async function loadWebMediaInternal(
throw new LocalMediaAccessError("invalid-file-url", (err as Error).message, { cause: err });
}
}
mediaUrl = resolveCanvasHttpPathToLocalPath(mediaUrl) ?? mediaUrl;
const optimizeAndClampImage = async (
buffer: Buffer,
@@ -319,13 +297,6 @@ async function loadWebMediaInternal(
if (mediaUrl.startsWith("~")) {
mediaUrl = resolveUserPath(mediaUrl);
}
// Resolve relative MEDIA: paths (e.g. "poker_profit.png", "./subdir/file.png")
// against the agent workspace directory so bare filenames written by agents
// are found on disk and pass the local-roots allowlist check.
if (workspaceDir && !path.isAbsolute(mediaUrl)) {
mediaUrl = path.resolve(workspaceDir, mediaUrl);
}
try {
assertNoWindowsNetworkPath(mediaUrl, "Local media path");
} catch (err) {
@@ -376,10 +347,11 @@ async function loadWebMediaInternal(
throw err;
}
}
const detectedMime = await detectMime({ buffer: data, filePath: mediaUrl });
const verifiedMime = hostReadCapability ? await detectMime({ buffer: data }) : detectedMime;
const mime = verifiedMime ?? detectedMime;
const mime = await detectMime({ buffer: data, filePath: mediaUrl });
const kind = kindFromMime(mime);
if (hostReadCapability) {
assertHostReadMediaAllowed({ contentType: mime, kind });
}
let fileName = path.basename(mediaUrl) || undefined;
if (fileName && !path.extname(fileName) && mime) {
const ext = extensionForMime(mime);
@@ -387,12 +359,6 @@ async function loadWebMediaInternal(
fileName = `${fileName}${ext}`;
}
}
if (hostReadCapability) {
assertHostReadMediaAllowed({
contentType: verifiedMime,
kind: kindFromMime(detectedMime ?? verifiedMime),
});
}
return await clampAndFinalize({
buffer: data,
contentType: mime,
@@ -472,10 +438,7 @@ export async function optimizeImageToJpeg(
quality,
};
}
} catch (error) {
if (isPixelLimitError(error)) {
throw error;
}
} catch {
// Continue trying other size/quality combinations
}
}

View File

@@ -51,21 +51,16 @@
/* Chat thread - scrollable middle section, transparent */
.chat-thread {
flex: 1;
flex: 1 1 0;
/* Grow, shrink, and use 0 base for proper scrolling */
overflow-y: auto;
overflow-x: hidden;
padding: 0 12px 16px;
padding: 0 6px 6px;
margin: 0 0 0 0;
min-height: 0;
/* Allow shrinking for flex scroll behavior */
border-radius: 0;
border: none;
border-radius: var(--radius-md);
background: transparent;
display: flex;
flex-direction: column;
gap: 12px;
min-width: 0;
}
.chat-thread-inner > :first-child {
@@ -305,18 +300,114 @@
justify-content: flex-end;
}
/* Embedded audio (e.g. gateway-injected TTS from slash commands) */
.chat-message-audio {
.chat-assistant-attachments {
display: flex;
flex-direction: column;
gap: 8px;
gap: 10px;
margin-bottom: 8px;
max-width: min(420px, 100%);
}
.chat-message-audio-el {
width: 100%;
min-height: 36px;
.chat-assistant-attachments img.chat-message-image {
display: block;
}
.chat-assistant-attachment-card {
display: flex;
align-items: center;
gap: 10px;
padding: 10px 12px;
border: 1px solid var(--border);
border-radius: var(--radius-md);
background: color-mix(in srgb, var(--card) 82%, var(--bg));
}
.chat-assistant-attachment-card--audio,
.chat-assistant-attachment-card--video {
display: flex;
flex-direction: column;
align-items: stretch;
}
.chat-assistant-attachment-card--audio audio,
.chat-assistant-attachment-card--video video {
width: min(100%, 360px);
max-width: 100%;
}
.chat-assistant-attachment-card--video video {
border-radius: var(--radius-sm);
background: #000;
}
.chat-assistant-attachment-card__header {
display: flex;
align-items: center;
justify-content: space-between;
gap: 8px;
margin-bottom: 8px;
}
.chat-assistant-attachment-card__title,
.chat-assistant-attachment-card__link {
color: var(--text);
font-size: 13px;
text-decoration: none;
word-break: break-word;
}
.chat-assistant-attachment-card__reason {
color: var(--muted);
font-size: 12px;
line-height: 1.4;
}
.chat-assistant-attachment-card__icon {
display: inline-flex;
width: 16px;
height: 16px;
color: var(--muted);
}
.chat-assistant-attachment-card__icon svg {
width: 16px;
height: 16px;
}
.chat-assistant-attachment-badge,
.chat-reply-pill {
display: inline-flex;
align-items: center;
gap: 6px;
width: fit-content;
max-width: 100%;
border-radius: 999px;
font-size: 12px;
}
.chat-assistant-attachment-badge {
padding: 3px 8px;
background: color-mix(in srgb, var(--accent) 14%, transparent);
color: var(--accent);
border: 1px solid color-mix(in srgb, var(--accent) 24%, transparent);
}
.chat-reply-pill {
margin-bottom: 8px;
padding: 5px 10px;
color: var(--muted);
border: 1px solid var(--border);
background: color-mix(in srgb, var(--bg) 70%, transparent);
}
.chat-reply-pill__icon {
display: inline-flex;
width: 14px;
height: 14px;
}
.chat-reply-pill__icon svg {
width: 14px;
height: 14px;
}
/* Compose input row - horizontal layout */
@@ -762,8 +853,8 @@
}
.chat-controls__session {
min-width: 98px;
max-width: 190px;
min-width: 140px;
max-width: 300px;
}
.chat-controls__session-row {
@@ -774,13 +865,8 @@
}
.chat-controls__model {
min-width: 124px;
max-width: 206px;
}
.chat-controls__thinking-select {
min-width: 88px;
max-width: 118px;
min-width: 170px;
max-width: 320px;
}
.chat-controls__thinking {
@@ -805,17 +891,13 @@
.chat-controls__session select {
padding: 6px 10px;
font-size: 13px;
max-width: 190px;
max-width: 300px;
overflow: hidden;
text-overflow: ellipsis;
}
.chat-controls__model select {
max-width: 206px;
}
.chat-controls__thinking-select select {
max-width: 118px;
max-width: 320px;
}
.chat-controls__thinking {
@@ -829,6 +911,10 @@
border: 1px solid var(--border);
}
.chat-controls__auto-expand {
padding: 8px;
}
/* Light theme thinking indicator override */
:root[data-theme-mode="light"] .chat-controls__thinking {
background: rgba(255, 255, 255, 0.9);
@@ -846,11 +932,6 @@
max-width: none;
}
.chat-controls__thinking-select {
min-width: 130px;
max-width: none;
}
.chat-controls {
gap: 8px;
}
@@ -858,14 +939,6 @@
.chat-compose__field textarea {
min-height: 64px;
}
.card.chat {
padding: 0;
}
.chat-thread {
padding: 0 0 16px;
}
}
@media (max-width: 640px) {
@@ -904,10 +977,6 @@
.chat-controls__model {
min-width: 150px;
}
.chat-controls__thinking-select {
min-width: 140px;
}
}
/* Chat loading skeleton */

View File

@@ -1,20 +1,28 @@
/* Tool Card Styles */
.chat-tool-card {
border: 1px solid var(--border);
border: 1px solid color-mix(in srgb, var(--border) 85%, transparent);
border-radius: var(--radius-md);
padding: 10px 12px;
padding: 12px 14px;
margin-top: 6px;
background: var(--card);
background: color-mix(in srgb, var(--card) 88%, var(--secondary) 12%);
box-shadow: inset 0 1px 0 color-mix(in srgb, var(--bg) 75%, transparent);
transition:
border-color var(--duration-fast) ease-out,
background var(--duration-fast) ease-out;
background var(--duration-fast) ease-out,
box-shadow var(--duration-fast) ease-out;
max-height: 120px;
overflow: hidden;
}
.chat-tool-card--expanded {
max-height: none;
overflow: visible;
}
.chat-tool-card:hover {
border-color: var(--border-strong);
background: var(--bg-hover);
border-color: color-mix(in srgb, var(--border-strong) 80%, transparent);
background: color-mix(in srgb, var(--card) 70%, var(--bg-hover) 30%);
box-shadow: inset 0 1px 0 color-mix(in srgb, var(--bg) 85%, transparent);
}
/* First tool card in a group - no top margin */
@@ -35,8 +43,16 @@
.chat-tool-card__header {
display: flex;
justify-content: space-between;
align-items: flex-start;
gap: 10px;
}
.chat-tool-card__actions {
display: inline-flex;
align-items: center;
gap: 8px;
gap: 6px;
flex-wrap: wrap;
justify-content: flex-end;
}
.chat-tool-card__title {
@@ -44,7 +60,7 @@
align-items: center;
gap: 6px;
font-weight: 600;
font-size: 13px;
font-size: 14px;
line-height: 1.2;
}
@@ -68,19 +84,41 @@
}
/* "View >" action link */
.chat-tool-card__action {
.chat-tool-card__action,
.chat-tool-card__action-btn {
display: inline-flex;
align-items: center;
justify-content: center;
gap: 4px;
font-size: 12px;
color: var(--accent);
opacity: 0.8;
transition: opacity 150ms ease-out;
font-size: 11px;
color: var(--muted);
opacity: 1;
transition:
color 150ms ease-out,
background 150ms ease-out,
border-color 150ms ease-out;
}
.chat-tool-card__action svg {
width: 12px;
height: 12px;
.chat-tool-card__action-btn {
width: 28px;
height: 28px;
padding: 0;
border: 1px solid color-mix(in srgb, var(--border) 80%, transparent);
border-radius: 999px;
background: color-mix(in srgb, var(--secondary) 55%, transparent);
cursor: pointer;
font: inherit;
}
.chat-tool-card__action-icon {
display: inline-flex;
align-items: center;
justify-content: center;
}
.chat-tool-card__action-icon svg {
width: 11px;
height: 11px;
stroke: currentColor;
fill: none;
stroke-width: 1.5px;
@@ -88,8 +126,12 @@
stroke-linejoin: round;
}
.chat-tool-card--clickable:hover .chat-tool-card__action {
opacity: 1;
.chat-tool-card--clickable:hover .chat-tool-card__action,
.chat-tool-card__action-btn:hover,
.chat-tool-card__action-btn:focus-visible {
color: var(--text);
background: color-mix(in srgb, var(--bg-hover) 70%, transparent);
border-color: color-mix(in srgb, var(--border-strong) 70%, transparent);
}
/* Status indicator for completed/empty results */
@@ -111,47 +153,212 @@
.chat-tool-card__status-text {
font-size: 11px;
margin-top: 4px;
margin-top: 10px;
}
.chat-tool-card__detail {
font-size: 12px;
color: var(--muted);
margin-top: 4px;
margin-top: 6px;
}
/* Collapsed preview - fixed height with truncation */
.chat-tools-inline {
display: grid;
gap: 6px;
}
.chat-tool-card__block {
margin-top: 12px;
}
.chat-tool-card__preview,
.chat-tool-card__raw {
margin-top: 12px;
}
.chat-tool-card__preview {
font-size: 11px;
color: var(--muted);
margin-top: 8px;
padding: 8px 10px;
background: var(--secondary);
border: 1px solid color-mix(in srgb, var(--border) 78%, transparent);
border-radius: var(--radius-md);
white-space: pre-wrap;
background: color-mix(in srgb, var(--secondary) 78%, transparent);
overflow: hidden;
max-height: 44px;
line-height: 1.4;
border: 1px solid var(--border);
}
.chat-tool-card--clickable:hover .chat-tool-card__preview {
background: var(--bg-hover);
border-color: var(--border-strong);
.chat-tool-card__preview-header {
display: flex;
align-items: center;
justify-content: space-between;
gap: 10px;
padding: 10px 12px;
border-bottom: 1px solid color-mix(in srgb, var(--border) 72%, transparent);
background: color-mix(in srgb, var(--card) 82%, transparent);
}
/* Short inline output */
.chat-tool-card__inline {
.chat-tool-card__preview-label {
font-size: 11px;
font-weight: 600;
letter-spacing: 0.04em;
text-transform: uppercase;
color: var(--muted);
}
.chat-tool-card__preview-tabs {
display: inline-flex;
align-items: center;
gap: 4px;
}
.chat-tool-card__preview-tab {
border: 1px solid color-mix(in srgb, var(--border) 76%, transparent);
background: color-mix(in srgb, var(--bg) 72%, transparent);
color: var(--muted);
border-radius: 999px;
padding: 4px 10px;
font: inherit;
font-size: 12px;
cursor: pointer;
transition:
color 150ms ease-out,
border-color 150ms ease-out,
background 150ms ease-out;
}
.chat-tool-card__preview-tab.is-active,
.chat-tool-card__preview-tab:hover,
.chat-tool-card__preview-tab:focus-visible {
color: var(--text);
margin-top: 6px;
padding: 6px 8px;
background: var(--secondary);
border-radius: var(--radius-sm);
border-color: color-mix(in srgb, var(--border-strong) 80%, transparent);
background: color-mix(in srgb, var(--card) 92%, transparent);
}
.chat-tool-card__preview-panel {
padding: 12px;
}
.chat-tool-card__preview-frame {
display: block;
width: 100%;
min-height: 420px;
border: 1px solid color-mix(in srgb, var(--border) 68%, transparent);
border-radius: var(--radius-md);
background: #fff;
}
.chat-tool-card__raw-toggle {
width: 100%;
display: inline-flex;
align-items: center;
justify-content: space-between;
gap: 8px;
padding: 10px 12px;
border: 1px solid color-mix(in srgb, var(--border) 75%, transparent);
border-radius: var(--radius-md);
background: color-mix(in srgb, var(--secondary) 68%, transparent);
color: var(--muted);
font: inherit;
font-size: 12px;
cursor: pointer;
transition:
color 150ms ease-out,
border-color 150ms ease-out,
background 150ms ease-out;
}
.chat-tool-card__raw-toggle:hover,
.chat-tool-card__raw-toggle:focus-visible {
color: var(--text);
border-color: color-mix(in srgb, var(--border-strong) 80%, transparent);
background: color-mix(in srgb, var(--card) 92%, transparent);
}
.chat-tool-card__raw-toggle[aria-expanded="true"] .chat-tool-card__raw-toggle-icon {
transform: rotate(180deg);
}
.chat-tool-card__raw-toggle-icon {
display: inline-flex;
align-items: center;
justify-content: center;
transition: transform 150ms ease-out;
}
.chat-tool-card__raw-toggle-icon svg {
width: 14px;
height: 14px;
stroke: currentColor;
fill: none;
stroke-width: 1.6px;
stroke-linecap: round;
stroke-linejoin: round;
}
.chat-tool-card__raw-body {
margin-top: 8px;
}
.chat-tool-card__block-header {
display: inline-flex;
align-items: center;
gap: 6px;
margin-bottom: 6px;
color: var(--muted);
}
.chat-tool-card__block-icon {
display: inline-flex;
align-items: center;
justify-content: center;
width: 14px;
height: 14px;
}
.chat-tool-card__block-icon svg {
width: 14px;
height: 14px;
stroke: currentColor;
fill: none;
stroke-width: 1.5px;
stroke-linecap: round;
stroke-linejoin: round;
}
.chat-tool-card__block-label {
font-size: 11px;
font-weight: 600;
letter-spacing: 0.04em;
text-transform: uppercase;
color: inherit;
}
.chat-tool-card__block-preview,
.chat-tool-card__block-content,
.chat-tool-card__block-empty {
margin: 0;
padding: 11px 12px;
border-radius: var(--radius-md);
border: 1px solid color-mix(in srgb, var(--border) 75%, transparent);
background: color-mix(in srgb, var(--secondary) 82%, transparent);
font-size: 11px;
line-height: 1.45;
white-space: pre-wrap;
word-break: break-word;
}
.chat-tool-card__block-preview,
.chat-tool-card__block-empty {
color: var(--muted);
}
.chat-tool-card__block-content {
color: var(--text);
overflow-x: auto;
}
.chat-tool-card__block-preview {
overflow: hidden;
max-height: 52px;
}
.chat-tools-summary {
display: flex;
align-items: center;
@@ -331,12 +538,21 @@
border: 1px solid color-mix(in srgb, var(--border) 75%, transparent);
border-radius: var(--radius-md);
background: color-mix(in srgb, var(--bg-hover) 35%, transparent);
width: 100%;
text-align: left;
appearance: none;
-webkit-appearance: none;
font: inherit;
transition:
color 150ms ease,
background 150ms ease,
border-color 150ms ease;
}
.chat-tool-msg-summary[type="button"] {
background: color-mix(in srgb, var(--bg-hover) 35%, transparent);
}
.chat-tool-msg-summary::-webkit-details-marker {
display: none;
}
@@ -348,6 +564,15 @@
transition: transform 150ms ease;
}
.chat-tool-msg-collapse--static > .chat-tool-msg-summary::before {
display: none;
}
.chat-tool-msg-collapse--manual.is-open > .chat-tool-msg-summary::before,
.chat-tool-msg-summary[aria-expanded="true"]::before {
transform: rotate(90deg);
}
.chat-tool-msg-collapse[open] > .chat-tool-msg-summary::before {
transform: rotate(90deg);
}
@@ -407,6 +632,42 @@
min-width: 0;
}
.chat-tool-msg-summary__spacer {
flex: 1 1 auto;
}
.chat-tool-msg-summary__btn {
width: 24px;
height: 24px;
padding: 0;
border: 1px solid color-mix(in srgb, var(--border) 70%, transparent);
border-radius: 999px;
background: color-mix(in srgb, var(--secondary) 55%, transparent);
color: var(--muted);
display: inline-flex;
align-items: center;
justify-content: center;
cursor: pointer;
flex-shrink: 0;
}
.chat-tool-msg-summary__btn:hover,
.chat-tool-msg-summary__btn:focus-visible {
color: var(--text);
background: color-mix(in srgb, var(--bg-hover) 70%, transparent);
border-color: color-mix(in srgb, var(--border-strong) 70%, transparent);
}
.chat-tool-msg-summary__btn svg {
width: 11px;
height: 11px;
stroke: currentColor;
fill: none;
stroke-width: 1.5px;
stroke-linecap: round;
stroke-linejoin: round;
}
.chat-tool-msg-body {
padding-top: 8px;
}

View File

@@ -9,7 +9,6 @@ const loadChatHistoryMock = vi.hoisted(() => vi.fn(async () => undefined));
type GatewayClientMock = {
start: ReturnType<typeof vi.fn>;
stop: ReturnType<typeof vi.fn>;
request: ReturnType<typeof vi.fn>;
options: { clientVersion?: string };
emitHello: (hello?: GatewayHelloOk) => void;
emitClose: (info: {
@@ -23,8 +22,8 @@ type GatewayClientMock = {
const gatewayClientInstances: GatewayClientMock[] = [];
vi.mock("./gateway.ts", async () => {
const actual = await vi.importActual<typeof import("./gateway.ts")>("./gateway.ts");
vi.mock("./gateway.ts", async (importOriginal) => {
const actual = await importOriginal<typeof import("./gateway.ts")>();
function resolveGatewayErrorDetailCode(
error: { details?: unknown } | null | undefined,
@@ -40,7 +39,6 @@ vi.mock("./gateway.ts", async () => {
class GatewayBrowserClient {
readonly start = vi.fn();
readonly stop = vi.fn();
readonly request = vi.fn(async () => ({}));
constructor(
private opts: {
@@ -58,7 +56,6 @@ vi.mock("./gateway.ts", async () => {
gatewayClientInstances.push({
start: this.start,
stop: this.stop,
request: this.request,
options: { clientVersion: this.opts.clientVersion },
emitHello: (hello) => {
this.opts.onHello?.(
@@ -89,9 +86,8 @@ vi.mock("./gateway.ts", async () => {
return { ...actual, GatewayBrowserClient, resolveGatewayErrorDetailCode };
});
vi.mock("./controllers/chat.ts", async () => {
const actual =
await vi.importActual<typeof import("./controllers/chat.ts")>("./controllers/chat.ts");
vi.mock("./controllers/chat.ts", async (importOriginal) => {
const actual = await importOriginal<typeof import("./controllers/chat.ts")>();
return {
...actual,
loadChatHistory: loadChatHistoryMock,
@@ -133,19 +129,15 @@ function createHost() {
assistantName: "OpenClaw",
assistantAvatar: null,
assistantAgentId: null,
localMediaPreviewRoots: [],
serverVersion: null,
sessionKey: "main",
basePath: "",
chatMessage: "",
chatMessages: [],
chatAttachments: [],
chatQueue: [],
chatToolMessages: [],
chatStreamSegments: [],
chatStream: null,
chatStreamStartedAt: null,
chatRunId: null,
chatSending: false,
toolStreamById: new Map(),
toolStreamOrder: [],
toolStreamSyncTimer: null,
@@ -187,7 +179,6 @@ describe("connectGateway", () => {
beforeEach(() => {
gatewayClientInstances.length = 0;
loadChatHistoryMock.mockClear();
vi.restoreAllMocks();
});
it("ignores stale client onGap callbacks after reconnect", () => {
@@ -205,72 +196,9 @@ describe("connectGateway", () => {
expect(host.lastError).toBeNull();
secondClient.emitGap(20, 24);
expect(gatewayClientInstances).toHaveLength(3);
expect(secondClient.stop).toHaveBeenCalledTimes(1);
expect(host.lastError).toBeNull();
});
it("preserves live approval prompts, clears stale run indicators, and resumes queued work after seq-gap reconnect", () => {
const now = 1_700_000_000_000;
vi.spyOn(Date, "now").mockReturnValue(now);
const host = createHost();
connectGateway(host);
const client = gatewayClientInstances[0];
expect(client).toBeDefined();
const chatHost = host as typeof host & {
chatRunId: string | null;
chatQueue: Array<{
id: string;
text: string;
createdAt: number;
pendingRunId?: string;
}>;
};
chatHost.chatRunId = "run-1";
chatHost.chatQueue = [
{
id: "pending",
text: "/steer tighten the plan",
createdAt: 1,
pendingRunId: "run-1",
},
{
id: "queued",
text: "follow up",
createdAt: 2,
},
];
host.execApprovalQueue = [
{
id: "approval-1",
kind: "exec",
request: { command: "rm -rf /tmp/demo" },
createdAtMs: now,
expiresAtMs: now + 60_000,
},
];
client.emitGap(20, 24);
expect(gatewayClientInstances).toHaveLength(2);
expect(host.execApprovalQueue).toHaveLength(1);
expect(host.execApprovalQueue[0]?.id).toBe("approval-1");
expect(chatHost.chatQueue).toHaveLength(1);
expect(chatHost.chatQueue[0]?.text).toBe("follow up");
const reconnectClient = gatewayClientInstances[1];
expect(reconnectClient).toBeDefined();
reconnectClient.emitHello();
expect(reconnectClient.request).toHaveBeenCalledWith("chat.send", {
sessionKey: "main",
message: "follow up",
deliver: false,
idempotencyKey: expect.any(String),
attachments: undefined,
});
expect(chatHost.chatQueue).toHaveLength(0);
expect(host.lastError).toBe(
"event gap detected (expected seq 20, got 24); refresh recommended",
);
});
it("ignores stale client onEvent callbacks after reconnect", () => {
@@ -640,6 +568,60 @@ describe("connectGateway", () => {
expect(loadChatHistoryMock).toHaveBeenCalledTimes(1);
});
it("reloads chat history after a final assistant event even when no live tool events were observed", () => {
const { client } = connectHostGateway();
client.emitEvent({
event: "chat",
payload: {
runId: "engine-run-1",
sessionKey: "main",
state: "final",
message: {
role: "assistant",
content: [{ type: "text", text: "Done" }],
},
},
});
expect(loadChatHistoryMock).toHaveBeenCalledTimes(1);
});
it("keeps live tool messages until history reload completes after a final tool run", async () => {
let resolveHistory: (() => void) | null = null;
loadChatHistoryMock.mockImplementationOnce(
() =>
new Promise<void>((resolve) => {
resolveHistory = resolve;
}),
);
const { client, host } = connectHostGateway();
emitToolResultEvent(client);
expect(host.chatToolMessages).toHaveLength(1);
client.emitEvent({
event: "chat",
payload: {
runId: "engine-run-1",
sessionKey: "main",
state: "final",
message: {
role: "assistant",
content: [{ type: "text", text: "Done" }],
},
},
});
expect(loadChatHistoryMock).toHaveBeenCalledTimes(1);
expect(host.chatToolMessages).toHaveLength(1);
resolveHistory?.();
await Promise.resolve();
expect(host.chatToolMessages).toEqual([]);
});
});
describe("resolveControlUiClientVersion", () => {

View File

@@ -2,11 +2,7 @@ import {
GATEWAY_EVENT_UPDATE_AVAILABLE,
type GatewayUpdateAvailableEventPayload,
} from "../../../src/gateway/events.js";
import {
CHAT_SESSIONS_ACTIVE_MINUTES,
clearPendingQueueItemsForRun,
flushChatQueueForEvent,
} from "./app-chat.ts";
import { CHAT_SESSIONS_ACTIVE_MINUTES, flushChatQueueForEvent } from "./app-chat.ts";
import type { EventLogEntry } from "./app-events.ts";
import {
applySettings,
@@ -29,7 +25,6 @@ import {
parseExecApprovalRequested,
parseExecApprovalResolved,
parsePluginApprovalRequested,
pruneExecApprovalQueue,
removeExecApproval,
} from "./controllers/exec-approval.ts";
import { loadHealthState } from "./controllers/health.ts";
@@ -43,7 +38,6 @@ import {
import { GatewayBrowserClient } from "./gateway.ts";
import type { Tab } from "./navigation.ts";
import type { UiSettings } from "./storage.ts";
import { normalizeOptionalString } from "./string-coerce.ts";
import type {
AgentsListResult,
PresenceEntry,
@@ -100,11 +94,6 @@ type SessionDefaultsSnapshot = {
type GatewayHostWithShutdownMessage = GatewayHost & {
pendingShutdownMessage?: string | null;
resumeChatQueueAfterReconnect?: boolean;
};
type ConnectGatewayOptions = {
reason?: "initial" | "seq-gap";
};
export function resolveControlUiClientVersion(params: {
@@ -112,7 +101,7 @@ export function resolveControlUiClientVersion(params: {
serverVersion: string | null;
pageUrl?: string;
}): string | undefined {
const serverVersion = normalizeOptionalString(params.serverVersion);
const serverVersion = params.serverVersion?.trim();
if (!serverVersion) {
return undefined;
}
@@ -138,16 +127,16 @@ function normalizeSessionKeyForDefaults(
value: string | undefined,
defaults: SessionDefaultsSnapshot,
): string {
const raw = normalizeOptionalString(value) ?? "";
const mainSessionKey = normalizeOptionalString(defaults.mainSessionKey);
const raw = (value ?? "").trim();
const mainSessionKey = defaults.mainSessionKey?.trim();
if (!mainSessionKey) {
return raw;
}
if (!raw) {
return mainSessionKey;
}
const mainKey = normalizeOptionalString(defaults.mainKey) ?? "main";
const defaultAgentId = normalizeOptionalString(defaults.defaultAgentId);
const mainKey = defaults.mainKey?.trim() || "main";
const defaultAgentId = defaults.defaultAgentId?.trim();
const isAlias =
raw === "main" ||
raw === mainKey ||
@@ -186,29 +175,14 @@ function applySessionDefaults(host: GatewayHost, defaults?: SessionDefaultsSnaps
}
}
export function connectGateway(host: GatewayHost, options?: ConnectGatewayOptions) {
export function connectGateway(host: GatewayHost) {
const shutdownHost = host as GatewayHostWithShutdownMessage;
const reconnectReason = options?.reason ?? "initial";
shutdownHost.pendingShutdownMessage = null;
shutdownHost.resumeChatQueueAfterReconnect = false;
host.lastError = null;
host.lastErrorCode = null;
host.hello = null;
host.connected = false;
if (reconnectReason === "seq-gap") {
// A seq gap means the socket stayed on the same gateway; preserve prompts
// that only arrived as ephemeral events and clear stale run-scoped indicators.
host.execApprovalQueue = pruneExecApprovalQueue(host.execApprovalQueue);
clearPendingQueueItemsForRun(
host as unknown as Parameters<typeof clearPendingQueueItemsForRun>[0],
host.chatRunId ?? undefined,
);
shutdownHost.resumeChatQueueAfterReconnect = true;
} else {
// Preserve any still-live approvals that were already staged in UI state.
// Initial connect can happen after a soft reload while an approval is pending.
host.execApprovalQueue = pruneExecApprovalQueue(host.execApprovalQueue);
}
host.execApprovalQueue = [];
host.execApprovalError = null;
const previousClient = host.client;
@@ -218,8 +192,8 @@ export function connectGateway(host: GatewayHost, options?: ConnectGatewayOption
});
const client = new GatewayBrowserClient({
url: host.settings.gatewayUrl,
token: normalizeOptionalString(host.settings.token) ? host.settings.token : undefined,
password: normalizeOptionalString(host.password) ? host.password : undefined,
token: host.settings.token.trim() ? host.settings.token : undefined,
password: host.password.trim() ? host.password : undefined,
clientName: "openclaw-control-ui",
clientVersion,
mode: "webchat",
@@ -240,14 +214,6 @@ export function connectGateway(host: GatewayHost, options?: ConnectGatewayOption
(host as unknown as { chatStream: string | null }).chatStream = null;
(host as unknown as { chatStreamStartedAt: number | null }).chatStreamStartedAt = null;
resetToolStream(host as unknown as Parameters<typeof resetToolStream>[0]);
if (shutdownHost.resumeChatQueueAfterReconnect) {
// The interrupted run will never emit its terminal event now that the
// old client is gone, so resume any deferred commands after hello.
shutdownHost.resumeChatQueueAfterReconnect = false;
void flushChatQueueForEvent(
host as unknown as Parameters<typeof flushChatQueueForEvent>[0],
);
}
void subscribeSessions(host as unknown as OpenClawApp);
void loadAssistantIdentity(host as unknown as OpenClawApp);
void loadAgents(host as unknown as OpenClawApp);
@@ -294,9 +260,8 @@ export function connectGateway(host: GatewayHost, options?: ConnectGatewayOption
if (host.client !== client) {
return;
}
host.lastError = `event gap detected (expected seq ${expected}, got ${received}); reconnecting`;
host.lastError = `event gap detected (expected seq ${expected}, got ${received}); refresh recommended`;
host.lastErrorCode = null;
connectGateway(host, { reason: "seq-gap" });
},
});
host.client = client;
@@ -323,12 +288,8 @@ function handleTerminalChatEvent(
// Check if tool events were seen before resetting (resetToolStream clears toolStreamOrder).
const toolHost = host as unknown as Parameters<typeof resetToolStream>[0];
const hadToolEvents = toolHost.toolStreamOrder.length > 0;
resetToolStream(toolHost);
clearPendingQueueItemsForRun(
host as unknown as Parameters<typeof clearPendingQueueItemsForRun>[0],
payload?.runId,
);
void flushChatQueueForEvent(host as unknown as Parameters<typeof flushChatQueueForEvent>[0]);
const flushQueue = () =>
void flushChatQueueForEvent(host as unknown as Parameters<typeof flushChatQueueForEvent>[0]);
const runId = payload?.runId;
if (runId && host.refreshSessionsAfterChat.has(runId)) {
host.refreshSessionsAfterChat.delete(runId);
@@ -341,9 +302,14 @@ function handleTerminalChatEvent(
// Reload history when tools were used so the persisted tool results
// replace the now-cleared streaming state.
if (hadToolEvents && state === "final") {
void loadChatHistory(host as unknown as OpenClawApp);
void loadChatHistory(host as unknown as OpenClawApp).finally(() => {
resetToolStream(toolHost);
flushQueue();
});
return true;
}
resetToolStream(toolHost);
flushQueue();
return false;
}
@@ -398,7 +364,10 @@ function handleGatewayEventUnsafe(host: GatewayHost, evt: GatewayEventFrame) {
if (evt.event === "shutdown") {
const payload = evt.payload as { reason?: unknown; restartExpectedMs?: unknown } | undefined;
const reason = normalizeOptionalString(payload?.reason) ?? "gateway stopping";
const reason =
payload && typeof payload.reason === "string" && payload.reason.trim()
? payload.reason.trim()
: "gateway stopping";
const shutdownMessage =
typeof payload?.restartExpectedMs === "number"
? `Restarting: ${reason}`

View File

@@ -11,6 +11,7 @@ function createHost() {
assistantName: "OpenClaw",
assistantAvatar: null,
assistantAgentId: null,
localMediaPreviewRoots: [],
chatHasAutoScrolled: false,
chatManualRefreshInFlight: false,
chatLoading: false,

View File

@@ -1,4 +1,9 @@
import { html, nothing } from "lit";
import {
buildAgentMainSessionKey,
parseAgentSessionKey,
resolveAgentIdFromSessionKey,
} from "../../../src/routing/session-key.js";
import { t } from "../i18n/index.ts";
import { getSafeLocalStorage } from "../local-storage.ts";
import { refreshChatAvatar } from "./app-chat.ts";
@@ -103,15 +108,10 @@ import {
updateSkillEdit,
updateSkillEnabled,
} from "./controllers/skills.ts";
import { buildExternalLinkRel, EXTERNAL_LINK_TARGET } from "./external-link.ts";
import "./components/dashboard-header.ts";
import { buildExternalLinkRel, EXTERNAL_LINK_TARGET } from "./external-link.ts";
import { icons } from "./icons.ts";
import { normalizeBasePath, TAB_GROUPS, subtitleForTab, titleForTab } from "./navigation.ts";
import {
buildAgentMainSessionKey,
parseAgentSessionKey,
resolveAgentIdFromSessionKey,
} from "./session-key.ts";
import { agentLogoUrl } from "./views/agents-utils.ts";
import {
resolveAgentConfig,
@@ -173,9 +173,9 @@ function formatDreamNextCycle(nextRunAtMs: number | undefined): string | null {
}
function resolveDreamingNextCycle(
status: { phases: Record<string, { enabled: boolean; nextRunAtMs?: number }> } | null,
status: { phases?: Record<string, { enabled: boolean; nextRunAtMs?: number }> } | null,
): string | null {
if (!status) {
if (!status?.phases) {
return null;
}
const nextRunAtMs = Object.values(status.phases)
@@ -1770,23 +1770,7 @@ export function renderApp(state: AppViewState) {
? renderChat({
sessionKey: state.sessionKey,
onSessionKeyChange: (next) => {
state.sessionKey = next;
state.chatMessage = "";
state.chatAttachments = [];
state.chatStream = null;
state.chatStreamStartedAt = null;
state.chatRunId = null;
state.chatQueue = [];
state.resetToolStream();
state.resetChatScroll();
state.applySettings({
...state.settings,
sessionKey: next,
lastActiveSessionKey: next,
});
void state.loadAssistantIdentity();
void loadChatHistory(state);
void refreshChatAvatar(state);
switchChatSession(state, next);
},
thinkingLevel: state.chatThinkingLevel,
showThinking,
@@ -1797,6 +1781,7 @@ export function renderApp(state: AppViewState) {
fallbackStatus: state.fallbackStatus,
assistantAvatarUrl: chatAvatarUrl,
messages: state.chatMessages,
sideResult: state.chatSideResult,
toolMessages: state.chatToolMessages,
streamSegments: state.chatStreamSegments,
stream: state.chatStream,
@@ -1809,7 +1794,11 @@ export function renderApp(state: AppViewState) {
error: state.lastError,
sessions: state.sessionsResult,
focusMode: chatFocus,
autoExpandToolCalls: state.onboarding
? false
: state.settings.chatAutoExpandToolCalls,
onRefresh: () => {
state.chatSideResult = null;
state.resetToolStream();
return Promise.all([loadChatHistory(state), refreshChatAvatar(state)]);
},
@@ -1832,6 +1821,9 @@ export function renderApp(state: AppViewState) {
canAbort: Boolean(state.chatRunId),
onAbort: () => void state.handleAbortChat(),
onQueueRemove: (id) => state.removeQueuedMessage(id),
onDismissSideResult: () => {
state.chatSideResult = null;
},
onNewSession: () => state.handleSendChat("/new", { restoreDraft: true }),
onClearHistory: async () => {
if (!state.client || !state.connected) {
@@ -1840,6 +1832,7 @@ export function renderApp(state: AppViewState) {
try {
await state.client.request("sessions.reset", { key: state.sessionKey });
state.chatMessages = [];
state.chatSideResult = null;
state.chatStream = null;
state.chatRunId = null;
await loadChatHistory(state);
@@ -1850,17 +1843,7 @@ export function renderApp(state: AppViewState) {
agentsList: state.agentsList,
currentAgentId: resolvedAgentId ?? "main",
onAgentChange: (agentId: string) => {
state.sessionKey = buildAgentMainSessionKey({ agentId });
state.chatMessages = [];
state.chatStream = null;
state.chatRunId = null;
state.applySettings({
...state.settings,
sessionKey: state.sessionKey,
lastActiveSessionKey: state.sessionKey,
});
void loadChatHistory(state);
void state.loadAssistantIdentity();
switchChatSession(state, buildAgentMainSessionKey({ agentId }));
},
onNavigateToAgent: () => {
state.agentsSelectedId = resolvedAgentId;
@@ -1876,11 +1859,15 @@ export function renderApp(state: AppViewState) {
sidebarContent: state.sidebarContent,
sidebarError: state.sidebarError,
splitRatio: state.splitRatio,
onOpenSidebar: (content: string) => state.handleOpenSidebar(content),
canvasHostUrl: state.hello?.canvasHostUrl ?? null,
onOpenSidebar: (content) => state.handleOpenSidebar(content),
onCloseSidebar: () => state.handleCloseSidebar(),
onSplitRatioChange: (ratio: number) => state.handleSplitRatioChange(ratio),
assistantName: state.assistantName,
assistantAvatar: state.assistantAvatar,
localMediaPreviewRoots: state.localMediaPreviewRoots,
embedSandboxMode: state.embedSandboxMode,
assistantAttachmentAuthToken: state.settings.token.trim() || null,
basePath: state.basePath ?? "",
})
: nothing}
@@ -1935,9 +1922,8 @@ export function renderApp(state: AppViewState) {
groundedSignalCount: state.dreamingStatus?.groundedSignalCount ?? 0,
totalSignalCount: state.dreamingStatus?.totalSignalCount ?? 0,
promotedCount: state.dreamingStatus?.promotedToday ?? 0,
phaseSignalCount: state.dreamingStatus?.phaseSignalCount ?? 0,
phases: state.dreamingStatus?.phases ?? undefined,
shortTermEntries: state.dreamingStatus?.shortTermEntries ?? [],
signalEntries: state.dreamingStatus?.signalEntries ?? [],
promotedEntries: state.dreamingStatus?.promotedEntries ?? [],
dreamingOf: null,
nextCycle: dreamingNextCycle,
@@ -1955,7 +1941,6 @@ export function renderApp(state: AppViewState) {
onBackfillDiary: () => backfillDreamDiary(state),
onResetDiary: () => resetDreamDiary(state),
onResetGroundedShortTerm: () => resetGroundedShortTerm(state),
onToggleEnabled: applyDreamingEnabled,
onRequestUpdate: requestHostUpdate,
})
: nothing}

View File

@@ -1,5 +1,6 @@
import type { EventLogEntry } from "./app-events.ts";
import type { CompactionStatus, FallbackStatus } from "./app-tool-stream.ts";
import type { ChatSideResult } from "./chat/side-result.ts";
import type { CronModelSuggestionsState, CronState } from "./controllers/cron.ts";
import type { DevicePairingList } from "./controllers/devices.ts";
import type { ExecApprovalRequest } from "./controllers/exec-approval.ts";
@@ -9,8 +10,10 @@ import type {
ClawHubSkillDetail,
SkillMessage,
} from "./controllers/skills.ts";
import type { EmbedSandboxMode } from "./embed-sandbox.ts";
import type { GatewayBrowserClient, GatewayHelloOk } from "./gateway.ts";
import type { Tab } from "./navigation.ts";
import type { SidebarContent } from "./sidebar-content.ts";
import type { UiSettings } from "./storage.ts";
import type { ThemeTransitionContext } from "./theme-transition.ts";
import type { ResolvedTheme, ThemeMode, ThemeName } from "./theme.ts";
@@ -33,6 +36,7 @@ import type {
CostUsageSummary,
SessionUsageTimeSeries,
SessionsListResult,
SessionCompactionCheckpoint,
SkillStatusReport,
StatusSummary,
ToolsCatalogResult,
@@ -61,6 +65,8 @@ export type AppViewState = {
assistantName: string;
assistantAvatar: string | null;
assistantAgentId: string | null;
localMediaPreviewRoots: string[];
embedSandboxMode: EmbedSandboxMode;
sessionKey: string;
chatLoading: boolean;
chatSending: boolean;
@@ -72,6 +78,8 @@ export type AppViewState = {
chatStream: string | null;
chatStreamStartedAt: number | null;
chatRunId: string | null;
chatSideResult: ChatSideResult | null;
chatSideResultTerminalRuns: Set<string>;
compactionStatus: CompactionStatus | null;
fallbackStatus: FallbackStatus | null;
chatAvatarUrl: string | null;
@@ -86,7 +94,7 @@ export type AppViewState = {
chatNewMessagesBelow: boolean;
navDrawerOpen: boolean;
sidebarOpen: boolean;
sidebarContent: string | null;
sidebarContent: SidebarContent | null;
sidebarError: string | null;
splitRatio: number;
scrollToBottom: (opts?: { smooth?: boolean }) => void;
@@ -199,6 +207,9 @@ export type AppViewState = {
sessionsLoading: boolean;
sessionsResult: SessionsListResult | null;
sessionsError: string | null;
threadsLoading: boolean;
threadsResult: SessionsListResult | null;
threadsError: string | null;
sessionsFilterActive: string;
sessionsFilterLimit: string;
sessionsIncludeGlobal: boolean;
@@ -211,7 +222,7 @@ export type AppViewState = {
sessionsPageSize: number;
sessionsSelectedKeys: Set<string>;
sessionsExpandedCheckpointKey: string | null;
sessionsCheckpointItemsByKey: Record<string, import("./types.ts").SessionCompactionCheckpoint[]>;
sessionsCheckpointItemsByKey: Record<string, SessionCompactionCheckpoint[]>;
sessionsCheckpointLoadingKey: string | null;
sessionsCheckpointBusyKey: string | null;
sessionsCheckpointErrorByKey: Record<string, string>;
@@ -401,7 +412,7 @@ export type AppViewState = {
resetChatScroll: () => void;
exportLogs: (lines: string[], label: string) => void;
handleLogsScroll: (event: Event) => void;
handleOpenSidebar: (content: string) => void;
handleOpenSidebar: (content: SidebarContent) => void;
handleCloseSidebar: () => void;
handleSplitRatioChange: (ratio: number) => void;
};

View File

@@ -1,5 +1,6 @@
import { LitElement } from "lit";
import { state } from "lit/decorators.js";
import { customElement, state } from "lit/decorators.js";
import { resolveAgentIdFromSessionKey } from "../../../src/routing/session-key.js";
import { i18n, I18nController, isSupportedLocale } from "../i18n/index.ts";
import {
handleChannelConfigReload as handleChannelConfigReloadInternal,
@@ -54,6 +55,7 @@ import {
import type { AppViewState } from "./app-view-state.ts";
import { normalizeAssistantIdentity } from "./assistant-identity.ts";
import { exportChatMarkdown } from "./chat/export.ts";
import type { ChatSideResult } from "./chat/side-result.ts";
import {
loadToolsEffective as loadToolsEffectiveInternal,
refreshVisibleToolsEffectiveForCurrentSession as refreshVisibleToolsEffectiveForCurrentSessionInternal,
@@ -70,7 +72,7 @@ import type {
} from "./controllers/skills.ts";
import type { GatewayBrowserClient, GatewayHelloOk } from "./gateway.ts";
import type { Tab } from "./navigation.ts";
import { resolveAgentIdFromSessionKey } from "./session-key.ts";
import type { SidebarContent } from "./sidebar-content.ts";
import { loadSettings, type UiSettings } from "./storage.ts";
import { VALID_THEME_NAMES, type ResolvedTheme, type ThemeMode, type ThemeName } from "./theme.ts";
import type {
@@ -122,6 +124,7 @@ function resolveOnboardingMode(): boolean {
return normalized === "1" || normalized === "true" || normalized === "yes" || normalized === "on";
}
@customElement("openclaw-app")
export class OpenClawApp extends LitElement {
private i18nController = new I18nController(this);
clientInstanceId = generateUUID();
@@ -154,6 +157,8 @@ export class OpenClawApp extends LitElement {
@state() assistantName = bootAssistantIdentity.name;
@state() assistantAvatar = bootAssistantIdentity.avatar;
@state() assistantAgentId = bootAssistantIdentity.agentId ?? null;
@state() localMediaPreviewRoots: string[] = [];
@state() embedSandboxMode: "powerful" | "isolated" = "powerful";
@state() serverVersion: string | null = null;
@state() sessionKey = this.settings.sessionKey;
@@ -166,6 +171,7 @@ export class OpenClawApp extends LitElement {
@state() chatStream: string | null = null;
@state() chatStreamStartedAt: number | null = null;
@state() chatRunId: string | null = null;
@state() chatSideResult: ChatSideResult | null = null;
@state() compactionStatus: CompactionStatus | null = null;
@state() fallbackStatus: FallbackStatus | null = null;
@state() chatAvatarUrl: string | null = null;
@@ -182,7 +188,7 @@ export class OpenClawApp extends LitElement {
// Sidebar state for tool output viewing
@state() sidebarOpen = false;
@state() sidebarContent: string | null = null;
@state() sidebarContent: SidebarContent | null = null;
@state() sidebarError: string | null = null;
@state() splitRatio = this.settings.splitRatio;
@@ -486,6 +492,7 @@ export class OpenClawApp extends LitElement {
private toolStreamById = new Map<string, ToolStreamEntry>();
private toolStreamOrder: string[] = [];
refreshSessionsAfterChat = new Set<string>();
chatSideResultTerminalRuns = new Set<string>();
basePath = "";
private popStateHandler = () =>
onPopStateInternal(this as unknown as Parameters<typeof onPopStateInternal>[0]);
@@ -754,7 +761,7 @@ export class OpenClawApp extends LitElement {
}
// Sidebar handlers for tool output viewing
handleOpenSidebar(content: string) {
handleOpenSidebar(content: SidebarContent) {
if (this.sidebarCloseTimer != null) {
window.clearTimeout(this.sidebarCloseTimer);
this.sidebarCloseTimer = null;
@@ -790,7 +797,3 @@ export class OpenClawApp extends LitElement {
return renderApp(this as unknown as AppViewState);
}
}
if (!customElements.get("openclaw-app")) {
customElements.define("openclaw-app", OpenClawApp);
}

View File

@@ -0,0 +1,33 @@
import { describe, expect, it } from "vitest";
import { resolveCanvasIframeUrl } from "./canvas-url.ts";
describe("resolveCanvasIframeUrl", () => {
it("allows same-origin hosted canvas document paths", () => {
expect(resolveCanvasIframeUrl("/__openclaw__/canvas/documents/cv_demo/index.html")).toBe(
"/__openclaw__/canvas/documents/cv_demo/index.html",
);
});
it("rewrites safe canvas paths through the scoped canvas host", () => {
expect(
resolveCanvasIframeUrl(
"/__openclaw__/canvas/documents/cv_demo/index.html",
"http://127.0.0.1:19003/__openclaw__/cap/cap_123",
),
).toBe(
"http://127.0.0.1:19003/__openclaw__/cap/cap_123/__openclaw__/canvas/documents/cv_demo/index.html",
);
});
it("rejects non-canvas same-origin paths", () => {
expect(resolveCanvasIframeUrl("/not-canvas/snake.html")).toBeUndefined();
});
it("rejects absolute external URLs", () => {
expect(resolveCanvasIframeUrl("https://example.com/evil.html")).toBeUndefined();
});
it("rejects file URLs", () => {
expect(resolveCanvasIframeUrl("file:///tmp/snake.html")).toBeUndefined();
});
});

63
ui/src/ui/canvas-url.ts Normal file
View File

@@ -0,0 +1,63 @@
const A2UI_PATH = "/__openclaw__/a2ui";
const CANVAS_HOST_PATH = "/__openclaw__/canvas";
const CANVAS_CAPABILITY_PATH_PREFIX = "/__openclaw__/cap";
function isCanvasHttpPath(pathname: string): boolean {
return (
pathname === CANVAS_HOST_PATH ||
pathname.startsWith(`${CANVAS_HOST_PATH}/`) ||
pathname === A2UI_PATH ||
pathname.startsWith(`${A2UI_PATH}/`)
);
}
function sanitizeCanvasEntryUrl(rawEntryUrl: string): string | undefined {
try {
const entry = new URL(rawEntryUrl, "http://localhost");
if (entry.origin !== "http://localhost") {
return undefined;
}
if (!isCanvasHttpPath(entry.pathname)) {
return undefined;
}
return `${entry.pathname}${entry.search}${entry.hash}`;
} catch {
return undefined;
}
}
export function resolveCanvasIframeUrl(
entryUrl: string | undefined,
canvasHostUrl?: string | null,
): string | undefined {
const rawEntryUrl = entryUrl?.trim();
if (!rawEntryUrl) {
return undefined;
}
const safeEntryUrl = sanitizeCanvasEntryUrl(rawEntryUrl);
if (!safeEntryUrl) {
return undefined;
}
if (!canvasHostUrl?.trim()) {
return safeEntryUrl;
}
try {
const scopedHostUrl = new URL(canvasHostUrl);
const scopedPrefix = scopedHostUrl.pathname.replace(/\/+$/, "");
if (!scopedPrefix.startsWith(CANVAS_CAPABILITY_PATH_PREFIX)) {
return safeEntryUrl;
}
const entry = new URL(safeEntryUrl, scopedHostUrl.origin);
if (!isCanvasHttpPath(entry.pathname)) {
return safeEntryUrl;
}
entry.protocol = scopedHostUrl.protocol;
entry.username = scopedHostUrl.username;
entry.password = scopedHostUrl.password;
entry.host = scopedHostUrl.host;
entry.pathname = `${scopedPrefix}${entry.pathname}`;
return entry.toString();
} catch {
return safeEntryUrl;
}
}

View File

@@ -23,7 +23,7 @@ describe("shouldReloadHistoryForFinalEvent", () => {
).toBe(true);
});
it("returns false when final event includes assistant payload", () => {
it("returns true when final event includes assistant payload", () => {
expect(
shouldReloadHistoryForFinalEvent({
runId: "run-1",
@@ -31,7 +31,7 @@ describe("shouldReloadHistoryForFinalEvent", () => {
state: "final",
message: { role: "assistant", content: [{ type: "text", text: "done" }] },
}),
).toBe(false);
).toBe(true);
});
it("returns true when final event message role is non-assistant", () => {

View File

@@ -1,17 +1,5 @@
import type { ChatEventPayload } from "./controllers/chat.ts";
import { normalizeLowercaseStringOrEmpty } from "./string-coerce.ts";
export function shouldReloadHistoryForFinalEvent(payload?: ChatEventPayload): boolean {
if (!payload || payload.state !== "final") {
return false;
}
if (!payload.message || typeof payload.message !== "object") {
return true;
}
const message = payload.message as Record<string, unknown>;
const role = normalizeLowercaseStringOrEmpty(message.role);
if (role && role !== "assistant") {
return true;
}
return false;
return Boolean(payload && payload.state === "final");
}

View File

@@ -2,12 +2,18 @@ import { html, nothing } from "lit";
import { unsafeHTML } from "lit/directives/unsafe-html.js";
import { getSafeLocalStorage } from "../../local-storage.ts";
import type { AssistantIdentity } from "../assistant-identity.ts";
import type { EmbedSandboxMode } from "../embed-sandbox.ts";
import { icons } from "../icons.ts";
import { toSanitizedMarkdownHtml } from "../markdown.ts";
import { openExternalUrlSafe } from "../open-external-url.ts";
import { normalizeLowercaseStringOrEmpty } from "../string-coerce.ts";
import type { SidebarContent } from "../sidebar-content.ts";
import { detectTextDirection } from "../text-direction.ts";
import type { MessageGroup, ToolCard } from "../types/chat-types.ts";
import type {
MessageContentItem,
MessageGroup,
NormalizedMessage,
ToolCard,
} from "../types/chat-types.ts";
import { agentLogoUrl } from "../views/agents-utils.ts";
import { renderCopyAsMarkdownButton } from "./copy-as-markdown.ts";
import {
@@ -15,19 +21,36 @@ import {
extractThinkingCached,
formatReasoningMarkdown,
} from "./message-extract.ts";
import { isToolResultMessage, normalizeRoleForGrouping } from "./message-normalizer.ts";
import {
isToolResultMessage,
normalizeMessage,
normalizeRoleForGrouping,
} from "./message-normalizer.ts";
import { isTtsSupported, speakText, stopTts, isTtsSpeaking } from "./speech.ts";
import { extractToolCards, renderToolCardSidebar } from "./tool-cards.ts";
import {
extractToolCards,
renderExpandedToolCardContent,
renderRawOutputToggle,
renderToolCard,
renderToolPreview,
} from "./tool-cards.ts";
type AssistantAttachmentAvailability =
| { status: "checking" }
| { status: "available" }
| { status: "unavailable"; reason: string };
const assistantAttachmentAvailabilityCache = new Map<string, AssistantAttachmentAvailability>();
export function resetAssistantAttachmentAvailabilityCacheForTest() {
assistantAttachmentAvailabilityCache.clear();
}
type ImageBlock = {
url: string;
alt?: string;
};
type AudioClip = {
url: string;
};
function extractImages(message: unknown): ImageBlock[] {
const m = message as Record<string, unknown>;
const content = m.content;
@@ -65,32 +88,6 @@ function extractImages(message: unknown): ImageBlock[] {
return images;
}
function extractAudioClips(message: unknown): AudioClip[] {
const m = message as Record<string, unknown>;
const content = m.content;
const clips: AudioClip[] = [];
if (!Array.isArray(content)) {
return clips;
}
for (const block of content) {
if (typeof block !== "object" || block === null) {
continue;
}
const b = block as Record<string, unknown>;
if (b.type !== "audio") {
continue;
}
const source = b.source as Record<string, unknown> | undefined;
if (source?.type === "base64" && typeof source.data === "string") {
const data = source.data;
const mediaType = (source.media_type as string) || "audio/mpeg";
const url = data.startsWith("data:") ? data : `data:${mediaType};base64,${data}`;
clips.push({ url });
}
}
return clips;
}
export function renderReadingIndicatorGroup(assistant?: AssistantIdentity, basePath?: string) {
return html`
<div class="chat-group assistant">
@@ -109,7 +106,7 @@ export function renderReadingIndicatorGroup(assistant?: AssistantIdentity, baseP
export function renderStreamingGroup(
text: string,
startedAt: number,
onOpenSidebar?: (content: string) => void,
onOpenSidebar?: (content: SidebarContent) => void,
assistant?: AssistantIdentity,
basePath?: string,
) {
@@ -129,6 +126,7 @@ export function renderStreamingGroup(
content: [{ type: "text", text }],
timestamp: startedAt,
},
`stream:${startedAt}`,
{ isStreaming: true, showReasoning: false },
onOpenSidebar,
)}
@@ -144,12 +142,22 @@ export function renderStreamingGroup(
export function renderMessageGroup(
group: MessageGroup,
opts: {
onOpenSidebar?: (content: string) => void;
onOpenSidebar?: (content: SidebarContent) => void;
showReasoning: boolean;
showToolCalls?: boolean;
autoExpandToolCalls?: boolean;
isToolMessageExpanded?: (messageId: string) => boolean;
onToggleToolMessageExpanded?: (messageId: string) => void;
isToolExpanded?: (toolCardId: string) => boolean;
onToggleToolExpanded?: (toolCardId: string) => void;
onRequestUpdate?: () => void;
assistantName?: string;
assistantAvatar?: string | null;
basePath?: string;
localMediaPreviewRoots?: readonly string[];
assistantAttachmentAuthToken?: string | null;
canvasHostUrl?: string | null;
embedSandboxMode?: EmbedSandboxMode;
contextWindow?: number | null;
onDelete?: () => void;
},
@@ -195,10 +203,22 @@ export function renderMessageGroup(
${group.messages.map((item, index) =>
renderGroupedMessage(
item.message,
item.key,
{
isStreaming: group.isStreaming && index === group.messages.length - 1,
showReasoning: opts.showReasoning,
showToolCalls: opts.showToolCalls ?? true,
autoExpandToolCalls: opts.autoExpandToolCalls ?? false,
isToolMessageExpanded: opts.isToolMessageExpanded,
onToggleToolMessageExpanded: opts.onToggleToolMessageExpanded,
isToolExpanded: opts.isToolExpanded,
onToggleToolExpanded: opts.onToggleToolExpanded,
onRequestUpdate: opts.onRequestUpdate,
canvasHostUrl: opts.canvasHostUrl,
basePath: opts.basePath,
localMediaPreviewRoots: opts.localMediaPreviewRoots,
assistantAttachmentAuthToken: opts.assistantAttachmentAuthToken,
embedSandboxMode: opts.embedSandboxMode,
},
opts.onOpenSidebar,
),
@@ -346,15 +366,9 @@ function extractGroupText(group: MessageGroup): string {
return parts.join("\n\n");
}
export const SKIP_DELETE_CONFIRM_KEY = "openclaw:skipDeleteConfirm";
const SKIP_DELETE_CONFIRM_KEY = "openclaw:skipDeleteConfirm";
type DeleteConfirmSide = "left" | "right";
type DeleteConfirmPopover = {
popover: HTMLDivElement;
cancel: HTMLButtonElement;
yes: HTMLButtonElement;
check: HTMLInputElement;
};
function shouldSkipDeleteConfirm(): boolean {
try {
@@ -364,45 +378,6 @@ function shouldSkipDeleteConfirm(): boolean {
}
}
function createDeleteConfirmPopover(side: DeleteConfirmSide): DeleteConfirmPopover {
const popover = document.createElement("div");
popover.className = `chat-delete-confirm chat-delete-confirm--${side}`;
const text = document.createElement("p");
text.className = "chat-delete-confirm__text";
text.textContent = "Delete this message?";
const remember = document.createElement("label");
remember.className = "chat-delete-confirm__remember";
const check = document.createElement("input");
check.className = "chat-delete-confirm__check";
check.type = "checkbox";
const rememberText = document.createElement("span");
rememberText.textContent = "Don't ask again";
remember.append(check, rememberText);
const actions = document.createElement("div");
actions.className = "chat-delete-confirm__actions";
const cancel = document.createElement("button");
cancel.className = "chat-delete-confirm__cancel";
cancel.type = "button";
cancel.textContent = "Cancel";
const yes = document.createElement("button");
yes.className = "chat-delete-confirm__yes";
yes.type = "button";
yes.textContent = "Delete";
actions.append(cancel, yes);
popover.append(text, remember, actions);
return { popover, cancel, yes, check };
}
function renderDeleteButton(onDelete: () => void, side: DeleteConfirmSide) {
return html`
<span class="chat-delete-wrap">
@@ -422,31 +397,43 @@ function renderDeleteButton(onDelete: () => void, side: DeleteConfirmSide) {
existing.remove();
return;
}
const { popover, cancel, yes, check } = createDeleteConfirmPopover(side);
const popover = document.createElement("div");
popover.className = `chat-delete-confirm chat-delete-confirm--${side}`;
popover.innerHTML = `
<p class="chat-delete-confirm__text">Delete this message?</p>
<label class="chat-delete-confirm__remember">
<input type="checkbox" class="chat-delete-confirm__check" />
<span>Don't ask again</span>
</label>
<div class="chat-delete-confirm__actions">
<button class="chat-delete-confirm__cancel" type="button">Cancel</button>
<button class="chat-delete-confirm__yes" type="button">Delete</button>
</div>
`;
wrap.appendChild(popover);
const removePopover = () => {
popover.remove();
document.removeEventListener("click", closeOnOutside, true);
};
const cancel = popover.querySelector(".chat-delete-confirm__cancel")!;
const yes = popover.querySelector(".chat-delete-confirm__yes")!;
const check = popover.querySelector(".chat-delete-confirm__check") as HTMLInputElement;
// Close on click outside.
const closeOnOutside = (evt: MouseEvent) => {
if (!popover.contains(evt.target as Node) && evt.target !== btn) {
removePopover();
}
};
cancel.addEventListener("click", removePopover);
cancel.addEventListener("click", () => popover.remove());
yes.addEventListener("click", () => {
if (check.checked) {
try {
getSafeLocalStorage()?.setItem(SKIP_DELETE_CONFIRM_KEY, "1");
} catch {}
}
removePopover();
popover.remove();
onDelete();
});
// Close on click outside
const closeOnOutside = (evt: MouseEvent) => {
if (!popover.contains(evt.target as Node) && evt.target !== btn) {
popover.remove();
document.removeEventListener("click", closeOnOutside, true);
}
};
requestAnimationFrame(() => document.addEventListener("click", closeOnOutside, true));
}}
>
@@ -611,52 +598,321 @@ function renderMessageImages(images: ImageBlock[]) {
`;
}
function renderMessageAudio(clips: AudioClip[]) {
if (clips.length === 0) {
function renderReplyPill(replyTarget: NormalizedMessage["replyTarget"]) {
if (!replyTarget) {
return nothing;
}
return html`
<div class="chat-message-audio">
${clips.map(
(clip) =>
html`<audio
class="chat-message-audio-el"
controls
preload="metadata"
src=${clip.url}
></audio>`,
)}
<div class="chat-reply-pill">
<span class="chat-reply-pill__icon">${icons.messageSquare}</span>
<span class="chat-reply-pill__label">
${replyTarget.kind === "current"
? "Replying to current message"
: `Replying to ${replyTarget.id}`}
</span>
</div>
`;
}
/** Render tool cards inside a collapsed `<details>` element. */
function renderCollapsedToolCards(
toolCards: ToolCard[],
onOpenSidebar?: (content: string) => void,
) {
const calls = toolCards.filter((c) => c.kind === "call");
const results = toolCards.filter((c) => c.kind === "result");
const totalTools = Math.max(calls.length, results.length) || toolCards.length;
const toolNames = [...new Set(toolCards.map((c) => c.name))];
const summaryLabel =
toolNames.length <= 3
? toolNames.join(", ")
: `${toolNames.slice(0, 2).join(", ")} +${toolNames.length - 2} more`;
function isLocalAssistantAttachmentSource(source: string): boolean {
const trimmed = source.trim();
if (/^\/(?:__openclaw__|media)\//.test(trimmed)) {
return false;
}
return (
trimmed.startsWith("file://") ||
trimmed.startsWith("~") ||
trimmed.startsWith("/") ||
/^[a-zA-Z]:[\\/]/.test(trimmed)
);
}
function normalizeLocalAttachmentPath(source: string): string | null {
const trimmed = source.trim();
if (!isLocalAssistantAttachmentSource(trimmed)) {
return null;
}
if (trimmed.startsWith("file://")) {
try {
const url = new URL(trimmed);
return decodeURIComponent(url.pathname);
} catch {
return null;
}
}
if (trimmed.startsWith("~")) {
return null;
}
return trimmed;
}
function isLocalAttachmentPreviewAllowed(
source: string,
localMediaPreviewRoots: readonly string[],
): boolean {
const normalizedSource = normalizeLocalAttachmentPath(source);
if (!normalizedSource) {
return false;
}
return localMediaPreviewRoots.some((root) => {
const normalizedRoot = root.trim().replace(/[\\/]+$/, "");
return (
normalizedRoot.length > 0 &&
(normalizedSource === normalizedRoot ||
normalizedSource.startsWith(`${normalizedRoot}/`) ||
normalizedSource.startsWith(`${normalizedRoot}\\`))
);
});
}
function buildAssistantAttachmentUrl(
source: string,
basePath?: string,
authToken?: string | null,
): string {
if (!isLocalAssistantAttachmentSource(source)) {
return source;
}
const normalizedBasePath =
basePath && basePath !== "/" ? (basePath.endsWith("/") ? basePath.slice(0, -1) : basePath) : "";
const params = new URLSearchParams({ source });
const normalizedToken = authToken?.trim();
if (normalizedToken) {
params.set("token", normalizedToken);
}
return `${normalizedBasePath}/__openclaw__/assistant-media?${params.toString()}`;
}
function buildAssistantAttachmentMetaUrl(
source: string,
basePath?: string,
authToken?: string | null,
): string {
const attachmentUrl = buildAssistantAttachmentUrl(source, basePath, authToken);
return `${attachmentUrl}${attachmentUrl.includes("?") ? "&" : "?"}meta=1`;
}
function resolveAssistantAttachmentAvailability(
source: string,
localMediaPreviewRoots: readonly string[],
basePath: string | undefined,
authToken: string | null | undefined,
onRequestUpdate: (() => void) | undefined,
): AssistantAttachmentAvailability {
if (!isLocalAssistantAttachmentSource(source)) {
return { status: "available" };
}
if (!isLocalAttachmentPreviewAllowed(source, localMediaPreviewRoots)) {
return { status: "unavailable", reason: "Outside allowed folders" };
}
const normalizedAuthToken = authToken?.trim() ?? "";
const cacheKey = `${basePath ?? ""}::${normalizedAuthToken}::${source}`;
const cached = assistantAttachmentAvailabilityCache.get(cacheKey);
if (cached) {
return cached;
}
assistantAttachmentAvailabilityCache.set(cacheKey, { status: "checking" });
if (typeof fetch === "function") {
void fetch(buildAssistantAttachmentMetaUrl(source, basePath, authToken), {
method: "GET",
headers: { Accept: "application/json" },
credentials: "same-origin",
})
.then(async (res) => {
const payload = (await res.json().catch(() => null)) as {
available?: boolean;
reason?: string;
} | null;
if (payload?.available === true) {
assistantAttachmentAvailabilityCache.set(cacheKey, { status: "available" });
} else {
assistantAttachmentAvailabilityCache.set(cacheKey, {
status: "unavailable",
reason: payload?.reason?.trim() || "Attachment unavailable",
});
}
})
.catch(() => {
assistantAttachmentAvailabilityCache.set(cacheKey, {
status: "unavailable",
reason: "Attachment unavailable",
});
})
.finally(() => {
onRequestUpdate?.();
});
}
return { status: "checking" };
}
function renderAssistantAttachmentStatusCard(params: {
kind: "image" | "audio" | "video" | "document";
label: string;
badge: string;
reason?: string;
}) {
const icon =
params.kind === "image"
? icons.image
: params.kind === "audio"
? icons.mic
: params.kind === "video"
? icons.monitor
: icons.paperclip;
return html`
<details class="chat-tools-collapse">
<summary class="chat-tools-summary">
<span class="chat-tools-summary__icon">${icons.zap}</span>
<span class="chat-tools-summary__count"
>${totalTools} tool${totalTools === 1 ? "" : "s"}</span
<div class="chat-assistant-attachment-card chat-assistant-attachment-card--blocked">
<div class="chat-assistant-attachment-card__header">
<span class="chat-assistant-attachment-card__icon">${icon}</span>
<span class="chat-assistant-attachment-card__title">${params.label}</span>
<span class="chat-assistant-attachment-badge chat-assistant-attachment-badge--muted"
>${params.badge}</span
>
<span class="chat-tools-summary__names">${summaryLabel}</span>
</summary>
<div class="chat-tools-collapse__body">
${toolCards.map((card) => renderToolCardSidebar(card, onOpenSidebar))}
</div>
</details>
${params.reason
? html`<div class="chat-assistant-attachment-card__reason">${params.reason}</div>`
: nothing}
</div>
`;
}
function renderAssistantAttachments(
attachments: Array<Extract<MessageContentItem, { type: "attachment" }>>,
localMediaPreviewRoots: readonly string[],
basePath?: string,
authToken?: string | null,
onRequestUpdate?: () => void,
) {
if (attachments.length === 0) {
return nothing;
}
return html`
<div class="chat-assistant-attachments">
${attachments.map(({ attachment }) => {
const availability = resolveAssistantAttachmentAvailability(
attachment.url,
localMediaPreviewRoots,
basePath,
authToken,
onRequestUpdate,
);
const attachmentUrl =
availability.status === "available"
? buildAssistantAttachmentUrl(attachment.url, basePath, authToken)
: null;
if (attachment.kind === "image") {
if (!attachmentUrl) {
return renderAssistantAttachmentStatusCard({
kind: "image",
label: attachment.label,
badge: availability.status === "checking" ? "Checking..." : "Unavailable",
reason: availability.status === "unavailable" ? availability.reason : undefined,
});
}
return html`
<img
src=${attachmentUrl}
alt=${attachment.label}
class="chat-message-image"
@click=${() => openExternalUrlSafe(attachmentUrl, { allowDataImage: true })}
/>
`;
}
if (attachment.kind === "audio") {
return html`
<div class="chat-assistant-attachment-card chat-assistant-attachment-card--audio">
<div class="chat-assistant-attachment-card__header">
<span class="chat-assistant-attachment-card__title">${attachment.label}</span>
${!attachmentUrl
? html`<span
class="chat-assistant-attachment-badge chat-assistant-attachment-badge--muted"
>${availability.status === "checking" ? "Checking..." : "Unavailable"}</span
>`
: attachment.isVoiceNote
? html`<span class="chat-assistant-attachment-badge">Voice note</span>`
: nothing}
</div>
${attachmentUrl
? html`<audio controls preload="metadata" src=${attachmentUrl}></audio>`
: availability.status === "unavailable"
? html`<div class="chat-assistant-attachment-card__reason">
${availability.reason}
</div>`
: nothing}
</div>
`;
}
if (attachment.kind === "video") {
if (!attachmentUrl) {
return renderAssistantAttachmentStatusCard({
kind: "video",
label: attachment.label,
badge: availability.status === "checking" ? "Checking..." : "Unavailable",
reason: availability.status === "unavailable" ? availability.reason : undefined,
});
}
return html`
<div class="chat-assistant-attachment-card chat-assistant-attachment-card--video">
<video controls preload="metadata" src=${attachmentUrl}></video>
<a
class="chat-assistant-attachment-card__link"
href=${attachmentUrl}
target="_blank"
rel="noreferrer"
>${attachment.label}</a
>
</div>
`;
}
if (!attachmentUrl) {
return renderAssistantAttachmentStatusCard({
kind: "document",
label: attachment.label,
badge: availability.status === "checking" ? "Checking..." : "Unavailable",
reason: availability.status === "unavailable" ? availability.reason : undefined,
});
}
return html`
<div class="chat-assistant-attachment-card">
<span class="chat-assistant-attachment-card__icon">${icons.paperclip}</span>
<a
class="chat-assistant-attachment-card__link"
href=${attachmentUrl}
target="_blank"
rel="noreferrer"
>${attachment.label}</a
>
</div>
`;
})}
</div>
`;
}
function renderInlineToolCards(
toolCards: ToolCard[],
opts: {
messageKey: string;
onOpenSidebar?: (content: SidebarContent) => void;
isToolExpanded?: (toolCardId: string) => boolean;
onToggleToolExpanded?: (toolCardId: string) => void;
canvasHostUrl?: string | null;
embedSandboxMode?: EmbedSandboxMode;
},
) {
return html`
<div class="chat-tools-inline">
${toolCards.map((card, index) =>
renderToolCard(card, {
expanded: opts.isToolExpanded?.(`${opts.messageKey}:toolcard:${index}`) ?? false,
onToggleExpanded: opts.onToggleToolExpanded
? () => opts.onToggleToolExpanded?.(`${opts.messageKey}:toolcard:${index}`)
: () => undefined,
onOpenSidebar: opts.onOpenSidebar,
canvasHostUrl: opts.canvasHostUrl,
embedSandboxMode: opts.embedSandboxMode ?? "powerful",
}),
)}
</div>
`;
}
@@ -705,14 +961,14 @@ function jsonSummaryLabel(parsed: unknown): string {
return "JSON";
}
function renderExpandButton(markdown: string, onOpenSidebar: (content: string) => void) {
function renderExpandButton(markdown: string, onOpenSidebar: (content: SidebarContent) => void) {
return html`
<button
class="btn btn--xs chat-expand-btn"
type="button"
title="Open in canvas"
aria-label="Open in canvas"
@click=${() => onOpenSidebar(markdown)}
@click=${() => onOpenSidebar({ kind: "markdown", content: markdown })}
>
<span class="chat-expand-btn__icon" aria-hidden="true">${icons.panelRightOpen}</span>
</button>
@@ -721,28 +977,53 @@ function renderExpandButton(markdown: string, onOpenSidebar: (content: string) =
function renderGroupedMessage(
message: unknown,
opts: { isStreaming: boolean; showReasoning: boolean; showToolCalls?: boolean },
onOpenSidebar?: (content: string) => void,
messageKey: string,
opts: {
isStreaming: boolean;
showReasoning: boolean;
showToolCalls?: boolean;
autoExpandToolCalls?: boolean;
isToolMessageExpanded?: (messageId: string) => boolean;
onToggleToolMessageExpanded?: (messageId: string) => void;
isToolExpanded?: (toolCardId: string) => boolean;
onToggleToolExpanded?: (toolCardId: string) => void;
onRequestUpdate?: () => void;
canvasHostUrl?: string | null;
basePath?: string;
localMediaPreviewRoots?: readonly string[];
assistantAttachmentAuthToken?: string | null;
embedSandboxMode?: EmbedSandboxMode;
},
onOpenSidebar?: (content: SidebarContent) => void,
) {
const m = message as Record<string, unknown>;
const role = typeof m.role === "string" ? m.role : "unknown";
const normalizedRole = normalizeRoleForGrouping(role);
const normalizedRawRole = normalizeLowercaseStringOrEmpty(role);
const isToolResult =
isToolResultMessage(message) ||
normalizedRawRole === "toolresult" ||
normalizedRawRole === "tool_result" ||
role.toLowerCase() === "toolresult" ||
role.toLowerCase() === "tool_result" ||
typeof m.toolCallId === "string" ||
typeof m.tool_call_id === "string";
const toolCards = (opts.showToolCalls ?? true) ? extractToolCards(message) : [];
const toolCards = (opts.showToolCalls ?? true) ? extractToolCards(message, messageKey) : [];
const hasToolCards = toolCards.length > 0;
const images = extractImages(message);
const hasImages = images.length > 0;
const audioClips = extractAudioClips(message);
const hasAudio = audioClips.length > 0;
const extractedText = extractTextCached(message);
const normalizedMessage = normalizeMessage(message);
const extractedText = normalizedMessage.content
.filter((item): item is Extract<MessageContentItem, { type: "text" }> => item.type === "text")
.map((item) => item.text ?? "")
.join("\n")
.trim();
const assistantAttachments = normalizedMessage.content.filter(
(item): item is Extract<MessageContentItem, { type: "attachment" }> =>
item.type === "attachment",
);
const assistantViewBlocks = normalizedMessage.content.filter(
(item): item is Extract<MessageContentItem, { type: "canvas" }> => item.type === "canvas",
);
const extractedThinking =
opts.showReasoning && role === "assistant" ? extractThinkingCached(message) : null;
const markdownBase = extractedText?.trim() ? extractedText : null;
@@ -754,26 +1035,26 @@ function renderGroupedMessage(
// Detect pure-JSON messages and render as collapsible block
const jsonResult = markdown && !opts.isStreaming ? detectJson(markdown) : null;
const bubbleClasses = [
"chat-bubble",
opts.isStreaming ? "streaming" : "",
"fade-in",
canCopyMarkdown ? "has-copy" : "",
]
const bubbleClasses = ["chat-bubble", opts.isStreaming ? "streaming" : "", "fade-in"]
.filter(Boolean)
.join(" ");
if (!markdown && hasToolCards && isToolResult) {
return renderCollapsedToolCards(toolCards, onOpenSidebar);
}
// Suppress empty bubbles when tool cards are the only content and toggle is off
const visibleToolCards = hasToolCards && (opts.showToolCalls ?? true);
if (!markdown && !visibleToolCards && !hasImages && !hasAudio) {
if (
!markdown &&
!visibleToolCards &&
!hasImages &&
assistantAttachments.length === 0 &&
assistantViewBlocks.length === 0 &&
!normalizedMessage.replyTarget
) {
return nothing;
}
const isToolMessage = normalizedRole === "tool" || isToolResult;
const toolMessageDisclosureId = `toolmsg:${messageKey}`;
const toolMessageExpanded = opts.isToolMessageExpanded?.(toolMessageDisclosureId) ?? false;
const toolNames = [...new Set(toolCards.map((c) => c.name))];
const toolSummaryLabel =
toolNames.length <= 3
@@ -781,11 +1062,19 @@ function renderGroupedMessage(
: `${toolNames.slice(0, 2).join(", ")} +${toolNames.length - 2} more`;
const toolPreview =
markdown && !toolSummaryLabel ? markdown.trim().replace(/\s+/g, " ").slice(0, 120) : "";
const singleToolCard = toolCards.length === 1 ? toolCards[0] : null;
const toolMessageLabel =
singleToolCard && !markdown && !hasImages
? singleToolCard.outputText?.trim()
? "Tool output"
: "Tool call"
: "Tool output";
const hasActions = canCopyMarkdown || canExpand;
return html`
<div class="${bubbleClasses}">
${renderReplyPill(normalizedMessage.replyTarget)}
${hasActions
? html`<div class="chat-bubble-actions">
${canExpand ? renderExpandButton(markdown!, onOpenSidebar!) : nothing}
@@ -794,47 +1083,106 @@ function renderGroupedMessage(
: nothing}
${isToolMessage
? html`
<details class="chat-tool-msg-collapse">
<summary class="chat-tool-msg-summary">
<div
class="chat-tool-msg-collapse chat-tool-msg-collapse--manual ${toolMessageExpanded
? "is-open"
: ""}"
>
<button
class="chat-tool-msg-summary"
type="button"
aria-expanded=${String(toolMessageExpanded)}
@click=${() => opts.onToggleToolMessageExpanded?.(toolMessageDisclosureId)}
>
<span class="chat-tool-msg-summary__icon">${icons.zap}</span>
<span class="chat-tool-msg-summary__label">Tool output</span>
<span class="chat-tool-msg-summary__label">${toolMessageLabel}</span>
${toolSummaryLabel
? html`<span class="chat-tool-msg-summary__names">${toolSummaryLabel}</span>`
: toolPreview
? html`<span class="chat-tool-msg-summary__preview">${toolPreview}</span>`
: nothing}
</summary>
<div class="chat-tool-msg-body">
${renderMessageImages(images)} ${renderMessageAudio(audioClips)}
${reasoningMarkdown
? html`<div class="chat-thinking">
${unsafeHTML(toSanitizedMarkdownHtml(reasoningMarkdown))}
</div>`
: nothing}
${jsonResult
? html`<details class="chat-json-collapse">
<summary class="chat-json-summary">
<span class="chat-json-badge">JSON</span>
<span class="chat-json-label">${jsonSummaryLabel(jsonResult.parsed)}</span>
</summary>
<pre class="chat-json-content"><code>${jsonResult.pretty}</code></pre>
</details>`
: markdown
? html`<div class="chat-text" dir="${detectTextDirection(markdown)}">
${unsafeHTML(toSanitizedMarkdownHtml(markdown))}
</div>`
: nothing}
${hasToolCards ? renderCollapsedToolCards(toolCards, onOpenSidebar) : nothing}
</div>
</details>
</button>
${toolMessageExpanded
? html`
<div class="chat-tool-msg-body">
${renderMessageImages(images)}
${renderAssistantAttachments(
assistantAttachments,
opts.localMediaPreviewRoots ?? [],
opts.basePath,
opts.assistantAttachmentAuthToken,
opts.onRequestUpdate,
)}
${reasoningMarkdown
? html`<div class="chat-thinking">
${unsafeHTML(toSanitizedMarkdownHtml(reasoningMarkdown))}
</div>`
: nothing}
${jsonResult
? html`<details
class="chat-json-collapse"
?open=${Boolean(opts.autoExpandToolCalls)}
>
<summary class="chat-json-summary">
<span class="chat-json-badge">JSON</span>
<span class="chat-json-label"
>${jsonSummaryLabel(jsonResult.parsed)}</span
>
</summary>
<pre class="chat-json-content"><code>${jsonResult.pretty}</code></pre>
</details>`
: markdown
? html`<div class="chat-text" dir="${detectTextDirection(markdown)}">
${unsafeHTML(toSanitizedMarkdownHtml(markdown))}
</div>`
: nothing}
${hasToolCards
? singleToolCard && !markdown && !hasImages
? renderExpandedToolCardContent(
singleToolCard,
onOpenSidebar,
opts.canvasHostUrl,
opts.embedSandboxMode ?? "powerful",
)
: renderInlineToolCards(toolCards, {
messageKey,
onOpenSidebar,
isToolExpanded: opts.isToolExpanded,
onToggleToolExpanded: opts.onToggleToolExpanded,
canvasHostUrl: opts.canvasHostUrl,
embedSandboxMode: opts.embedSandboxMode ?? "powerful",
})
: nothing}
</div>
`
: nothing}
</div>
`
: html`
${renderMessageImages(images)} ${renderMessageAudio(audioClips)}
${renderMessageImages(images)}
${renderAssistantAttachments(
assistantAttachments,
opts.localMediaPreviewRoots ?? [],
opts.basePath,
opts.assistantAttachmentAuthToken,
opts.onRequestUpdate,
)}
${reasoningMarkdown
? html`<div class="chat-thinking">
${unsafeHTML(toSanitizedMarkdownHtml(reasoningMarkdown))}
</div>`
: nothing}
${normalizedRole === "assistant" && assistantViewBlocks.length > 0
? html`${assistantViewBlocks.map(
(block) => html`${renderToolPreview(block.preview, "chat_message", {
onOpenSidebar,
rawText: block.rawText ?? null,
canvasHostUrl: opts.canvasHostUrl,
embedSandboxMode: opts.embedSandboxMode ?? "powerful",
})}
${block.rawText ? renderRawOutputToggle(block.rawText) : nothing}`,
)}`
: nothing}
${jsonResult
? html`<details class="chat-json-collapse">
<summary class="chat-json-summary">
@@ -848,7 +1196,16 @@ function renderGroupedMessage(
${unsafeHTML(toSanitizedMarkdownHtml(markdown))}
</div>`
: nothing}
${hasToolCards ? renderCollapsedToolCards(toolCards, onOpenSidebar) : nothing}
${hasToolCards
? renderInlineToolCards(toolCards, {
messageKey,
onOpenSidebar,
isToolExpanded: opts.isToolExpanded,
onToggleToolExpanded: opts.onToggleToolExpanded,
canvasHostUrl: opts.canvasHostUrl,
embedSandboxMode: opts.embedSandboxMode ?? "powerful",
})
: nothing}
`}
</div>
`;

View File

@@ -68,6 +68,172 @@ describe("message-normalizer", () => {
expect(result.content).toEqual([{ type: "text", text: "Alternative format" }]);
});
it("expands [canvas] shortcodes into canvas blocks", () => {
const result = normalizeMessage({
role: "assistant",
content: 'Here.\n[canvas ref="cv_status" title="Status" height="320" /]',
});
expect(result.content).toEqual([
{ type: "text", text: "Here." },
{
type: "canvas",
preview: {
kind: "canvas",
surface: "assistant_message",
render: "url",
viewId: "cv_status",
url: "/__openclaw__/canvas/documents/cv_status/index.html",
title: "Status",
preferredHeight: 320,
},
rawText: null,
},
]);
});
it("ignores [canvas] shortcodes inside fenced code blocks", () => {
const result = normalizeMessage({
role: "assistant",
content: '```text\n[canvas ref="cv_status" /]\n```',
});
expect(result.content).toEqual([
{
type: "text",
text: '```text\n[canvas ref="cv_status" /]\n```',
},
]);
});
it("leaves block-form inline html canvas shortcodes as plain text", () => {
const result = normalizeMessage({
role: "assistant",
content: '[canvas content_type="html" title="Status"]\n<div>Ready</div>\n[/canvas]',
});
expect(result.content).toEqual([
{
type: "text",
text: '[canvas content_type="html" title="Status"]\n<div>Ready</div>\n[/canvas]',
},
]);
});
it("extracts MEDIA attachments and reply metadata from assistant text", () => {
const result = normalizeMessage({
role: "assistant",
content:
"[[reply_to:thread-123]]Intro\nMEDIA:https://example.com/image.png\nOutro\nMEDIA:https://example.com/voice.ogg\n[[audio_as_voice]]",
});
expect(result.replyTarget).toEqual({ kind: "id", id: "thread-123" });
expect(result.audioAsVoice).toBe(true);
expect(result.content).toEqual([
{ type: "text", text: "Intro" },
{
type: "attachment",
attachment: {
url: "https://example.com/image.png",
kind: "image",
label: "image.png",
mimeType: "image/png",
},
},
{ type: "text", text: "Outro" },
{
type: "attachment",
attachment: {
url: "https://example.com/voice.ogg",
kind: "audio",
label: "voice.ogg",
mimeType: "audio/ogg",
isVoiceNote: true,
},
},
]);
});
it("marks media-only audio attachments as voice notes when audio_as_voice is present", () => {
const result = normalizeMessage({
role: "assistant",
content: "MEDIA:https://example.com/voice.ogg\n[[audio_as_voice]]",
});
expect(result.audioAsVoice).toBe(true);
expect(result.content).toEqual([
{
type: "attachment",
attachment: {
url: "https://example.com/voice.ogg",
kind: "audio",
label: "voice.ogg",
mimeType: "audio/ogg",
isVoiceNote: true,
},
},
]);
});
it("keeps valid local MEDIA paths as assistant attachments", () => {
const result = normalizeMessage({
role: "assistant",
content: "Hello\nMEDIA:/tmp/openclaw/test-image.png\nWorld",
});
expect(result.content).toEqual([
{ type: "text", text: "Hello" },
{
type: "attachment",
attachment: {
url: "/tmp/openclaw/test-image.png",
kind: "image",
label: "test-image.png",
mimeType: "image/png",
},
},
{ type: "text", text: "World" },
]);
});
it("keeps spaced local filenames together instead of leaking suffix text", () => {
const result = normalizeMessage({
role: "assistant",
content: "MEDIA:/tmp/openclaw/shinkansen kato - Google Shopping.pdf",
});
expect(result.content).toEqual([
{
type: "attachment",
attachment: {
url: "/tmp/openclaw/shinkansen kato - Google Shopping.pdf",
kind: "document",
label: "shinkansen kato - Google Shopping.pdf",
mimeType: "application/pdf",
},
},
]);
});
it("does not fall back to raw text when an invalid MEDIA line is stripped", () => {
const result = normalizeMessage({
role: "assistant",
content: "MEDIA:~/Pictures/My File.png",
});
expect(result.content).toEqual([]);
});
it("strips reply_to_current without rendering a quoted preview", () => {
const result = normalizeMessage({
role: "assistant",
content: "[[reply_to_current]]\nReply body",
});
expect(result.replyTarget).toEqual({ kind: "current" });
expect(result.content).toEqual([{ type: "text", text: "Reply body" }]);
});
it("detects tool result by toolCallId", () => {
const result = normalizeMessage({
role: "assistant",

View File

@@ -3,14 +3,219 @@
*/
import { stripInboundMetadata } from "../../../../src/auto-reply/reply/strip-inbound-meta.js";
import { extractCanvasShortcodes } from "../../../../src/chat/canvas-render.js";
import {
isToolCallContentType,
isToolResultContentType,
resolveToolBlockArgs,
} from "../../../../src/chat/tool-content.js";
import { normalizeLowercaseStringOrEmpty } from "../string-coerce.ts";
import { mediaKindFromMime } from "../../../../src/media/constants.js";
import { splitMediaFromOutput } from "../../../../src/media/parse.js";
import { parseInlineDirectives } from "../../../../src/utils/directive-tags.js";
import type { NormalizedMessage, MessageContentItem } from "../types/chat-types.ts";
function coerceCanvasPreview(
value: unknown,
):
| Extract<NonNullable<NormalizedMessage["content"][number]>, { type: "canvas" }>["preview"]
| null {
if (!value || typeof value !== "object" || Array.isArray(value)) {
return null;
}
const preview = value as Record<string, unknown>;
if (preview.kind !== "canvas" || preview.surface === "tool_card") {
return null;
}
const render = preview.render === "url" ? "url" : null;
if (!render) {
return null;
}
return {
kind: "canvas",
surface: "assistant_message",
render,
...(typeof preview.title === "string" ? { title: preview.title } : {}),
...(typeof preview.preferredHeight === "number"
? { preferredHeight: preview.preferredHeight }
: {}),
...(typeof preview.url === "string" ? { url: preview.url } : {}),
...(typeof preview.viewId === "string" ? { viewId: preview.viewId } : {}),
...(typeof preview.className === "string" ? { className: preview.className } : {}),
...(typeof preview.style === "string" ? { style: preview.style } : {}),
};
}
function isRenderableAssistantAttachment(url: string): boolean {
const trimmed = url.trim();
return (
/^https?:\/\//i.test(trimmed) ||
/^data:(?:image|audio|video)\//i.test(trimmed) ||
/^\/(?:__openclaw__|media)\//.test(trimmed) ||
trimmed.startsWith("file://") ||
trimmed.startsWith("~") ||
trimmed.startsWith("/") ||
/^[a-zA-Z]:[\\/]/.test(trimmed)
);
}
const MIME_BY_EXT: Record<string, string> = {
png: "image/png",
jpg: "image/jpeg",
jpeg: "image/jpeg",
webp: "image/webp",
gif: "image/gif",
heic: "image/heic",
heif: "image/heif",
ogg: "audio/ogg",
oga: "audio/ogg",
mp3: "audio/mpeg",
wav: "audio/wav",
flac: "audio/flac",
aac: "audio/aac",
opus: "audio/opus",
m4a: "audio/mp4",
mp4: "video/mp4",
mov: "video/quicktime",
pdf: "application/pdf",
txt: "text/plain",
md: "text/markdown",
csv: "text/csv",
json: "application/json",
zip: "application/zip",
};
function getFileExtension(url: string): string | undefined {
const trimmed = url.trim();
if (!trimmed) {
return undefined;
}
const source = (() => {
try {
if (/^https?:\/\//i.test(trimmed)) {
return new URL(trimmed).pathname;
}
} catch {}
return trimmed;
})();
const fileName = source.split(/[\\/]/).pop() ?? source;
const match = /\.([a-zA-Z0-9]+)$/.exec(fileName);
return match?.[1]?.toLowerCase();
}
function mimeTypeFromUrl(url: string): string | undefined {
const ext = getFileExtension(url);
return ext ? MIME_BY_EXT[ext] : undefined;
}
function inferAttachmentKind(url: string): {
kind: "image" | "audio" | "video" | "document";
mimeType?: string;
label: string;
} {
const mimeType = mimeTypeFromUrl(url);
const kind = mediaKindFromMime(mimeType) ?? "document";
const label = (() => {
try {
if (/^https?:\/\//i.test(url)) {
const parsed = new URL(url);
const name = parsed.pathname.split("/").pop()?.trim();
return name || parsed.hostname || url;
}
} catch {}
const name = url.split(/[\\/]/).pop()?.trim();
return name || url;
})();
return { kind, mimeType, label };
}
function mergeAdjacentTextItems(items: MessageContentItem[]): MessageContentItem[] {
const merged: MessageContentItem[] = [];
for (const item of items) {
const previous = merged[merged.length - 1];
if (item.type === "text" && previous?.type === "text") {
previous.text = [previous.text, item.text].filter((value) => value !== undefined).join("\n");
continue;
}
merged.push(item);
}
return merged.filter((item) => item.type !== "text" || Boolean(item.text?.trim()));
}
function expandTextContent(text: string): {
content: MessageContentItem[];
audioAsVoice: boolean;
replyTarget: NormalizedMessage["replyTarget"];
} {
const extracted = extractCanvasShortcodes(text);
const parsed = splitMediaFromOutput(extracted.text);
const parts: MessageContentItem[] = [];
let audioAsVoice = parsed.audioAsVoice === true;
let replyTarget: NormalizedMessage["replyTarget"] = null;
const segments = parsed.segments ?? [{ type: "text" as const, text: parsed.text }];
for (const segment of segments) {
if (segment.type === "media") {
if (!isRenderableAssistantAttachment(segment.url)) {
continue;
}
const inferred = inferAttachmentKind(segment.url);
parts.push({
type: "attachment",
attachment: {
url: segment.url,
kind: inferred.kind,
label: inferred.label,
mimeType: inferred.mimeType,
},
});
continue;
}
const directives = parseInlineDirectives(segment.text, {
stripAudioTag: true,
stripReplyTags: true,
});
audioAsVoice = audioAsVoice || directives.audioAsVoice;
if (directives.replyToExplicitId) {
replyTarget = { kind: "id", id: directives.replyToExplicitId };
} else if (directives.replyToCurrent && replyTarget === null) {
replyTarget = { kind: "current" };
}
if (directives.text) {
parts.push({ type: "text", text: directives.text });
}
}
for (const preview of extracted.previews) {
parts.push({ type: "canvas", preview, rawText: null });
}
const content = mergeAdjacentTextItems(
parts.map((item) => {
if (item.type === "attachment" && item.attachment.kind === "audio" && audioAsVoice) {
return {
...item,
attachment: {
...item.attachment,
isVoiceNote: true,
},
};
}
return item;
}),
);
return {
content:
content.length > 0
? content
: parsed.text.trim().length > 0
? [{ type: "text", text: parsed.text }]
: [],
audioAsVoice,
replyTarget,
};
}
/**
* Normalize a raw message object into a consistent structure.
*/
@@ -39,18 +244,62 @@ export function normalizeMessage(message: unknown): NormalizedMessage {
// Extract content
let content: MessageContentItem[] = [];
let audioAsVoice = false;
let replyTarget: NormalizedMessage["replyTarget"] = null;
if (typeof m.content === "string") {
content = [{ type: "text", text: m.content }];
const expanded = expandTextContent(m.content);
content = expanded.content;
audioAsVoice = expanded.audioAsVoice;
replyTarget = expanded.replyTarget;
} else if (Array.isArray(m.content)) {
content = m.content.map((item: Record<string, unknown>) => ({
type: (item.type as MessageContentItem["type"]) || "text",
text: item.text as string | undefined,
name: item.name as string | undefined,
args: resolveToolBlockArgs(item),
}));
content = m.content.flatMap((item: Record<string, unknown>) => {
if (
item.type === "canvas" &&
item.preview &&
typeof item.preview === "object" &&
!Array.isArray(item.preview)
) {
const preview = coerceCanvasPreview(item.preview);
if (!preview) {
return [];
}
return [
{
type: "canvas" as const,
preview,
rawText: typeof item.rawText === "string" ? item.rawText : null,
},
];
}
if (item.type === "text" && typeof item.text === "string") {
const expanded = expandTextContent(item.text);
audioAsVoice = audioAsVoice || expanded.audioAsVoice;
if (expanded.replyTarget?.kind === "id") {
replyTarget = expanded.replyTarget;
} else if (expanded.replyTarget?.kind === "current" && replyTarget === null) {
replyTarget = expanded.replyTarget;
}
return expanded.content;
}
return [
{
type:
(item.type as Extract<
MessageContentItem,
{ type: "text" | "tool_call" | "tool_result" }
>["type"]) || "text",
text: item.text as string | undefined,
name: item.name as string | undefined,
args: resolveToolBlockArgs(item),
},
];
});
} else if (typeof m.text === "string") {
content = [{ type: "text", text: m.text }];
const expanded = expandTextContent(m.text);
content = expanded.content;
audioAsVoice = expanded.audioAsVoice;
replyTarget = expanded.replyTarget;
}
const timestamp = typeof m.timestamp === "number" ? m.timestamp : Date.now();
@@ -68,14 +317,22 @@ export function normalizeMessage(message: unknown): NormalizedMessage {
});
}
return { role, content, timestamp, id, senderLabel };
return {
role,
content,
timestamp,
id,
senderLabel,
...(audioAsVoice ? { audioAsVoice: true } : {}),
...(replyTarget ? { replyTarget } : {}),
};
}
/**
* Normalize role for grouping purposes.
*/
export function normalizeRoleForGrouping(role: string): string {
const lower = normalizeLowercaseStringOrEmpty(role);
const lower = role.toLowerCase();
// Preserve original casing when it's already a core role.
if (role === "user" || role === "User") {
return role;
@@ -103,6 +360,6 @@ export function normalizeRoleForGrouping(role: string): string {
*/
export function isToolResultMessage(message: unknown): boolean {
const m = message as Record<string, unknown>;
const role = normalizeLowercaseStringOrEmpty(m.role);
const role = typeof m.role === "string" ? m.role.toLowerCase() : "";
return role === "toolresult" || role === "tool_result";
}

View File

@@ -1,34 +1,401 @@
/* @vitest-environment jsdom */
import { render } from "lit";
import { describe, expect, it } from "vitest";
import { extractToolCards, renderToolCardSidebar } from "./tool-cards.ts";
import { describe, expect, it, vi } from "vitest";
import { buildToolCardSidebarContent, extractToolCards, renderToolCard } from "./tool-cards.ts";
describe("tool cards", () => {
it("renders anthropic tool_use input details in tool cards", () => {
const cards = extractToolCards({
role: "assistant",
content: [
{
type: "tool_use",
id: "toolu_123",
name: "Bash",
input: { command: 'time claude -p "say ok"' },
},
],
});
describe("tool-cards", () => {
it("pretty-prints structured args and pairs tool output onto the same card", () => {
const cards = extractToolCards(
{
role: "assistant",
toolCallId: "call-1",
content: [
{
type: "toolcall",
id: "call-1",
name: "browser.open",
arguments: { url: "https://example.com", retry: 0 },
},
{
type: "toolresult",
id: "call-1",
name: "browser.open",
text: "Opened page",
},
],
},
"msg:1",
);
expect(cards).toHaveLength(1);
expect(cards[0]).toMatchObject({
kind: "call",
name: "Bash",
args: { command: 'time claude -p "say ok"' },
id: "msg:1:call-1",
name: "browser.open",
outputText: "Opened page",
});
expect(cards[0]?.inputText).toContain('"url": "https://example.com"');
expect(cards[0]?.inputText).toContain('"retry": 0');
});
it("preserves string args verbatim and keeps empty-output cards", () => {
const cards = extractToolCards(
{
role: "assistant",
toolCallId: "call-2",
content: [
{
type: "toolcall",
name: "deck_manage",
arguments: "with Example Deck",
},
],
},
"msg:2",
);
expect(cards).toHaveLength(1);
expect(cards[0]?.inputText).toBe("with Example Deck");
expect(cards[0]?.outputText).toBeUndefined();
});
it("builds sidebar content with input and empty output status", () => {
const [card] = extractToolCards(
{
role: "assistant",
toolCallId: "call-3",
content: [
{
type: "toolcall",
name: "deck_manage",
arguments: "with Example Deck",
},
],
},
"msg:3",
);
const sidebar = buildToolCardSidebarContent(card);
expect(sidebar).toContain("## Deck Manage");
expect(sidebar).toContain("### Tool input");
expect(sidebar).toContain("with Example Deck");
expect(sidebar).toContain("### Tool output");
expect(sidebar).toContain("No output");
});
it("extracts canvas handle payloads into canvas previews", () => {
const [card] = extractToolCards(
{
role: "tool",
toolName: "canvas_render",
content: JSON.stringify({
kind: "canvas",
view: {
backend: "canvas",
id: "cv_inline",
url: "/__openclaw__/canvas/documents/cv_inline/index.html",
},
presentation: {
target: "assistant_message",
title: "Inline demo",
preferred_height: 420,
},
}),
},
"msg:view:1",
);
expect(card?.preview).toMatchObject({
kind: "canvas",
surface: "assistant_message",
render: "url",
viewId: "cv_inline",
url: "/__openclaw__/canvas/documents/cv_inline/index.html",
title: "Inline demo",
preferredHeight: 420,
});
});
it("drops tool_card-targeted canvas payloads", () => {
const [card] = extractToolCards(
{
role: "tool",
toolName: "canvas_render",
content: JSON.stringify({
kind: "canvas",
view: {
backend: "canvas",
id: "cv_tool_card",
url: "/__openclaw__/canvas/documents/cv_tool_card/index.html",
},
presentation: {
target: "tool_card",
title: "Tool card demo",
},
}),
},
"msg:view:2",
);
expect(card?.preview).toBeUndefined();
});
it("does not extract inline-html canvas payloads into canvas previews", () => {
const [card] = extractToolCards(
{
role: "tool",
toolName: "canvas_render",
content: JSON.stringify({
kind: "canvas",
source: {
type: "html",
content: "<div>hello</div>",
},
presentation: {
target: "assistant_message",
title: "Status",
preferred_height: 300,
},
}),
},
"msg:view:3",
);
expect(card?.preview).toBeUndefined();
});
it("does not create a view preview for malformed json output", () => {
const [card] = extractToolCards(
{
role: "tool",
toolName: "canvas_render",
content: '{"kind":"present_view","view":{"id":"broken"}',
},
"msg:view:4",
);
expect(card?.preview).toBeUndefined();
});
it("does not create a view preview for generic tool text output", () => {
const [card] = extractToolCards(
{
role: "tool",
toolName: "browser.open",
content: "present_view: cv_widget",
},
"msg:view:5",
);
expect(card?.preview).toBeUndefined();
});
it("renders expanded cards with inline input and output sections", () => {
const container = document.createElement("div");
render(renderToolCardSidebar(cards[0]), container);
const toggle = vi.fn();
render(
renderToolCard(
{
id: "msg:4:call-4",
name: "browser.open",
args: { url: "https://example.com" },
inputText: '{\n "url": "https://example.com"\n}',
outputText: "Opened page",
},
{ expanded: true, onToggleExpanded: toggle },
),
container,
);
expect(container.textContent).toContain('time claude -p "say ok"');
expect(container.textContent).toContain("Bash");
expect(container.textContent).toContain("Tool input");
expect(container.textContent).toContain("Tool output");
expect(container.textContent).toContain("https://example.com");
expect(container.textContent).toContain("Opened page");
});
it("renders expanded tool calls without an inline output block when no output is present", () => {
const container = document.createElement("div");
render(
renderToolCard(
{
id: "msg:4b:call-4b",
name: "sessions_spawn",
args: { mode: "session", thread: true },
inputText: '{\n "mode": "session",\n "thread": true\n}',
},
{ expanded: true, onToggleExpanded: vi.fn() },
),
container,
);
expect(container.textContent).toContain("Tool input");
expect(container.textContent).toContain('"thread": true');
expect(container.textContent).not.toContain("Tool output");
expect(container.textContent).not.toContain("No output");
});
it("labels collapsed tool calls as tool call", () => {
const container = document.createElement("div");
render(
renderToolCard(
{
id: "msg:5:call-5",
name: "sessions_spawn",
args: { mode: "run" },
inputText: '{\n "mode": "run"\n}',
},
{ expanded: false, onToggleExpanded: vi.fn() },
),
container,
);
expect(container.textContent).toContain("Tool call");
expect(container.textContent).not.toContain("Tool input");
const summaryButton = container.querySelector("button.chat-tool-msg-summary");
expect(summaryButton).not.toBeNull();
expect(summaryButton?.getAttribute("aria-expanded")).toBe("false");
});
it("does not render inline preview frames inside tool rows anymore", () => {
const container = document.createElement("div");
render(
renderToolCard(
{
id: "msg:view:6",
name: "canvas_render",
outputText: JSON.stringify({
kind: "canvas",
source: {
type: "html",
content: '<div onclick="alert(1)">front<script>window.bad = true;</script></div>',
},
presentation: {
target: "tool_card",
title: "Status view",
},
}),
preview: {
kind: "canvas",
surface: "assistant_message",
render: "url",
url: "/__openclaw__/canvas/documents/cv_status/index.html",
title: "Status view",
},
},
{ expanded: true, onToggleExpanded: vi.fn() },
),
container,
);
const rawToggle = container.querySelector<HTMLButtonElement>(".chat-tool-card__raw-toggle");
const rawBody = container.querySelector<HTMLElement>(".chat-tool-card__raw-body");
expect(container.querySelector(".chat-tool-card__preview-frame")).toBeNull();
expect(rawToggle?.getAttribute("aria-expanded")).toBe("false");
expect(rawBody?.hidden).toBe(true);
rawToggle?.dispatchEvent(new MouseEvent("click", { bubbles: true }));
expect(rawToggle?.getAttribute("aria-expanded")).toBe("true");
expect(rawBody?.hidden).toBe(false);
});
it("keeps raw details for legacy canvas tool output without rendering tool-row previews", () => {
const container = document.createElement("div");
render(
renderToolCard(
{
id: "msg:view:7",
name: "canvas_render",
outputText: JSON.stringify({
kind: "canvas",
view: {
backend: "canvas",
id: "cv_counter",
url: "/__openclaw__/canvas/documents/cv_counter/index.html",
title: "Counter demo",
preferred_height: 480,
},
presentation: {
target: "tool_card",
},
}),
preview: {
kind: "canvas",
surface: "assistant_message",
render: "url",
viewId: "cv_counter",
title: "Counter demo",
url: "/__openclaw__/canvas/documents/cv_counter/index.html",
preferredHeight: 480,
},
},
{ expanded: true, onToggleExpanded: vi.fn() },
),
container,
);
const rawToggle = container.querySelector<HTMLButtonElement>(".chat-tool-card__raw-toggle");
const rawBody = container.querySelector<HTMLElement>(".chat-tool-card__raw-body");
expect(container.textContent).toContain("Counter demo");
expect(container.querySelector(".chat-tool-card__preview-frame")).toBeNull();
expect(rawToggle?.getAttribute("aria-expanded")).toBe("false");
expect(rawBody?.hidden).toBe(true);
rawToggle?.dispatchEvent(new MouseEvent("click", { bubbles: true }));
expect(rawToggle?.getAttribute("aria-expanded")).toBe("true");
expect(rawBody?.hidden).toBe(false);
expect(rawBody?.textContent).toContain('"kind":"canvas"');
});
it("opens assistant-surface canvas payloads in the sidebar when explicitly requested", () => {
const container = document.createElement("div");
const onOpenSidebar = vi.fn();
render(
renderToolCard(
{
id: "msg:view:8",
name: "canvas_render",
outputText: JSON.stringify({
kind: "canvas",
view: {
backend: "canvas",
id: "cv_sidebar",
url: "/__openclaw__/canvas/documents/cv_sidebar/index.html",
title: "Player",
preferred_height: 360,
},
presentation: {
target: "assistant_message",
},
}),
preview: {
kind: "canvas",
surface: "assistant_message",
render: "url",
viewId: "cv_sidebar",
url: "/__openclaw__/canvas/documents/cv_sidebar/index.html",
title: "Player",
preferredHeight: 360,
},
},
{ expanded: true, onToggleExpanded: vi.fn(), onOpenSidebar },
),
container,
);
const sidebarButton = container.querySelector<HTMLButtonElement>(".chat-tool-card__action-btn");
sidebarButton?.dispatchEvent(new MouseEvent("click", { bubbles: true }));
expect(sidebarButton).not.toBeNull();
expect(onOpenSidebar).toHaveBeenCalledWith(
expect.objectContaining({
kind: "canvas",
docId: "cv_sidebar",
entryUrl: "/__openclaw__/canvas/documents/cv_sidebar/index.html",
}),
);
});
});

View File

@@ -1,118 +1,19 @@
import { html, nothing } from "lit";
import {
isToolCallContentType,
isToolResultContentType,
resolveToolBlockArgs,
} from "../../../../src/chat/tool-content.js";
import { extractCanvasFromText } from "../../../../src/chat/canvas-render.js";
import { resolveCanvasIframeUrl } from "../canvas-url.ts";
import { resolveEmbedSandbox, type EmbedSandboxMode } from "../embed-sandbox.ts";
import { icons } from "../icons.ts";
import type { SidebarContent } from "../sidebar-content.ts";
import { formatToolDetail, resolveToolDisplay } from "../tool-display.ts";
import type { ToolCard } from "../types/chat-types.ts";
import { TOOL_INLINE_THRESHOLD } from "./constants.ts";
import { extractTextCached } from "./message-extract.ts";
import { isToolResultMessage } from "./message-normalizer.ts";
import { formatToolOutputForSidebar, getTruncatedPreview } from "./tool-helpers.ts";
export function extractToolCards(message: unknown): ToolCard[] {
const m = message as Record<string, unknown>;
const content = normalizeContent(m.content);
const cards: ToolCard[] = [];
export type ToolPreview = NonNullable<ToolCard["preview"]>;
for (const item of content) {
const isToolCall =
isToolCallContentType(item.type) ||
(typeof item.name === "string" && resolveToolBlockArgs(item) != null);
if (isToolCall) {
cards.push({
kind: "call",
name: (item.name as string) ?? "tool",
args: coerceArgs(resolveToolBlockArgs(item)),
});
}
}
for (const item of content) {
if (!isToolResultContentType(item.type)) {
continue;
}
const text = extractToolText(item);
const name = typeof item.name === "string" ? item.name : "tool";
cards.push({ kind: "result", name, text });
}
if (isToolResultMessage(message) && !cards.some((card) => card.kind === "result")) {
const name =
(typeof m.toolName === "string" && m.toolName) ||
(typeof m.tool_name === "string" && m.tool_name) ||
"tool";
const text = extractTextCached(message) ?? undefined;
cards.push({ kind: "result", name, text });
}
return cards;
}
export function renderToolCardSidebar(card: ToolCard, onOpenSidebar?: (content: string) => void) {
const display = resolveToolDisplay({ name: card.name, args: card.args });
const detail = formatToolDetail(display);
const hasText = Boolean(card.text?.trim());
const canClick = Boolean(onOpenSidebar);
const handleClick = canClick
? () => {
if (hasText) {
onOpenSidebar!(formatToolOutputForSidebar(card.text!));
return;
}
const info = `## ${display.label}\n\n${
detail ? `**Command:** \`${detail}\`\n\n` : ""
}*No output — tool completed successfully.*`;
onOpenSidebar!(info);
}
: undefined;
const isShort = hasText && (card.text?.length ?? 0) <= TOOL_INLINE_THRESHOLD;
const showCollapsed = hasText && !isShort;
const showInline = hasText && isShort;
const isEmpty = !hasText;
return html`
<div
class="chat-tool-card ${canClick ? "chat-tool-card--clickable" : ""}"
@click=${handleClick}
role=${canClick ? "button" : nothing}
tabindex=${canClick ? "0" : nothing}
@keydown=${canClick
? (e: KeyboardEvent) => {
if (e.key !== "Enter" && e.key !== " ") {
return;
}
e.preventDefault();
handleClick?.();
}
: nothing}
>
<div class="chat-tool-card__header">
<div class="chat-tool-card__title">
<span class="chat-tool-card__icon">${icons[display.icon]}</span>
<span>${display.label}</span>
</div>
${canClick
? html`<span class="chat-tool-card__action"
>${hasText ? "View" : ""} ${icons.check}</span
>`
: nothing}
${isEmpty && !canClick
? html`<span class="chat-tool-card__status">${icons.check}</span>`
: nothing}
</div>
${detail ? html`<div class="chat-tool-card__detail">${detail}</div>` : nothing}
${isEmpty ? html` <div class="chat-tool-card__status-text muted">Completed</div> ` : nothing}
${showCollapsed
? html`<div class="chat-tool-card__preview mono">${getTruncatedPreview(card.text!)}</div>`
: nothing}
${showInline ? html`<div class="chat-tool-card__inline mono">${card.text}</div>` : nothing}
</div>
`;
function resolveCanvasPreviewSandbox(preview: ToolPreview): string {
return resolveEmbedSandbox(preview.kind === "canvas" ? "powerful" : "powerful");
}
function normalizeContent(content: unknown): Array<Record<string, unknown>> {
@@ -149,3 +50,472 @@ function extractToolText(item: Record<string, unknown>): string | undefined {
}
return undefined;
}
export function extractToolPreview(
outputText: string | undefined,
toolName: string | undefined,
): ToolCard["preview"] | undefined {
return extractCanvasFromText(outputText, toolName);
}
function resolveToolCardId(
item: Record<string, unknown>,
message: Record<string, unknown>,
index: number,
prefix = "tool",
): string {
const explicitId =
(typeof item.id === "string" && item.id.trim()) ||
(typeof item.toolCallId === "string" && item.toolCallId.trim()) ||
(typeof item.tool_call_id === "string" && item.tool_call_id.trim()) ||
(typeof item.callId === "string" && item.callId.trim()) ||
(typeof message.toolCallId === "string" && message.toolCallId.trim()) ||
(typeof message.tool_call_id === "string" && message.tool_call_id.trim()) ||
"";
if (explicitId) {
return `${prefix}:${explicitId}`;
}
const name =
(typeof item.name === "string" && item.name.trim()) ||
(typeof message.toolName === "string" && message.toolName.trim()) ||
(typeof message.tool_name === "string" && message.tool_name.trim()) ||
"tool";
return `${prefix}:${name}:${index}`;
}
function serializeToolInput(args: unknown): string | undefined {
if (args === undefined || args === null) {
return undefined;
}
if (typeof args === "string") {
return args;
}
try {
return JSON.stringify(args, null, 2);
} catch {
if (typeof args === "number" || typeof args === "boolean" || typeof args === "bigint") {
return String(args);
}
if (typeof args === "symbol") {
return args.description ? `Symbol(${args.description})` : "Symbol()";
}
return Object.prototype.toString.call(args);
}
}
function formatPayloadForSidebar(
text: string | undefined,
language: "json" | "text" = "text",
): string {
if (!text?.trim()) {
return "";
}
if (language === "json") {
return `\`\`\`json
${text}
\`\`\``;
}
const formatted = formatToolOutputForSidebar(text);
if (formatted.includes("```")) {
return formatted;
}
return `\`\`\`text
${text}
\`\`\``;
}
function findLatestCard(cards: ToolCard[], id: string, name: string): ToolCard | undefined {
for (let i = cards.length - 1; i >= 0; i--) {
const card = cards[i];
if (!card) {
continue;
}
if (card.id === id || (card.name === name && !card.outputText)) {
return card;
}
}
return undefined;
}
export function extractToolCards(message: unknown, prefix = "tool"): ToolCard[] {
const m = message as Record<string, unknown>;
const content = normalizeContent(m.content);
const cards: ToolCard[] = [];
for (let index = 0; index < content.length; index++) {
const item = content[index] ?? {};
const kind = (typeof item.type === "string" ? item.type : "").toLowerCase();
const isToolCall =
["toolcall", "tool_call", "tooluse", "tool_use"].includes(kind) ||
(typeof item.name === "string" && item.arguments != null);
if (!isToolCall) {
continue;
}
const args = coerceArgs(item.arguments ?? item.args);
cards.push({
id: resolveToolCardId(item, m, index, prefix),
name: (item.name as string) ?? "tool",
args,
inputText: serializeToolInput(args),
});
}
for (let index = 0; index < content.length; index++) {
const item = content[index] ?? {};
const kind = (typeof item.type === "string" ? item.type : "").toLowerCase();
if (kind !== "toolresult" && kind !== "tool_result") {
continue;
}
const name = typeof item.name === "string" ? item.name : "tool";
const cardId = resolveToolCardId(item, m, index, prefix);
const existing = findLatestCard(cards, cardId, name);
const text = extractToolText(item);
const preview = extractToolPreview(text, name);
if (existing) {
existing.outputText = text;
existing.preview = preview;
continue;
}
cards.push({
id: cardId,
name,
outputText: text,
preview,
});
}
const role = typeof m.role === "string" ? m.role.toLowerCase() : "";
const isStandaloneToolMessage =
isToolResultMessage(message) ||
role === "tool" ||
role === "function" ||
typeof m.toolName === "string" ||
typeof m.tool_name === "string";
if (isStandaloneToolMessage && cards.length === 0) {
const name =
(typeof m.toolName === "string" && m.toolName) ||
(typeof m.tool_name === "string" && m.tool_name) ||
"tool";
const text = extractTextCached(message) ?? undefined;
cards.push({
id: resolveToolCardId({}, m, 0, prefix),
name,
outputText: text,
preview: extractToolPreview(text, name),
});
}
return cards;
}
export function buildToolCardSidebarContent(card: ToolCard): string {
const display = resolveToolDisplay({ name: card.name, args: card.args });
const detail = formatToolDetail(display);
const sections = [`## ${display.label}`, `**Tool:** \`${display.name}\``];
if (detail) {
sections.push(`**Summary:** ${detail}`);
}
if (card.inputText?.trim()) {
const inputIsJson = typeof card.args === "object" && card.args !== null;
sections.push(
`### Tool input\n${formatPayloadForSidebar(card.inputText, inputIsJson ? "json" : "text")}`,
);
}
if (card.outputText?.trim()) {
sections.push(`### Tool output\n${formatPayloadForSidebar(card.outputText)}`);
} else {
sections.push(`### Tool output\n*No output — tool completed successfully.*`);
}
return sections.join("\n\n");
}
function handleRawDetailsToggle(event: Event) {
const button = event.currentTarget as HTMLButtonElement | null;
const root = button?.closest(".chat-tool-card__raw");
const body = root?.querySelector<HTMLElement>(".chat-tool-card__raw-body");
if (!button || !body) {
return;
}
const expanded = button.getAttribute("aria-expanded") === "true";
button.setAttribute("aria-expanded", String(!expanded));
body.hidden = expanded;
}
function renderPreviewFrame(params: {
title: string;
src?: string;
height?: number;
sandbox?: string;
}) {
return html`
<iframe
class="chat-tool-card__preview-frame"
title=${params.title}
sandbox=${params.sandbox ?? ""}
src=${params.src ?? nothing}
style=${params.height ? `height:${params.height}px` : ""}
></iframe>
`;
}
export function renderToolPreview(
preview: ToolPreview | undefined,
surface: "chat_tool" | "chat_message" | "sidebar",
options?: {
onOpenSidebar?: (content: SidebarContent) => void;
rawText?: string | null;
canvasHostUrl?: string | null;
embedSandboxMode?: EmbedSandboxMode;
},
) {
if (!preview) {
return nothing;
}
if (preview.kind !== "canvas" || surface === "chat_tool") {
return nothing;
}
if (preview.surface !== "assistant_message") {
return nothing;
}
return html`
<div class="chat-tool-card__preview" data-kind="canvas" data-surface=${surface}>
<div class="chat-tool-card__preview-header">
<span class="chat-tool-card__preview-label">${preview.title?.trim() || "Canvas"}</span>
</div>
<div class="chat-tool-card__preview-panel" data-side="canvas">
${renderPreviewFrame({
title: preview.title?.trim() || "Canvas",
src: resolveCanvasIframeUrl(preview.url, options?.canvasHostUrl),
height: preview.preferredHeight,
sandbox:
preview.kind === "canvas"
? resolveEmbedSandbox(options?.embedSandboxMode ?? "powerful")
: resolveCanvasPreviewSandbox(preview),
})}
</div>
</div>
`;
}
export function buildSidebarContent(value: string): SidebarContent {
return {
kind: "markdown",
content: value,
};
}
export function buildPreviewSidebarContent(
preview: ToolPreview,
rawText?: string | null,
): SidebarContent | null {
if (preview.kind !== "canvas" || preview.render !== "url" || !preview.viewId || !preview.url) {
return null;
}
return {
kind: "canvas",
docId: preview.viewId,
entryUrl: preview.url,
...(preview.title ? { title: preview.title } : {}),
...(preview.preferredHeight ? { preferredHeight: preview.preferredHeight } : {}),
...(rawText ? { rawText } : {}),
};
}
export function renderRawOutputToggle(text: string) {
return html`
<div class="chat-tool-card__raw">
<button
class="chat-tool-card__raw-toggle"
type="button"
aria-expanded="false"
@click=${handleRawDetailsToggle}
>
<span>Raw details</span>
<span class="chat-tool-card__raw-toggle-icon">${icons.chevronDown}</span>
</button>
<div class="chat-tool-card__raw-body" hidden>
${renderToolDataBlock({
label: "Tool output",
text,
expanded: true,
})}
</div>
</div>
`;
}
function renderToolDataBlock(params: {
label: string;
text: string;
expanded: boolean;
empty?: boolean;
}) {
const { label, text, expanded, empty } = params;
return html`
<div class="chat-tool-card__block ${expanded ? "chat-tool-card__block--expanded" : ""}">
<div class="chat-tool-card__block-header">
<span class="chat-tool-card__block-icon">${icons.zap}</span>
<span class="chat-tool-card__block-label">${label}</span>
</div>
${empty
? html`<div class="chat-tool-card__block-empty muted">${text}</div>`
: expanded
? html`<pre class="chat-tool-card__block-content"><code>${text}</code></pre>`
: html`<div class="chat-tool-card__block-preview mono">
${getTruncatedPreview(text)}
</div>`}
</div>
`;
}
function renderCollapsedToolSummary(params: {
label: string;
name: string;
expanded: boolean;
onToggleExpanded: () => void;
}) {
const { label, name, expanded, onToggleExpanded } = params;
return html`
<button
class="chat-tool-msg-summary"
type="button"
aria-expanded=${String(expanded)}
@click=${() => onToggleExpanded()}
>
<span class="chat-tool-msg-summary__icon">${icons.zap}</span>
<span class="chat-tool-msg-summary__label">${label}</span>
<span class="chat-tool-msg-summary__names">${name}</span>
</button>
`;
}
export function renderToolCard(
card: ToolCard,
opts: {
expanded: boolean;
onToggleExpanded: (id: string) => void;
onOpenSidebar?: (content: SidebarContent) => void;
canvasHostUrl?: string | null;
embedSandboxMode?: EmbedSandboxMode;
},
) {
const hasOutput = Boolean(card.outputText?.trim());
const previewLabel = hasOutput ? "Tool output" : "Tool call";
return html`
<div
class="chat-tool-msg-collapse chat-tool-msg-collapse--manual ${opts.expanded
? "is-open"
: ""}"
>
${renderCollapsedToolSummary({
label: previewLabel,
name: card.name,
expanded: opts.expanded,
onToggleExpanded: () => opts.onToggleExpanded(card.id),
})}
${opts.expanded
? html`
<div class="chat-tool-msg-body">
${renderExpandedToolCardContent(
card,
opts.onOpenSidebar,
opts.canvasHostUrl,
opts.embedSandboxMode ?? "powerful",
)}
</div>
`
: nothing}
</div>
`;
}
export function renderExpandedToolCardContent(
card: ToolCard,
onOpenSidebar?: (content: SidebarContent) => void,
canvasHostUrl?: string | null,
embedSandboxMode: EmbedSandboxMode = "powerful",
) {
const display = resolveToolDisplay({ name: card.name, args: card.args });
const detail = formatToolDetail(display);
const hasOutput = Boolean(card.outputText?.trim());
const hasInput = Boolean(card.inputText?.trim());
const canOpenSidebar = Boolean(onOpenSidebar);
const previewSidebarContent =
card.preview?.kind === "canvas"
? buildPreviewSidebarContent(card.preview, card.outputText)
: null;
const sidebarActionContent =
previewSidebarContent ?? buildSidebarContent(buildToolCardSidebarContent(card));
const visiblePreview = card.preview
? renderToolPreview(card.preview, "chat_tool", {
onOpenSidebar,
rawText: card.outputText,
canvasHostUrl,
embedSandboxMode,
})
: nothing;
return html`
<div class="chat-tool-card chat-tool-card--expanded">
<div class="chat-tool-card__header">
<div class="chat-tool-card__title">
<span class="chat-tool-card__icon">${icons[display.icon]}</span>
<span>${display.label}</span>
</div>
${canOpenSidebar
? html`
<div class="chat-tool-card__actions">
<button
class="chat-tool-card__action-btn"
type="button"
@click=${() => onOpenSidebar?.(sidebarActionContent)}
title="Open in the side panel"
aria-label="Open tool details in side panel"
>
<span class="chat-tool-card__action-icon">${icons.panelRightOpen}</span>
</button>
</div>
`
: nothing}
</div>
${detail ? html`<div class="chat-tool-card__detail">${detail}</div>` : nothing}
${hasInput
? renderToolDataBlock({
label: "Tool input",
text: card.inputText!,
expanded: true,
})
: nothing}
${hasOutput
? card.preview
? html`${visiblePreview} ${renderRawOutputToggle(card.outputText!)}`
: renderToolDataBlock({
label: "Tool output",
text: card.outputText!,
expanded: true,
})
: nothing}
</div>
`;
}
export function renderToolCardSidebar(
card: ToolCard,
onOpenSidebar?: (content: SidebarContent) => void,
canvasHostUrl?: string | null,
embedSandboxMode: EmbedSandboxMode = "powerful",
) {
return renderToolCard(card, {
expanded: false,
onToggleExpanded: () => undefined,
onOpenSidebar,
canvasHostUrl,
embedSandboxMode,
});
}

View File

@@ -12,6 +12,10 @@ describe("loadControlUiBootstrapConfig", () => {
basePath: "/openclaw",
assistantName: "Ops",
assistantAvatar: "O",
assistantAgentId: "main",
serverVersion: "2026.3.7",
localMediaPreviewRoots: ["/tmp/openclaw"],
embedSandbox: "isolated",
}),
});
vi.stubGlobal("fetch", fetchMock as unknown as typeof fetch);
@@ -21,6 +25,8 @@ describe("loadControlUiBootstrapConfig", () => {
assistantName: "Assistant",
assistantAvatar: null,
assistantAgentId: null,
localMediaPreviewRoots: [],
embedSandboxMode: "powerful" as const,
serverVersion: null,
};
@@ -32,8 +38,10 @@ describe("loadControlUiBootstrapConfig", () => {
);
expect(state.assistantName).toBe("Ops");
expect(state.assistantAvatar).toBe("O");
expect(state.assistantAgentId).toBeNull();
expect(state.serverVersion).toBeNull();
expect(state.assistantAgentId).toBe("main");
expect(state.serverVersion).toBe("2026.3.7");
expect(state.localMediaPreviewRoots).toEqual(["/tmp/openclaw"]);
expect(state.embedSandboxMode).toBe("isolated");
vi.unstubAllGlobals();
});
@@ -47,6 +55,8 @@ describe("loadControlUiBootstrapConfig", () => {
assistantName: "Assistant",
assistantAvatar: null,
assistantAgentId: null,
localMediaPreviewRoots: [],
embedSandboxMode: "powerful" as const,
serverVersion: null,
};
@@ -57,8 +67,7 @@ describe("loadControlUiBootstrapConfig", () => {
expect.objectContaining({ method: "GET" }),
);
expect(state.assistantName).toBe("Assistant");
expect(state.assistantAgentId).toBeNull();
expect(state.serverVersion).toBeNull();
expect(state.embedSandboxMode).toBe("powerful");
vi.unstubAllGlobals();
});
@@ -72,6 +81,8 @@ describe("loadControlUiBootstrapConfig", () => {
assistantName: "Assistant",
assistantAvatar: null,
assistantAgentId: null,
localMediaPreviewRoots: [],
embedSandboxMode: "powerful" as const,
serverVersion: null,
};
@@ -81,8 +92,6 @@ describe("loadControlUiBootstrapConfig", () => {
`/openclaw${CONTROL_UI_BOOTSTRAP_CONFIG_PATH}`,
expect.objectContaining({ method: "GET" }),
);
expect(state.assistantAgentId).toBeNull();
expect(state.serverVersion).toBeNull();
vi.unstubAllGlobals();
});

View File

@@ -1,6 +1,7 @@
import {
CONTROL_UI_BOOTSTRAP_CONFIG_PATH,
type ControlUiBootstrapConfig,
type ControlUiEmbedSandboxMode,
} from "../../../../src/gateway/control-ui-contract.js";
import { normalizeAssistantIdentity } from "../assistant-identity.ts";
import { normalizeBasePath } from "../navigation.ts";
@@ -11,6 +12,8 @@ export type ControlUiBootstrapState = {
assistantAvatar: string | null;
assistantAgentId: string | null;
serverVersion: string | null;
localMediaPreviewRoots: string[];
embedSandboxMode: ControlUiEmbedSandboxMode;
};
export async function loadControlUiBootstrapConfig(state: ControlUiBootstrapState) {
@@ -37,11 +40,18 @@ export async function loadControlUiBootstrapConfig(state: ControlUiBootstrapStat
}
const parsed = (await res.json()) as ControlUiBootstrapConfig;
const normalized = normalizeAssistantIdentity({
agentId: parsed.assistantAgentId ?? null,
name: parsed.assistantName,
avatar: parsed.assistantAvatar ?? null,
});
state.assistantName = normalized.name;
state.assistantAvatar = normalized.avatar;
state.assistantAgentId = normalized.agentId ?? null;
state.serverVersion = parsed.serverVersion ?? null;
state.localMediaPreviewRoots = Array.isArray(parsed.localMediaPreviewRoots)
? parsed.localMediaPreviewRoots.filter((value): value is string => typeof value === "string")
: [];
state.embedSandboxMode = parsed.embedSandbox === "isolated" ? "isolated" : "powerful";
} catch {
// Ignore bootstrap failures; UI will update identity after connecting.
}

View File

@@ -0,0 +1,7 @@
import type { ControlUiEmbedSandboxMode } from "../../../src/gateway/control-ui-contract.js";
export type EmbedSandboxMode = ControlUiEmbedSandboxMode;
export function resolveEmbedSandbox(mode: EmbedSandboxMode | null | undefined): string {
return mode === "isolated" ? "allow-scripts" : "allow-scripts allow-same-origin";
}

View File

@@ -12,7 +12,6 @@ import {
} from "../../../src/gateway/protocol/connect-error-details.js";
import { clearDeviceAuthToken, loadDeviceAuthToken, storeDeviceAuthToken } from "./device-auth.ts";
import { loadOrCreateDeviceIdentity, signDevicePayload } from "./device-identity.ts";
import { normalizeLowercaseStringOrEmpty, normalizeOptionalString } from "./string-coerce.ts";
import { generateUUID } from "./uuid.ts";
export type GatewayEventFrame = {
@@ -83,7 +82,7 @@ export function isNonRecoverableAuthError(error: GatewayErrorInfo | undefined):
function isTrustedRetryEndpoint(url: string): boolean {
try {
const gatewayUrl = new URL(url, window.location.href);
const host = normalizeLowercaseStringOrEmpty(gatewayUrl.hostname);
const host = gatewayUrl.hostname.trim().toLowerCase();
const isLoopbackHost =
host === "localhost" || host === "::1" || host === "[::1]" || host === "127.0.0.1";
const isLoopbackIPv4 = host.startsWith("127.");
@@ -112,6 +111,7 @@ export type GatewayHelloOk = {
scopes?: string[];
issuedAtMs?: number;
};
canvasHostUrl?: string;
policy?: { tickIntervalMs?: number };
};
@@ -295,10 +295,6 @@ export class GatewayBrowserClient {
stop() {
this.closed = true;
if (this.connectTimer !== null) {
window.clearTimeout(this.connectTimer);
this.connectTimer = null;
}
this.ws?.close();
this.ws = null;
this.pendingConnectError = undefined;
@@ -387,8 +383,8 @@ export class GatewayBrowserClient {
const role = CONTROL_UI_OPERATOR_ROLE;
const scopes = [...CONTROL_UI_OPERATOR_SCOPES];
const client = this.buildConnectClient();
const explicitGatewayToken = normalizeOptionalString(this.opts.token);
const explicitPassword = normalizeOptionalString(this.opts.password);
const explicitGatewayToken = this.opts.token?.trim() || undefined;
const explicitPassword = this.opts.password?.trim() || undefined;
// crypto.subtle is only available in secure contexts (HTTPS, localhost).
// Over plain HTTP, we skip device identity and fall back to token-only auth.
@@ -565,8 +561,8 @@ export class GatewayBrowserClient {
}
private selectConnectAuth(params: { role: string; deviceId: string }): SelectedConnectAuth {
const explicitGatewayToken = normalizeOptionalString(this.opts.token);
const authPassword = normalizeOptionalString(this.opts.password);
const explicitGatewayToken = this.opts.token?.trim() || undefined;
const authPassword = this.opts.password?.trim() || undefined;
const storedEntry = loadDeviceAuthToken({
deviceId: params.deviceId,
role: params.role,

View File

@@ -0,0 +1,15 @@
export type MarkdownSidebarContent = {
kind: "markdown";
content: string;
};
export type CanvasSidebarContent = {
kind: "canvas";
docId: string;
title?: string;
entryUrl: string;
preferredHeight?: number;
rawText?: string | null;
};
export type SidebarContent = MarkdownSidebarContent | CanvasSidebarContent;

View File

@@ -21,12 +21,28 @@ export type MessageGroup = {
};
/** Content item types in a normalized message */
export type MessageContentItem = {
type: "text" | "tool_call" | "tool_result";
text?: string;
name?: string;
args?: unknown;
};
export type MessageContentItem =
| {
type: "text" | "tool_call" | "tool_result";
text?: string;
name?: string;
args?: unknown;
}
| {
type: "attachment";
attachment: {
url: string;
kind: "image" | "audio" | "video" | "document";
label: string;
mimeType?: string;
isVoiceNote?: boolean;
};
}
| {
type: "canvas";
preview: Extract<NonNullable<ToolCard["preview"]>, { kind: "canvas" }>;
rawText?: string | null;
};
/** Normalized message structure for rendering */
export type NormalizedMessage = {
@@ -35,12 +51,34 @@ export type NormalizedMessage = {
timestamp: number;
id?: string;
senderLabel?: string | null;
audioAsVoice?: boolean;
replyTarget?:
| {
kind: "current";
}
| {
kind: "id";
id: string;
}
| null;
};
/** Tool card representation for tool calls and results */
/** Tool card representation for inline tool call/result rendering */
export type ToolCard = {
kind: "call" | "result";
id: string;
name: string;
args?: unknown;
text?: string;
inputText?: string;
outputText?: string;
preview?: {
kind: "canvas";
surface: "assistant_message";
render: "url";
title?: string;
preferredHeight?: number;
url?: string;
viewId?: string;
className?: string;
style?: string;
};
};

File diff suppressed because it is too large Load Diff

View File

@@ -1,10 +1,7 @@
import { html, nothing, type TemplateResult } from "lit";
import { ref } from "lit/directives/ref.js";
import { repeat } from "lit/directives/repeat.js";
import type {
CompactionStatus as CompactionIndicatorStatus,
FallbackStatus as FallbackIndicatorStatus,
} from "../app-tool-stream.ts";
import { unsafeHTML } from "lit/directives/unsafe-html.js";
import {
CHAT_ATTACHMENT_ACCEPT,
isSupportedChatAttachmentMimeType,
@@ -17,11 +14,17 @@ import {
renderStreamingGroup,
} from "../chat/grouped-render.ts";
import { InputHistory } from "../chat/input-history.ts";
import { normalizeMessage, normalizeRoleForGrouping } from "../chat/message-normalizer.ts";
import { extractTextCached } from "../chat/message-extract.ts";
import {
isToolResultMessage,
normalizeMessage,
normalizeRoleForGrouping,
} from "../chat/message-normalizer.ts";
import { PinnedMessages } from "../chat/pinned-messages.ts";
import { getPinnedMessageSummary } from "../chat/pinned-summary.ts";
import { messageMatchesSearchQuery } from "../chat/search-match.ts";
import { getOrCreateSessionCacheValue } from "../chat/session-cache.ts";
import type { ChatSideResult } from "../chat/side-result.ts";
import {
CATEGORY_LABELS,
SLASH_COMMANDS,
@@ -30,16 +33,35 @@ import {
type SlashCommandDef,
} from "../chat/slash-commands.ts";
import { isSttSupported, startStt, stopStt } from "../chat/speech.ts";
import { buildSidebarContent, extractToolCards, extractToolPreview } from "../chat/tool-cards.ts";
import type { EmbedSandboxMode } from "../embed-sandbox.ts";
import { icons } from "../icons.ts";
import { normalizeLowercaseStringOrEmpty } from "../string-coerce.ts";
import { toSanitizedMarkdownHtml } from "../markdown.ts";
import type { SidebarContent } from "../sidebar-content.ts";
import { detectTextDirection } from "../text-direction.ts";
import type { GatewaySessionRow, SessionsListResult } from "../types.ts";
import type { ChatItem, MessageGroup } from "../types/chat-types.ts";
import type { ChatItem, MessageGroup, ToolCard } from "../types/chat-types.ts";
import type { ChatAttachment, ChatQueueItem } from "../ui-types.ts";
import { agentLogoUrl, resolveAgentAvatarUrl } from "./agents-utils.ts";
import { renderMarkdownSidebar } from "./markdown-sidebar.ts";
import "../components/resizable-divider.ts";
export type CompactionIndicatorStatus = {
active: boolean;
startedAt: number | null;
completedAt: number | null;
};
export type FallbackIndicatorStatus = {
phase?: "active" | "cleared";
selected: string;
active: string;
previous?: string;
reason?: string;
attempts: string[];
occurredAt: number;
};
export type ChatProps = {
sessionKey: string;
onSessionKeyChange: (next: string) => void;
@@ -52,6 +74,7 @@ export type ChatProps = {
compactionStatus?: CompactionIndicatorStatus | null;
fallbackStatus?: FallbackIndicatorStatus | null;
messages: unknown[];
sideResult?: ChatSideResult | null;
toolMessages: unknown[];
streamSegments: Array<{ text: string; ts: number }>;
stream: string | null;
@@ -66,11 +89,16 @@ export type ChatProps = {
sessions: SessionsListResult | null;
focusMode: boolean;
sidebarOpen?: boolean;
sidebarContent?: string | null;
sidebarContent?: SidebarContent | null;
sidebarError?: string | null;
splitRatio?: number;
canvasHostUrl?: string | null;
embedSandboxMode?: EmbedSandboxMode;
assistantName: string;
assistantAvatar: string | null;
localMediaPreviewRoots?: string[];
assistantAttachmentAuthToken?: string | null;
autoExpandToolCalls?: boolean;
attachments?: ChatAttachment[];
onAttachmentsChange?: (attachments: ChatAttachment[]) => void;
showNewMessages?: boolean;
@@ -83,6 +111,7 @@ export type ChatProps = {
onSend: () => void;
onAbort?: () => void;
onQueueRemove: (id: string) => void;
onDismissSideResult?: () => void;
onNewSession: () => void;
onClearHistory?: () => void;
agentsList: {
@@ -93,7 +122,7 @@ export type ChatProps = {
onAgentChange: (agentId: string) => void;
onNavigateToAgent?: () => void;
onSessionSelect?: (sessionKey: string) => void;
onOpenSidebar?: (content: string) => void;
onOpenSidebar?: (content: SidebarContent) => void;
onCloseSidebar?: () => void;
onSplitRatioChange?: (ratio: number) => void;
onChatScroll?: (event: Event) => void;
@@ -107,6 +136,9 @@ const FALLBACK_TOAST_DURATION_MS = 8000;
const inputHistories = new Map<string, InputHistory>();
const pinnedMessagesMap = new Map<string, PinnedMessages>();
const deletedMessagesMap = new Map<string, DeletedMessages>();
const expandedToolCardsBySession = new Map<string, Map<string, boolean>>();
const initializedToolCardsBySession = new Map<string, Set<string>>();
const lastAutoExpandPrefBySession = new Map<string, boolean>();
function getInputHistory(sessionKey: string): InputHistory {
return getOrCreateSessionCacheValue(inputHistories, sessionKey, () => new InputHistory());
@@ -128,6 +160,143 @@ function getDeletedMessages(sessionKey: string): DeletedMessages {
);
}
function getExpandedToolCards(sessionKey: string): Map<string, boolean> {
return getOrCreateSessionCacheValue(expandedToolCardsBySession, sessionKey, () => new Map());
}
function getInitializedToolCards(sessionKey: string): Set<string> {
return getOrCreateSessionCacheValue(initializedToolCardsBySession, sessionKey, () => new Set());
}
function appendCanvasBlockToAssistantMessage(
message: unknown,
preview: Extract<NonNullable<ToolCard["preview"]>, { kind: "canvas" }>,
rawText: string | null,
) {
const raw = message as Record<string, unknown>;
const existingContent = Array.isArray(raw.content)
? [...raw.content]
: typeof raw.content === "string"
? [{ type: "text", text: raw.content }]
: typeof raw.text === "string"
? [{ type: "text", text: raw.text }]
: [];
const alreadyHasArtifact = existingContent.some((block) => {
if (!block || typeof block !== "object") {
return false;
}
const typed = block as {
type?: unknown;
preview?: { kind?: unknown; viewId?: unknown; url?: unknown };
};
return (
typed.type === "canvas" &&
typed.preview?.kind === "canvas" &&
((preview.viewId && typed.preview.viewId === preview.viewId) ||
(preview.url && typed.preview.url === preview.url))
);
});
if (alreadyHasArtifact) {
return message;
}
return {
...raw,
content: [
...existingContent,
{
type: "canvas",
preview,
...(rawText ? { rawText } : {}),
},
],
};
}
function extractChatMessagePreview(toolMessage: unknown): {
preview: Extract<NonNullable<ToolCard["preview"]>, { kind: "canvas" }>;
text: string | null;
timestamp: number | null;
} | null {
const normalized = normalizeMessage(toolMessage);
const cards = extractToolCards(toolMessage, "preview");
for (let index = cards.length - 1; index >= 0; index--) {
const card = cards[index];
if (card?.preview?.kind === "canvas") {
return {
preview: card.preview,
text: card.outputText ?? null,
timestamp: normalized.timestamp ?? null,
};
}
}
const text = extractTextCached(toolMessage) ?? undefined;
const toolRecord = toolMessage as Record<string, unknown>;
const toolName =
typeof toolRecord.toolName === "string"
? toolRecord.toolName
: typeof toolRecord.tool_name === "string"
? toolRecord.tool_name
: undefined;
const preview = extractToolPreview(text, toolName);
if (preview?.kind !== "canvas") {
return null;
}
return { preview, text: text ?? null, timestamp: normalized.timestamp ?? null };
}
function findNearestAssistantMessageIndex(
items: ChatItem[],
toolTimestamp: number | null,
): number | null {
const assistantEntries = items
.map((item, index) => {
if (item.kind !== "message") {
return null;
}
const message = item.message as Record<string, unknown>;
const role = typeof message.role === "string" ? message.role.toLowerCase() : "";
if (role !== "assistant") {
return null;
}
return {
index,
timestamp: normalizeMessage(item.message).timestamp ?? null,
};
})
.filter(Boolean) as Array<{ index: number; timestamp: number | null }>;
if (assistantEntries.length === 0) {
return null;
}
if (toolTimestamp == null) {
return assistantEntries[assistantEntries.length - 1]?.index ?? null;
}
let previous: { index: number; timestamp: number } | null = null;
let next: { index: number; timestamp: number } | null = null;
for (const entry of assistantEntries) {
if (entry.timestamp == null) {
continue;
}
if (entry.timestamp <= toolTimestamp) {
previous = { index: entry.index, timestamp: entry.timestamp };
continue;
}
next = { index: entry.index, timestamp: entry.timestamp };
break;
}
if (previous && next) {
const previousDelta = toolTimestamp - previous.timestamp;
const nextDelta = next.timestamp - toolTimestamp;
return nextDelta < previousDelta ? next.index : previous.index;
}
if (previous) {
return previous.index;
}
if (next) {
return next.index;
}
return assistantEntries[assistantEntries.length - 1]?.index ?? null;
}
interface ChatEphemeralState {
sttRecording: boolean;
sttInterimText: string;
@@ -178,11 +347,65 @@ function adjustTextareaHeight(el: HTMLTextAreaElement) {
el.style.height = `${Math.min(el.scrollHeight, 150)}px`;
}
function syncToolCardExpansionState(
sessionKey: string,
items: Array<ChatItem | MessageGroup>,
autoExpandToolCalls: boolean,
) {
const expanded = getExpandedToolCards(sessionKey);
const initialized = getInitializedToolCards(sessionKey);
const previousAutoExpand = lastAutoExpandPrefBySession.get(sessionKey) ?? false;
const currentToolCardIds = new Set<string>();
for (const item of items) {
if (item.kind !== "group") {
continue;
}
for (const entry of item.messages) {
const cards = extractToolCards(entry.message, entry.key);
for (let cardIndex = 0; cardIndex < cards.length; cardIndex++) {
const disclosureId = `${entry.key}:toolcard:${cardIndex}`;
currentToolCardIds.add(disclosureId);
if (initialized.has(disclosureId)) {
continue;
}
expanded.set(disclosureId, autoExpandToolCalls);
initialized.add(disclosureId);
}
const messageRecord = entry.message as Record<string, unknown>;
const role = typeof messageRecord.role === "string" ? messageRecord.role : "unknown";
const normalizedRole = normalizeRoleForGrouping(role);
const isToolMessage =
isToolResultMessage(entry.message) ||
normalizedRole === "tool" ||
role.toLowerCase() === "toolresult" ||
role.toLowerCase() === "tool_result" ||
typeof messageRecord.toolCallId === "string" ||
typeof messageRecord.tool_call_id === "string";
if (!isToolMessage) {
continue;
}
const disclosureId = `toolmsg:${entry.key}`;
currentToolCardIds.add(disclosureId);
if (initialized.has(disclosureId)) {
continue;
}
expanded.set(disclosureId, autoExpandToolCalls);
initialized.add(disclosureId);
}
}
if (autoExpandToolCalls && !previousAutoExpand) {
for (const toolCardId of currentToolCardIds) {
expanded.set(toolCardId, true);
}
}
lastAutoExpandPrefBySession.set(sessionKey, autoExpandToolCalls);
}
function renderCompactionIndicator(status: CompactionIndicatorStatus | null | undefined) {
if (!status) {
return nothing;
}
if (status.phase === "active") {
if (status.active) {
return html`
<div
class="compaction-indicator compaction-indicator--active"
@@ -193,18 +416,7 @@ function renderCompactionIndicator(status: CompactionIndicatorStatus | null | un
</div>
`;
}
if (status.phase === "retrying") {
return html`
<div
class="compaction-indicator compaction-indicator--active"
role="status"
aria-live="polite"
>
${icons.loader} Retrying after compaction...
</div>
`;
}
if (status.phase === "complete" && status.completedAt) {
if (status.completedAt) {
const elapsed = Date.now() - status.completedAt;
if (elapsed < COMPACTION_TOAST_DURATION_MS) {
return html`
@@ -255,6 +467,43 @@ function renderFallbackIndicator(status: FallbackIndicatorStatus | null | undefi
`;
}
function renderSideResult(
sideResult: ChatSideResult | null | undefined,
onDismiss?: () => void,
): TemplateResult | typeof nothing {
if (!sideResult) {
return nothing;
}
return html`
<section
class=${`chat-side-result ${sideResult.isError ? "chat-side-result--error" : ""}`}
role="status"
aria-live="polite"
aria-label="BTW side result"
>
<div class="chat-side-result__header">
<div class="chat-side-result__label-row">
<span class="chat-side-result__label">BTW</span>
<span class="chat-side-result__meta">Not saved to chat history</span>
</div>
<button
class="btn chat-side-result__dismiss"
type="button"
aria-label="Dismiss BTW result"
title="Dismiss"
@click=${() => onDismiss?.()}
>
${icons.x}
</button>
</div>
<div class="chat-side-result__question">${sideResult.question}</div>
<div class="chat-side-result__body" dir=${detectTextDirection(sideResult.text)}>
${unsafeHTML(toSanitizedMarkdownHtml(sideResult.text))}
</div>
</section>
`;
}
/**
* Compact notice when context usage reaches 85%+.
* Progressively shifts from amber (85%) to red (90%+).
@@ -324,8 +573,8 @@ function renderContextNotice(
<div class="context-notice" role="status" style="--ctx-color:${color};--ctx-bg:${bg}">
<svg
class="context-notice__icon"
width="16"
height="16"
width="24"
height="24"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
@@ -496,12 +745,12 @@ function updateSlashMenu(value: string, requestUpdate: () => void): void {
// Arg mode: /command <partial-arg>
const argMatch = value.match(/^\/(\S+)\s(.*)$/);
if (argMatch) {
const cmdName = normalizeLowercaseStringOrEmpty(argMatch[1]);
const argFilter = normalizeLowercaseStringOrEmpty(argMatch[2]);
const cmdName = argMatch[1].toLowerCase();
const argFilter = argMatch[2].toLowerCase();
const cmd = SLASH_COMMANDS.find((c) => c.name === cmdName);
if (cmd?.argOptions?.length) {
const filtered = argFilter
? cmd.argOptions.filter((opt) => normalizeLowercaseStringOrEmpty(opt).startsWith(argFilter))
? cmd.argOptions.filter((opt) => opt.toLowerCase().startsWith(argFilter))
: cmd.argOptions;
if (filtered.length > 0) {
vs.slashMenuMode = "args";
@@ -884,7 +1133,7 @@ function renderSlashMenu(
export function renderChat(props: ChatProps) {
const canCompose = props.connected;
const isBusy = props.sending || props.stream !== null || props.canAbort;
const isBusy = props.sending || props.stream !== null;
const canAbort = Boolean(props.canAbort && props.onAbort);
const activeSession = props.sessions?.sessions?.find((row) => row.key === props.sessionKey);
const reasoningLevel = activeSession?.reasoningLevel ?? "off";
@@ -933,6 +1182,12 @@ export function renderChat(props: ChatProps) {
};
const chatItems = buildChatItems(props);
syncToolCardExpansionState(props.sessionKey, chatItems, Boolean(props.autoExpandToolCalls));
const expandedToolCards = getExpandedToolCards(props.sessionKey);
const toggleToolCardExpanded = (toolCardId: string) => {
expandedToolCards.set(toolCardId, !expandedToolCards.get(toolCardId));
requestUpdate();
};
const isEmpty = chatItems.length === 0 && !props.loading;
const thread = html`
@@ -1020,9 +1275,23 @@ export function renderChat(props: ChatProps) {
onOpenSidebar: props.onOpenSidebar,
showReasoning,
showToolCalls: props.showToolCalls,
autoExpandToolCalls: Boolean(props.autoExpandToolCalls),
isToolMessageExpanded: (messageId: string) =>
expandedToolCards.get(messageId) ?? false,
onToggleToolMessageExpanded: (messageId: string) => {
expandedToolCards.set(messageId, !expandedToolCards.get(messageId));
requestUpdate();
},
isToolExpanded: (toolCardId: string) => expandedToolCards.get(toolCardId) ?? false,
onToggleToolExpanded: toggleToolCardExpanded,
onRequestUpdate: requestUpdate,
assistantName: props.assistantName,
assistantAvatar: assistantIdentity.avatar,
basePath: props.basePath,
localMediaPreviewRoots: props.localMediaPreviewRoots ?? [],
assistantAttachmentAuthToken: props.assistantAttachmentAuthToken ?? null,
canvasHostUrl: props.canvasHostUrl,
embedSandboxMode: props.embedSandboxMode ?? "powerful",
contextWindow:
activeSession?.contextTokens ?? props.sessions?.defaults?.contextTokens ?? null,
onDelete: () => {
@@ -1101,6 +1370,12 @@ export function renderChat(props: ChatProps) {
}
}
if (e.key === "Escape" && props.sideResult && !vs.searchOpen) {
e.preventDefault();
props.onDismissSideResult?.();
return;
}
// Input history (only when input is empty)
if (!props.draft.trim()) {
if (e.key === "ArrowUp") {
@@ -1197,12 +1472,24 @@ export function renderChat(props: ChatProps) {
${renderMarkdownSidebar({
content: props.sidebarContent ?? null,
error: props.sidebarError ?? null,
canvasHostUrl: props.canvasHostUrl,
embedSandboxMode: props.embedSandboxMode ?? "powerful",
onClose: props.onCloseSidebar!,
onViewRawText: () => {
if (!props.sidebarContent || !props.onOpenSidebar) {
return;
}
props.onOpenSidebar(`\`\`\`\n${props.sidebarContent}\n\`\`\``);
if (props.sidebarContent.kind === "markdown") {
props.onOpenSidebar(
buildSidebarContent(`\`\`\`\n${props.sidebarContent.content}\n\`\`\``),
);
return;
}
if (props.sidebarContent.rawText?.trim()) {
props.onOpenSidebar(
buildSidebarContent(`\`\`\`json\n${props.sidebarContent.rawText}\n\`\`\``),
);
}
},
})}
</div>
@@ -1237,6 +1524,7 @@ export function renderChat(props: ChatProps) {
</div>
`
: nothing}
${renderSideResult(props.sideResult, props.onDismissSideResult)}
${renderFallbackIndicator(props.fallbackStatus)}
${renderCompactionIndicator(props.compactionStatus)}
${renderContextNotice(activeSession, props.sessions?.defaults?.contextTokens ?? null)}
@@ -1422,14 +1710,13 @@ function groupMessages(items: ChatItem[]): Array<ChatItem | MessageGroup> {
const normalized = normalizeMessage(item.message);
const role = normalizeRoleForGrouping(normalized.role);
const senderLabel =
normalizeLowercaseStringOrEmpty(role) === "user" ? (normalized.senderLabel ?? null) : null;
const senderLabel = role.toLowerCase() === "user" ? (normalized.senderLabel ?? null) : null;
const timestamp = normalized.timestamp || Date.now();
if (
!currentGroup ||
currentGroup.role !== role ||
(normalizeLowercaseStringOrEmpty(role) === "user" && currentGroup.senderLabel !== senderLabel)
(role.toLowerCase() === "user" && currentGroup.senderLabel !== senderLabel)
) {
if (currentGroup) {
result.push(currentGroup);
@@ -1488,7 +1775,7 @@ function buildChatItems(props: ChatProps): Array<ChatItem | MessageGroup> {
continue;
}
if (!props.showToolCalls && normalizeLowercaseStringOrEmpty(normalized.role) === "toolresult") {
if (!props.showToolCalls && normalized.role.toLowerCase() === "toolresult") {
continue;
}
@@ -1503,6 +1790,31 @@ function buildChatItems(props: ChatProps): Array<ChatItem | MessageGroup> {
message: msg,
});
}
const liftedCanvasSources = tools
.map((tool) => extractChatMessagePreview(tool))
.filter((entry) => Boolean(entry)) as Array<{
preview: Extract<NonNullable<ToolCard["preview"]>, { kind: "canvas" }>;
text: string | null;
timestamp: number | null;
}>;
for (const liftedCanvasSource of liftedCanvasSources) {
const assistantIndex = findNearestAssistantMessageIndex(items, liftedCanvasSource.timestamp);
if (assistantIndex == null) {
continue;
}
const item = items[assistantIndex];
if (!item || item.kind !== "message") {
continue;
}
items[assistantIndex] = {
...item,
message: appendCanvasBlockToAssistantMessage(
item.message as Record<string, unknown>,
liftedCanvasSource.preview,
liftedCanvasSource.text,
),
};
}
// Interleave stream segments and tool cards in order. Each segment
// contains text that was streaming before the corresponding tool started.
// This ensures correct visual ordering: text → tool → text → tool → ...
@@ -1547,7 +1859,20 @@ function messageKey(message: unknown, index: number): string {
const m = message as Record<string, unknown>;
const toolCallId = typeof m.toolCallId === "string" ? m.toolCallId : "";
if (toolCallId) {
return `tool:${toolCallId}`;
const role = typeof m.role === "string" ? m.role : "unknown";
const id = typeof m.id === "string" ? m.id : "";
if (id) {
return `tool:${role}:${toolCallId}:${id}`;
}
const messageId = typeof m.messageId === "string" ? m.messageId : "";
if (messageId) {
return `tool:${role}:${toolCallId}:${messageId}`;
}
const timestamp = typeof m.timestamp === "number" ? m.timestamp : null;
if (timestamp != null) {
return `tool:${role}:${toolCallId}:${timestamp}:${index}`;
}
return `tool:${role}:${toolCallId}:${index}`;
}
const id = typeof m.id === "string" ? m.id : "";
if (id) {

View File

@@ -1,20 +1,35 @@
import { html } from "lit";
import { html, nothing } from "lit";
import { unsafeHTML } from "lit/directives/unsafe-html.js";
import { resolveCanvasIframeUrl } from "../canvas-url.ts";
import { resolveEmbedSandbox, type EmbedSandboxMode } from "../embed-sandbox.ts";
import { icons } from "../icons.ts";
import { toSanitizedMarkdownHtml } from "../markdown.ts";
import type { SidebarContent } from "../sidebar-content.ts";
function resolveSidebarCanvasSandbox(
content: SidebarContent,
embedSandboxMode: EmbedSandboxMode,
): string {
return content.kind === "canvas" ? resolveEmbedSandbox(embedSandboxMode) : "allow-scripts";
}
export type MarkdownSidebarProps = {
content: string | null;
content: SidebarContent | null;
error: string | null;
onClose: () => void;
onViewRawText: () => void;
canvasHostUrl?: string | null;
embedSandboxMode?: EmbedSandboxMode;
};
export function renderMarkdownSidebar(props: MarkdownSidebarProps) {
const content = props.content;
return html`
<div class="sidebar-panel">
<div class="sidebar-header">
<div class="sidebar-title">Tool Output</div>
<div class="sidebar-title">
${content?.kind === "canvas" ? content.title?.trim() || "Render Preview" : "Tool Details"}
</div>
<button @click=${props.onClose} class="btn" title="Close sidebar">${icons.x}</button>
</div>
<div class="sidebar-content">
@@ -26,9 +41,41 @@ export function renderMarkdownSidebar(props: MarkdownSidebarProps) {
</button>
`
: props.content
? html`<div class="sidebar-markdown">
${unsafeHTML(toSanitizedMarkdownHtml(props.content))}
</div>`
? content.kind === "canvas"
? html`
<div class="chat-tool-card__preview" data-kind="canvas">
<div class="chat-tool-card__preview-panel" data-side="front">
<iframe
class="chat-tool-card__preview-frame"
title=${content.title?.trim() || "Render preview"}
sandbox=${resolveSidebarCanvasSandbox(
content,
props.embedSandboxMode ?? "powerful",
)}
src=${resolveCanvasIframeUrl(content.entryUrl, props.canvasHostUrl) ??
nothing}
style=${content.preferredHeight
? `height:${content.preferredHeight}px`
: ""}
></iframe>
</div>
${content.rawText?.trim()
? html`
<div style="margin-top: 12px;">
<button @click=${props.onViewRawText} class="btn">View Raw Text</button>
</div>
`
: nothing}
</div>
`
: html`<div class="sidebar-markdown">
${unsafeHTML(
toSanitizedMarkdownHtml(content.content, {
expandJsonBlocks: true,
disableTruncation: true,
}),
)}
</div>`
: html` <div class="muted">No content available</div> `}
</div>
</div>