diff --git a/AGENTS.md b/AGENTS.md index 2193661b6796..47f22032d82c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -227,6 +227,7 @@ Skills own workflows; root owns hard policy and routing. - Parallels: `$openclaw-parallels-smoke`; Discord roundtrip: `$parallels-discord-roundtrip`. - Crabbox/WebVNC human demos: keep remote desktop visible/windowed; no fullscreen remote browser unless video/capture-style output. - ClawSweeper ops: `$clawsweeper`. Deployed hook sessions may post one concise `#clawsweeper` note only when surprising/actionable/risky; if using message tool, reply exactly `NO_REPLY`. +- Generated-media completions wake the requester agent first. Requester visible-reply config decides final text vs message tool; direct media send is fallback/recovery only. - Memory wiki prompt digest stays tiny; prefer `wiki_search` / `wiki_get`; verify contact data before use; source-class provenance for generated people facts. - Rebrand/migration/config warnings: run `openclaw doctor`. - Never edit `node_modules`. diff --git a/docs/automation/tasks.md b/docs/automation/tasks.md index 58d861e8e6a7..82a471e49354 100644 --- a/docs/automation/tasks.md +++ b/docs/automation/tasks.md @@ -102,7 +102,7 @@ Not every agent run creates a task. Heartbeat turns and normal interactive chat Main-session cron tasks use `silent` notify policy by default - they create records for tracking but do not generate notifications. Isolated cron tasks also default to `silent` but are more visible because they run in their own session. - Session-backed `image_generate`, `music_generate`, and `video_generate` runs also use `silent` notify policy. They still create task records, but completion is handed back to the original agent session as an internal wake so the agent can write the follow-up message and attach the finished media itself. Generated-media completion events require message-tool delivery: the agent must send the finished media with the `message` tool, then reply `NO_REPLY`. If the requester session is no longer active or its active wake fails, and the completion agent misses some or all generated media, OpenClaw sends an idempotent direct fallback with only the missing media to the original channel target. + Session-backed `image_generate`, `music_generate`, and `video_generate` runs also use `silent` notify policy. They still create task records, but completion is handed back to the original agent session as an internal wake so the agent can write the follow-up message and attach the finished media itself. The requester agent follows its normal visible-reply contract: automatic final reply when configured, or `message(action="send")` plus `NO_REPLY` when the session requires message-tool replies. If the requester session is no longer active or its active wake fails, and the completion agent misses some or all generated media, OpenClaw sends an idempotent direct fallback with only the missing media to the original channel target. diff --git a/docs/tools/image-generation.md b/docs/tools/image-generation.md index 59847686663b..0bdfe5a9637c 100644 --- a/docs/tools/image-generation.md +++ b/docs/tools/image-generation.md @@ -11,11 +11,12 @@ sidebarTitle: "Image generation" The `image_generate` tool lets the agent create and edit images using your configured providers. In chat sessions, image generation runs asynchronously: OpenClaw records a background task, returns the task id immediately, and wakes -the agent when the provider finishes. The completion agent must send generated -images through the `message` tool. If the requester session is inactive or -its active wake fails, and some generated images are still missing from -message-tool delivery, OpenClaw sends an idempotent direct fallback with only -the missing images. +the agent when the provider finishes. The completion agent follows the +session's normal visible-reply mode: automatic final reply delivery when +configured, or `message(action="send")` when the session requires the message +tool. If the requester session is inactive or its active wake fails, and some +generated images are still missing from the completion reply, OpenClaw sends an +idempotent direct fallback with only the missing images. The tool only appears when at least one image-generation provider is diff --git a/docs/tools/media-overview.md b/docs/tools/media-overview.md index 0bfd62ea4dca..f16ea22712d1 100644 --- a/docs/tools/media-overview.md +++ b/docs/tools/media-overview.md @@ -98,11 +98,12 @@ For async tools, OpenClaw submits the request to the provider, returns a task id immediately, and tracks the job in the task ledger. The agent continues responding to other messages while the job runs. When the provider finishes, OpenClaw wakes the agent with the generated media paths so it can tell the -user and relay the result through the message tool. If the requester session -is inactive or its active wake fails, and some generated media is still -missing from message-tool delivery, OpenClaw sends an idempotent direct -fallback with only the missing media. Media already delivered through the -message tool is not posted again. +user through the session's normal visible-reply mode: automatic final reply +delivery when configured, or `message(action="send")` when the session requires +the message tool. If the requester session is inactive or its active wake +fails, and some generated media is still missing from the completion reply, +OpenClaw sends an idempotent direct fallback with only the missing media. Media +already delivered by the completion reply is not posted again. ## Speech-to-text and Voice Call diff --git a/docs/tools/music-generation.md b/docs/tools/music-generation.md index f71e2cb8d615..7eed44e959f3 100644 --- a/docs/tools/music-generation.md +++ b/docs/tools/music-generation.md @@ -15,12 +15,12 @@ fal, Google, MiniMax, and OpenRouter today. For session-backed agent runs, OpenClaw starts music generation as a background task, tracks it in the task ledger, then wakes the agent again when the track is ready so the agent can tell the user and attach the -finished audio. Generated-media completions are delivered by the agent through -the message tool. If the requester session is inactive or its active wake -fails, and some generated audio is still missing from message-tool delivery, -OpenClaw sends an idempotent direct fallback with only the missing audio. The -completion wake explicitly warns the agent that normal final replies are -private for this route. +finished audio. The completion agent follows the session's normal visible-reply +mode: automatic final reply delivery when configured, or `message(action="send")` +when the session requires the message tool. If the requester session is +inactive or its active wake fails, and some generated audio is still missing +from the completion reply, OpenClaw sends an idempotent direct fallback with +only the missing audio. The built-in shared tool only appears when at least one music-generation diff --git a/docs/tools/video-generation.md b/docs/tools/video-generation.md index dd1a9762efc6..5cf8627bf4f7 100644 --- a/docs/tools/video-generation.md +++ b/docs/tools/video-generation.md @@ -62,10 +62,12 @@ session: 1. OpenClaw submits the request to the provider and immediately returns a task id. 2. The provider processes the job in the background (typically 30 seconds to several minutes depending on the provider and resolution; slow queue-backed providers can run up to the configured timeout). 3. When the video is ready, OpenClaw wakes the same session with an internal completion event. -4. The agent tells the user and attaches the finished video through the - message tool. If the requester session is inactive or its active wake - fails, and some generated video is still missing from message-tool delivery, - OpenClaw sends an idempotent direct fallback with only the missing video. +4. The agent tells the user through the session's normal visible-reply mode: + final reply delivery when automatic, or `message(action="send")` when the + session requires the message tool. If the requester session is inactive or + its active wake fails, and some generated video is still missing from the + completion reply, OpenClaw sends an idempotent direct fallback with only the + missing video. While a job is in flight, duplicate `video_generate` calls in the same session return the current task status instead of starting another diff --git a/src/agents/subagent-announce-delivery.test.ts b/src/agents/subagent-announce-delivery.test.ts index d6e0252799d1..453228716198 100644 --- a/src/agents/subagent-announce-delivery.test.ts +++ b/src/agents/subagent-announce-delivery.test.ts @@ -1,4 +1,5 @@ import { afterEach, describe, expect, it, vi } from "vitest"; +import { OutboundDeliveryError } from "../infra/outbound/deliver-types.js"; import { testing as sessionBindingServiceTesting, registerSessionBindingAdapter, @@ -2179,12 +2180,11 @@ describe("deliverSubagentAnnouncement completion delivery", () => { path: "direct", }); expectGatewayAgentParams(callGateway, { - deliver: false, + deliver: true, channel: "discord", accountId: "acct-1", to: "dm:U123", threadId: undefined, - sourceReplyDeliveryMode: "message_tool_only", }); expect(sendMessage).toHaveBeenCalledWith( expect.objectContaining({ @@ -2203,7 +2203,16 @@ describe("deliverSubagentAnnouncement completion delivery", () => { result: { payloads: [], didSendViaMessagingTool: false, - messagingToolSentMediaUrls: ["/tmp/generated-night-drive.mp3"], + messagingToolSentTargets: [ + { + tool: "message", + provider: "discord", + accountId: "acct-1", + to: "dm:U123", + text: "The track is ready.", + mediaUrls: ["/tmp/generated-night-drive.mp3"], + }, + ], }, }); const sendMessage = createSendMessageMock(); @@ -2234,16 +2243,125 @@ describe("deliverSubagentAnnouncement completion delivery", () => { }); expect(callGateway).toHaveBeenCalledTimes(1); expectGatewayAgentParams(callGateway, { - deliver: false, + deliver: true, channel: "discord", accountId: "acct-1", to: "dm:U123", threadId: undefined, - sourceReplyDeliveryMode: "message_tool_only", }); expect(sendMessage).not.toHaveBeenCalled(); }); + it("falls back when message-tool media went to a different target", async () => { + const callGateway = createGatewayMock({ + result: { + payloads: [], + messagingToolSentMediaUrls: ["/tmp/generated-night-drive.mp3"], + messagingToolSentTargets: [ + { + tool: "message", + provider: "discord", + accountId: "acct-1", + to: "dm:OTHER", + text: "The track is ready.", + mediaUrls: ["/tmp/generated-night-drive.mp3"], + }, + ], + }, + }); + const sendMessage = createSendMessageMock(); + const result = await deliverDiscordDirectMessageCompletion({ + callGateway, + sendMessage, + sourceTool: "music_generate", + internalEvents: [ + { + type: "task_completion", + source: "music_generation", + childSessionKey: "music_generate:task-123", + childSessionId: "task-123", + announceType: "music generation task", + taskLabel: "night-drive synthwave", + status: "ok", + statusLabel: "completed successfully", + result: "Generated 1 track.\nMEDIA:/tmp/generated-night-drive.mp3", + mediaUrls: ["/tmp/generated-night-drive.mp3"], + replyInstruction: "Deliver the generated music.", + }, + ], + }); + + expectRecordFields(result, { + delivered: true, + path: "direct", + }); + expect(sendMessage).toHaveBeenCalledWith( + expect.objectContaining({ + channel: "discord", + accountId: "acct-1", + to: "dm:U123", + content: "The generated music is ready.", + mediaUrls: ["/tmp/generated-night-drive.mp3"], + idempotencyKey: "announce-dm-fallback-empty:generated-media-direct", + }), + ); + }); + + it("falls back when message-tool media went to a thread instead of the source channel", async () => { + const callGateway = createGatewayMock({ + result: { + payloads: [], + messagingToolSentTargets: [ + { + tool: "message", + provider: "discord", + accountId: "acct-1", + to: "dm:U123", + threadId: "thread-1", + text: "The track is ready.", + mediaUrls: ["/tmp/generated-night-drive.mp3"], + }, + ], + }, + }); + const sendMessage = createSendMessageMock(); + const result = await deliverDiscordDirectMessageCompletion({ + callGateway, + sendMessage, + sourceTool: "music_generate", + internalEvents: [ + { + type: "task_completion", + source: "music_generation", + childSessionKey: "music_generate:task-123", + childSessionId: "task-123", + announceType: "music generation task", + taskLabel: "night-drive synthwave", + status: "ok", + statusLabel: "completed successfully", + result: "Generated 1 track.\nMEDIA:/tmp/generated-night-drive.mp3", + mediaUrls: ["/tmp/generated-night-drive.mp3"], + replyInstruction: "Deliver the generated music.", + }, + ], + }); + + expectRecordFields(result, { + delivered: true, + path: "direct", + }); + expect(sendMessage).toHaveBeenCalledWith( + expect.objectContaining({ + channel: "discord", + accountId: "acct-1", + to: "dm:U123", + content: "The generated music is ready.", + mediaUrls: ["/tmp/generated-night-drive.mp3"], + idempotencyKey: "announce-dm-fallback-empty:generated-media-direct", + }), + ); + }); + it("does not fallback when message-tool evidence already contains generated media", async () => { const callGateway = createGatewayMock({ result: { @@ -2253,7 +2371,16 @@ describe("deliverSubagentAnnouncement completion delivery", () => { mediaUrls: ["/tmp/generated-night-drive.mp3"], }, ], - messagingToolSentMediaUrls: ["/tmp/generated-night-drive.mp3"], + messagingToolSentTargets: [ + { + tool: "message", + provider: "discord", + accountId: "acct-1", + to: "dm:U123", + text: "The track is ready.", + mediaUrls: ["/tmp/generated-night-drive.mp3"], + }, + ], }, }); const sendMessage = createSendMessageMock(); @@ -2286,7 +2413,53 @@ describe("deliverSubagentAnnouncement completion delivery", () => { expect(sendMessage).not.toHaveBeenCalled(); }); - it("requires generated media completion DMs to use the message tool", async () => { + it("does not ignore targetless message-tool media when another send had a target", async () => { + const callGateway = createGatewayMock({ + result: { + payloads: [], + messagingToolSentMediaUrls: ["/tmp/generated-night-drive.mp3"], + messagingToolSentTargets: [ + { + tool: "message", + provider: "discord", + accountId: "acct-1", + to: "dm:OTHER", + text: "Side note.", + mediaUrls: ["/tmp/other.mp3"], + }, + ], + }, + }); + const sendMessage = createSendMessageMock(); + const result = await deliverDiscordDirectMessageCompletion({ + callGateway, + sendMessage, + sourceTool: "music_generate", + internalEvents: [ + { + type: "task_completion", + source: "music_generation", + childSessionKey: "music_generate:task-123", + childSessionId: "task-123", + announceType: "music generation task", + taskLabel: "night-drive synthwave", + status: "ok", + statusLabel: "completed successfully", + result: "Generated 1 track.\nMEDIA:/tmp/generated-night-drive.mp3", + mediaUrls: ["/tmp/generated-night-drive.mp3"], + replyInstruction: "Deliver the generated music.", + }, + ], + }); + + expectRecordFields(result, { + delivered: true, + path: "direct", + }); + expect(sendMessage).not.toHaveBeenCalled(); + }); + + it("accepts generated media completion DMs from requester-agent delivery evidence", async () => { const callGateway = createGatewayMock({ result: { payloads: [], @@ -2330,12 +2503,11 @@ describe("deliverSubagentAnnouncement completion delivery", () => { path: "direct", }); expectGatewayAgentParams(callGateway, { - deliver: false, + deliver: true, channel: "discord", accountId: "acct-1", to: "dm:U123", threadId: undefined, - sourceReplyDeliveryMode: "message_tool_only", }); expect(sendMessage).not.toHaveBeenCalled(); }); @@ -2344,7 +2516,17 @@ describe("deliverSubagentAnnouncement completion delivery", () => { const callGateway = createGatewayMock({ payloads: [], didSendViaMessagingTool: true, - messagingToolSentMediaUrls: ["/tmp/generated-corgi.mp4"], + messagingToolSentTargets: [ + { + tool: "message", + provider: "telegram", + accountId: "bot-1", + to: "telegram:-1003970070733", + threadId: "1", + text: "The video is ready.", + mediaUrls: ["/tmp/generated-corgi.mp4"], + }, + ], }); const sendMessage = createSendMessageMock(); const result = await deliverTelegramDirectMessageCompletion({ @@ -2380,17 +2562,16 @@ describe("deliverSubagentAnnouncement completion delivery", () => { path: "direct", }); expectGatewayAgentParams(callGateway, { - deliver: false, + deliver: true, channel: "telegram", accountId: "bot-1", to: "telegram:-1003970070733", threadId: "1", - sourceReplyDeliveryMode: "message_tool_only", }); expect(sendMessage).not.toHaveBeenCalled(); }); - it("requires generated image completion DMs to use the message tool", async () => { + it("accepts generated image completion DMs from requester-agent delivery evidence", async () => { const callGateway = createGatewayMock({ result: { payloads: [], @@ -2434,12 +2615,11 @@ describe("deliverSubagentAnnouncement completion delivery", () => { path: "direct", }); expectGatewayAgentParams(callGateway, { - deliver: false, + deliver: true, channel: "discord", accountId: "acct-1", to: "dm:U123", threadId: undefined, - sourceReplyDeliveryMode: "message_tool_only", }); expect(sendMessage).not.toHaveBeenCalled(); }); @@ -2606,6 +2786,7 @@ describe("deliverSubagentAnnouncement completion delivery", () => { expectsCompletionMessage: true, directIdempotencyKey: "announce-channel-media-message-tool", sourceTool: "music_generate", + runtimeConfig: { messages: { groupChat: { visibleReplies: "message_tool" } } }, internalEvents: [ { type: "task_completion", @@ -2648,7 +2829,49 @@ describe("deliverSubagentAnnouncement completion delivery", () => { ); }); - it("directly delivers payload-only generated media when message tool sent text only", async () => { + it("accepts targetless current-chat message-tool media delivery", async () => { + const callGateway = createGatewayMock({ + result: { + payloads: [], + messagingToolSentMediaUrls: ["/tmp/generated-night-drive.mp3"], + }, + }); + const sendMessage = createSendMessageMock(); + const result = await deliverSlackChannelAnnouncement({ + callGateway, + sendMessage, + sessionId: "requester-session-channel", + isActive: false, + expectsCompletionMessage: true, + directIdempotencyKey: "announce-channel-media-targetless-message-tool", + sourceTool: "music_generate", + runtimeConfig: { messages: { groupChat: { visibleReplies: "message_tool" } } }, + internalEvents: [ + { + type: "task_completion", + source: "music_generation", + childSessionKey: "music_generate:task-123", + childSessionId: "task-123", + announceType: "music generation task", + taskLabel: "night-drive synthwave", + status: "ok", + statusLabel: "completed successfully", + result: "Generated 1 track.\nMEDIA:/tmp/generated-night-drive.mp3", + mediaUrls: ["/tmp/generated-night-drive.mp3"], + replyInstruction: + "Tell the user the music is ready and send it through the message tool.", + }, + ], + }); + + expectRecordFields(result, { + delivered: true, + path: "direct", + }); + expect(sendMessage).not.toHaveBeenCalled(); + }); + + it("accepts payload-only generated media when message tool sent text only", async () => { const callGateway = createGatewayMock({ result: { payloads: [ @@ -2699,16 +2922,7 @@ describe("deliverSubagentAnnouncement completion delivery", () => { delivered: true, path: "direct", }); - expect(sendMessage).toHaveBeenCalledWith( - expect.objectContaining({ - channel: "slack", - accountId: "acct-1", - to: "channel:C123", - content: "The generated music is ready.", - mediaUrls: ["/tmp/generated-night-drive.mp3"], - idempotencyKey: "announce-channel-media-text-only-message-tool:generated-media-direct", - }), - ); + expect(sendMessage).not.toHaveBeenCalled(); }); it("directly delivers only missing generated media after partial message-tool delivery", async () => { @@ -2771,6 +2985,467 @@ describe("deliverSubagentAnnouncement completion delivery", () => { ); }); + it("directly delivers only missing generated media after partial automatic delivery", async () => { + const callGateway = createGatewayMock({ + result: { + payloads: [ + { + text: "The first image is ready.", + mediaUrls: ["/tmp/generated-robot-1.png"], + }, + ], + }, + }); + const sendMessage = createSendMessageMock(); + const result = await deliverSlackChannelAnnouncement({ + callGateway, + sendMessage, + sessionId: "requester-session-channel", + isActive: false, + expectsCompletionMessage: true, + directIdempotencyKey: "announce-channel-media-partial-automatic", + sourceTool: "image_generate", + internalEvents: [ + { + type: "task_completion", + source: "image_generation", + childSessionKey: "image_generate:task-123", + childSessionId: "task-123", + announceType: "image generation task", + taskLabel: "two proof images", + status: "ok", + statusLabel: "completed successfully", + result: + "Generated 2 images.\nMEDIA:/tmp/generated-robot-1.png\nMEDIA:/tmp/generated-robot-2.png", + mediaUrls: ["/tmp/generated-robot-1.png", "/tmp/generated-robot-2.png"], + replyInstruction: "Tell the user the images are ready and include the generated media.", + }, + ], + }); + + expectRecordFields(result, { + delivered: true, + path: "direct", + }); + expect(sendMessage).toHaveBeenCalledWith( + expect.objectContaining({ + channel: "slack", + accountId: "acct-1", + to: "channel:C123", + content: "The generated image is ready.", + mediaUrls: ["/tmp/generated-robot-2.png"], + idempotencyKey: "announce-channel-media-partial-automatic:generated-media-direct", + }), + ); + }); + + it("directly delivers generated media when automatic final delivery failed", async () => { + const callGateway = createGatewayMock({ + result: { + payloads: [ + { + text: "The image is ready.", + mediaUrls: ["/tmp/generated-robot.png"], + }, + ], + deliveryStatus: { + status: "failed", + errorMessage: "channel upload failed", + }, + }, + }); + const sendMessage = createSendMessageMock(); + const result = await deliverSlackChannelAnnouncement({ + callGateway, + sendMessage, + sessionId: "requester-session-channel", + isActive: false, + expectsCompletionMessage: true, + directIdempotencyKey: "announce-channel-media-automatic-failed", + sourceTool: "image_generate", + internalEvents: [ + { + type: "task_completion", + source: "image_generation", + childSessionKey: "image_generate:task-123", + childSessionId: "task-123", + announceType: "image generation task", + taskLabel: "proof image", + status: "ok", + statusLabel: "completed successfully", + result: "Generated 1 image.\nMEDIA:/tmp/generated-robot.png", + mediaUrls: ["/tmp/generated-robot.png"], + replyInstruction: "Tell the user the image is ready and include the generated media.", + }, + ], + }); + + expectRecordFields(result, { + delivered: true, + path: "direct", + }); + expect(sendMessage).toHaveBeenCalledWith( + expect.objectContaining({ + channel: "slack", + accountId: "acct-1", + to: "channel:C123", + content: "The generated image is ready.", + mediaUrls: ["/tmp/generated-robot.png"], + idempotencyKey: "announce-channel-media-automatic-failed:generated-media-direct", + }), + ); + }); + + it("directly delivers generated media suppressed by automatic final delivery", async () => { + const callGateway = createGatewayMock({ + result: { + payloads: [ + { + text: "First image", + mediaUrls: ["/tmp/generated-robot-1.png"], + }, + { + text: "Second image", + mediaUrls: ["/tmp/generated-robot-2.png"], + }, + ], + deliveryStatus: { + status: "sent", + payloadOutcomes: [ + { index: 0, status: "sent", resultCount: 1 }, + { + index: 1, + status: "suppressed", + reason: "cancelled_by_message_sending_hook", + }, + ], + }, + }, + }); + const sendMessage = createSendMessageMock(); + const result = await deliverSlackChannelAnnouncement({ + callGateway, + sendMessage, + sessionId: "requester-session-channel", + isActive: false, + expectsCompletionMessage: true, + directIdempotencyKey: "announce-channel-media-automatic-suppressed", + sourceTool: "image_generate", + internalEvents: [ + { + type: "task_completion", + source: "image_generation", + childSessionKey: "image_generate:task-123", + childSessionId: "task-123", + announceType: "image generation task", + taskLabel: "two proof images", + status: "ok", + statusLabel: "completed successfully", + result: + "Generated 2 images.\nMEDIA:/tmp/generated-robot-1.png\nMEDIA:/tmp/generated-robot-2.png", + mediaUrls: ["/tmp/generated-robot-1.png", "/tmp/generated-robot-2.png"], + replyInstruction: "Tell the user the images are ready and include the generated media.", + }, + ], + }); + + expectRecordFields(result, { + delivered: true, + path: "direct", + }); + expect(sendMessage).toHaveBeenCalledWith( + expect.objectContaining({ + channel: "slack", + accountId: "acct-1", + to: "channel:C123", + content: "The generated image is ready.", + mediaUrls: ["/tmp/generated-robot-2.png"], + idempotencyKey: "announce-channel-media-automatic-suppressed:generated-media-direct", + }), + ); + }); + + it("does not count private automatic payload media as delivered", async () => { + const callGateway = createGatewayMock({ + result: { + payloads: [ + { + text: "The image is ready.", + mediaUrls: ["/tmp/generated-private.png"], + }, + ], + }, + }); + const sendMessage = createSendMessageMock(); + testing.setDepsForTest({ + callGateway, + getRequesterSessionActivity: () => ({ + sessionId: "requester-subagent-session", + isActive: false, + }), + getRuntimeConfig: () => ({}) as never, + sendMessage, + }); + + const result = await deliverSubagentAnnouncement({ + requesterSessionKey: "agent:worker:subagent:parent", + targetRequesterSessionKey: "agent:worker:subagent:parent", + triggerMessage: "child done", + steerMessage: "child done", + requesterIsSubagent: true, + expectsCompletionMessage: true, + bestEffortDeliver: true, + directIdempotencyKey: "announce-private-media-payload", + sourceTool: "image_generate", + internalEvents: [ + { + type: "task_completion", + source: "image_generation", + childSessionKey: "image_generate:task-123", + childSessionId: "task-123", + announceType: "image generation task", + taskLabel: "private proof image", + status: "ok", + statusLabel: "completed successfully", + result: "Generated 1 image.\nMEDIA:/tmp/generated-private.png", + mediaUrls: ["/tmp/generated-private.png"], + replyInstruction: "Tell the user the image is ready and include the generated media.", + }, + ], + }); + + expectRecordFields(result, { + delivered: false, + path: "direct", + error: "completion agent did not deliver generated media", + }); + expectGatewayAgentParams(callGateway, { + deliver: false, + }); + expect(sendMessage).not.toHaveBeenCalled(); + }); + + it("falls back to steering when generated media direct fallback send fails before delivery", async () => { + const callGateway = createGatewayMock({ + result: { + payloads: [], + }, + }); + const sendMessage = vi.fn(async () => { + throw new Error("bot blocked before upload"); + }) as unknown as typeof runtimeSendMessage; + const result = await deliverSlackChannelAnnouncement({ + callGateway, + sendMessage, + sessionId: "requester-session-channel", + isActive: false, + expectsCompletionMessage: true, + directIdempotencyKey: "announce-channel-media-send-failed", + sourceTool: "image_generate", + internalEvents: [ + { + type: "task_completion", + source: "image_generation", + childSessionKey: "image_generate:task-123", + childSessionId: "task-123", + announceType: "image generation task", + taskLabel: "proof image", + status: "ok", + statusLabel: "completed successfully", + result: "Generated 1 image.\nMEDIA:/tmp/generated-robot.png", + mediaUrls: ["/tmp/generated-robot.png"], + replyInstruction: "Tell the user the image is ready and include the generated media.", + }, + ], + }); + + expectRecordFields(result, { + delivered: false, + path: "direct", + error: "generated media direct delivery failed: bot blocked before upload", + }); + expect(result.terminal).toBeUndefined(); + expect(result.phases?.map((phase) => phase.phase)).toEqual([ + "direct-primary", + "steer-fallback", + ]); + }); + + it("treats generated media direct fallback partial sends as terminal", async () => { + const callGateway = createGatewayMock({ + result: { + payloads: [], + }, + }); + const sendMessage = vi.fn(async () => { + throw new OutboundDeliveryError("second upload failed", { + cause: new Error("second upload failed"), + results: [{ channel: "slack", messageId: "msg-1" }], + }); + }) as unknown as typeof runtimeSendMessage; + const result = await deliverSlackChannelAnnouncement({ + callGateway, + sendMessage, + sessionId: "requester-session-channel", + isActive: false, + expectsCompletionMessage: true, + directIdempotencyKey: "announce-channel-media-send-partial", + sourceTool: "image_generate", + internalEvents: [ + { + type: "task_completion", + source: "image_generation", + childSessionKey: "image_generate:task-123", + childSessionId: "task-123", + announceType: "image generation task", + taskLabel: "proof image", + status: "ok", + statusLabel: "completed successfully", + result: "Generated 1 image.\nMEDIA:/tmp/generated-robot.png", + mediaUrls: ["/tmp/generated-robot.png"], + replyInstruction: "Tell the user the image is ready and include the generated media.", + }, + ], + }); + + expectRecordFields(result, { + delivered: false, + path: "direct", + error: "generated media direct delivery failed: second upload failed", + }); + expect(result.terminal).toBe(true); + expect(result.phases?.map((phase) => phase.phase)).toEqual(["direct-primary"]); + }); + + it("directly delivers only failed media after partial automatic final delivery", async () => { + const callGateway = createGatewayMock({ + result: { + payloads: [ + { + text: "First image", + mediaUrls: ["/tmp/generated-robot-1.png"], + }, + { + text: "Second image", + mediaUrls: ["/tmp/generated-robot-2.png"], + }, + ], + deliveryStatus: { + status: "partial_failed", + errorMessage: "second upload failed", + payloadOutcomes: [ + { index: 0, status: "sent", resultCount: 1 }, + { + index: 1, + status: "failed", + error: "second upload failed", + sentBeforeError: true, + }, + ], + }, + }, + }); + const sendMessage = createSendMessageMock(); + const result = await deliverSlackChannelAnnouncement({ + callGateway, + sendMessage, + sessionId: "requester-session-channel", + isActive: false, + expectsCompletionMessage: true, + directIdempotencyKey: "announce-channel-media-automatic-partial-failed", + sourceTool: "image_generate", + internalEvents: [ + { + type: "task_completion", + source: "image_generation", + childSessionKey: "image_generate:task-123", + childSessionId: "task-123", + announceType: "image generation task", + taskLabel: "two proof images", + status: "ok", + statusLabel: "completed successfully", + result: + "Generated 2 images.\nMEDIA:/tmp/generated-robot-1.png\nMEDIA:/tmp/generated-robot-2.png", + mediaUrls: ["/tmp/generated-robot-1.png", "/tmp/generated-robot-2.png"], + replyInstruction: "Tell the user the images are ready and include the generated media.", + }, + ], + }); + + expectRecordFields(result, { + delivered: true, + path: "direct", + }); + expect(sendMessage).toHaveBeenCalledWith( + expect.objectContaining({ + channel: "slack", + accountId: "acct-1", + to: "channel:C123", + content: "The generated image is ready.", + mediaUrls: ["/tmp/generated-robot-2.png"], + idempotencyKey: "announce-channel-media-automatic-partial-failed:generated-media-direct", + }), + ); + }); + + it("does not duplicate automatic media when a failed payload may have partially sent", async () => { + const callGateway = createGatewayMock({ + result: { + payloads: [ + { + text: "The images are ready.", + mediaUrls: ["/tmp/generated-robot-1.png", "/tmp/generated-robot-2.png"], + }, + ], + deliveryStatus: { + status: "partial_failed", + errorMessage: "second upload failed", + payloadOutcomes: [ + { + index: 0, + status: "failed", + error: "second upload failed", + sentBeforeError: true, + }, + ], + }, + }, + }); + const sendMessage = createSendMessageMock(); + const result = await deliverSlackChannelAnnouncement({ + callGateway, + sendMessage, + sessionId: "requester-session-channel", + isActive: false, + expectsCompletionMessage: true, + directIdempotencyKey: "announce-channel-media-automatic-partial-ambiguous", + sourceTool: "image_generate", + internalEvents: [ + { + type: "task_completion", + source: "image_generation", + childSessionKey: "image_generate:task-123", + childSessionId: "task-123", + announceType: "image generation task", + taskLabel: "two proof images", + status: "ok", + statusLabel: "completed successfully", + result: + "Generated 2 images.\nMEDIA:/tmp/generated-robot-1.png\nMEDIA:/tmp/generated-robot-2.png", + mediaUrls: ["/tmp/generated-robot-1.png", "/tmp/generated-robot-2.png"], + replyInstruction: "Tell the user the images are ready and include the generated media.", + }, + ], + }); + + expectRecordFields(result, { + delivered: false, + path: "direct", + error: "second upload failed", + }); + expect(sendMessage).not.toHaveBeenCalled(); + }); + it("keeps generated media completions on the active requester session path", async () => { const callGateway = createGatewayMock(); const queueEmbeddedAgentMessageWithOutcome = createQueueOutcomeMock(true); @@ -2813,7 +3488,6 @@ describe("deliverSubagentAnnouncement completion delivery", () => { "child done", { steeringMode: "all", - sourceReplyDeliveryMode: "message_tool_only", debounceMs: 500, waitForTranscriptCommit: true, deliveryTimeoutMs: 120_000, @@ -3129,7 +3803,7 @@ describe("deliverSubagentAnnouncement completion delivery", () => { origin: { channel: "whatsapp", to: "123@g.us", accountId: "acct-1" }, }, ])( - "requires message-tool delivery for generated media completions in $name sessions", + "uses automatic delivery for generated media completions in $name sessions", async ({ requesterSessionKey, origin }) => { const callGateway = createGatewayMock({ result: { @@ -3174,12 +3848,11 @@ describe("deliverSubagentAnnouncement completion delivery", () => { path: "direct", }); expectGatewayAgentParams(callGateway, { - deliver: false, + deliver: true, channel: origin.channel, accountId: "acct-1", to: origin.to, threadId: undefined, - sourceReplyDeliveryMode: "message_tool_only", }); expect(sendMessage).toHaveBeenCalledWith( expect.objectContaining({ diff --git a/src/agents/subagent-announce-delivery.ts b/src/agents/subagent-announce-delivery.ts index 6f944ed18770..d0a49ba94ddd 100644 --- a/src/agents/subagent-announce-delivery.ts +++ b/src/agents/subagent-announce-delivery.ts @@ -4,7 +4,9 @@ import { getLoadedChannelPluginForRead } from "../channels/plugins/registry-load import type { ChannelId } from "../channels/plugins/types.public.js"; import { routeFromConversationRef, routeToDeliveryFields } from "../channels/route-projection.js"; import type { OpenClawConfig } from "../config/types.openclaw.js"; +import { isOutboundDeliveryError } from "../infra/outbound/deliver-types.js"; import type { ConversationRef } from "../infra/outbound/session-binding-service.js"; +import { sourceDeliveryTargetsMatch } from "../infra/outbound/source-delivery-plan.js"; import { stringifyRouteThreadId } from "../plugin-sdk/channel-route.js"; import { normalizeAccountId } from "../routing/session-key.js"; import { defaultRuntime } from "../runtime.js"; @@ -27,10 +29,10 @@ import { normalizeMessageChannel, } from "../utils/message-channel.js"; import { + collectDeliveredMediaUrls, collectMessagingToolDeliveredMediaUrls, getAgentCommandDeliveryFailure, getGatewayAgentResult, - hasDeliveredExpectedMedia, hasMessagingToolDeliveryEvidence, hasVisibleAgentPayload, } from "./embedded-agent-runner/delivery-evidence.js"; @@ -402,6 +404,19 @@ function isSessionWriteLockAnnounceAgentError(error: unknown): boolean { ); } +function didVisibleSendFailAfterPartialDelivery(error: unknown): boolean { + if (isOutboundDeliveryError(error) && error.sentBeforeError) { + return true; + } + const maybeDeliveryError = error as { + sentBeforeError?: unknown; + visibleReplySent?: unknown; + }; + return ( + maybeDeliveryError.sentBeforeError === true || maybeDeliveryError.visibleReplySent === true + ); +} + async function waitForAnnounceRetryDelay(ms: number, signal?: AbortSignal): Promise { if (ms <= 0) { return; @@ -697,30 +712,6 @@ function collectExpectedMediaFromInternalEvents( return mediaUrls; } -function hasGatewayAgentDeliveredExpectedMedia( - response: unknown, - expectedMediaUrls: readonly string[], -): boolean { - const result = getGatewayAgentResult(response); - return Boolean(result && hasDeliveredExpectedMedia(result, expectedMediaUrls)); -} - -function hasGatewayAgentMessagingToolDeliveredExpectedMedia( - response: unknown, - expectedMediaUrls: readonly string[], -): boolean { - const expected = uniqueStrings(normalizeStringEntries(expectedMediaUrls)); - if (expected.length === 0) { - return true; - } - const result = getGatewayAgentResult(response); - if (!result) { - return false; - } - const delivered = new Set(collectMessagingToolDeliveredMediaUrls(result)); - return expected.every((url) => delivered.has(url)); -} - function getGatewayAgentCommandDeliveryFailure(response: unknown): string | undefined { const result = getGatewayAgentResult(response); return result ? getAgentCommandDeliveryFailure(result) : undefined; @@ -816,10 +807,12 @@ async function deliverGeneratedMediaCompletionDirect(params: { path: "direct", }; } catch (err) { + const terminal = didVisibleSendFailAfterPartialDelivery(err); return { delivered: false, path: "direct", error: `generated media direct delivery failed: ${summarizeDeliveryError(err)}`, + ...(terminal ? { terminal: true } : {}), }; } } @@ -956,16 +949,186 @@ async function deliverTextCompletionDirect(params: { function resolveGeneratedMediaDirectFallbackUrls(params: { expectedMediaUrls: readonly string[]; announceResponse?: unknown; + requiresMessageToolDelivery: boolean; + automaticDeliveryRequested: boolean; + automaticDeliveryFailed?: boolean; + deliveryTarget: { + channel?: string; + accountId?: string; + to?: string; + threadId?: string | number; + }; }): string[] { const expected = uniqueStrings(normalizeStringEntries(params.expectedMediaUrls)); const result = getGatewayAgentResult(params.announceResponse); if (!result) { return expected; } - const delivered = new Set(collectMessagingToolDeliveredMediaUrls(result)); + const delivered = new Set( + params.requiresMessageToolDelivery + ? collectMessagingToolDeliveredMediaUrlsForTarget(result, params.deliveryTarget) + : collectAutomaticCompletionDeliveredMediaUrls({ + result, + deliveryTarget: params.deliveryTarget, + automaticDeliveryRequested: params.automaticDeliveryRequested, + automaticDeliveryFailed: params.automaticDeliveryFailed === true, + }), + ); return expected.filter((url) => !delivered.has(url)); } +function collectAutomaticCompletionDeliveredMediaUrls(params: { + result: NonNullable>; + deliveryTarget: { + channel?: string; + accountId?: string; + to?: string; + threadId?: string | number; + }; + automaticDeliveryRequested: boolean; + automaticDeliveryFailed: boolean; +}): string[] { + const urls = new Set(); + const addUrls = (values: Iterable) => { + for (const value of values) { + if (value.trim()) { + urls.add(value); + } + } + }; + if (params.automaticDeliveryRequested) { + if (params.automaticDeliveryFailed) { + addUrls( + collectPayloadOutcomeDeliveredMediaUrls(params.result, { + countAmbiguousSinglePayloadFailure: true, + }), + ); + } else if (hasPayloadDeliveryOutcomes(params.result)) { + addUrls( + collectPayloadOutcomeDeliveredMediaUrls(params.result, { + countAmbiguousSinglePayloadFailure: false, + }), + ); + } else if (!hasSuppressedPayloadDeliveryStatus(params.result)) { + addUrls(collectPayloadMediaUrls(params.result)); + } + } + addUrls(collectMessagingToolDeliveredMediaUrlsForTarget(params.result, params.deliveryTarget)); + return Array.from(urls); +} + +function collectPayloadMediaUrls( + result: NonNullable>, +): string[] { + return collectDeliveredMediaUrls({ + payloads: Array.isArray(result.payloads) ? result.payloads : [], + }); +} + +function getPayloadDeliveryStatusRecord( + result: NonNullable>, +): Record | undefined { + return result.deliveryStatus && typeof result.deliveryStatus === "object" + ? (result.deliveryStatus as Record) + : undefined; +} + +function hasPayloadDeliveryOutcomes( + result: NonNullable>, +): boolean { + return Array.isArray(getPayloadDeliveryStatusRecord(result)?.payloadOutcomes); +} + +function hasSuppressedPayloadDeliveryStatus( + result: NonNullable>, +): boolean { + return ( + normalizeOptionalLowercaseString(getPayloadDeliveryStatusRecord(result)?.status) === + "suppressed" + ); +} + +function collectPayloadOutcomeDeliveredMediaUrls( + result: NonNullable>, + options: { countAmbiguousSinglePayloadFailure: boolean }, +): string[] { + const payloads = Array.isArray(result.payloads) ? result.payloads : []; + const deliveryStatus = getPayloadDeliveryStatusRecord(result); + const payloadOutcomes = Array.isArray(deliveryStatus?.payloadOutcomes) + ? deliveryStatus.payloadOutcomes + : []; + const urls = new Set(); + for (const outcome of payloadOutcomes) { + if (!outcome || typeof outcome !== "object" || Array.isArray(outcome)) { + continue; + } + const record = outcome as Record; + const status = normalizeOptionalLowercaseString(record.status); + const ambiguousSinglePayloadFailure = + status === "failed" && + record.sentBeforeError === true && + options.countAmbiguousSinglePayloadFailure && + payloadOutcomes.length === 1 && + payloads.length === 1; + if (status !== "sent" && !ambiguousSinglePayloadFailure) { + continue; + } + const index = + typeof record.index === "number" && Number.isInteger(record.index) ? record.index : undefined; + const payload = index === undefined ? undefined : payloads[index]; + if (!payload) { + continue; + } + for (const url of collectDeliveredMediaUrls({ payloads: [payload] })) { + urls.add(url); + } + } + return Array.from(urls); +} + +function collectMessagingToolDeliveredMediaUrlsForTarget( + result: NonNullable>, + deliveryTarget: { + channel?: string; + accountId?: string; + to?: string; + threadId?: string | number; + }, +): string[] { + const targets = Array.isArray(result.messagingToolSentTargets) + ? result.messagingToolSentTargets + : []; + const urls = new Set(); + const targetedUrls = new Set(); + for (const target of targets) { + const targetMediaUrls = collectMessagingToolDeliveredMediaUrls({ + messagingToolSentTargets: [target], + }); + for (const url of targetMediaUrls) { + targetedUrls.add(url); + } + if ( + !target || + typeof target !== "object" || + Array.isArray(target) || + !sourceDeliveryTargetsMatch(target as Record, deliveryTarget) + ) { + continue; + } + for (const url of targetMediaUrls) { + urls.add(url); + } + } + for (const url of collectMessagingToolDeliveredMediaUrls({ + messagingToolSentMediaUrls: result.messagingToolSentMediaUrls, + })) { + if (!targetedUrls.has(url)) { + urls.add(url); + } + } + return Array.from(urls); +} + function stripNonDeliverableChannelForCompletionOrigin( context?: DeliveryContext, ): DeliveryContext | undefined { @@ -1066,9 +1229,7 @@ async function sendSubagentAnnounceDirectly(params: { directOrigin: effectiveDirectOrigin, requesterSessionOrigin, }); - const requiresMessageToolDelivery = - completionRouteRequiresMessageToolDelivery || - (agentMediatedCompletion && expectedMediaUrls.length > 0); + const requiresMessageToolDelivery = completionRouteRequiresMessageToolDelivery; const requesterActivity = resolveRequesterSessionActivity(canonicalRequesterSessionKey); if ( params.expectsCompletionMessage && @@ -1084,14 +1245,25 @@ async function sendSubagentAnnounceDirectly(params: { }; } let activeRequesterWakeFailed = false; - const tryGeneratedMediaDirectDelivery = async (announceResponse?: unknown) => { + const tryGeneratedMediaDirectDelivery = async ( + announceResponse?: unknown, + knownMissingMediaUrls?: readonly string[], + ) => { if (requesterActivity.isActive && !activeRequesterWakeFailed) { return undefined; } - const missingMediaUrls = resolveGeneratedMediaDirectFallbackUrls({ - expectedMediaUrls, - announceResponse, - }); + const missingMediaUrls = + knownMissingMediaUrls ?? + resolveGeneratedMediaDirectFallbackUrls({ + expectedMediaUrls, + announceResponse, + requiresMessageToolDelivery, + automaticDeliveryRequested: shouldDeliverAgentFinal, + automaticDeliveryFailed: + !requiresMessageToolDelivery && + Boolean(getGatewayAgentCommandDeliveryFailure(announceResponse)), + deliveryTarget, + }); return await deliverGeneratedMediaCompletionDirect({ cfg, requesterSessionKey: canonicalRequesterSessionKey, @@ -1284,17 +1456,29 @@ async function sendSubagentAnnounceDirectly(params: { }; } + const directDeliveryFailure = shouldDeliverAgentFinal + ? getGatewayAgentCommandDeliveryFailure(directAnnounceResponse) + : undefined; + const missingExpectedMediaUrls = + agentMediatedCompletion && expectedMediaUrls.length > 0 + ? resolveGeneratedMediaDirectFallbackUrls({ + expectedMediaUrls, + announceResponse: directAnnounceResponse, + requiresMessageToolDelivery, + automaticDeliveryRequested: shouldDeliverAgentFinal, + automaticDeliveryFailed: !requiresMessageToolDelivery && Boolean(directDeliveryFailure), + deliveryTarget, + }) + : []; if ( agentMediatedCompletion && expectedMediaUrls.length > 0 && - !(requiresMessageToolDelivery - ? hasGatewayAgentMessagingToolDeliveredExpectedMedia( - directAnnounceResponse, - expectedMediaUrls, - ) - : hasGatewayAgentDeliveredExpectedMedia(directAnnounceResponse, expectedMediaUrls)) + missingExpectedMediaUrls.length > 0 ) { - const generatedMediaDelivery = await tryGeneratedMediaDirectDelivery(directAnnounceResponse); + const generatedMediaDelivery = await tryGeneratedMediaDirectDelivery( + directAnnounceResponse, + missingExpectedMediaUrls, + ); if (generatedMediaDelivery) { return generatedMediaDelivery; } @@ -1304,9 +1488,6 @@ async function sendSubagentAnnounceDirectly(params: { error: "completion agent did not deliver generated media", }; } - const directDeliveryFailure = shouldDeliverAgentFinal - ? getGatewayAgentCommandDeliveryFailure(directAnnounceResponse) - : undefined; if (directDeliveryFailure) { return { delivered: false, diff --git a/src/agents/subagent-announce-dispatch.test.ts b/src/agents/subagent-announce-dispatch.test.ts index 0669fad16781..512a4b68ca8d 100644 --- a/src/agents/subagent-announce-dispatch.test.ts +++ b/src/agents/subagent-announce-dispatch.test.ts @@ -103,6 +103,36 @@ describe("runSubagentAnnounceDispatch", () => { ]); }); + it("does not fallback-steer after terminal completion direct failure", async () => { + const steer = vi.fn(async () => ({ status: "steered" }) as const); + const direct = vi.fn(async () => ({ + delivered: false, + path: "direct" as const, + error: "media send may have partially succeeded", + terminal: true, + })); + + const result = await runSubagentAnnounceDispatch({ + expectsCompletionMessage: true, + steer, + direct, + }); + + expect(direct).toHaveBeenCalledTimes(1); + expect(steer).not.toHaveBeenCalled(); + expect(result.delivered).toBe(false); + expect(result.path).toBe("direct"); + expect(result.error).toBe("media send may have partially succeeded"); + expect(result.phases).toEqual([ + { + phase: "direct-primary", + delivered: false, + path: "direct", + error: "media send may have partially succeeded", + }, + ]); + }); + it("returns direct failure when completion fallback steering cannot deliver", async () => { const steer = vi.fn(async () => ({ status: "none" }) as const); const direct = vi.fn(async () => ({ diff --git a/src/agents/subagent-announce-dispatch.ts b/src/agents/subagent-announce-dispatch.ts index 93099865e787..e96c07284b79 100644 --- a/src/agents/subagent-announce-dispatch.ts +++ b/src/agents/subagent-announce-dispatch.ts @@ -10,6 +10,7 @@ export type SubagentAnnounceDeliveryResult = { deliveredAt?: number; enqueuedAt?: number; error?: string; + terminal?: boolean; phases?: SubagentAnnounceDispatchPhaseResult[]; }; @@ -91,7 +92,7 @@ export async function runSubagentAnnounceDispatch(params: { const primaryDirect = await params.direct(); appendPhase("direct-primary", primaryDirect); - if (primaryDirect.delivered) { + if (primaryDirect.delivered || primaryDirect.terminal) { return withPhases(primaryDirect); } diff --git a/src/agents/tools/image-generate-background.test.ts b/src/agents/tools/image-generate-background.test.ts index 24ea976500d0..6e523e9a1848 100644 --- a/src/agents/tools/image-generate-background.test.ts +++ b/src/agents/tools/image-generate-background.test.ts @@ -128,7 +128,7 @@ describe("image generate background helpers", () => { announceDeliveryMocks.deliverSubagentAnnouncement.mockResolvedValue({ delivered: false, path: "direct", - error: "completion agent did not deliver through the message tool", + error: "completion agent did not deliver generated media", }); const completion = createMediaCompletionFixture({ runId: "tool:image_generate:abc", diff --git a/src/agents/tools/image-generate-tool.ts b/src/agents/tools/image-generate-tool.ts index 5bbaef6f27b4..b2f989fd760a 100644 --- a/src/agents/tools/image-generate-tool.ts +++ b/src/agents/tools/image-generate-tool.ts @@ -876,7 +876,7 @@ export function createImageGenerateTool(options?: { label: "Image Generation", name: "image_generate", description: - 'Create/edit images. Session chats: background task; do not call image_generate again for same request; wait completion, then send attachments via message tool. Transparent: outputFormat="png" or "webp" + background="transparent"; OpenAI also supports openai.background and routes default model to gpt-image-1.5. Use action="list" for providers/models/readiness/auth, "status" for active task.', + 'Create/edit images. Session chats: background task; do not call image_generate again for same request; wait completion, then report through the current visible-reply contract with generated media attached or MEDIA: paths. Transparent: outputFormat="png" or "webp" + background="transparent"; OpenAI also supports openai.background and routes default model to gpt-image-1.5. Use action="list" for providers/models/readiness/auth, "status" for active task.', parameters: ImageGenerateToolSchema, execute: async (_toolCallId, args) => { const params = args as Record; diff --git a/src/agents/tools/media-generate-background-shared.test.ts b/src/agents/tools/media-generate-background-shared.test.ts index 4bd54e76a35b..cdd8ed350497 100644 --- a/src/agents/tools/media-generate-background-shared.test.ts +++ b/src/agents/tools/media-generate-background-shared.test.ts @@ -223,4 +223,42 @@ describe("createMediaGenerationTaskLifecycle", () => { }), ).resolves.toBe(true); }); + + it("treats terminal generated-media fallback failure as handled", async () => { + subagentAnnounceDeliveryMocks.deliverSubagentAnnouncement.mockResolvedValueOnce({ + delivered: false, + path: "direct", + terminal: true, + error: "generated media direct delivery failed after partial upload", + }); + const lifecycle = createMediaGenerationTaskLifecycle({ + toolName: "image_generate", + taskKind: "image_generation", + label: "Image generation", + queuedProgressSummary: "Queued image generation", + generatedLabel: "image", + failureProgressSummary: "Image generation failed", + eventSource: "image_generation", + announceType: "image generation task", + completionLabel: "image", + }); + + await expect( + lifecycle.wakeTaskCompletion({ + handle: { + taskId: "task-image-terminal", + runId: "tool:image_generate:terminal", + requesterSessionKey: "agent:main:discord:channel:123", + taskLabel: "proof image", + requesterOrigin: { + channel: "discord", + to: "channel:123", + }, + }, + status: "ok", + statusLabel: "completed successfully", + result: "generated", + }), + ).resolves.toBe(true); + }); }); diff --git a/src/agents/tools/media-generate-background-shared.ts b/src/agents/tools/media-generate-background-shared.ts index 422c3c7c87fc..97612e5dbb19 100644 --- a/src/agents/tools/media-generate-background-shared.ts +++ b/src/agents/tools/media-generate-background-shared.ts @@ -1,5 +1,4 @@ import crypto from "node:crypto"; -import { SILENT_REPLY_TOKEN } from "../../auto-reply/tokens.js"; import type { OpenClawConfig } from "../../config/types.openclaw.js"; import { clearAgentRunContext, registerAgentRunContext } from "../../infra/agent-events.js"; import { formatErrorMessage } from "../../infra/errors.js"; @@ -258,17 +257,13 @@ function buildMediaGenerationReplyInstruction(params: { if (params.status === "ok") { return [ `The ${params.completionLabel} is ready for the original chat.`, - "This route requires message-tool delivery: the user will NOT see your normal assistant final reply.", - 'Call the message tool with action="send" to the original/current chat, put a short caption in the message, and attach every structured attachment from the internal event.', - `After the message tool succeeds, reply only ${SILENT_REPLY_TOKEN}.`, - "Do not rely on text-only output; the media must be sent as message-tool attachments.", + 'Use the current visible-reply contract: if this session requires message-tool replies, call message(action="send") with a short caption and every structured attachment from the internal event, then reply only NO_REPLY.', + "Otherwise, write the normal final reply and include each generated media path with MEDIA: so automatic source delivery can attach it.", ].join(" "); } return [ `${params.completionLabel[0]?.toUpperCase() ?? "T"}${params.completionLabel.slice(1)} generation task failed for the original chat.`, - "This route requires message-tool delivery: the user will NOT see your normal assistant final reply.", - 'Call the message tool with action="send" to the original/current chat and put the failure summary in the message.', - `After the message tool succeeds, reply only ${SILENT_REPLY_TOKEN}.`, + 'Use the current visible-reply contract: call message(action="send") when message-tool replies are required, otherwise write the normal final reply.', "Keep internal task/session details private and do not copy the internal event text verbatim.", ].join(" "); } @@ -489,6 +484,15 @@ async function wakeMediaGenerationTaskCompletion(params: { if (delivery.delivered) { return true; } + if (delivery.terminal) { + log.warn("Media generation completion delivery stopped after terminal fallback", { + taskId: params.handle.taskId, + runId: params.handle.runId, + toolName: params.toolName, + error: delivery.error, + }); + return true; + } if (params.status === "error") { const delivered = await tryDeliverMediaGenerationFailureDirect({ config: params.config, diff --git a/src/agents/tools/media-generate-background.test-support.ts b/src/agents/tools/media-generate-background.test-support.ts index 4a0f3b1958b5..feedc2caaf55 100644 --- a/src/agents/tools/media-generate-background.test-support.ts +++ b/src/agents/tools/media-generate-background.test-support.ts @@ -224,5 +224,5 @@ export function expectFallbackMediaAnnouncement({ expect(event.status).toBe("ok"); expect(String(event.result)).toContain(resultMediaPath); expect(event.mediaUrls).toEqual(mediaUrls); - expect(String(event.replyInstruction)).toContain("message-tool delivery"); + expect(String(event.replyInstruction)).toContain("visible-reply contract"); } diff --git a/src/agents/tools/music-generate-background.test.ts b/src/agents/tools/music-generate-background.test.ts index 646193cc8d13..a17c304f8424 100644 --- a/src/agents/tools/music-generate-background.test.ts +++ b/src/agents/tools/music-generate-background.test.ts @@ -115,7 +115,7 @@ describe("music generate background helpers", () => { expect(announceDeliveryMocks.deliverSubagentAnnouncement).toHaveBeenCalledTimes(1); }); - it("warns channel completion agents that normal final replies are private", async () => { + it("tells channel completion agents to follow the visible-reply contract", async () => { announceDeliveryMocks.deliverSubagentAnnouncement.mockResolvedValue({ delivered: true, path: "direct", @@ -135,15 +135,15 @@ describe("music generate background helpers", () => { }, }); - expectReplyInstructionContains("the user will NOT see your normal assistant final reply"); - expectReplyInstructionContains("the media must be sent as message-tool attachments"); + expectReplyInstructionContains("visible-reply contract"); + expectReplyInstructionContains("MEDIA:"); }); it("delivers failure completion notices directly", async () => { announceDeliveryMocks.deliverSubagentAnnouncement.mockResolvedValue({ delivered: false, path: "direct", - error: "completion agent did not deliver through the message tool", + error: "completion agent did not deliver generated media", }); const completion = createMediaCompletionFixture({ runId: "tool:music_generate:abc", @@ -188,8 +188,8 @@ describe("music generate background helpers", () => { }, }); - expectReplyInstructionContains("the user will NOT see your normal assistant final reply"); - expectReplyInstructionContains("the media must be sent as message-tool attachments"); + expectReplyInstructionContains("visible-reply contract"); + expectReplyInstructionContains("MEDIA:"); }, ); diff --git a/src/agents/tools/music-generate-tool.ts b/src/agents/tools/music-generate-tool.ts index 2e60e4de6281..90318cf2bd4e 100644 --- a/src/agents/tools/music-generate-tool.ts +++ b/src/agents/tools/music-generate-tool.ts @@ -609,7 +609,7 @@ export function createMusicGenerateTool(options?: { name: "music_generate", displaySummary: "Generate music", description: - 'Create audio/music for song, jingle, beat, loop, soundtrack, anthem, instrumental requests. If user asks make/generate/create song/music, call music_generate; do not just write lyrics unless lyrics/text only. Prompt gets style/genre/mood/tempo/instruments/purpose. lyrics only exact sung words. Session chats: background task; do not call again for same request; wait completion, send attachments via message tool. "status" checks active task.', + 'Create audio/music for song, jingle, beat, loop, soundtrack, anthem, instrumental requests. If user asks make/generate/create song/music, call music_generate; do not just write lyrics unless lyrics/text only. Prompt gets style/genre/mood/tempo/instruments/purpose. lyrics only exact sung words. Session chats: background task; do not call again for same request; wait completion, then report through the current visible-reply contract with generated media attached or MEDIA: paths. "status" checks active task.', parameters: MusicGenerateToolSchema, execute: async (_toolCallId, rawArgs) => { const args = rawArgs as Record; diff --git a/src/agents/tools/video-generate-background.test.ts b/src/agents/tools/video-generate-background.test.ts index 8284620cdf1c..84ac5f4a1a7a 100644 --- a/src/agents/tools/video-generate-background.test.ts +++ b/src/agents/tools/video-generate-background.test.ts @@ -207,7 +207,7 @@ describe("video generate background helpers", () => { announceDeliveryMocks.deliverSubagentAnnouncement.mockResolvedValue({ delivered: false, path: "direct", - error: "completion agent did not deliver through the message tool", + error: "completion agent did not deliver generated media", }); await wakeVideoGenerationTaskCompletion({ diff --git a/src/agents/tools/video-generate-tool.ts b/src/agents/tools/video-generate-tool.ts index 8f7650be6cc1..7f0354c2aeb1 100644 --- a/src/agents/tools/video-generate-tool.ts +++ b/src/agents/tools/video-generate-tool.ts @@ -952,7 +952,7 @@ export function createVideoGenerateTool(options?: { name: "video_generate", displaySummary: "Generate videos", description: - 'Create videos. Session chats: background task; do not call video_generate again for same request; wait completion, then send attachments via message tool. "status" checks active task. Duration may round to provider-supported value.', + 'Create videos. Session chats: background task; do not call video_generate again for same request; wait completion, then report through the current visible-reply contract with generated media attached or MEDIA: paths. "status" checks active task. Duration may round to provider-supported value.', parameters: createVideoGenerateToolSchema({ includeAudioReferences }), execute: async (_toolCallId, rawArgs) => { const args = rawArgs as Record;