Compare commits

...

7 Commits

Author SHA1 Message Date
scoootscooob
f0635aeca7 plugins: match installed plugin records by root 2026-05-15 20:30:35 -07:00
scoootscooob
2171e714f5 plugins: keep model-profile runtime llm compat 2026-05-15 20:28:33 -07:00
scoootscooob
9dc496986a docs: align runtime llm profile override docs 2026-05-15 20:28:33 -07:00
Eva (agent)
80c54f8288 Normalize same-model profile selection in runtime LLM 2026-05-15 20:28:33 -07:00
Eva (agent)
074621dfd9 Test auth-profile override gates across runtime LLM lanes 2026-05-15 20:28:33 -07:00
Eva (agent)
9588d72156 Gate structured LLM auth profiles consistently 2026-05-15 20:28:33 -07:00
Eva (agent)
9c0e21f239 plugin-sdk: add host-owned structured runtime llm 2026-05-15 20:28:32 -07:00
21 changed files with 1163 additions and 76 deletions

View File

@@ -414,6 +414,7 @@ Docs: https://docs.openclaw.ai
- Plugin SDK: remove the owner-specific `provider-auth-login` public subpath after moving Chutes, GitHub Copilot, and OpenAI Codex auth flows back to provider-owned modules.
- Plugin SDK: remove provider-specific model, stream, and xAI compatibility helpers from public exports after moving bundled callers to provider-owned modules.
- Plugin SDK: expose runtime-supplied active model metadata to native plugin tool factories for diagnostics and plugin-owned policy decisions. Fixes #77857. Thanks @jamiezigelbaum.
- Plugin SDK/runtime: add `api.runtime.llm.completeStructured(...)` for host-owned structured plugin inference with optional image inputs, JSON/schema validation, auth-profile selection, and the same model/agent override trust gates as `api.runtime.llm.complete`.
- QA/Mantis: add Telegram live PR evidence automation with Convex-leased credentials, Crabbox transcript capture, motion GIF previews, and inline PR comments.
- QA/Mantis: add a Telegram desktop scenario builder that leases Crabbox, installs native Telegram Desktop, configures an OpenClaw Telegram gateway with leased bot credentials, and records VNC screenshot/video artifacts.
- Discord/voice: add realtime voice diagnostics for speaker turns, playback resets, barge-in detection, and audio cutoff analysis.

View File

@@ -1,2 +1,2 @@
eed14a427f34d1531d63e5f1065ef2325b46f77e5a16ce809e5e845cd6700769 plugin-sdk-api-baseline.json
d8d090e4858f8d619b2151d69b8dc992132bc8930e04b990e4a69a433fb19d41 plugin-sdk-api-baseline.jsonl
55aab1bccac852ddd34e529fedfc1e51be0bd51b49fa75cd758fe11fa9d63255 plugin-sdk-api-baseline.json
71297a69c418a5090ac4f5007da677ef35e8d9ac26e8cfb3a8084c2f898aedb9 plugin-sdk-api-baseline.jsonl

View File

@@ -228,9 +228,10 @@ See [MCP](/cli/mcp#openclaw-as-an-mcp-client-registry) and
- `plugins.entries.<id>.hooks.allowConversationAccess`: when `true`, trusted non-bundled plugins may read raw conversation content from typed hooks such as `llm_input`, `llm_output`, `before_model_resolve`, `before_agent_reply`, `before_agent_run`, `before_agent_finalize`, and `agent_end`.
- `plugins.entries.<id>.subagent.allowModelOverride`: explicitly trust this plugin to request per-run `provider` and `model` overrides for background subagent runs.
- `plugins.entries.<id>.subagent.allowedModels`: optional allowlist of canonical `provider/model` targets for trusted subagent overrides. Use `"*"` only when you intentionally want to allow any model.
- `plugins.entries.<id>.llm.allowModelOverride`: explicitly trust this plugin to request model overrides for `api.runtime.llm.complete`.
- `plugins.entries.<id>.llm.allowedModels`: optional allowlist of canonical `provider/model` targets for trusted plugin LLM completion overrides. Use `"*"` only when you intentionally want to allow any model.
- `plugins.entries.<id>.llm.allowAgentIdOverride`: explicitly trust this plugin to run `api.runtime.llm.complete` against a non-default agent id.
- `plugins.entries.<id>.llm.allowModelOverride`: explicitly trust this plugin to request model overrides for `api.runtime.llm.complete` and `api.runtime.llm.completeStructured`.
- `plugins.entries.<id>.llm.allowedModels`: optional allowlist of canonical `provider/model` targets for trusted plugin runtime LLM overrides. Use `"*"` only when you intentionally want to allow any model.
- `plugins.entries.<id>.llm.allowAgentIdOverride`: explicitly trust this plugin to run `api.runtime.llm.complete` / `completeStructured` against a non-default agent id.
- `plugins.entries.<id>.llm.allowProfileOverride`: explicitly trust this plugin to select a non-default auth profile for runtime LLM completions through either a `profile` field or a `provider/model@profile` model ref. For compatibility, a `provider/model@profile` ref is also accepted when the plugin already has model-override trust for that selected model.
- `plugins.entries.<id>.config`: plugin-defined config object (validated by native OpenClaw plugin schema when available).
- Channel plugin account/runtime settings live under `channels.<id>` and should be described by the owning plugin's manifest `channelConfigs` metadata, not by a central OpenClaw option registry.

View File

@@ -195,8 +195,49 @@ two-party event loops that do not go through the shared channel-turn kernel.
result includes provider/model/agent attribution plus normalized token,
cache, and estimated cost usage when available.
For bounded structured work, use `api.runtime.llm.completeStructured(...)`.
```typescript
const structured = await api.runtime.llm.completeStructured({
instructions: "Extract vendor, total, and searchable tags.",
input: [
{
type: "image",
buffer: receiptBuffer,
mimeType: "image/png",
fileName: "receipt.png",
},
{ type: "text", text: "Prefer the printed total over handwritten notes." },
],
schemaName: "receipt.evidence",
jsonSchema: {
type: "object",
properties: {
vendor: { type: "string" },
total: { type: "number" },
tags: { type: "array", items: { type: "string" } },
},
required: ["vendor", "total"],
},
purpose: "receipts.extract",
profile: "openai-codex:work",
maxTokens: 512,
timeoutMs: 30000,
});
```
`completeStructured(...)` keeps auth, provider routing, and runtime execution
host-owned. Plugins provide instructions, optional text/image inputs, and an
optional JSON Schema; the host returns the raw text plus parsed JSON only
when `jsonMode: true` or `jsonSchema` is provided.
For provider/model-explicit media capability routing, use
`api.runtime.mediaUnderstanding.extractStructuredWithModel(...)` instead.
`completeStructured(...)` is the generic agent-bound runtime lane; the
media-understanding helper is the narrower provider-owned image/media lane.
<Warning>
Model overrides require operator opt-in via `plugins.entries.<id>.llm.allowModelOverride: true` in config. Use `plugins.entries.<id>.llm.allowedModels` to restrict trusted plugins to specific canonical `provider/model` targets. Cross-agent completions require `plugins.entries.<id>.llm.allowAgentIdOverride: true`.
Model overrides require operator opt-in via `plugins.entries.<id>.llm.allowModelOverride: true` in config. Use `plugins.entries.<id>.llm.allowedModels` to restrict trusted plugins to specific canonical `provider/model` targets. Cross-agent completions require `plugins.entries.<id>.llm.allowAgentIdOverride: true`. New code should use `plugins.entries.<id>.llm.allowProfileOverride: true` for auth-profile selection through either the `profile` field or a `provider/model@profile` model ref. For compatibility, a model ref with a profile suffix also remains valid when the plugin is already trusted to override that selected model. The same host-owned trust gate applies across runtime LLM surfaces.
</Warning>
</Accordion>

View File

@@ -80,6 +80,22 @@ export function resolveContextEngineCapabilities(
},
}).complete(request);
},
completeStructured: async (request) => {
const { createRuntimeLlm } = await import("../../plugins/runtime/runtime-llm.runtime.js");
return await createRuntimeLlm({
getConfig: () => params.config,
authority: {
caller: { kind: "context-engine", id: params.purpose },
requiresBoundAgent: true,
...(sessionKey ? { sessionKey } : {}),
...(agentId ? { agentId } : {}),
...(contextEnginePluginId ? { pluginIdForPolicy: contextEnginePluginId } : {}),
allowAgentIdOverride: false,
allowModelOverride: false,
allowComplete: true,
},
}).completeStructured(request);
},
},
};
}

View File

@@ -380,6 +380,7 @@ const TARGET_KEYS = [
"plugins.entries.*.llm.allowModelOverride",
"plugins.entries.*.llm.allowedModels",
"plugins.entries.*.llm.allowAgentIdOverride",
"plugins.entries.*.llm.allowProfileOverride",
"plugins.entries.*.apiKey",
"plugins.entries.*.env",
"plugins.entries.*.config",

View File

@@ -1340,13 +1340,15 @@ export const FIELD_HELP: Record<string, string> = {
"plugins.entries.*.subagent.allowedModels":
'Allowed override targets for trusted plugin subagent runs as canonical "provider/model" refs. Use "*" only when you intentionally allow any model.',
"plugins.entries.*.llm":
"Per-plugin api.runtime.llm.complete controls for model and agent override trust. Keep this unset unless a plugin must explicitly steer host-owned completion calls.",
"Per-plugin api.runtime.llm.complete and api.runtime.llm.completeStructured controls for model, auth-profile, and agent override trust. Keep this unset unless a plugin must explicitly steer host-owned completion calls.",
"plugins.entries.*.llm.allowModelOverride":
"Explicitly allows this plugin to request model overrides in api.runtime.llm.complete. Keep false unless the plugin is trusted to steer model selection.",
"Explicitly allows this plugin to request model overrides in api.runtime.llm.complete or api.runtime.llm.completeStructured. Keep false unless the plugin is trusted to steer model selection.",
"plugins.entries.*.llm.allowedModels":
'Allowed override targets for trusted plugin LLM completions as canonical "provider/model" refs. Use "*" only when you intentionally allow any model.',
"plugins.entries.*.llm.allowAgentIdOverride":
"Explicitly allows this plugin to request api.runtime.llm.complete against a non-default agent id. Keep false unless the plugin is trusted for cross-agent model access.",
"Explicitly allows this plugin to request runtime.llm completions against a non-default agent id. Keep false unless the plugin is trusted for cross-agent model access.",
"plugins.entries.*.llm.allowProfileOverride":
"Explicitly allows this plugin to request a non-default auth profile in runtime.llm completions. Keep false unless the plugin is trusted to steer which stored provider credentials/account the host should use.",
"plugins.entries.*.apiKey":
"Optional API key field consumed by plugins that accept direct key configuration in entry settings. Use secret/env substitution and avoid committing real credentials into config files.",
"plugins.entries.*.env":

View File

@@ -1008,6 +1008,7 @@ export const FIELD_LABELS: Record<string, string> = {
"plugins.entries.*.llm.allowModelOverride": "Allow Plugin LLM Model Override",
"plugins.entries.*.llm.allowedModels": "Plugin LLM Allowed Models",
"plugins.entries.*.llm.allowAgentIdOverride": "Allow Plugin LLM Agent Override",
"plugins.entries.*.llm.allowProfileOverride": "Allow Plugin LLM Auth Profile Override",
"plugins.entries.*.apiKey": "Plugin API Key", // pragma: allowlist secret
"plugins.entries.*.env": "Plugin Environment Variables",
"plugins.entries.*.config": "Plugin Config",

View File

@@ -25,15 +25,17 @@ export type PluginEntryConfig = {
allowedModels?: string[];
};
llm?: {
/** Explicitly allow this plugin to request a model override for api.runtime.llm.complete. */
/** Explicitly allow this plugin to request a model override for api.runtime.llm.complete / completeStructured. */
allowModelOverride?: boolean;
/**
* Allowed completion model override targets as canonical provider/model refs.
* Use "*" to explicitly allow any model for this plugin.
*/
allowedModels?: string[];
/** Explicitly allow this plugin to run completions against a non-default agent id. */
/** Explicitly allow this plugin to run runtime.llm calls against a non-default agent id. */
allowAgentIdOverride?: boolean;
/** Explicitly allow this plugin to request a specific auth profile for runtime.llm calls. */
allowProfileOverride?: boolean;
};
config?: Record<string, unknown>;
};

View File

@@ -214,6 +214,7 @@ const PluginEntrySchema = z
allowModelOverride: z.boolean().optional(),
allowedModels: z.array(z.string()).optional(),
allowAgentIdOverride: z.boolean().optional(),
allowProfileOverride: z.boolean().optional(),
})
.strict()
.optional(),

View File

@@ -178,6 +178,9 @@ export type ContextEngineRuntimeContext = Record<string, unknown> & {
complete: (
params: import("../plugins/runtime/types-core.js").LlmCompleteParams,
) => Promise<import("../plugins/runtime/types-core.js").LlmCompleteResult>;
completeStructured: (
params: import("../plugins/runtime/types-core.js").LlmCompleteStructuredParams,
) => Promise<import("../plugins/runtime/types-core.js").LlmCompleteStructuredResult>;
};
};

View File

@@ -752,6 +752,7 @@ export function createPluginRuntimeMock(overrides: DeepPartial<PluginRuntime> =
},
llm: {
complete: vi.fn(),
completeStructured: vi.fn(),
},
nodes: {
list: vi.fn(async () => ({ nodes: [] })),

View File

@@ -35,6 +35,7 @@ export type NormalizedPluginsConfig = {
allowedModels?: string[];
hasAllowedModelsConfig?: boolean;
allowAgentIdOverride?: boolean;
allowProfileOverride?: boolean;
};
config?: unknown;
}
@@ -185,6 +186,8 @@ function normalizePluginEntries(
: undefined,
allowAgentIdOverride: (llmRaw as { allowAgentIdOverride?: unknown })
.allowAgentIdOverride,
allowProfileOverride: (llmRaw as { allowProfileOverride?: unknown })
.allowProfileOverride,
}
: undefined;
const normalizedLlm =
@@ -192,7 +195,8 @@ function normalizePluginEntries(
(typeof llm.allowModelOverride === "boolean" ||
llm.hasAllowedModelsConfig ||
(Array.isArray(llm.allowedModels) && llm.allowedModels.length > 0) ||
typeof llm.allowAgentIdOverride === "boolean")
typeof llm.allowAgentIdOverride === "boolean" ||
typeof llm.allowProfileOverride === "boolean")
? {
...(typeof llm.allowModelOverride === "boolean"
? { allowModelOverride: llm.allowModelOverride }
@@ -204,6 +208,9 @@ function normalizePluginEntries(
...(typeof llm.allowAgentIdOverride === "boolean"
? { allowAgentIdOverride: llm.allowAgentIdOverride }
: {}),
...(typeof llm.allowProfileOverride === "boolean"
? { allowProfileOverride: llm.allowProfileOverride }
: {}),
}
: undefined;
normalized[normalizedKey] = {

View File

@@ -154,6 +154,7 @@ describe("normalizePluginsConfig", () => {
allowModelOverride: true,
allowedModels: [" openai/gpt-5.4 ", "", "anthropic/claude-sonnet-4-6"],
allowAgentIdOverride: false,
allowProfileOverride: true,
},
})?.llm,
).toEqual({
@@ -161,6 +162,7 @@ describe("normalizePluginsConfig", () => {
hasAllowedModelsConfig: true,
allowedModels: ["openai/gpt-5.4", "anthropic/claude-sonnet-4-6"],
allowAgentIdOverride: false,
allowProfileOverride: true,
});
});

View File

@@ -747,6 +747,8 @@ function matchesInstalledPluginRecord(params: {
if (!record) {
return false;
}
const resolvedCandidateRoot = resolveUserPath(params.candidate.rootDir, params.env);
const candidateRoot = safeRealpathSync(resolvedCandidateRoot) ?? resolvedCandidateRoot;
const resolvedCandidateSource = resolveUserPath(params.candidate.source, params.env);
const candidateSource = safeRealpathSync(resolvedCandidateSource) ?? resolvedCandidateSource;
const trackedPaths = [record.installPath, record.sourcePath]
@@ -759,7 +761,12 @@ function matchesInstalledPluginRecord(params: {
return false;
}
return trackedPaths.some((trackedPath) => {
return candidateSource === trackedPath || isPathInside(trackedPath, candidateSource);
return (
candidateRoot === trackedPath ||
candidateSource === trackedPath ||
isPathInside(trackedPath, candidateRoot) ||
isPathInside(trackedPath, candidateSource)
);
});
}

View File

@@ -2425,6 +2425,8 @@ export function createPluginRegistry(registryParams: PluginRegistryParams) {
return {
complete: (params) =>
withPluginRuntimePluginIdScope(pluginId, () => llm.complete(params)),
completeStructured: (params) =>
withPluginRuntimePluginIdScope(pluginId, () => llm.completeStructured(params)),
} satisfies PluginRuntime["llm"];
}
if (prop !== "subagent") {

View File

@@ -327,6 +327,16 @@ describe("plugin runtime command execution", () => {
]);
},
},
{
name: "exposes runtime.llm completion helpers",
assert: (runtime: ReturnType<typeof createPluginRuntime>) => {
expect(runtime.llm).toMatchObject({
complete: expect.any(Function),
completeStructured: expect.any(Function),
});
expectFunctionKeys(runtime.llm, ["complete", "completeStructured"]);
},
},
] as const)("$name", ({ assert }) => {
expectRuntimeShape(assert);
});

View File

@@ -114,6 +114,10 @@ function createRuntimeLlmFacade(): PluginRuntime["llm"] {
const llm = await loadLlm();
return llm.complete(params);
},
completeStructured: async (params) => {
const llm = await loadLlm();
return llm.completeStructured(params);
},
};
}

View File

@@ -9,6 +9,7 @@ const hoisted = vi.hoisted(() => ({
prepareSimpleCompletionModelForAgent: vi.fn(),
completeWithPreparedSimpleCompletionModel: vi.fn(),
resolveSimpleCompletionSelectionForAgent: vi.fn(),
resolveAutoImageModel: vi.fn(),
}));
vi.mock("../../agents/simple-completion-runtime.js", () => ({
@@ -17,6 +18,10 @@ vi.mock("../../agents/simple-completion-runtime.js", () => ({
resolveSimpleCompletionSelectionForAgent: hoisted.resolveSimpleCompletionSelectionForAgent,
}));
vi.mock("../../media-understanding/runner.js", () => ({
resolveAutoImageModel: hoisted.resolveAutoImageModel,
}));
const cfg = {
agents: {
defaults: {
@@ -25,7 +30,7 @@ const cfg = {
},
} satisfies OpenClawConfig;
function createPreparedModel(modelId = "gpt-5.5") {
function createPreparedModel(modelId = "gpt-5.5", input: string[] = ["text"]) {
return {
selection: {
provider: "openai",
@@ -37,7 +42,7 @@ function createPreparedModel(modelId = "gpt-5.5") {
id: modelId,
name: modelId,
api: "openai",
input: ["text"],
input,
reasoning: false,
contextWindow: 128_000,
maxTokens: 4096,
@@ -137,6 +142,7 @@ function primeCompletionMocks() {
cost: { total: 0.0042 },
},
});
hoisted.resolveAutoImageModel.mockResolvedValue(null);
}
describe("runtime.llm.complete", () => {
@@ -144,6 +150,7 @@ describe("runtime.llm.complete", () => {
hoisted.prepareSimpleCompletionModelForAgent.mockReset();
hoisted.completeWithPreparedSimpleCompletionModel.mockReset();
hoisted.resolveSimpleCompletionSelectionForAgent.mockReset();
hoisted.resolveAutoImageModel.mockReset();
primeCompletionMocks();
});
@@ -172,6 +179,46 @@ describe("runtime.llm.complete", () => {
});
});
it("binds context-engine structured completions to the active session agent", async () => {
hoisted.completeWithPreparedSimpleCompletionModel.mockResolvedValueOnce({
content: [{ type: "text", text: '{"summary":"ok"}' }],
usage: {
input: 4,
output: 3,
cacheRead: 0,
cacheWrite: 0,
total: 7,
cost: { total: 0.001 },
},
});
const runtimeContext = resolveContextEngineCapabilities({
config: cfg,
sessionKey: "agent:ada:session:abc",
purpose: "context-engine.after-turn",
});
const result = await runtimeContext.llm!.completeStructured({
instructions: "Extract a short summary.",
input: [{ type: "text", text: "Customer said the rollout worked." }],
jsonMode: true,
purpose: "memory-maintenance",
});
expect(hoisted.prepareSimpleCompletionModelForAgent).toHaveBeenCalledWith(
expect.objectContaining({
cfg,
agentId: "ada",
}),
);
expect(result.agentId).toBe("ada");
expect(result.audit).toMatchObject({
caller: { kind: "context-engine", id: "context-engine.after-turn" },
purpose: "memory-maintenance",
sessionKey: "agent:ada:session:abc",
});
expect(result.parsed).toEqual({ summary: "ok" });
});
it("uses trusted context-engine attribution inside plugin runtime scope", async () => {
const runtimeContext = resolveContextEngineCapabilities({
config: cfg,
@@ -412,10 +459,10 @@ describe("runtime.llm.complete", () => {
await expect(
llm.complete({
model: "openai/gpt-5.5",
model: "openai/gpt-5.6",
messages: [{ role: "user", content: "Ping" }],
}),
).rejects.toThrow('model override "openai/gpt-5.5" is not allowlisted');
).rejects.toThrow('model override "openai/gpt-5.6" is not allowlisted');
});
it("uses runtime-scoped config and the host preparation/dispatch path", async () => {
@@ -609,11 +656,534 @@ describe("runtime.llm.complete", () => {
await expect(
withPluginRuntimePluginIdScope("trusted-plugin", () =>
llm.complete({
model: "openai/gpt-5.5",
model: "openai/gpt-5.6",
messages: [{ role: "user", content: "Ping" }],
}),
),
).rejects.toThrow('model override "openai/gpt-5.5" is not allowlisted');
).rejects.toThrow('model override "openai/gpt-5.6" is not allowlisted');
});
it("rejects auth-profile suffixes on complete without profile or model override trust", async () => {
const llm = createRuntimeLlm({
getConfig: () => cfg,
authority: {
caller: { kind: "host", id: "runtime-test" },
allowComplete: true,
},
});
await expect(
llm.complete({
model: "openai/gpt-5.5@openai-codex:work",
messages: [{ role: "user", content: "Ping" }],
}),
).rejects.toThrow("cannot override the target model");
});
it("preserves auth-profile suffix compatibility when the model override is trusted", async () => {
const llm = createRuntimeLlm({
getConfig: () => cfg,
authority: {
caller: { kind: "host", id: "runtime-test" },
allowComplete: true,
allowModelOverride: true,
},
});
await llm.complete({
model: "openai/gpt-5.4@openai-codex:work",
messages: [{ role: "user", content: "Ping" }],
});
expect(hoisted.prepareSimpleCompletionModelForAgent).toHaveBeenCalledWith(
expect.objectContaining({
modelRef: "openai/gpt-5.4",
preferredProfile: "openai-codex:work",
}),
);
});
it("treats same-as-default auth-profile suffixes on complete as profile-only overrides", async () => {
const llm = createRuntimeLlm({
getConfig: () => cfg,
authority: {
caller: { kind: "host", id: "runtime-test" },
allowComplete: true,
allowProfileOverride: true,
},
});
await llm.complete({
model: "openai/gpt-5.5@openai-codex:work",
messages: [{ role: "user", content: "Ping" }],
});
expect(hoisted.prepareSimpleCompletionModelForAgent).toHaveBeenCalledWith(
expect.objectContaining({
modelRef: undefined,
preferredProfile: "openai-codex:work",
}),
);
});
it("returns parsed structured JSON for text-only completion and validates the schema", async () => {
const llm = createRuntimeLlm({
getConfig: () => cfg,
authority: {
caller: { kind: "host", id: "runtime-test" },
allowComplete: true,
},
});
hoisted.completeWithPreparedSimpleCompletionModel.mockResolvedValueOnce({
content: [{ type: "text", text: '```json\n{"summary":"ok"}\n```' }],
usage: {
input: 4,
output: 3,
cacheRead: 0,
cacheWrite: 0,
total: 7,
cost: { total: 0.001 },
},
});
const result = await llm.completeStructured({
instructions: "Extract a short summary.",
input: [{ type: "text", text: "Customer said the rollout worked." }],
schemaName: "support.summary",
jsonSchema: {
type: "object",
properties: { summary: { type: "string" } },
required: ["summary"],
},
systemPrompt: "Be precise.",
purpose: "structured.summary",
});
expect(result).toMatchObject({
text: '```json\n{"summary":"ok"}\n```',
parsed: { summary: "ok" },
contentType: "json",
provider: "openai",
model: "gpt-5.5",
});
expect(hoisted.completeWithPreparedSimpleCompletionModel).toHaveBeenCalledWith(
expect.objectContaining({
context: expect.objectContaining({
systemPrompt: "Be precise.",
messages: [
expect.objectContaining({
role: "user",
content: [
expect.objectContaining({
type: "text",
text: expect.stringContaining("Schema name: support.summary"),
}),
expect.objectContaining({
type: "text",
text: "Customer said the rollout worked.",
}),
],
}),
],
}),
}),
);
});
it("supports image-plus-text structured completion with the host-owned llm runtime", async () => {
const llm = createRuntimeLlm({
getConfig: () => cfg,
authority: {
caller: { kind: "host", id: "runtime-test" },
allowComplete: true,
},
});
hoisted.prepareSimpleCompletionModelForAgent.mockResolvedValueOnce(
createPreparedModel("gpt-5.5", ["text", "image"]),
);
hoisted.completeWithPreparedSimpleCompletionModel.mockResolvedValueOnce({
content: [{ type: "text", text: '{"caption":"receipt","tags":["finance"]}' }],
usage: {
input: 10,
output: 6,
cacheRead: 0,
cacheWrite: 0,
total: 16,
cost: { total: 0.002 },
},
});
const result = await llm.completeStructured({
instructions: "Extract searchable receipt metadata.",
input: [
{
type: "image",
buffer: Buffer.from("hello"),
mimeType: "image/png",
fileName: "receipt.png",
},
{ type: "text", text: "Prefer the printed total over handwritten notes." },
],
jsonMode: true,
});
expect(result.parsed).toEqual({ caption: "receipt", tags: ["finance"] });
expect(hoisted.completeWithPreparedSimpleCompletionModel).toHaveBeenCalledWith(
expect.objectContaining({
context: expect.objectContaining({
messages: [
expect.objectContaining({
role: "user",
content: expect.arrayContaining([
expect.objectContaining({
type: "image",
mimeType: "image/png",
data: Buffer.from("hello").toString("base64"),
}),
expect.objectContaining({
type: "text",
text: "Prefer the printed total over handwritten notes.",
}),
]),
}),
],
}),
}),
);
});
it("rejects structured auth-profile overrides without explicit trust", async () => {
const llm = createRuntimeLlm({
getConfig: () => cfg,
authority: {
caller: { kind: "host", id: "runtime-test" },
allowComplete: true,
},
});
await expect(
llm.completeStructured({
instructions: "Extract summary.",
input: [{ type: "text", text: "Hello" }],
profile: "openai-codex:work",
jsonMode: false,
}),
).rejects.toThrow("cannot override the auth profile");
});
it("forwards preferred auth profiles into structured completion prep when trusted", async () => {
const llm = createRuntimeLlm({
getConfig: () => cfg,
authority: {
caller: { kind: "host", id: "runtime-test" },
allowComplete: true,
allowProfileOverride: true,
},
});
await llm.completeStructured({
instructions: "Extract summary.",
input: [{ type: "text", text: "Hello" }],
profile: "openai-codex:work",
jsonMode: false,
});
expect(hoisted.prepareSimpleCompletionModelForAgent).toHaveBeenCalledWith(
expect.objectContaining({
preferredProfile: "openai-codex:work",
}),
);
});
it("rejects auth-profile suffixes in structured model refs without profile or model override trust", async () => {
const llm = createRuntimeLlm({
getConfig: () => cfg,
authority: {
caller: { kind: "host", id: "runtime-test" },
allowComplete: true,
},
});
await expect(
llm.completeStructured({
model: "openai/gpt-5.5@openai-codex:work",
instructions: "Extract summary.",
input: [{ type: "text", text: "Hello" }],
jsonMode: false,
}),
).rejects.toThrow("cannot override the target model");
});
it("preserves structured auth-profile suffix compatibility when the model override is trusted", async () => {
const llm = createRuntimeLlm({
getConfig: () => cfg,
authority: {
caller: { kind: "host", id: "runtime-test" },
allowComplete: true,
allowModelOverride: true,
},
});
await llm.completeStructured({
model: "openai/gpt-5.4@openai-codex:work",
instructions: "Extract summary.",
input: [{ type: "text", text: "Hello" }],
jsonMode: false,
});
expect(hoisted.prepareSimpleCompletionModelForAgent).toHaveBeenCalledWith(
expect.objectContaining({
modelRef: "openai/gpt-5.4",
preferredProfile: "openai-codex:work",
}),
);
});
it("treats same-as-default auth-profile suffixes in structured model refs as profile-only overrides when trusted", async () => {
const llm = createRuntimeLlm({
getConfig: () => cfg,
authority: {
caller: { kind: "host", id: "runtime-test" },
allowComplete: true,
allowProfileOverride: true,
},
});
await llm.completeStructured({
model: "openai/gpt-5.5@openai-codex:work",
instructions: "Extract summary.",
input: [{ type: "text", text: "Hello" }],
jsonMode: false,
});
expect(hoisted.prepareSimpleCompletionModelForAgent).toHaveBeenCalledWith(
expect.objectContaining({
modelRef: undefined,
preferredProfile: "openai-codex:work",
}),
);
});
it("allows same-as-default auth-profile suffixes through plugin llm policy", async () => {
const llm = createRuntimeLlm({
getConfig: () => ({
...cfg,
plugins: {
entries: {
"trusted-plugin": {
llm: {
allowProfileOverride: true,
},
},
},
},
}),
authority: {
allowComplete: true,
},
});
await withPluginRuntimePluginIdScope("trusted-plugin", () =>
llm.completeStructured({
model: "openai/gpt-5.5@openai-codex:work",
instructions: "Extract summary.",
input: [{ type: "text", text: "Hello" }],
jsonMode: false,
}),
);
expect(hoisted.prepareSimpleCompletionModelForAgent).toHaveBeenCalledWith(
expect.objectContaining({
modelRef: undefined,
preferredProfile: "openai-codex:work",
}),
);
});
it("requires both model and profile trust when the structured model ref changes the target model", async () => {
const llm = createRuntimeLlm({
getConfig: () => cfg,
authority: {
caller: { kind: "host", id: "runtime-test" },
allowComplete: true,
allowModelOverride: true,
allowedModels: ["openai/gpt-5.4"],
allowProfileOverride: true,
},
});
await llm.completeStructured({
model: "openai/gpt-5.4@openai-codex:work",
instructions: "Extract summary.",
input: [{ type: "text", text: "Hello" }],
jsonMode: false,
});
expect(hoisted.prepareSimpleCompletionModelForAgent).toHaveBeenCalledWith(
expect.objectContaining({
modelRef: "openai/gpt-5.4",
preferredProfile: "openai-codex:work",
}),
);
});
it("rejects conflicting explicit and embedded structured auth-profile overrides", async () => {
const llm = createRuntimeLlm({
getConfig: () => cfg,
authority: {
caller: { kind: "host", id: "runtime-test" },
allowComplete: true,
allowModelOverride: true,
allowProfileOverride: true,
},
});
await expect(
llm.completeStructured({
model: "openai/gpt-5.5@openai-codex:work",
profile: "openai-codex:other",
instructions: "Extract summary.",
input: [{ type: "text", text: "Hello" }],
jsonMode: false,
}),
).rejects.toThrow("conflicting auth profiles");
});
it("falls back to the configured image model for structured image input", async () => {
const llm = createRuntimeLlm({
getConfig: () => cfg,
authority: {
caller: { kind: "host", id: "runtime-test" },
allowComplete: true,
},
});
hoisted.resolveAutoImageModel.mockResolvedValueOnce({
provider: "google",
model: "gemini-3.1-flash-image-preview",
});
hoisted.prepareSimpleCompletionModelForAgent.mockResolvedValueOnce(
createPreparedModel("gemini-3.1-flash-image-preview", ["text", "image"]),
);
hoisted.completeWithPreparedSimpleCompletionModel.mockResolvedValueOnce({
content: [{ type: "text", text: '{"summary":"ok"}' }],
usage: {
input: 2,
output: 2,
cacheRead: 0,
cacheWrite: 0,
total: 4,
cost: { total: 0.001 },
},
});
const result = await llm.completeStructured({
instructions: "Extract receipt fields.",
input: [{ type: "image", buffer: Buffer.from("hello"), mimeType: "image/png" }],
jsonMode: true,
});
expect(result.parsed).toEqual({ summary: "ok" });
expect(hoisted.resolveAutoImageModel).toHaveBeenCalledWith(expect.objectContaining({ cfg }));
expect(hoisted.prepareSimpleCompletionModelForAgent).toHaveBeenCalledWith(
expect.objectContaining({
modelRef: "google/gemini-3.1-flash-image-preview",
}),
);
});
it("rejects structured image input when neither the active nor fallback image model supports images", async () => {
const llm = createRuntimeLlm({
getConfig: () => cfg,
authority: {
caller: { kind: "host", id: "runtime-test" },
allowComplete: true,
},
});
hoisted.resolveAutoImageModel.mockResolvedValueOnce(null);
await expect(
llm.completeStructured({
instructions: "Extract receipt fields.",
input: [{ type: "image", buffer: Buffer.from("hello"), mimeType: "image/png" }],
}),
).rejects.toThrow("does not support image input");
});
it("returns controlled errors for invalid structured JSON", async () => {
const llm = createRuntimeLlm({
getConfig: () => cfg,
authority: {
caller: { kind: "host", id: "runtime-test" },
allowComplete: true,
},
});
hoisted.completeWithPreparedSimpleCompletionModel.mockResolvedValueOnce({
content: [{ type: "text", text: "not json" }],
usage: {
input: 2,
output: 2,
cacheRead: 0,
cacheWrite: 0,
total: 4,
cost: { total: 0.001 },
},
});
await expect(
llm.completeStructured({
instructions: "Extract summary.",
input: [{ type: "text", text: "Hello" }],
jsonMode: true,
}),
).rejects.toThrow("returned invalid JSON");
});
it("aborts structured completions when timeoutMs elapses", async () => {
const llm = createRuntimeLlm({
getConfig: () => cfg,
authority: {
caller: { kind: "host", id: "runtime-test" },
allowComplete: true,
},
});
hoisted.completeWithPreparedSimpleCompletionModel.mockImplementationOnce(
async ({ options }: { options?: { signal?: AbortSignal } }) =>
await new Promise((_, reject) => {
options?.signal?.addEventListener(
"abort",
() => reject(options.signal?.reason ?? new Error("aborted")),
{ once: true },
);
}),
);
await expect(
llm.completeStructured({
instructions: "Extract summary.",
input: [{ type: "text", text: "Hello" }],
jsonMode: false,
timeoutMs: 1,
}),
).rejects.toThrow("timed out");
});
it("applies the same model-override trust rules to structured completion", async () => {
const llm = createRuntimeLlm({
getConfig: () => cfg,
authority: {
allowComplete: true,
},
});
await expect(
withPluginRuntimePluginIdScope("plain-plugin", () =>
llm.completeStructured({
model: "openai/gpt-5.4",
instructions: "Extract summary.",
input: [{ type: "text", text: "Ping" }],
}),
),
).rejects.toThrow("cannot override the target model");
});
it("denies completions when runtime authority disables the capability", async () => {

View File

@@ -1,9 +1,14 @@
import type { Api, Message } from "@earendil-works/pi-ai";
import { splitTrailingAuthProfile } from "../../agents/model-ref-profile.js";
import { normalizeModelRef } from "../../agents/model-selection.js";
import type { NormalizedUsage, UsageLike } from "../../agents/usage.js";
import { normalizeUsage } from "../../agents/usage.js";
import type { OpenClawConfig } from "../../config/types.openclaw.js";
import { getChildLogger } from "../../logging.js";
import {
type JsonSchemaObject,
validateJsonSchemaValue,
} from "../../plugin-sdk/json-schema-runtime.js";
import { normalizeAgentId } from "../../routing/session-key.js";
import { normalizeOptionalString } from "../../shared/string-coerce.js";
import { estimateUsageCost, resolveModelCostConfig } from "../../utils/usage-format.js";
@@ -13,6 +18,9 @@ import type {
LlmCompleteCaller,
LlmCompleteParams,
LlmCompleteResult,
LlmCompleteStructuredInput,
LlmCompleteStructuredParams,
LlmCompleteStructuredResult,
LlmCompleteUsage,
PluginRuntimeCore,
RuntimeLogger,
@@ -27,6 +35,7 @@ export type RuntimeLlmAuthority = {
requiresBoundAgent?: boolean;
allowAgentIdOverride?: boolean;
allowModelOverride?: boolean;
allowProfileOverride?: boolean;
allowedModels?: readonly string[];
allowComplete?: boolean;
denyReason?: string;
@@ -41,6 +50,7 @@ export type CreateRuntimeLlmOptions = {
type RuntimeLlmOverridePolicy = {
allowAgentIdOverride: boolean;
allowModelOverride: boolean;
allowProfileOverride: boolean;
hasConfiguredAllowedModels: boolean;
allowAnyModel: boolean;
allowedModels: Set<string>;
@@ -93,7 +103,7 @@ function resolveRuntimeConfig(options: CreateRuntimeLlmOptions): OpenClawConfig
}
async function resolveAgentId(params: {
request: LlmCompleteParams;
request: Pick<LlmCompleteParams, "agentId">;
cfg: OpenClawConfig;
authority?: RuntimeLlmAuthority;
allowAgentIdOverride: boolean;
@@ -131,6 +141,126 @@ function buildSystemPrompt(params: LlmCompleteParams): string | undefined {
return segments.length > 0 ? segments.join("\n\n") : undefined;
}
function stripCodeFences(raw: string): string {
const trimmed = raw.trim();
const match = trimmed.match(/^```(?:json)?\s*([\s\S]*?)\s*```$/i);
return match ? (match[1] ?? "").trim() : trimmed;
}
function buildStructuredInstructions(params: LlmCompleteStructuredParams): string {
const sections = [params.instructions.trim()];
if (normalizeOptionalString(params.schemaName)) {
sections.push(`Schema name: ${params.schemaName!.trim()}`);
}
if (params.jsonSchema !== undefined) {
sections.push(`JSON schema:\n${JSON.stringify(params.jsonSchema)}`);
}
if (shouldUseJsonMode(params)) {
sections.push("Return valid JSON only. Do not wrap the JSON in Markdown fences.");
}
return sections.join("\n\n");
}
function shouldUseJsonMode(
params: Pick<LlmCompleteStructuredParams, "jsonMode" | "jsonSchema">,
): boolean {
return params.jsonMode === true || params.jsonSchema !== undefined;
}
function hasImageInput(input: LlmCompleteStructuredInput[]): boolean {
return input.some((entry) => entry.type === "image");
}
function buildStructuredMessages(params: { request: LlmCompleteStructuredParams }): Message[] {
const now = Date.now();
return [
{
role: "user" as const,
timestamp: now,
content: [
{ type: "text" as const, text: buildStructuredInstructions(params.request) },
...params.request.input.map((entry) =>
entry.type === "text"
? { type: "text" as const, text: entry.text }
: {
type: "image" as const,
data: entry.buffer.toString("base64"),
mimeType: normalizeOptionalString(entry.mimeType) ?? "image/png",
},
),
],
},
];
}
function createCompletionSignal(
signal: AbortSignal | undefined,
timeoutMs: number | undefined,
): { signal: AbortSignal | undefined; cleanup: () => void } {
if (timeoutMs === undefined || !Number.isFinite(timeoutMs) || timeoutMs <= 0) {
return { signal, cleanup: () => undefined };
}
const controller = new AbortController();
const timer = setTimeout(
() => controller.abort(new Error("Plugin LLM completion timed out")),
timeoutMs,
);
timer.unref?.();
let detachParentAbort: (() => void) | undefined;
if (signal) {
if (signal.aborted) {
controller.abort(signal.reason);
} else {
const onAbort = () => controller.abort(signal.reason);
signal.addEventListener("abort", onAbort, { once: true });
detachParentAbort = () => signal.removeEventListener("abort", onAbort);
}
}
return {
signal:
typeof AbortSignal.any === "function"
? AbortSignal.any([controller.signal, ...(signal ? [signal] : [])])
: controller.signal,
cleanup: () => {
clearTimeout(timer);
detachParentAbort?.();
},
};
}
function parseStructuredText(params: { text: string; jsonMode: boolean; jsonSchema?: unknown }): {
parsed?: unknown;
contentType: "json" | "text";
} {
if (!params.jsonMode) {
return { contentType: "text" };
}
let parsed: unknown;
try {
parsed = JSON.parse(stripCodeFences(params.text));
} catch {
throw new Error("Plugin LLM structured completion returned invalid JSON.");
}
if (
params.jsonSchema &&
typeof params.jsonSchema === "object" &&
!Array.isArray(params.jsonSchema)
) {
const validation = validateJsonSchemaValue({
schema: params.jsonSchema as JsonSchemaObject,
cacheKey: "runtime.llm.completeStructured",
value: parsed,
cache: false,
});
if (!validation.ok) {
const message =
validation.errors.map((entry) => entry.text).join("; ") || "invalid structured JSON";
throw new Error(`Plugin LLM structured completion JSON did not match schema: ${message}`);
}
}
return { parsed, contentType: "json" };
}
function buildMessages(params: {
request: LlmCompleteParams;
provider: string;
@@ -217,6 +347,26 @@ function finiteOption(value: number | undefined): number | undefined {
return typeof value === "number" && Number.isFinite(value) ? value : undefined;
}
function modelSupportsImageInput(model: { input?: unknown }): boolean {
return Array.isArray(model.input) && model.input.includes("image");
}
function normalizeResolvedSelectionModelRef(
selection:
| {
provider: string;
modelId: string;
}
| null
| undefined,
): string | null {
if (!selection) {
return null;
}
const normalized = normalizeModelRef(selection.provider, selection.modelId);
return `${normalized.provider}/${normalized.model}`;
}
function normalizeAllowedModelRef(raw: string): string | null {
const trimmed = raw.trim();
if (!trimmed) {
@@ -241,6 +391,7 @@ function normalizeAllowedModelRef(raw: string): string | null {
function buildPolicyFromEntry(entry: {
allowAgentIdOverride?: boolean;
allowModelOverride?: boolean;
allowProfileOverride?: boolean;
hasAllowedModelsConfig?: boolean;
allowedModels?: readonly string[];
}): RuntimeLlmOverridePolicy {
@@ -260,6 +411,7 @@ function buildPolicyFromEntry(entry: {
return {
allowAgentIdOverride: entry.allowAgentIdOverride === true,
allowModelOverride: entry.allowModelOverride === true,
allowProfileOverride: entry.allowProfileOverride === true,
hasConfiguredAllowedModels: entry.hasAllowedModelsConfig === true,
allowAnyModel,
allowedModels,
@@ -298,6 +450,7 @@ function resolveAuthorityModelPolicy(
if (
authority?.allowAgentIdOverride !== true &&
authority?.allowModelOverride !== true &&
authority?.allowProfileOverride !== true &&
authority?.allowedModels === undefined
) {
return undefined;
@@ -305,11 +458,41 @@ function resolveAuthorityModelPolicy(
return buildPolicyFromEntry({
allowAgentIdOverride: authority.allowAgentIdOverride,
allowModelOverride: authority.allowModelOverride,
allowProfileOverride: authority.allowProfileOverride,
hasAllowedModelsConfig: authority.allowedModels !== undefined,
allowedModels: authority.allowedModels,
});
}
function assertAllowedProfileOverride(params: {
requestedProfile: string | undefined;
pluginPolicyId: string | undefined;
authorityPolicy: RuntimeLlmOverridePolicy | undefined;
pluginPolicy: RuntimeLlmOverridePolicy | undefined;
}): void {
if (!params.requestedProfile) {
return;
}
if (params.authorityPolicy?.allowProfileOverride) {
return;
}
if (params.pluginPolicy?.allowProfileOverride) {
return;
}
const owner = params.pluginPolicyId ? ` for plugin "${params.pluginPolicyId}"` : "";
throw new Error(`Plugin LLM completion cannot override the auth profile${owner}.`);
}
function hasAllowedProfileOverride(params: {
authorityPolicy: RuntimeLlmOverridePolicy | undefined;
pluginPolicy: RuntimeLlmOverridePolicy | undefined;
}): boolean {
return (
params.authorityPolicy?.allowProfileOverride === true ||
params.pluginPolicy?.allowProfileOverride === true
);
}
function assertAllowedModelOverride(params: {
resolvedModelRef: string | null;
pluginPolicyId: string | undefined;
@@ -354,74 +537,159 @@ function assertAllowedModelOverride(params: {
*/
export function createRuntimeLlm(options: CreateRuntimeLlmOptions = {}): PluginRuntimeCore["llm"] {
const logger = options.logger ?? toRuntimeLogger(defaultLogger);
return {
complete: async (params: LlmCompleteParams): Promise<LlmCompleteResult> => {
const caller = resolveTrustedCaller(options.authority);
if (options.authority?.allowComplete === false) {
const reason = options.authority.denyReason ?? "capability denied";
logger.warn("plugin llm completion denied", {
caller,
purpose: params.purpose,
reason,
});
throw new Error(`Plugin LLM completion denied: ${reason}`);
}
const [
{
prepareSimpleCompletionModelForAgent,
completeWithPreparedSimpleCompletionModel,
resolveSimpleCompletionSelectionForAgent,
},
async function prepareRuntimeCall(params: {
model?: string;
agentId?: string;
profile?: string;
preferImageModel?: boolean;
}) {
const caller = resolveTrustedCaller(options.authority);
const [
{
prepareSimpleCompletionModelForAgent,
completeWithPreparedSimpleCompletionModel,
resolveSimpleCompletionSelectionForAgent,
},
cfg,
] = await Promise.all([
import("../../agents/simple-completion-runtime.js"),
Promise.resolve(resolveRuntimeConfig(options)),
]);
const pluginPolicyId = resolvePluginPolicyId(options.authority, caller);
const pluginPolicy = resolvePluginLlmOverridePolicy(cfg, pluginPolicyId);
const authorityPolicy = resolveAuthorityModelPolicy(options.authority);
const agentId = await resolveAgentId({
request: params,
cfg,
authority: options.authority,
allowAgentIdOverride:
options.authority?.allowAgentIdOverride === false
? false
: authorityPolicy?.allowAgentIdOverride === true ||
pluginPolicy?.allowAgentIdOverride === true,
});
const requestedModel = normalizeOptionalString(params.model);
const splitRequestedModel = requestedModel
? splitTrailingAuthProfile(requestedModel)
: { model: undefined, profile: undefined };
const requestedModelRef = normalizeOptionalString(splitRequestedModel.model);
const requestedProfileFromModel = normalizeOptionalString(splitRequestedModel.profile);
const requestedProfile = normalizeOptionalString(params.profile);
if (
requestedProfile &&
requestedProfileFromModel &&
requestedProfile !== requestedProfileFromModel
) {
throw new Error(
"Plugin LLM completion received conflicting auth profiles in model and profile fields.",
);
}
const effectiveRequestedProfile = requestedProfile ?? requestedProfileFromModel;
let effectiveRequestedModelRef = requestedModelRef;
let modelRefProfileCoveredByModelOverride = false;
if (requestedModelRef) {
const selection = resolveSimpleCompletionSelectionForAgent({
cfg,
] = await Promise.all([
import("../../agents/simple-completion-runtime.js"),
Promise.resolve(resolveRuntimeConfig(options)),
]);
const pluginPolicyId = resolvePluginPolicyId(options.authority, caller);
const pluginPolicy = resolvePluginLlmOverridePolicy(cfg, pluginPolicyId);
const authorityPolicy = resolveAuthorityModelPolicy(options.authority);
const agentId = await resolveAgentId({
request: params,
cfg,
authority: options.authority,
allowAgentIdOverride:
options.authority?.allowAgentIdOverride === false
? false
: authorityPolicy?.allowAgentIdOverride === true ||
pluginPolicy?.allowAgentIdOverride === true,
agentId,
modelRef: requestedModelRef,
});
const requestedModel = normalizeOptionalString(params.model);
if (requestedModel) {
const selection = resolveSimpleCompletionSelectionForAgent({
cfg,
agentId,
modelRef: requestedModel,
});
const normalizedSelection = selection
? normalizeModelRef(selection.provider, selection.modelId)
: null;
const resolvedModelRef = normalizedSelection
? `${normalizedSelection.provider}/${normalizedSelection.model}`
: null;
const defaultSelection = resolveSimpleCompletionSelectionForAgent({
cfg,
agentId,
});
const resolvedModelRef = normalizeResolvedSelectionModelRef(selection);
const defaultResolvedModelRef = normalizeResolvedSelectionModelRef(defaultSelection);
const changesTargetModel =
defaultResolvedModelRef === null || resolvedModelRef !== defaultResolvedModelRef;
if (changesTargetModel) {
assertAllowedModelOverride({
resolvedModelRef,
pluginPolicyId,
authorityPolicy,
pluginPolicy,
});
modelRefProfileCoveredByModelOverride = requestedProfileFromModel !== undefined;
} else {
if (
requestedProfileFromModel &&
!hasAllowedProfileOverride({ authorityPolicy, pluginPolicy })
) {
assertAllowedModelOverride({
resolvedModelRef,
pluginPolicyId,
authorityPolicy,
pluginPolicy,
});
modelRefProfileCoveredByModelOverride = true;
}
effectiveRequestedModelRef = undefined;
}
const prepared = await prepareSimpleCompletionModelForAgent({
cfg,
agentId,
modelRef: params.model,
allowMissingApiKeyModes: ["aws-sdk"],
}
if (requestedProfile) {
assertAllowedProfileOverride({
requestedProfile,
pluginPolicyId,
authorityPolicy,
pluginPolicy,
});
} else if (requestedProfileFromModel && !modelRefProfileCoveredByModelOverride) {
assertAllowedProfileOverride({
requestedProfile: requestedProfileFromModel,
pluginPolicyId,
authorityPolicy,
pluginPolicy,
});
}
if ("error" in prepared) {
throw new Error(`Plugin LLM completion failed: ${prepared.error}`);
let hostResolvedModelRef = effectiveRequestedModelRef;
if (!hostResolvedModelRef && params.preferImageModel) {
const [{ resolveAutoImageModel }, { resolveAgentDir }] = await Promise.all([
import("../../media-understanding/runner.js"),
import("../../agents/agent-scope.js"),
]);
const imageModel = await resolveAutoImageModel({
cfg,
agentDir: resolveAgentDir(cfg, agentId),
});
if (imageModel?.provider && imageModel?.model) {
hostResolvedModelRef = `${imageModel.provider}/${imageModel.model}`;
}
}
const prepared = await prepareSimpleCompletionModelForAgent({
cfg,
agentId,
modelRef: hostResolvedModelRef,
preferredProfile: effectiveRequestedProfile,
allowMissingApiKeyModes: ["aws-sdk"],
});
if ("error" in prepared) {
throw new Error(`Plugin LLM completion failed: ${prepared.error}`);
}
return {
caller,
cfg,
agentId,
prepared,
completeWithPreparedSimpleCompletionModel,
};
}
return {
complete: async (params: LlmCompleteParams): Promise<LlmCompleteResult> => {
if (options.authority?.allowComplete === false) {
const reason = options.authority.denyReason ?? "capability denied";
logger.warn("plugin llm completion denied", {
caller: resolveTrustedCaller(options.authority),
purpose: params.purpose,
reason,
});
throw new Error(`Plugin LLM completion denied: ${reason}`);
}
const { caller, cfg, agentId, prepared, completeWithPreparedSimpleCompletionModel } =
await prepareRuntimeCall(params);
const context = {
systemPrompt: buildSystemPrompt(params),
@@ -481,5 +749,104 @@ export function createRuntimeLlm(options: CreateRuntimeLlmOptions = {}): PluginR
},
};
},
completeStructured: async (
params: LlmCompleteStructuredParams,
): Promise<LlmCompleteStructuredResult> => {
if (options.authority?.allowComplete === false) {
const reason = options.authority.denyReason ?? "capability denied";
logger.warn("plugin llm structured completion denied", {
caller: resolveTrustedCaller(options.authority),
purpose: params.purpose,
reason,
});
throw new Error(`Plugin LLM structured completion denied: ${reason}`);
}
if (params.input.length === 0) {
throw new Error("Plugin LLM structured completion requires at least one input.");
}
if (!params.instructions.trim()) {
throw new Error("Plugin LLM structured completion requires instructions.");
}
const imageInput = hasImageInput(params.input);
const { caller, cfg, agentId, prepared, completeWithPreparedSimpleCompletionModel } =
await prepareRuntimeCall({
...params,
preferImageModel: imageInput,
});
if (imageInput && !modelSupportsImageInput(prepared.model)) {
throw new Error(
`Plugin LLM structured completion model does not support image input: ${prepared.selection.provider}/${prepared.selection.modelId}`,
);
}
const { signal, cleanup } = createCompletionSignal(
params.signal,
finiteOption(params.timeoutMs),
);
try {
const result = await completeWithPreparedSimpleCompletionModel({
model: prepared.model,
auth: prepared.auth,
cfg,
context: {
systemPrompt: normalizeOptionalString(params.systemPrompt),
messages: buildStructuredMessages({ request: params }),
},
options: {
maxTokens: finiteOption(params.maxTokens),
temperature: finiteOption(params.temperature),
signal,
},
});
const text = result.content
.filter((c): c is { type: "text"; text: string } => c.type === "text")
.map((c) => c.text)
.join("");
const normalizedUsage = normalizeUsage(result.usage as UsageLike | undefined);
const usage = buildUsage({
rawUsage: result.usage,
normalized: normalizedUsage,
cfg,
provider: prepared.selection.provider,
model: prepared.selection.modelId,
});
const structured = parseStructuredText({
text,
jsonMode: shouldUseJsonMode(params),
jsonSchema: params.jsonSchema,
});
logger.info("plugin llm structured completion", {
caller,
purpose: params.purpose,
sessionKey: options.authority?.sessionKey,
agentId,
provider: prepared.selection.provider,
model: prepared.selection.modelId,
usage,
contentType: structured.contentType,
imageInput,
});
return {
text,
provider: prepared.selection.provider,
model: prepared.selection.modelId,
agentId,
usage,
parsed: structured.parsed,
contentType: structured.contentType,
audit: {
caller,
...(params.purpose ? { purpose: params.purpose } : {}),
...(options.authority?.sessionKey ? { sessionKey: options.authority.sessionKey } : {}),
},
};
} finally {
cleanup();
}
},
};
}

View File

@@ -108,6 +108,22 @@ export type LlmCompleteUsage = {
costUsd?: number;
};
export type LlmCompleteStructuredTextInput = {
type: "text";
text: string;
};
export type LlmCompleteStructuredImageInput = {
type: "image";
buffer: Buffer;
mimeType?: string;
fileName?: string;
};
export type LlmCompleteStructuredInput =
| LlmCompleteStructuredTextInput
| LlmCompleteStructuredImageInput;
export type LlmCompleteParams = {
messages: LlmCompleteMessage[];
/** Model ref (e.g. "anthropic/claude-sonnet-4-6"); defaults to the target agent's configured model. */
@@ -122,6 +138,30 @@ export type LlmCompleteParams = {
agentId?: string;
};
export type LlmCompleteStructuredParams = {
input: LlmCompleteStructuredInput[];
instructions: string;
/** Model ref (e.g. "anthropic/claude-sonnet-4-6"); defaults to the target agent's configured model. */
model?: string;
maxTokens?: number;
temperature?: number;
systemPrompt?: string;
signal?: AbortSignal;
timeoutMs?: number;
/** Human-readable reason for audit/debug output. */
purpose?: string;
/** Agent whose model/credentials to use. Session-bound capabilities may disallow overrides. */
agentId?: string;
/**
* Preferred auth profile id when the selected provider supports profile-backed auth.
* Requires host trust via plugins.entries.<id>.llm.allowProfileOverride or an equivalent authority policy.
*/
profile?: string;
schemaName?: string;
jsonSchema?: unknown;
jsonMode?: boolean;
};
export type LlmCompleteResult = {
text: string;
provider: string;
@@ -135,6 +175,11 @@ export type LlmCompleteResult = {
};
};
export type LlmCompleteStructuredResult = LlmCompleteResult & {
parsed?: unknown;
contentType: "json" | "text";
};
type RuntimeRunEmbeddedPiAgent = (
params: import("../../agents/pi-embedded-runner/run/params.js").RunEmbeddedPiAgentParams,
) => Promise<import("../../agents/pi-embedded-runner/types.js").EmbeddedPiRunResult>;
@@ -313,6 +358,9 @@ export type PluginRuntimeCore = {
taskFlow: import("./runtime-taskflow.types.js").PluginRuntimeTaskFlow;
llm: {
complete: (params: LlmCompleteParams) => Promise<LlmCompleteResult>;
completeStructured: (
params: LlmCompleteStructuredParams,
) => Promise<LlmCompleteStructuredResult>;
};
modelAuth: {
/** Resolve auth for a model. Only provider/model, optional cfg, and workspaceDir are used. */