liuyu/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-06-06 05:51:15 +08:00
Code Issues Packages Projects Releases Wiki Activity

Add live transport scenario planning

Browse Source
This commit is contained in:
Dallin Romney
2026-06-04 20:11:36 -07:00
parent 1de46bb425
commit 71984842c4
52 changed files with 1573 additions and 477 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/.generated/plugin-sdk-api-baseline.sha256
View File

@@ -1,2 +1,2 @@
9ce72d763de6c95566e0167f99f5454b07c7c67940675533cb24c07058619a63 plugin-sdk-api-baseline.json
e4dfccb85b985fe865145e24978255b729cdcbca0e26650a363a11bfcfc2e27b plugin-sdk-api-baseline.jsonl
58e91672fcf00ebb6947081a5a5ba2d379c2bdbde8f8e5322a045e840dce2f7a plugin-sdk-api-baseline.json
15b24cc1d41fde8206a82f319669d43410c633b92a37600a99f84c9ae6af7490 plugin-sdk-api-baseline.jsonl

2
docs/concepts/personal-agent-benchmark-pack.md
View File

@@ -34,7 +34,7 @@ The machine-readable pack metadata lives in
`--pack personal-agent`:
```bash
OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 pnpm openclaw qa suite \
OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI=1 pnpm openclaw qa suite \
--provider-mode mock-openai \
--pack personal-agent \
--concurrency 1

2
docs/concepts/qa-matrix.md
View File

@@ -9,7 +9,7 @@ title: "Matrix QA"
The Matrix QA lane runs the bundled `@openclaw/matrix` plugin against a disposable Tuwunel homeserver in Docker, with temporary driver, SUT, and observer accounts plus seeded rooms. It is the live transport-real coverage for Matrix.
This is maintainer-only tooling. Packaged OpenClaw releases intentionally omit `qa-lab`, so `openclaw qa` is only available from a source checkout. Source checkouts load the bundled runner directly - no plugin install step is needed.
This is maintainer-only tooling. The Matrix live lane still requires a source checkout because `qa-matrix` and its Docker-backed fixtures are not packaged with OpenClaw. Source checkouts load the bundled runner directly - no plugin install step is needed.
For broader QA framework context, see [QA overview](/concepts/qa-e2e-automation).

2
docs/help/testing.md
View File

@@ -333,7 +333,7 @@ gh workflow run package-acceptance.yml --ref main \
- Starts only the local AIMock provider server for direct protocol smoke
testing.
- `pnpm openclaw qa matrix`
- Runs the Matrix live QA lane against a disposable Docker-backed Tuwunel homeserver. Source-checkout only - packaged installs do not ship `qa-lab`.
- Runs the Matrix live QA lane against a disposable Docker-backed Tuwunel homeserver. Source-checkout only - packaged installs do not ship `qa-matrix` and its Docker-backed fixtures.
- Full CLI, profile/scenario catalog, env vars, and artifact layout: [Matrix QA](/concepts/qa-matrix).
- `pnpm openclaw qa telegram`
- Runs the Telegram live QA lane against a real private group using the driver and SUT bot tokens from env.

12
docs/plugins/plugin-inventory.md
View File

@@ -180,10 +180,10 @@ commands.
| [zalo](/plugins/reference/zalo) | OpenClaw Zalo channel plugin for bot and webhook chats. | `@openclaw/zalo`<br />npm; ClawHub | channels: zalo |
| [zalouser](/plugins/reference/zalouser) | OpenClaw Zalo Personal Account plugin via native zca-js integration. | `@openclaw/zalouser`<br />npm; ClawHub | channels: zalouser; contracts: tools |
## Source checkout only
## QA Tooling
| Plugin | Description | Distribution | Surface |
| ------------------------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------ | -------------------- |
| [qa-channel](/plugins/reference/qa-channel) | Adds the QA Channel surface for sending and receiving OpenClaw messages. | `@openclaw/qa-channel`<br />source checkout only | channels: qa-channel |
| [qa-lab](/plugins/reference/qa-lab) | OpenClaw QA lab plugin with private debugger UI and scenario runner. | `@openclaw/qa-lab`<br />source checkout only | plugin |
| [qa-matrix](/plugins/reference/qa-matrix) | Matrix QA transport runner and substrate. | `@openclaw/qa-matrix`<br />source checkout only | plugin |
| Plugin | Description | Distribution | Surface |
| ------------------------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------- | -------------------- |
| [qa-channel](/plugins/reference/qa-channel) | Adds the QA Channel surface for sending and receiving OpenClaw messages. | bundled in OpenClaw | channels: qa-channel |
| [qa-lab](/plugins/reference/qa-lab) | OpenClaw QA lab plugin with private debugger UI and scenario runner. | bundled in OpenClaw; experimental CLI gate | plugin |
| [qa-matrix](/plugins/reference/qa-matrix) | Matrix QA transport runner and substrate. | `@openclaw/qa-matrix`<br />source checkout only | plugin |

4
docs/plugins/reference.md
View File

@@ -103,8 +103,8 @@ pnpm plugins:inventory:gen
| [perplexity](/plugins/reference/perplexity) | Adds web search provider support. | `@openclaw/perplexity-plugin`<br />included in OpenClaw | contracts: webSearchProviders |
| [pixverse](/plugins/reference/pixverse) | OpenClaw PixVerse video generation provider plugin. | `@openclaw/pixverse-provider`<br />npm; ClawHub: `clawhub:@openclaw/pixverse-provider` | contracts: videoGenerationProviders |
| [policy](/plugins/reference/policy) | Adds policy-backed doctor checks for workspace conformance. | `@openclaw/policy`<br />included in OpenClaw | plugin |
| [qa-channel](/plugins/reference/qa-channel) | Adds the QA Channel surface for sending and receiving OpenClaw messages. | `@openclaw/qa-channel`<br />source checkout only | channels: qa-channel |
| [qa-lab](/plugins/reference/qa-lab) | OpenClaw QA lab plugin with private debugger UI and scenario runner. | `@openclaw/qa-lab`<br />source checkout only | plugin |
| [qa-channel](/plugins/reference/qa-channel) | Adds the QA Channel surface for sending and receiving OpenClaw messages. | bundled in OpenClaw | channels: qa-channel |
| [qa-lab](/plugins/reference/qa-lab) | OpenClaw QA lab plugin with private debugger UI and scenario runner. | bundled in OpenClaw; experimental CLI gate | plugin |
| [qa-matrix](/plugins/reference/qa-matrix) | Matrix QA transport runner and substrate. | `@openclaw/qa-matrix`<br />source checkout only | plugin |
| [qianfan](/plugins/reference/qianfan) | Adds Qianfan model provider support to OpenClaw. | `@openclaw/qianfan-provider`<br />included in OpenClaw | providers: qianfan |
| [qqbot](/plugins/reference/qqbot) | OpenClaw QQ Bot channel plugin for group and direct-message workflows. | `@openclaw/qqbot`<br />npm; ClawHub | channels: qqbot; contracts: tools; skills |

4
docs/plugins/reference/qa-channel.md
View File

@@ -11,8 +11,8 @@ Adds the QA Channel surface for sending and receiving OpenClaw messages.
## Distribution
- Package: `@openclaw/qa-channel`
- Install route: source checkout only
- Package: bundled in OpenClaw as `@openclaw/qa-channel`
- Install route: included with OpenClaw for QA scenarios and experimental user-flow execution
## Surface

4
docs/plugins/reference/qa-lab.md
View File

@@ -11,8 +11,8 @@ OpenClaw QA lab plugin with private debugger UI and scenario runner.
## Distribution
- Package: `@openclaw/qa-lab`
- Install route: source checkout only
- Package: bundled in OpenClaw as `@openclaw/qa-lab`
- Install route: included with OpenClaw; CLI access requires `OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI=1`
## Surface

5
docs/plugins/sdk-subpaths.md
View File

@@ -229,7 +229,12 @@ usage endpoint failed or returned no usable usage data.
| `plugin-sdk/lazy-runtime` | Lazy runtime import/binding helpers such as `createLazyRuntimeModule`, `createLazyRuntimeMethod`, and `createLazyRuntimeSurface` |
| `plugin-sdk/process-runtime` | Process exec helpers |
| `plugin-sdk/cli-runtime` | CLI formatting, wait, version, argument-invocation, and lazy command-group helpers |
| `plugin-sdk/qa-channel` | QA Channel facade helpers for synthetic user-flow and suite execution |
| `plugin-sdk/qa-channel-protocol` | QA Channel bus protocol helpers shared by the synthetic QA channel runtime |
| `plugin-sdk/qa-lab` | Gated QA Lab CLI facade used by experimental QA command registration |
| `plugin-sdk/qa-runtime` | QA Lab runtime facade helpers for scenario and user-flow execution |
| `plugin-sdk/qa-live-transport-scenarios` | Shared live transport QA scenario ids, baseline coverage helpers, and scenario-selection helper |
| `plugin-sdk/qa-user-flows` | Shared user-flow surfaces, capability mappings, standard flow catalog, and planner helpers for QA runners |
| `plugin-sdk/gateway-method-runtime` | Reserved Gateway method dispatch helper for plugin HTTP routes that declare `contracts.gatewayMethodDispatch: ["authenticated-request"]` |
| `plugin-sdk/gateway-runtime` | Gateway client, event-loop-ready client start helper, gateway CLI RPC, gateway protocol errors, and channel-status patch helpers |
| `plugin-sdk/config-contracts` | Focused type-only config surface for plugin config shapes such as `OpenClawConfig` and channel/provider config types |

70
extensions/qa-lab/src/cli.test.ts
View File

@@ -162,6 +162,76 @@ describe("qa cli registration", () => {
expect(commandNames).toContain("mantis");
expect(commandNames).toContain("credentials");
expect(commandNames).toContain("coverage");
expect(commandNames).toContain("user-flows");
});
it("runs selected user flows through existing QA suite scenarios", async () => {
const stdoutChunks: string[] = [];
const stdoutWrite = vi
.spyOn(process.stdout, "write")
.mockImplementation((chunk: string | Uint8Array) => {
stdoutChunks.push(String(chunk));
return true;
});
try {
await program.parseAsync([
"node",
"openclaw",
"qa",
"user-flows",
"run",
"--flow",
"messaging.direct-reply",
"--capability",
"messaging.inbound-message",
"--capability",
"messaging.outbound-final-reply",
"--repo-root",
"/tmp/openclaw-repo",
"--output-dir",
".artifacts/qa-user-flows",
"--transport",
"qa-channel",
"--provider-mode",
"mock-openai",
"--concurrency",
"1",
"--allow-failures",
"--json",
]);
} finally {
stdoutWrite.mockRestore();
}
expect(runQaSuiteCommand).toHaveBeenCalledWith({
repoRoot: "/tmp/openclaw-repo",
outputDir: ".artifacts/qa-user-flows",
transportId: "qa-channel",
providerMode: "mock-openai",
primaryModel: undefined,
alternateModel: undefined,
fastMode: false,
allowFailures: true,
concurrency: 1,
scenarioIds: ["channel-chat-baseline", "dm-chat-baseline"],
});
const output = JSON.parse(stdoutChunks.join("")) as {
execution: { runner: string; scenarioIds: string[] };
plan: { selected: Array<{ id: string }>; skipped: Array<{ id: string; reason: string }> };
status: string;
};
expect(output).toMatchObject({
status: "pass",
execution: {
runner: "qa-suite",
scenarioIds: ["channel-chat-baseline", "dm-chat-baseline"],
},
});
expect(output.plan.selected.map((flow) => flow.id)).toEqual(["messaging.direct-reply"]);
expect(output.plan.skipped.find((flow) => flow.id === "memory.scoped-recall")).toMatchObject({
id: "memory.scoped-recall",
reason: "not-requested",
});
});
it("does not expose a control-ui token flag on qa ui", () => {

2
extensions/qa-lab/src/cli.ts
View File

@@ -15,6 +15,7 @@ import {
} from "./providers/live-frontier/parity.js";
import type { QaProviderMode, QaProviderModeInput } from "./run-config.js";
import { hasQaScenarioPack } from "./scenario-catalog.js";
import { registerQaUserFlowCli } from "./user-flow-cli.js";
type QaLabCliRuntime = typeof import("./cli.runtime.js");
@@ -285,6 +286,7 @@ export function registerQaLabCli(program: Command) {
.command("qa")
.description("Run private QA automation flows and launch the QA debugger");
registerMantisCli(qa);
registerQaUserFlowCli(qa, { runSuite: runQaSuite });
qa.command("run")
.description("Run the bundled QA self-check and write a Markdown report")

243
extensions/qa-lab/src/user-flow-cli.ts Normal file
View File

@@ -0,0 +1,243 @@
// Qa Lab plugin module exposes user-flow planning commands.
import type { Command } from "commander";
import {
QA_USER_FLOW_STANDARD_FLOWS,
planQaUserFlows,
type QaStandardUserFlowCapabilityId,
type QaStandardUserFlowId,
} from "openclaw/plugin-sdk/qa-user-flows";
type QaUserFlowSelectionOptions = {
surface?: string[];
};
type QaUserFlowsPlanOptions = QaUserFlowSelectionOptions & {
capability?: string[];
driverFlow?: string[];
flow?: string[];
};
type QaUserFlowsRunOptions = QaUserFlowsPlanOptions & {
allowFailures?: boolean;
altModel?: string;
concurrency?: number;
fast?: boolean;
json?: boolean;
model?: string;
outputDir?: string;
providerMode?: string;
repoRoot?: string;
transport?: string;
};
type QaUserFlowSuiteRunOptions = {
allowFailures?: boolean;
alternateModel?: string;
concurrency?: number;
fastMode?: boolean;
outputDir?: string;
primaryModel?: string;
providerMode?: string;
repoRoot?: string;
scenarioIds?: string[];
transportId?: string;
};
type QaUserFlowCliRuntime = {
runSuite(opts: QaUserFlowSuiteRunOptions): Promise<void>;
};
function writeJson(value: unknown) {
process.stdout.write(`${JSON.stringify(value, null, 2)}\n`);
}
function normalizeRepeatedStringValues(values: readonly string[] | undefined) {
const result: string[] = [];
const seen = new Set<string>();
for (const value of values ?? []) {
for (const entry of value.split(",")) {
const trimmed = entry.trim();
if (!trimmed || seen.has(trimmed)) {
continue;
}
seen.add(trimmed);
result.push(trimmed);
}
}
return result;
}
function collectQaUserFlowCliString(value: string, previous: string[]) {
const trimmed = value.trim();
return trimmed ? [...previous, trimmed] : previous;
}
function filterStandardFlowsBySurface(surfaces: readonly string[]) {
if (surfaces.length === 0) {
return [...QA_USER_FLOW_STANDARD_FLOWS];
}
const requested = new Set(surfaces);
return QA_USER_FLOW_STANDARD_FLOWS.filter((flow) => requested.has(flow.surface));
}
function buildQaUserFlowsPlanOutput(opts: QaUserFlowsPlanOptions) {
const surfaces = normalizeRepeatedStringValues(opts.surface);
const flows = filterStandardFlowsBySurface(surfaces);
const availableCapabilities = normalizeRepeatedStringValues(
opts.capability,
) as QaStandardUserFlowCapabilityId[];
const driverSupportedFlowIds = normalizeRepeatedStringValues(
opts.driverFlow,
) as QaStandardUserFlowId[];
const requestedFlowIds = normalizeRepeatedStringValues(opts.flow) as QaStandardUserFlowId[];
const plan = planQaUserFlows({
flows,
availableCapabilities,
...(driverSupportedFlowIds.length > 0 ? { driverSupportedFlowIds } : {}),
...(requestedFlowIds.length > 0 ? { requestedFlowIds } : {}),
});
return {
version: 1,
filters: {
surfaces,
requestedFlowIds,
driverSupportedFlowIds,
},
availableCapabilities,
...plan,
};
}
function resolveQaUserFlowRunScenarioIds(plan: ReturnType<typeof buildQaUserFlowsPlanOutput>) {
const unsupportedRunnerFlowIds = plan.selected
.filter((flow) => flow.execution.runner !== "qa-lab-flow" || !flow.qaScenarioIds?.length)
.map((flow) => `${flow.id} (${flow.execution.runner})`);
if (unsupportedRunnerFlowIds.length > 0) {
throw new Error(
`selected user flow(s) do not have a QA suite execution mapping yet: ${unsupportedRunnerFlowIds.join(", ")}`,
);
}
return normalizeRepeatedStringValues(plan.selected.flatMap((flow) => flow.qaScenarioIds ?? []));
}
function buildQaUserFlowRunOutput(params: {
error?: string;
plan: ReturnType<typeof buildQaUserFlowsPlanOutput>;
scenarioIds: readonly string[];
status: "fail" | "pass";
}) {
return {
version: 1,
status: params.status,
plan: {
selected: params.plan.selected,
skipped: params.plan.skipped,
},
execution: {
runner: "qa-suite",
scenarioIds: [...params.scenarioIds],
},
...(params.error ? { error: params.error } : {}),
};
}
export function registerQaUserFlowCli(qa: Command, runtime?: QaUserFlowCliRuntime) {
const userFlows = qa.command("user-flows").description("Run OpenClaw-owned QA user flows");
// Future: list and plan commands could expose the standard catalog and the
// selected/skipped run plan for Kova/debugging without executing scenarios.
userFlows
.command("run")
.description("Run selected QA user flows through the existing QA suite runner")
.option("--repo-root <path>", "Repository root to target when running from a neutral cwd")
.option("--output-dir <path>", "Suite artifact directory")
.option("--transport <id>", "QA transport id", "qa-channel")
.option("--provider-mode <mode>", "QA provider mode", "mock-openai")
.option("--model <ref>", "Primary provider/model ref")
.option("--alt-model <ref>", "Alternate provider/model ref")
.option("--fast", "Enable provider fast mode where supported", false)
.option(
"--allow-failures",
"Write artifacts without setting a failing exit code when scenarios fail",
false,
)
.option("--concurrency <count>", "Scenario worker concurrency", (value: string) => {
const parsed = Number.parseInt(value, 10);
if (!Number.isSafeInteger(parsed) || parsed < 1 || String(parsed) !== value.trim()) {
throw new Error("--concurrency must be a positive integer.");
}
return parsed;
})
.option(
"--surface <id>",
"Filter by user-flow surface (repeatable or comma-separated)",
collectQaUserFlowCliString,
[],
)
.option(
"--flow <id>",
"Request one user-flow id (repeatable or comma-separated)",
collectQaUserFlowCliString,
[],
)
.option(
"--capability <id>",
"Declare one available capability id (repeatable or comma-separated)",
collectQaUserFlowCliString,
[],
)
.option(
"--driver-flow <id>",
"Declare one concretely driver-supported flow id (repeatable or comma-separated)",
collectQaUserFlowCliString,
[],
)
.option("--json", "Emit selected/skipped user-flow plan and execution summary as JSON", false)
.action(async (opts: QaUserFlowsRunOptions) => {
if (!runtime) {
throw new Error("QA user-flow run command is missing its suite runner.");
}
const plan = buildQaUserFlowsPlanOutput(opts);
const scenarioIds = resolveQaUserFlowRunScenarioIds(plan);
if (scenarioIds.length === 0) {
throw new Error("No runnable QA suite scenarios selected for the requested user flows.");
}
try {
await runtime.runSuite({
repoRoot: opts.repoRoot,
outputDir: opts.outputDir,
transportId: opts.transport,
providerMode: opts.providerMode,
primaryModel: opts.model,
alternateModel: opts.altModel,
fastMode: opts.fast,
allowFailures: opts.allowFailures,
concurrency: opts.concurrency,
scenarioIds,
});
if (opts.json) {
writeJson(buildQaUserFlowRunOutput({ plan, scenarioIds, status: "pass" }));
}
} catch (error) {
if (opts.json) {
writeJson(
buildQaUserFlowRunOutput({
plan,
scenarioIds,
status: "fail",
error: error instanceof Error ? error.message : String(error),
}),
);
}
throw error;
}
});
}
export const qaUserFlowCliTesting = {
buildQaUserFlowsPlanOutput,
buildQaUserFlowRunOutput,
normalizeRepeatedStringValues,
resolveQaUserFlowRunScenarioIds,
};

36
package.json
View File

@@ -99,8 +99,6 @@
"!dist/extensions/nostr/**",
"!dist/extensions/qqbot/**",
"!dist/extensions/pixverse/**",
"!dist/extensions/qa-channel/**",
"!dist/extensions/qa-lab/**",
"!dist/extensions/qa-matrix/**",
"!dist/extensions/openshell/**",
"!dist/extensions/slack/**",
@@ -112,25 +110,15 @@
"!dist/extensions/whatsapp/**",
"!dist/extensions/zalo/**",
"!dist/extensions/zalouser/**",
"!dist/plugin-sdk/extensions/qa-channel/**",
"!dist/plugin-sdk/extensions/qa-lab/**",
"!dist/plugin-sdk/qa-channel.*",
"!dist/plugin-sdk/qa-channel-protocol.*",
"!dist/plugin-sdk/qa-lab.*",
"!dist/plugin-sdk/qa-runtime.*",
"!dist/plugin-sdk/src/**",
"!dist/plugin-sdk/src/plugin-sdk/qa-channel.d.ts",
"!dist/plugin-sdk/src/plugin-sdk/qa-channel-protocol.d.ts",
"!dist/plugin-sdk/src/plugin-sdk/qa-lab.d.ts",
"!dist/plugin-sdk/src/plugin-sdk/qa-runtime.d.ts",
"!dist/qa-runtime-*.js",
"docs/",
"!docs/.generated/**",
"!docs/channels/qa-channel.md",
"!docs/assets/**",
"!docs/images/**",
"!docs/**/*.jpg",
"!docs/**/*.png",
"qa/scenarios.md",
"qa/scenarios/",
"src/agents/templates/",
"scripts/crabbox-wrapper.mjs",
"patches/",
@@ -1073,6 +1061,22 @@
"types": "./dist/plugin-sdk/keyed-async-queue.d.ts",
"default": "./dist/plugin-sdk/keyed-async-queue.js"
},
"./plugin-sdk/qa-channel": {
"types": "./dist/plugin-sdk/qa-channel.d.ts",
"default": "./dist/plugin-sdk/qa-channel.js"
},
"./plugin-sdk/qa-channel-protocol": {
"types": "./dist/plugin-sdk/qa-channel-protocol.d.ts",
"default": "./dist/plugin-sdk/qa-channel-protocol.js"
},
"./plugin-sdk/qa-lab": {
"types": "./dist/plugin-sdk/qa-lab.d.ts",
"default": "./dist/plugin-sdk/qa-lab.js"
},
"./plugin-sdk/qa-runtime": {
"types": "./dist/plugin-sdk/qa-runtime.d.ts",
"default": "./dist/plugin-sdk/qa-runtime.js"
},
"./plugin-sdk/qa-runner-runtime": {
"types": "./dist/plugin-sdk/qa-runner-runtime.d.ts",
"default": "./dist/plugin-sdk/qa-runner-runtime.js"
@@ -1081,6 +1085,10 @@
"types": "./dist/plugin-sdk/qa-live-transport-scenarios.d.ts",
"default": "./dist/plugin-sdk/qa-live-transport-scenarios.js"
},
"./plugin-sdk/qa-user-flows": {
"types": "./dist/plugin-sdk/qa-user-flows.d.ts",
"default": "./dist/plugin-sdk/qa-user-flows.js"
},
"./plugin-sdk/memory-core": {
"types": "./dist/plugin-sdk/memory-core.d.ts",
"default": "./dist/plugin-sdk/memory-core.js"

2
scripts/check-gateway-cpu-scenarios.mjs
View File

@@ -195,7 +195,7 @@ function buildPrivateQaEnv(env, qaState) {
}
: {}),
OPENCLAW_BUILD_PRIVATE_QA: "1",
OPENCLAW_ENABLE_PRIVATE_QA_CLI: "1",
OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: "1",
OPENCLAW_RUN_NODE_SKIP_DTS_BUILD: env.OPENCLAW_RUN_NODE_SKIP_DTS_BUILD ?? "1",
OPENCLAW_TEST_DISABLE_UPDATE_CHECK: env.OPENCLAW_TEST_DISABLE_UPDATE_CHECK ?? "1",
};

37
scripts/check-openclaw-package-tarball.mjs
View File

@@ -86,32 +86,11 @@ const LEGACY_LOCAL_BUILD_METADATA_COMPAT_MAX = { year: 2026, month: 4, day: 26 }
const LEGACY_SHRINKWRAP_COMPAT_MAX = { year: 2026, month: 5, day: 20 };
const FORBIDDEN_LOCAL_BUILD_METADATA_FILES = new Set(LOCAL_BUILD_METADATA_DIST_PATHS);
const LEGACY_OMITTED_PRIVATE_QA_INVENTORY_PREFIXES = [
"dist/extensions/qa-channel/",
"dist/extensions/qa-lab/",
"dist/extensions/qa-matrix/",
"dist/plugin-sdk/extensions/qa-channel/",
"dist/plugin-sdk/extensions/qa-lab/",
];
const LEGACY_OMITTED_PRIVATE_QA_INVENTORY_FILES = new Set([
"dist/plugin-sdk/qa-channel.d.ts",
"dist/plugin-sdk/qa-channel.js",
"dist/plugin-sdk/qa-channel-protocol.d.ts",
"dist/plugin-sdk/qa-channel-protocol.js",
"dist/plugin-sdk/qa-lab.d.ts",
"dist/plugin-sdk/qa-lab.js",
"dist/plugin-sdk/qa-runtime.d.ts",
"dist/plugin-sdk/qa-runtime.js",
"dist/plugin-sdk/src/plugin-sdk/qa-channel.d.ts",
"dist/plugin-sdk/src/plugin-sdk/qa-channel-protocol.d.ts",
"dist/plugin-sdk/src/plugin-sdk/qa-lab.d.ts",
"dist/plugin-sdk/src/plugin-sdk/qa-runtime.d.ts",
]);
const LEGACY_OMITTED_QA_MATRIX_INVENTORY_PREFIXES = ["dist/extensions/qa-matrix/"];
function isLegacyOmittedPrivateQaInventoryEntry(relativePath) {
return (
LEGACY_OMITTED_PRIVATE_QA_INVENTORY_FILES.has(relativePath) ||
LEGACY_OMITTED_PRIVATE_QA_INVENTORY_PREFIXES.some((prefix) => relativePath.startsWith(prefix))
function isLegacyOmittedQaMatrixInventoryEntry(relativePath) {
return LEGACY_OMITTED_QA_MATRIX_INVENTORY_PREFIXES.some((prefix) =>
relativePath.startsWith(prefix),
);
}
@@ -256,7 +235,7 @@ if (!entrySet.has("dist/postinstall-inventory.json")) {
let packageDistImports = null;
if (entrySet.has("dist/postinstall-inventory.json")) {
try {
const allowLegacyPrivateQaInventoryOmissions =
const allowLegacyQaMatrixInventoryOmissions =
isLegacyPackageAcceptanceCompatVersion(packageVersion);
const inventory = JSON.parse(readTarEntry("dist/postinstall-inventory.json"));
if (!Array.isArray(inventory) || inventory.some((entry) => typeof entry !== "string")) {
@@ -274,11 +253,11 @@ if (entrySet.has("dist/postinstall-inventory.json")) {
const normalizedEntry = inventoryEntry.replace(/\\/gu, "/");
if (!entrySet.has(normalizedEntry)) {
if (
allowLegacyPrivateQaInventoryOmissions &&
isLegacyOmittedPrivateQaInventoryEntry(normalizedEntry)
allowLegacyQaMatrixInventoryOmissions &&
isLegacyOmittedQaMatrixInventoryEntry(normalizedEntry)
) {
warnings.push(
`legacy inventory references omitted private QA tar entry ${normalizedEntry}`,
`legacy inventory references omitted QA Matrix tar entry ${normalizedEntry}`,
);
continue;
}

2
scripts/e2e/telegram-user-crabbox-proof.ts
View File

@@ -450,7 +450,7 @@ function childProcessBaseEnv() {
"LC_ALL",
"NODE_OPTIONS",
"OPENCLAW_BUILD_PRIVATE_QA",
"OPENCLAW_ENABLE_PRIVATE_QA_CLI",
"OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI",
"PATH",
"PNPM_HOME",
"SHELL",

4
scripts/lib/bundled-plugin-build-entries.mjs
View File

@@ -10,8 +10,8 @@ import {
import { shouldBuildBundledCluster } from "./optional-bundled-clusters.mjs";
const TOP_LEVEL_PUBLIC_SURFACE_EXTENSIONS = new Set([".ts", ".js", ".mts", ".cts", ".mjs", ".cjs"]);
/** Bundled plugin directories built with core but not packaged as standalone npm plugins. */
export const NON_PACKAGED_BUNDLED_PLUGIN_DIRS = new Set(["qa-channel", "qa-lab", "qa-matrix"]);
/** Bundled plugin directories built with core but not packaged in the root npm package. */
export const NON_PACKAGED_BUNDLED_PLUGIN_DIRS = new Set(["qa-matrix"]);
const EXCLUDED_CORE_BUNDLED_PLUGIN_DIRS = new Set(["qqbot", "whatsapp"]);
const BUNDLED_PLUGIN_BUILD_IDS_ENV = "OPENCLAW_BUNDLED_PLUGIN_BUILD_IDS";
const TOP_LEVEL_PRIVATE_TEST_SURFACE_RE =

2
scripts/lib/bundled-runtime-sidecar-paths.json
View File

@@ -11,6 +11,8 @@
"dist/extensions/memory-core/runtime-api.js",
"dist/extensions/ollama/runtime-api.js",
"dist/extensions/open-prose/runtime-api.js",
"dist/extensions/qa-channel/runtime-api.js",
"dist/extensions/qa-lab/runtime-api.js",
"dist/extensions/signal/runtime-api.js",
"dist/extensions/telegram/runtime-api.js",
"dist/extensions/telegram/runtime-setter-api.js",

2
scripts/lib/plugin-gateway-gauntlet.mjs
View File

@@ -386,7 +386,7 @@ function buildGauntletPrebuildEnv(env, options = {}) {
PNPM_CONFIG_VERIFY_DEPS_BEFORE_RUN: env.PNPM_CONFIG_VERIFY_DEPS_BEFORE_RUN ?? "false",
...runtimeOnlyPrebuildEnv,
OPENCLAW_BUILD_PRIVATE_QA: "1",
OPENCLAW_ENABLE_PRIVATE_QA_CLI: "1",
OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: "1",
...(buildIds.size > 0
? {
OPENCLAW_BUNDLED_PLUGIN_BUILD_IDS: [...buildIds]

15
scripts/lib/plugin-sdk-doc-metadata.ts
View File

@@ -108,9 +108,24 @@ export const pluginSdkDocMetadata = {
"runtime-store": {
category: "runtime",
},
"qa-channel": {
category: "utilities",
},
"qa-channel-protocol": {
category: "utilities",
},
"qa-lab": {
category: "utilities",
},
"qa-runtime": {
category: "utilities",
},
"qa-live-transport-scenarios": {
category: "utilities",
},
"qa-user-flows": {
category: "utilities",
},
"agent-runtime": {
category: "runtime",
},

5
scripts/lib/plugin-sdk-entrypoints.json
View File

@@ -247,8 +247,13 @@
"json-store",
"persistent-dedupe",
"keyed-async-queue",
"qa-channel",
"qa-channel-protocol",
"qa-lab",
"qa-runtime",
"qa-runner-runtime",
"qa-live-transport-scenarios",
"qa-user-flows",
"memory-core",
"memory-core-engine-runtime",
"memory-core-host-embedding-registry",

4
scripts/lib/plugin-sdk-private-local-only-subpaths.json
View File

@@ -10,10 +10,6 @@
"plugin-test-runtime",
"provider-http-test-mocks",
"provider-test-contracts",
"qa-channel",
"qa-channel-protocol",
"qa-lab",
"qa-runtime",
"reply-payload-testing",
"ssrf-runtime-internal",
"test-env",

6
scripts/openclaw-cross-os-release-checks.ts
View File

@@ -150,11 +150,7 @@ function buildReleaseProviderConfigOverride(providerMeta) {
const PACKAGE_DIST_INVENTORY_RELATIVE_PATH = "dist/postinstall-inventory.json";
const INSTALL_STAGE_DEBRIS_DIR_PATTERN = /^\.openclaw-install-stage(?:-[^/]+)?$/iu;
const OMITTED_QA_EXTENSION_PREFIXES = [
"dist/extensions/qa-channel/",
"dist/extensions/qa-lab/",
"dist/extensions/qa-matrix/",
];
const OMITTED_QA_EXTENSION_PREFIXES = ["dist/extensions/qa-matrix/"];
export const CROSS_OS_DASHBOARD_SMOKE_TIMEOUT_MS = 120_000;
export const CROSS_OS_DASHBOARD_FETCH_TIMEOUT_MS = 10_000;
export const CROSS_OS_DISCORD_FETCH_TIMEOUT_MS = parsePositiveIntegerEnv(

90
scripts/openclaw-npm-release-check.ts
View File

@@ -3,7 +3,6 @@
import { execFileSync } from "node:child_process";
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { pathToFileURL } from "node:url";
import {
LOCAL_BUILD_METADATA_DIST_PATHS,
@@ -85,61 +84,7 @@ const FORBIDDEN_PACKED_PATH_RULES = [
describe: (packedPath: string) =>
`npm package must not include generated docs artifact "${packedPath}".`,
},
{
prefix: "docs/channels/qa-channel.md",
describe: (packedPath: string) =>
`npm package must not include private QA channel docs "${packedPath}".`,
},
{
prefix: "dist/extensions/qa-channel/",
describe: (packedPath: string) =>
`npm package must not include private QA channel artifact "${packedPath}".`,
},
{
prefix: "dist/extensions/qa-lab/",
describe: (packedPath: string) =>
`npm package must not include private QA lab artifact "${packedPath}".`,
},
{
prefix: "dist/plugin-sdk/extensions/qa-channel/",
describe: (packedPath: string) =>
`npm package must not include private QA channel type artifact "${packedPath}".`,
},
{
prefix: "dist/plugin-sdk/extensions/qa-lab/",
describe: (packedPath: string) =>
`npm package must not include private QA lab type artifact "${packedPath}".`,
},
{
prefix: "dist/plugin-sdk/qa-channel.",
describe: (packedPath: string) =>
`npm package must not include private QA channel SDK artifact "${packedPath}".`,
},
{
prefix: "dist/plugin-sdk/qa-channel-protocol.",
describe: (packedPath: string) =>
`npm package must not include private QA channel SDK artifact "${packedPath}".`,
},
{
prefix: "dist/qa-runtime-",
describe: (packedPath: string) =>
`npm package must not include private QA runtime chunk "${packedPath}".`,
},
{
prefix: "qa/",
describe: (packedPath: string) =>
`npm package must not include private QA suite artifact "${packedPath}".`,
},
] as const;
const FORBIDDEN_PRIVATE_QA_CONTENT_MARKERS = [
"//#region extensions/qa-lab/",
"qa-channel/runtime-api.js",
"qa-channel.js",
"qa-channel-protocol.js",
"qa-lab/cli.js",
"qa-lab/runtime-api.js",
] as const;
const FORBIDDEN_PRIVATE_QA_CONTENT_SCAN_PREFIXES = ["dist/"] as const;
const PACKED_TEST_CARGO_DIRECTORY_SEGMENTS = new Set([
"__snapshots__",
"__tests__",
@@ -688,7 +633,6 @@ function collectPackedTarballErrors(): string[] {
return [
...collectControlUiPackErrors(packedPaths),
...collectForbiddenPackedPathErrors(packedPaths),
...collectForbiddenPackedContentErrors(packedPaths),
...collectPackedTestCargoErrors(packedPaths),
];
}
@@ -723,40 +667,6 @@ export function collectForbiddenPackedPathErrors(paths: Iterable<string>): strin
return errors.toSorted((left, right) => left.localeCompare(right));
}
export function collectForbiddenPackedContentErrors(
paths: Iterable<string>,
rootDir = process.cwd(),
): string[] {
const textPathPattern = /\.(?:[cm]?js|d\.ts|json|md|mjs|cjs)$/u;
const errors: string[] = [];
for (const packedPath of paths) {
if (
!FORBIDDEN_PRIVATE_QA_CONTENT_SCAN_PREFIXES.some((prefix) => packedPath.startsWith(prefix))
) {
continue;
}
if (!textPathPattern.test(packedPath)) {
continue;
}
let content: string;
try {
content = readFileSync(pathToFileURL(join(rootDir, packedPath)), "utf8");
} catch {
continue;
}
const matchedMarker = FORBIDDEN_PRIVATE_QA_CONTENT_MARKERS.find((marker) =>
content.includes(marker),
);
if (!matchedMarker) {
continue;
}
errors.push(
`npm package must not include private QA lab marker "${matchedMarker}" in "${packedPath}".`,
);
}
return errors.toSorted((left, right) => left.localeCompare(right));
}
export function collectPackedTestCargoErrors(paths: Iterable<string>): string[] {
const errors: string[] = [];
for (const packedPath of paths) {

2
scripts/qa-e2e.ts
View File

@@ -3,7 +3,7 @@ import { pathToFileURL } from "node:url";
export function enablePrivateQaScriptEnv(env: NodeJS.ProcessEnv = process.env) {
env.OPENCLAW_BUILD_PRIVATE_QA = "1";
env.OPENCLAW_ENABLE_PRIVATE_QA_CLI = "1";
env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI = "1";
env.OPENCLAW_DISABLE_BUNDLED_PLUGINS = "0";
}

34
scripts/release-check.ts
View File

@@ -104,41 +104,18 @@ const forbiddenPrefixes = [
...LOCAL_BUILD_METADATA_DIST_PATHS,
"dist-runtime/",
"dist/OpenClaw.app/",
"dist/extensions/qa-channel/",
"dist/extensions/qa-lab/",
"dist/plugin-sdk/extensions/qa-channel/",
"dist/plugin-sdk/extensions/qa-lab/",
"dist/plugin-sdk/qa-channel.",
"dist/plugin-sdk/qa-channel-protocol.",
"dist/plugin-sdk/qa-lab.",
"dist/plugin-sdk/qa-runtime.",
"dist/plugin-sdk/src/",
"dist/plugin-sdk/src/plugin-sdk/qa-channel.d.ts",
"dist/plugin-sdk/src/plugin-sdk/qa-channel-protocol.d.ts",
"dist/plugin-sdk/src/plugin-sdk/qa-lab.d.ts",
"dist/plugin-sdk/src/plugin-sdk/qa-runtime.d.ts",
...listPrivateLocalOnlyPluginSdkDistArtifacts(),
"dist/qa-runtime-",
"dist/plugin-sdk/.tsbuildinfo",
"docs/.generated/",
"docs/channels/qa-channel.md",
"qa/",
];
const forbiddenPrivateQaContentMarkers = [
"//#region extensions/qa-lab/",
"qa-channel/runtime-api.js",
"qa-channel.js",
"qa-channel-protocol.js",
"qa-lab/cli.js",
"qa-lab/runtime-api.js",
] as const;
const forbiddenPrivatePluginSdkDeclarationMarkers = [
"//#region src/agents/test-helpers/",
"//#region src/plugin-sdk/test-helpers/",
"//#region src/test-helpers/",
"//#region src/test-utils/",
] as const;
const forbiddenPrivateQaContentScanPrefixes = ["dist/"] as const;
const forbiddenPackContentScanPrefixes = ["dist/"] as const;
const forbiddenPluginSdkRootAliasMinifiedExportPattern = /\bmod\.[A-Za-z_$]\b/u;
const appcastPath = resolve("appcast.xml");
const laneBuildMin = 1_000_000_000;
@@ -924,7 +901,7 @@ export function collectForbiddenPackContentPaths(
const textPathPattern = /\.(?:[cm]?js|d\.ts|json|md|mjs|cjs)$/u;
return [...paths]
.filter((packedPath) => {
if (!forbiddenPrivateQaContentScanPrefixes.some((prefix) => packedPath.startsWith(prefix))) {
if (!forbiddenPackContentScanPrefixes.some((prefix) => packedPath.startsWith(prefix))) {
return false;
}
if (!textPathPattern.test(packedPath)) {
@@ -936,10 +913,7 @@ export function collectForbiddenPackContentPaths(
} catch {
return false;
}
return (
forbiddenPrivateQaContentMarkers.some((marker) => content.includes(marker)) ||
forbiddenPrivatePluginSdkDeclarationMarkers.some((marker) => content.includes(marker))
);
return forbiddenPrivatePluginSdkDeclarationMarkers.some((marker) => content.includes(marker));
})
.toSorted((left, right) => left.localeCompare(right));
}
@@ -1202,7 +1176,7 @@ async function main() {
}
}
if (forbiddenContent.length > 0) {
console.error("release-check: forbidden private QA markers in npm pack:");
console.error("release-check: forbidden package content markers in npm pack:");
for (const path of forbiddenContent) {
console.error(` - ${path}`);
}

2
scripts/run-node.mjs
View File

@@ -1412,7 +1412,7 @@ export async function runNodeMain(params = {}) {
deps.privateQaRequiredDistEntries = resolvePrivateQaRequiredDistEntries(deps.distRoot);
if (deps.args[0] === "qa") {
deps.env.OPENCLAW_BUILD_PRIVATE_QA = "1";
deps.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI = "1";
deps.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI = "1";
deps.env.OPENCLAW_DISABLE_BUNDLED_PLUGINS ??= "0";
}
deps.outputTee = createRunNodeOutputTee(deps);

74
src/cli/program/private-qa-cli.test.ts
View File

@@ -1,34 +1,30 @@
// Private QA CLI tests cover private QA command registration and filesystem behavior.
// Experimental QA CLI tests cover QA command registration and filesystem behavior.
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import { loadPrivateQaCliModule } from "./private-qa-cli.js";
import { loadExperimentalQaCliModule } from "./private-qa-cli.js";
describe("private-qa-cli", () => {
describe("experimental QA CLI loader", () => {
const tempDirs: string[] = [];
const originalPrivateQaCli = process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI;
const originalExperimentalQaCli = process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI;
afterEach(() => {
for (const dir of tempDirs.splice(0)) {
fs.rmSync(dir, { recursive: true, force: true });
}
if (originalPrivateQaCli === undefined) {
delete process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI;
if (originalExperimentalQaCli === undefined) {
delete process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI;
} else {
process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI = originalPrivateQaCli;
process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI = originalExperimentalQaCli;
}
});
it("loads the private QA CLI from a source checkout path", async () => {
process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI = "1";
const repoRoot = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-private-qa-source-"));
it("loads the bundled QA CLI module when experimental QA is enabled", async () => {
process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI = "1";
const repoRoot = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-qa-bundle-"));
tempDirs.push(repoRoot);
const expectedPaths = new Set([
path.join(repoRoot, ".git"),
path.join(repoRoot, "src"),
path.join(repoRoot, "dist", "plugin-sdk", "qa-lab.js"),
]);
const expectedPaths = new Set([path.join(repoRoot, "dist", "plugin-sdk", "qa-lab.js")]);
let importedSpecifier: string | undefined;
const isQaLabCliAvailable = vi.fn();
const registerQaLabCli = vi.fn();
@@ -40,7 +36,7 @@ describe("private-qa-cli", () => {
};
});
const module = await loadPrivateQaCliModule({
const module = await loadExperimentalQaCliModule({
importModule,
resolvePackageRootSync: () => repoRoot,
existsSync: (filePath) => typeof filePath === "string" && expectedPaths.has(filePath),
@@ -52,55 +48,31 @@ describe("private-qa-cli", () => {
expect(module.registerQaLabCli).toBe(registerQaLabCli);
});
it("loads the private QA CLI from a raw synced source checkout path", async () => {
process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI = "1";
const repoRoot = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-private-qa-raw-source-"));
it("rejects when the bundled QA CLI module is missing", () => {
process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI = "1";
const repoRoot = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-qa-missing-bundle-"));
tempDirs.push(repoRoot);
const expectedPaths = new Set([
path.join(repoRoot, "pnpm-workspace.yaml"),
path.join(repoRoot, "src"),
path.join(repoRoot, "dist", "plugin-sdk", "qa-lab.js"),
]);
const importModule = vi.fn(async () => ({
isQaLabCliAvailable: vi.fn(),
registerQaLabCli: vi.fn(),
}));
await expect(
loadPrivateQaCliModule({
expect(() =>
loadExperimentalQaCliModule({
importModule,
resolvePackageRootSync: () => repoRoot,
existsSync: (filePath) => typeof filePath === "string" && expectedPaths.has(filePath),
existsSync: () => false,
}),
).resolves.toMatchObject({
isQaLabCliAvailable: expect.any(Function),
registerQaLabCli: expect.any(Function),
});
expect(importModule).toHaveBeenCalledTimes(1);
});
it("rejects non-source package roots even when private QA is enabled", () => {
process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI = "1";
const root = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-private-qa-"));
tempDirs.push(root);
fs.writeFileSync(path.join(root, "package.json"), JSON.stringify({ name: "openclaw" }), "utf8");
const importModule = vi.fn(async () => ({}));
expect(() =>
loadPrivateQaCliModule({
resolvePackageRootSync: () => root,
importModule,
}),
).toThrow("Private QA CLI is only available from an OpenClaw source checkout.");
).toThrow("bundled QA Lab CLI module");
expect(importModule).not.toHaveBeenCalled();
});
it("rejects when the private QA env flag is disabled", () => {
delete process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI;
it("rejects when the experimental QA env flag is disabled", () => {
delete process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI;
const importModule = vi.fn(async () => ({}));
expect(() => loadPrivateQaCliModule({ importModule })).toThrow(
"Private QA CLI is only available from an OpenClaw source checkout.",
expect(() => loadExperimentalQaCliModule({ importModule })).toThrow(
"OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI=1",
);
expect(importModule).not.toHaveBeenCalled();
});

42
src/cli/program/private-qa-cli.ts
View File

@@ -1,18 +1,17 @@
// Private QA CLI loader, enabled only from source checkouts and explicit env opt-in.
// Experimental QA CLI loader, enabled by explicit env opt-in.
import fs from "node:fs";
import path from "node:path";
import { pathToFileURL } from "node:url";
import { resolveOpenClawPackageRootSync } from "../../infra/openclaw-root.js";
const PRIVATE_QA_DIST_RELATIVE_PATH = path.join("dist", "plugin-sdk", "qa-lab.js");
const SOURCE_CHECKOUT_MARKER_RELATIVE_PATHS = [".git", "pnpm-workspace.yaml"] as const;
const QA_LAB_DIST_RELATIVE_PATH = path.join("dist", "plugin-sdk", "qa-lab.js");
/** Return true when private QA CLI routes should be exposed. */
export function isPrivateQaCliEnabled(env: NodeJS.ProcessEnv = process.env): boolean {
return env.OPENCLAW_ENABLE_PRIVATE_QA_CLI === "1";
/** Return true when experimental QA CLI routes should be exposed. */
export function isExperimentalQaCliEnabled(env: NodeJS.ProcessEnv = process.env): boolean {
return env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI === "1";
}
function resolvePrivateQaSourceModuleSpecifier(params?: {
function resolveQaLabModuleSpecifier(params?: {
env?: NodeJS.ProcessEnv;
cwd?: string;
argv1?: string;
@@ -21,7 +20,7 @@ function resolvePrivateQaSourceModuleSpecifier(params?: {
existsSync?: typeof fs.existsSync;
}): string | null {
const env = params?.env ?? process.env;
if (!isPrivateQaCliEnabled(env)) {
if (!isExperimentalQaCliEnabled(env)) {
return null;
}
const resolvePackageRootSync = params?.resolvePackageRootSync ?? resolveOpenClawPackageRootSync;
@@ -34,28 +33,21 @@ function resolvePrivateQaSourceModuleSpecifier(params?: {
return null;
}
const existsSync = params?.existsSync ?? fs.existsSync;
const sourceModulePath = path.join(packageRoot, PRIVATE_QA_DIST_RELATIVE_PATH);
const hasSourceCheckoutMarker = SOURCE_CHECKOUT_MARKER_RELATIVE_PATHS.some((relativePath) =>
existsSync(path.join(packageRoot, relativePath)),
);
if (
!hasSourceCheckoutMarker ||
!existsSync(path.join(packageRoot, "src")) ||
!existsSync(sourceModulePath)
) {
const modulePath = path.join(packageRoot, QA_LAB_DIST_RELATIVE_PATH);
if (!existsSync(modulePath)) {
return null;
}
return pathToFileURL(sourceModulePath).href;
return pathToFileURL(modulePath).href;
}
async function dynamicImportPrivateQaCliModule(
async function dynamicImportExperimentalQaCliModule(
specifier: string,
): Promise<Record<string, unknown>> {
return (await import(specifier)) as Record<string, unknown>;
}
/** Load the private QA module from a source checkout or throw a user-facing availability error. */
export function loadPrivateQaCliModule(params?: {
/** Load the experimental QA module or throw a user-facing availability error. */
export function loadExperimentalQaCliModule(params?: {
env?: NodeJS.ProcessEnv;
cwd?: string;
argv1?: string;
@@ -64,9 +56,11 @@ export function loadPrivateQaCliModule(params?: {
existsSync?: typeof fs.existsSync;
importModule?: (specifier: string) => Promise<Record<string, unknown>>;
}): Promise<Record<string, unknown>> {
const specifier = resolvePrivateQaSourceModuleSpecifier(params);
const specifier = resolveQaLabModuleSpecifier(params);
if (!specifier) {
throw new Error("Private QA CLI is only available from an OpenClaw source checkout.");
throw new Error(
"Experimental QA CLI requires OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI=1 and a bundled QA Lab CLI module.",
);
}
return (params?.importModule ?? dynamicImportPrivateQaCliModule)(specifier);
return (params?.importModule ?? dynamicImportExperimentalQaCliModule)(specifier);
}

4
src/cli/program/register.subclis-core.ts
View File

@@ -13,7 +13,7 @@ import {
type CommandGroupDescriptorSpec,
} from "./command-group-descriptors.js";
import { removeCommandByName } from "./command-tree.js";
import { loadPrivateQaCliModule } from "./private-qa-cli.js";
import { loadExperimentalQaCliModule } from "./private-qa-cli.js";
import {
registerCommandGroupByName,
registerCommandGroups,
@@ -182,7 +182,7 @@ const entrySpecs: readonly CommandGroupDescriptorSpec<SubCliRegistrar>[] = [
},
{
commandNames: ["qa"],
loadModule: loadPrivateQaCliModule,
loadModule: loadExperimentalQaCliModule,
exportName: "registerQaLabCli",
},
{

22
src/cli/program/register.subclis.test.ts
View File

@@ -26,8 +26,8 @@ const { registerQaLabCli } = vi.hoisted(() => ({
qa.command("run").action(() => undefined);
}),
}));
const { loadPrivateQaCliModule } = vi.hoisted(() => ({
loadPrivateQaCliModule: vi.fn(async () => ({ registerQaLabCli })),
const { loadExperimentalQaCliModule } = vi.hoisted(() => ({
loadExperimentalQaCliModule: vi.fn(async () => ({ registerQaLabCli })),
}));
const { inferAction, registerCapabilityCli } = vi.hoisted(() => {
@@ -79,14 +79,14 @@ vi.mock("./private-qa-cli.js", async () => {
const actual = await vi.importActual<typeof import("./private-qa-cli.js")>("./private-qa-cli.js");
return {
...actual,
loadPrivateQaCliModule,
loadExperimentalQaCliModule,
};
});
describe("registerSubCliCommands", () => {
const originalArgv = process.argv;
const originalDisableLazySubcommands = process.env.OPENCLAW_DISABLE_LAZY_SUBCOMMANDS;
const originalEnablePrivateQaCli = process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI;
const originalEnableExperimentalQaCli = process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI;
const createRegisteredProgram = (argv: string[], name?: string) => {
process.argv = argv;
@@ -104,13 +104,13 @@ describe("registerSubCliCommands", () => {
} else {
process.env.OPENCLAW_DISABLE_LAZY_SUBCOMMANDS = originalDisableLazySubcommands;
}
process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI = "1";
process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI = "1";
registerAcpCli.mockClear();
acpAction.mockClear();
registerNodesCli.mockClear();
nodesAction.mockClear();
registerQaLabCli.mockClear();
loadPrivateQaCliModule.mockClear();
loadExperimentalQaCliModule.mockClear();
registerCapabilityCli.mockClear();
inferAction.mockClear();
registerPluginsCli.mockClear();
@@ -128,10 +128,10 @@ describe("registerSubCliCommands", () => {
} else {
process.env.OPENCLAW_DISABLE_LAZY_SUBCOMMANDS = originalDisableLazySubcommands;
}
if (originalEnablePrivateQaCli === undefined) {
delete process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI;
if (originalEnableExperimentalQaCli === undefined) {
delete process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI;
} else {
process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI = originalEnablePrivateQaCli;
process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI = originalEnableExperimentalQaCli;
}
});
@@ -157,8 +157,8 @@ describe("registerSubCliCommands", () => {
expect(registerAcpCli).not.toHaveBeenCalled();
});
it("omits the qa placeholder when the private qa cli is disabled", () => {
delete process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI;
it("omits the qa placeholder when the experimental QA CLI is disabled", () => {
delete process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI;
const program = createRegisteredProgram(["node", "openclaw"]);

20
src/cli/program/subcli-descriptors.test.ts
View File

@@ -11,19 +11,19 @@ function descriptorNames(descriptors: ReadonlyArray<{ name: string }>): string[]
}
describe("sub-cli descriptors", () => {
const originalPrivateQaCli = process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI;
const originalExperimentalQaCli = process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI;
afterEach(() => {
if (originalPrivateQaCli === undefined) {
delete process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI;
if (originalExperimentalQaCli === undefined) {
delete process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI;
} else {
process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI = originalPrivateQaCli;
process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI = originalExperimentalQaCli;
}
vi.resetModules();
});
it("keeps the exported descriptor list aligned with private QA visibility when disabled (#83927)", async () => {
delete process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI;
it("keeps the exported descriptor list aligned with experimental QA visibility when disabled", async () => {
delete process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI;
const { SUB_CLI_DESCRIPTORS, getSubCliEntries } = await importSubCliDescriptors();
const exportedNames = descriptorNames(SUB_CLI_DESCRIPTORS);
@@ -32,8 +32,8 @@ describe("sub-cli descriptors", () => {
expect(exportedNames).not.toContain("qa");
});
it("keeps all sub-cli filter surfaces aligned when private QA is disabled (#83926)", async () => {
delete process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI;
it("keeps all sub-cli filter surfaces aligned when experimental QA is disabled", async () => {
delete process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI;
const {
SUB_CLI_DESCRIPTORS,
@@ -47,8 +47,8 @@ describe("sub-cli descriptors", () => {
expect(getSubCliParentDefaultHelpCommands()).not.toContain("qa");
});
it("includes qa in the exported descriptor list when private QA is enabled", async () => {
process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI = "1";
it("includes qa in the exported descriptor list when experimental QA is enabled", async () => {
process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI = "1";
const {
SUB_CLI_DESCRIPTORS,

18
src/cli/program/subcli-descriptors.ts
View File

@@ -1,7 +1,7 @@
// Sub-CLI descriptor catalog used for root help placeholders and lazy registration.
import { defineCommandDescriptorCatalog } from "./command-descriptor-utils.js";
import type { NamedCommandDescriptor } from "./command-group-descriptors.js";
import { isPrivateQaCliEnabled } from "./private-qa-cli.js";
import { isExperimentalQaCliEnabled } from "./private-qa-cli.js";
/** Descriptor shape for root-level sub-CLI commands. */
export type SubCliDescriptor = NamedCommandDescriptor;
@@ -104,7 +104,7 @@ const subCliCommandCatalog = defineCommandDescriptorCatalog([
},
{
name: "qa",
description: "Run QA scenarios and launch the private QA debugger UI",
description: "Run experimental QA scenarios and user flows",
hasSubcommands: true,
},
{
@@ -181,25 +181,25 @@ const subCliCommandCatalog = defineCommandDescriptorCatalog([
},
] as const satisfies ReadonlyArray<SubCliDescriptor>);
function filterPrivateQaItems<T>(
function filterExperimentalQaItems<T>(
items: ReadonlyArray<T>,
getName: (item: T) => string,
): ReadonlyArray<T> {
if (isPrivateQaCliEnabled()) {
if (isExperimentalQaCliEnabled()) {
return items;
}
return items.filter((item) => getName(item) !== "qa");
}
/** Visible sub-CLI descriptors after private QA gating. */
export const SUB_CLI_DESCRIPTORS = filterPrivateQaItems(
/** Visible sub-CLI descriptors after experimental QA gating. */
export const SUB_CLI_DESCRIPTORS = filterExperimentalQaItems(
subCliCommandCatalog.descriptors,
(descriptor) => descriptor.name,
);
/** Return visible sub-CLI descriptors in help/registration order. */
export function getSubCliEntries(): ReadonlyArray<SubCliDescriptor> {
return filterPrivateQaItems(
return filterExperimentalQaItems(
subCliCommandCatalog.getDescriptors(),
(descriptor) => descriptor.name,
);
@@ -208,7 +208,7 @@ export function getSubCliEntries(): ReadonlyArray<SubCliDescriptor> {
/** Return visible sub-CLI names that own child subcommands. */
export function getSubCliCommandsWithSubcommands(): string[] {
return [
...filterPrivateQaItems(
...filterExperimentalQaItems(
subCliCommandCatalog.getCommandsWithSubcommands(),
(command) => command,
),
@@ -218,7 +218,7 @@ export function getSubCliCommandsWithSubcommands(): string[] {
/** Return visible sub-CLI names whose parent command should show help by default. */
export function getSubCliParentDefaultHelpCommands(): string[] {
return [
...filterPrivateQaItems(
...filterExperimentalQaItems(
subCliCommandCatalog.getParentDefaultHelpCommands(),
(command) => command,
),

41
src/infra/package-dist-inventory.test.ts
View File

@@ -40,7 +40,7 @@ describe("package dist inventory", () => {
});
});
it("keeps npm-omitted dist artifacts out of the inventory", async () => {
it("keeps omitted dist artifacts out of the inventory while tracking packaged QA facades", async () => {
await withTempDir({ prefix: "openclaw-dist-inventory-pack-" }, async (packageRoot) => {
const packagedQaChannelRuntime = path.join(
packageRoot,
@@ -56,8 +56,8 @@ describe("package dist inventory", () => {
"qa-lab",
"runtime-api.js",
);
const omittedQaChunk = path.join(packageRoot, "dist", "extensions", "qa-channel", "cli.js");
const omittedQaLabChunk = path.join(packageRoot, "dist", "extensions", "qa-lab", "cli.js");
const packagedQaChunk = path.join(packageRoot, "dist", "extensions", "qa-channel", "cli.js");
const packagedQaLabChunk = path.join(packageRoot, "dist", "extensions", "qa-lab", "cli.js");
const omittedQaMatrixChunk = path.join(
packageRoot,
"dist",
@@ -65,20 +65,20 @@ describe("package dist inventory", () => {
"qa-matrix",
"index.js",
);
const omittedQaLabPluginSdk = path.join(packageRoot, "dist", "plugin-sdk", "qa-lab.js");
const omittedQaChannelPluginSdk = path.join(
const packagedQaLabPluginSdk = path.join(packageRoot, "dist", "plugin-sdk", "qa-lab.js");
const packagedQaChannelPluginSdk = path.join(
packageRoot,
"dist",
"plugin-sdk",
"qa-channel.js",
);
const omittedQaChannelProtocolPluginSdk = path.join(
const packagedQaChannelProtocolPluginSdk = path.join(
packageRoot,
"dist",
"plugin-sdk",
"qa-channel-protocol.js",
);
const omittedQaLabTypes = path.join(
const packagedQaLabTypes = path.join(
packageRoot,
"dist",
"plugin-sdk",
@@ -100,7 +100,7 @@ describe("package dist inventory", () => {
"plugin-sdk",
"provider-entry.d.ts",
);
const omittedQaRuntimeChunk = path.join(packageRoot, "dist", "qa-runtime-B9LDtssJ.js");
const packagedQaRuntimeChunk = path.join(packageRoot, "dist", "qa-runtime-B9LDtssJ.js");
const [omittedBuildStamp, omittedRuntimePostBuildStamp] = LOCAL_BUILD_METADATA_DIST_PATHS.map(
(relativePath) => path.join(packageRoot, relativePath),
);
@@ -108,27 +108,36 @@ describe("package dist inventory", () => {
await fs.mkdir(path.dirname(packagedQaChannelRuntime), { recursive: true });
await fs.mkdir(path.dirname(packagedQaLabRuntime), { recursive: true });
await fs.mkdir(path.dirname(omittedQaMatrixChunk), { recursive: true });
await fs.mkdir(path.dirname(omittedQaLabTypes), { recursive: true });
await fs.mkdir(path.dirname(packagedQaLabTypes), { recursive: true });
await fs.mkdir(path.join(packageRoot, "dist", "plugin-sdk"), { recursive: true });
await fs.mkdir(path.dirname(omittedDeepPluginSdkDeclaration), { recursive: true });
await fs.writeFile(packagedQaChannelRuntime, "export {};\n", "utf8");
await fs.writeFile(packagedQaLabRuntime, "export {};\n", "utf8");
await fs.writeFile(omittedQaChunk, "export {};\n", "utf8");
await fs.writeFile(omittedQaLabChunk, "export {};\n", "utf8");
await fs.writeFile(packagedQaChunk, "export {};\n", "utf8");
await fs.writeFile(packagedQaLabChunk, "export {};\n", "utf8");
await fs.writeFile(omittedQaMatrixChunk, "export {};\n", "utf8");
await fs.writeFile(omittedQaLabPluginSdk, "export {};\n", "utf8");
await fs.writeFile(omittedQaChannelPluginSdk, "export {};\n", "utf8");
await fs.writeFile(omittedQaChannelProtocolPluginSdk, "export {};\n", "utf8");
await fs.writeFile(omittedQaLabTypes, "export {};\n", "utf8");
await fs.writeFile(packagedQaLabPluginSdk, "export {};\n", "utf8");
await fs.writeFile(packagedQaChannelPluginSdk, "export {};\n", "utf8");
await fs.writeFile(packagedQaChannelProtocolPluginSdk, "export {};\n", "utf8");
await fs.writeFile(packagedQaLabTypes, "export {};\n", "utf8");
await fs.writeFile(omittedDeepPluginSdkDeclaration, "export {};\n", "utf8");
await fs.writeFile(flatPluginSdkDeclaration, "export {};\n", "utf8");
await fs.writeFile(omittedQaRuntimeChunk, "export {};\n", "utf8");
await fs.writeFile(packagedQaRuntimeChunk, "export {};\n", "utf8");
await fs.writeFile(omittedBuildStamp, "{}\n", "utf8");
await fs.writeFile(omittedRuntimePostBuildStamp, "{}\n", "utf8");
await fs.writeFile(omittedMap, "{}", "utf8");
await expect(writePackageDistInventory(packageRoot)).resolves.toStrictEqual([
"dist/extensions/qa-channel/cli.js",
"dist/extensions/qa-channel/runtime-api.js",
"dist/extensions/qa-lab/cli.js",
"dist/extensions/qa-lab/runtime-api.js",
"dist/plugin-sdk/extensions/qa-lab/cli.d.ts",
"dist/plugin-sdk/provider-entry.d.ts",
"dist/plugin-sdk/qa-channel-protocol.js",
"dist/plugin-sdk/qa-channel.js",
"dist/plugin-sdk/qa-lab.js",
"dist/qa-runtime-B9LDtssJ.js",
]);
});
});

36
src/infra/package-dist-inventory.ts
View File

@@ -9,35 +9,10 @@ export { LOCAL_BUILD_METADATA_DIST_PATHS } from "../../scripts/lib/local-build-m
export const PACKAGE_DIST_INVENTORY_RELATIVE_PATH = "dist/postinstall-inventory.json";
const PACKAGE_DIST_INVENTORY_SCAN_CONCURRENCY = 32;
const LEGACY_QA_CHANNEL_DIR = ["qa", "channel"].join("-");
const LEGACY_QA_LAB_DIR = ["qa", "lab"].join("-");
const OMITTED_QA_EXTENSION_PREFIXES = [
`dist/extensions/${LEGACY_QA_CHANNEL_DIR}/`,
`dist/extensions/${LEGACY_QA_LAB_DIR}/`,
"dist/extensions/qa-matrix/",
];
const OMITTED_PRIVATE_QA_PLUGIN_SDK_PREFIXES = [
`dist/plugin-sdk/extensions/${LEGACY_QA_CHANNEL_DIR}/`,
`dist/plugin-sdk/extensions/${LEGACY_QA_LAB_DIR}/`,
];
const OMITTED_PRIVATE_QA_PLUGIN_SDK_FILES = new Set([
`dist/plugin-sdk/${LEGACY_QA_CHANNEL_DIR}.d.ts`,
`dist/plugin-sdk/${LEGACY_QA_CHANNEL_DIR}.js`,
`dist/plugin-sdk/${LEGACY_QA_CHANNEL_DIR}-protocol.d.ts`,
`dist/plugin-sdk/${LEGACY_QA_CHANNEL_DIR}-protocol.js`,
`dist/plugin-sdk/${LEGACY_QA_LAB_DIR}.d.ts`,
`dist/plugin-sdk/${LEGACY_QA_LAB_DIR}.js`,
"dist/plugin-sdk/qa-runtime.d.ts",
"dist/plugin-sdk/qa-runtime.js",
`dist/plugin-sdk/src/plugin-sdk/${LEGACY_QA_CHANNEL_DIR}.d.ts`,
`dist/plugin-sdk/src/plugin-sdk/${LEGACY_QA_CHANNEL_DIR}-protocol.d.ts`,
`dist/plugin-sdk/src/plugin-sdk/${LEGACY_QA_LAB_DIR}.d.ts`,
"dist/plugin-sdk/src/plugin-sdk/qa-runtime.d.ts",
]);
const OMITTED_QA_EXTENSION_PREFIXES = ["dist/extensions/qa-matrix/"];
// The build keeps source-shaped SDK declarations for local boundary projects,
// but the npm package ships flat declarations and must not inventory the old tree.
const OMITTED_DEEP_PLUGIN_SDK_DECLARATION_PREFIX = "dist/plugin-sdk/src/";
const OMITTED_PRIVATE_QA_DIST_PREFIXES = ["dist/qa-runtime-"];
const OMITTED_PLUGIN_SDK_TEST_FILES = new Set([
"dist/plugin-sdk/agent-runtime-test-contracts.d.ts",
"dist/plugin-sdk/agent-runtime-test-contracts.js",
@@ -77,8 +52,6 @@ const OMITTED_DIST_SUBTREE_PATTERNS = [
/^dist\/extensions\/[^/]+\/node_modules(?:\/|$)/u,
/^dist\/extensions\/qa-matrix(?:\/|$)/u,
/^dist\/plugin-sdk\/src(?:\/|$)/u,
new RegExp(`^dist/plugin-sdk/extensions/${LEGACY_QA_CHANNEL_DIR}(?:/|$)`, "u"),
new RegExp(`^dist/plugin-sdk/extensions/${LEGACY_QA_LAB_DIR}(?:/|$)`, "u"),
] as const;
const INSTALL_STAGE_DEBRIS_DIR_PATTERN = /^\.openclaw-install-stage(?:-[^/]+)?$/iu;
type ExternalizedBundledExtensionIds = ReadonlySet<string>;
@@ -314,13 +287,6 @@ function isPackagedDistPath(relativePath: string, rules: PackageDistInventoryRul
if (relativePath.startsWith(OMITTED_DEEP_PLUGIN_SDK_DECLARATION_PREFIX)) {
return false;
}
if (
OMITTED_PRIVATE_QA_PLUGIN_SDK_PREFIXES.some((prefix) => relativePath.startsWith(prefix)) ||
OMITTED_PRIVATE_QA_PLUGIN_SDK_FILES.has(relativePath) ||
OMITTED_PRIVATE_QA_DIST_PREFIXES.some((prefix) => relativePath.startsWith(prefix))
) {
return false;
}
if (OMITTED_QA_EXTENSION_PREFIXES.some((prefix) => relativePath.startsWith(prefix))) {
return false;
}

2
src/infra/run-node.test.ts
View File

@@ -1060,7 +1060,7 @@ describe("run-node script", () => {
| undefined;
expect(postBuildParams?.cwd).toBe(tmp);
expect(postBuildParams?.env?.OPENCLAW_BUILD_PRIVATE_QA).toBe("1");
expect(postBuildParams?.env?.OPENCLAW_ENABLE_PRIVATE_QA_CLI).toBe("1");
expect(postBuildParams?.env?.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI).toBe("1");
expect(postBuildParams?.env?.OPENCLAW_DISABLE_BUNDLED_PLUGINS).toBe("0");
});
});

4
src/plugin-sdk/entrypoints.ts
View File

@@ -59,7 +59,10 @@ export const supportedBundledFacadeSdkEntrypoints = [
"mattermost",
"memory-core-engine-runtime",
"provider-zai-endpoint",
"qa-channel",
"qa-lab",
"qa-runner-runtime",
"qa-runtime",
"telegram-account",
"tts-runtime",
"zalouser",
@@ -89,6 +92,7 @@ export const publicPluginOwnedSdkEntrypoints = [
"memory-host-markdown",
"memory-host-search",
"memory-host-status",
"qa-channel-protocol",
"speech-core",
"telegram-command-config",
"video-generation-core",

2
src/plugin-sdk/private-qa-bundled-env.ts
View File

@@ -9,7 +9,7 @@ import { resolveOpenClawPackageRootSync } from "../infra/openclaw-root.js";
export function resolvePrivateQaBundledPluginsEnv(
env: NodeJS.ProcessEnv = process.env,
): NodeJS.ProcessEnv | undefined {
if (env.OPENCLAW_ENABLE_PRIVATE_QA_CLI !== "1") {
if (env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI !== "1") {
return undefined;
}
const packageRoot = resolveOpenClawPackageRootSync({

198
src/plugin-sdk/qa-live-transport-scenarios.ts
View File

@@ -1,3 +1,11 @@
import {
planQaUserFlows,
type QaStandardUserFlowId,
type QaUserFlowDefinition,
type QaUserFlowPlan,
type QaUserFlowSkipReason,
} from "./qa-user-flows.js";
/** Standard live-transport behavior buckets used to compare channel QA suites. */
export type LiveTransportStandardScenarioId =
| "canary"
@@ -10,6 +18,19 @@ export type LiveTransportStandardScenarioId =
| "reaction-observation"
| "help-command";
/** Channel capability atoms used to plan standard live transport scenarios. */
export type LiveTransportScenarioCapabilityId =
| "inbound-message"
| "inbound-reaction"
| "inbound-thread"
| "mention-gating"
| "native-help-command"
| "outbound-final-reply"
| "outbound-thread-reply"
| "sender-allowlist"
| "top-level-reply"
| "runtime-restart";
/** Transport-specific live QA scenario with optional mapping to a standard behavior bucket. */
export type LiveTransportScenarioDefinition<TId extends string = string> = {
/** Transport-specific scenario id accepted by CLI scenario filters. */
@@ -22,59 +43,121 @@ export type LiveTransportScenarioDefinition<TId extends string = string> = {
title: string;
};
type LiveTransportStandardScenarioDefinition = {
export type LiveTransportStandardScenarioDefinition = {
description: string;
id: LiveTransportStandardScenarioId;
requiredCapabilities: readonly LiveTransportScenarioCapabilityId[];
title: string;
userFlowId?: QaStandardUserFlowId;
} & Pick<QaUserFlowDefinition, "action" | "contracts" | "surface">;
export type LiveTransportStandardScenarioPlanEntry = LiveTransportStandardScenarioDefinition & {
missingCapabilities: readonly LiveTransportScenarioCapabilityId[];
};
const LIVE_TRANSPORT_STANDARD_SCENARIOS: readonly LiveTransportStandardScenarioDefinition[] = [
{
id: "canary",
title: "Transport canary",
description: "The lane can trigger one known-good reply on the real transport.",
},
{
id: "mention-gating",
title: "Mention gating",
description: "Messages without the required mention do not trigger a reply.",
},
{
id: "allowlist-block",
title: "Sender allowlist block",
description: "Non-allowlisted senders do not trigger a reply.",
},
{
id: "top-level-reply-shape",
title: "Top-level reply shape",
description: "Top-level replies stay top-level when the lane is configured that way.",
},
{
id: "restart-resume",
title: "Restart resume",
description: "The lane still responds after a gateway restart.",
},
{
id: "thread-follow-up",
title: "Thread follow-up",
description: "Threaded prompts receive threaded replies with the expected relation metadata.",
},
{
id: "thread-isolation",
title: "Thread isolation",
description: "Fresh top-level prompts stay out of prior threads.",
},
{
id: "reaction-observation",
title: "Reaction observation",
description: "Reaction events are observed and normalized correctly.",
},
{
id: "help-command",
title: "Help command",
description: "The transport-specific help command path replies successfully.",
},
] as const;
export type LiveTransportStandardScenarioSkipReason = QaUserFlowSkipReason;
export type LiveTransportStandardScenarioPlan =
QaUserFlowPlan<LiveTransportStandardScenarioDefinition>;
export const LIVE_TRANSPORT_STANDARD_SCENARIOS: readonly LiveTransportStandardScenarioDefinition[] =
[
{
id: "canary",
title: "Transport canary",
description: "The lane can trigger one known-good reply on the real transport.",
surface: "messaging",
action: { actor: "user", verb: "send", object: "message" },
contracts: [{ family: "channel", name: "base channel plugin contract" }],
requiredCapabilities: ["inbound-message", "outbound-final-reply"],
userFlowId: "messaging.direct-reply",
},
{
id: "mention-gating",
title: "Mention gating",
description: "Messages without the required mention do not trigger a reply.",
surface: "messaging",
action: { actor: "user", verb: "send", object: "unmentioned group message" },
contracts: [{ family: "channel", name: "mention gating contract" }],
requiredCapabilities: ["inbound-message", "mention-gating"],
userFlowId: "messaging.mention-gating",
},
{
id: "allowlist-block",
title: "Sender allowlist block",
description: "Non-allowlisted senders do not trigger a reply.",
surface: "security",
action: { actor: "user", verb: "send", object: "blocked sender message" },
contracts: [{ family: "channel", name: "sender allowlist contract" }],
requiredCapabilities: ["inbound-message", "sender-allowlist"],
userFlowId: "security.sender-allowlist-block",
},
{
id: "top-level-reply-shape",
title: "Top-level reply shape",
description: "Top-level replies stay top-level when the lane is configured that way.",
surface: "messaging",
action: { actor: "user", verb: "send", object: "top-level message" },
contracts: [{ family: "channel", name: "reply shape contract" }],
requiredCapabilities: ["inbound-message", "outbound-final-reply", "top-level-reply"],
userFlowId: "messaging.direct-reply",
},
{
id: "restart-resume",
title: "Restart resume",
description: "The lane still responds after a gateway restart.",
surface: "recovery",
action: { actor: "system", verb: "restart", object: "gateway" },
contracts: [{ family: "gateway", name: "restart recovery contract" }],
requiredCapabilities: ["inbound-message", "outbound-final-reply", "runtime-restart"],
userFlowId: "recovery.restart-resume",
},
{
id: "thread-follow-up",
title: "Thread follow-up",
description: "Threaded prompts receive threaded replies with the expected relation metadata.",
surface: "messaging",
action: { actor: "user", verb: "reply", object: "thread" },
contracts: [{ family: "channel", name: "thread binding contract" }],
requiredCapabilities: ["inbound-message", "inbound-thread", "outbound-thread-reply"],
userFlowId: "messaging.thread-follow-up",
},
{
id: "thread-isolation",
title: "Thread isolation",
description: "Fresh top-level prompts stay out of prior threads.",
surface: "messaging",
action: { actor: "user", verb: "send", object: "fresh top-level message" },
contracts: [{ family: "channel", name: "thread isolation contract" }],
requiredCapabilities: [
"inbound-message",
"inbound-thread",
"outbound-final-reply",
"top-level-reply",
],
userFlowId: "messaging.thread-follow-up",
},
{
id: "reaction-observation",
title: "Reaction observation",
description: "Reaction events are observed and normalized correctly.",
surface: "messaging",
action: { actor: "user", verb: "react", object: "message" },
contracts: [{ family: "channel", name: "message actions contract" }],
requiredCapabilities: ["inbound-reaction"],
userFlowId: "messaging.reaction-edit-delete",
},
{
id: "help-command",
title: "Help command",
description: "The transport-specific help command path replies successfully.",
surface: "setup",
action: { actor: "user", verb: "request", object: "native help command" },
contracts: [{ family: "channel", name: "native command contract" }],
requiredCapabilities: ["inbound-message", "native-help-command", "outbound-final-reply"],
userFlowId: "setup.native-help-command",
},
] as const;
/** Minimum standard scenarios expected from baseline live transport suites. */
export const LIVE_TRANSPORT_BASELINE_STANDARD_SCENARIO_IDS: readonly LiveTransportStandardScenarioId[] =
@@ -158,3 +241,22 @@ export function findMissingLiveTransportStandardScenarios(params: {
const covered = new Set(params.coveredStandardScenarioIds);
return params.expectedStandardScenarioIds.filter((id) => !covered.has(id));
}
/** Plans standard live transport scenarios from channel capabilities and driver support. */
export function planLiveTransportStandardScenarios(params: {
availableCapabilities: readonly LiveTransportScenarioCapabilityId[];
driverSupportedScenarioIds?: readonly LiveTransportStandardScenarioId[];
requestedScenarioIds?: readonly LiveTransportStandardScenarioId[];
}): LiveTransportStandardScenarioPlan {
assertKnownStandardScenarioIds(params.driverSupportedScenarioIds ?? []);
assertKnownStandardScenarioIds(params.requestedScenarioIds ?? []);
return planQaUserFlows({
flows: LIVE_TRANSPORT_STANDARD_SCENARIOS,
availableCapabilities: params.availableCapabilities,
...(params.driverSupportedScenarioIds
? { driverSupportedFlowIds: params.driverSupportedScenarioIds }
: {}),
...(params.requestedScenarioIds ? { requestedFlowIds: params.requestedScenarioIds } : {}),
});
}

28
src/plugin-sdk/qa-runner-runtime.test.ts
View File

@@ -6,10 +6,10 @@ import type { Command } from "commander";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import {
cleanupTempDirs,
expectPrivateQaLabRuntimeSurfaceLoad,
expectExperimentalQaLabRuntimeSurfaceLoad,
expectQaLabRuntimeSurfaceLoad,
makePrivateQaSourceRoot,
restorePrivateQaCliEnv,
makeExperimentalQaSourceRoot,
restoreExperimentalQaCliEnv,
} from "./qa-runtime.test-helpers.js";
const loadPluginManifestRegistry = vi.hoisted(() => vi.fn());
@@ -49,7 +49,7 @@ function firstPublicSurfaceCall(): PublicSurfaceCall | undefined {
describe("plugin-sdk qa-runner-runtime", () => {
const tempDirs: string[] = [];
const originalPrivateQaCli = process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI;
const originalExperimentalQaCli = process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI;
beforeEach(() => {
loadPluginManifestRegistry.mockReset().mockReturnValue({
@@ -59,12 +59,12 @@ describe("plugin-sdk qa-runner-runtime", () => {
loadBundledPluginPublicSurfaceModuleSync.mockReset();
tryLoadActivatedBundledPluginPublicSurfaceModuleSync.mockReset();
resolveOpenClawPackageRootSync.mockReset().mockReturnValue(null);
delete process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI;
delete process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI;
});
afterEach(() => {
cleanupTempDirs(tempDirs);
restorePrivateQaCliEnv(originalPrivateQaCli);
restoreExperimentalQaCliEnv(originalExperimentalQaCli);
});
it("stays cold until runner discovery is requested", async () => {
@@ -82,8 +82,8 @@ describe("plugin-sdk qa-runner-runtime", () => {
});
});
it("uses the source bundled tree for qa-lab runtime loading in private qa mode", async () => {
await expectPrivateQaLabRuntimeSurfaceLoad({
it("uses the source bundled tree for qa-lab runtime loading with the experimental QA source override", async () => {
await expectExperimentalQaLabRuntimeSurfaceLoad({
tempDirs,
importRuntime: () => import("./qa-runner-runtime.js"),
loadBundledPluginPublicSurfaceModuleSync,
@@ -91,8 +91,8 @@ describe("plugin-sdk qa-runner-runtime", () => {
});
});
it("loads bundled plugin test APIs with the private QA source tree override", async () => {
const sourceRoot = makePrivateQaSourceRoot(tempDirs, "openclaw-qa-test-api-root-");
it("loads bundled plugin test APIs with the experimental QA source tree override", async () => {
const sourceRoot = makeExperimentalQaSourceRoot(tempDirs, "openclaw-qa-test-api-root-");
resolveOpenClawPackageRootSync.mockReturnValue(sourceRoot);
const testApi = { marker: "matrix-test-api" };
@@ -104,7 +104,7 @@ describe("plugin-sdk qa-runner-runtime", () => {
const testApiCall = firstPublicSurfaceCall();
expect(testApiCall?.dirName).toBe("matrix");
expect(testApiCall?.artifactBasename).toBe("test-api.js");
expect(testApiCall?.env?.OPENCLAW_ENABLE_PRIVATE_QA_CLI).toBe("1");
expect(testApiCall?.env?.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI).toBe("1");
expect(testApiCall?.env?.OPENCLAW_BUNDLED_PLUGINS_DIR).toBe(
path.join(sourceRoot, "extensions"),
);
@@ -188,7 +188,7 @@ describe("plugin-sdk qa-runner-runtime", () => {
});
it("prefers the source bundled tree for private qa discovery in repo checkouts", async () => {
const sourceRoot = makePrivateQaSourceRoot(tempDirs, "openclaw-qa-runner-root-");
const sourceRoot = makeExperimentalQaSourceRoot(tempDirs, "openclaw-qa-runner-root-");
resolveOpenClawPackageRootSync.mockReturnValue(sourceRoot);
const register = vi.fn((qa: Command) => qa);
@@ -221,7 +221,7 @@ describe("plugin-sdk qa-runner-runtime", () => {
},
]);
const manifestCall = firstManifestRegistryCall();
expect(manifestCall?.env?.OPENCLAW_ENABLE_PRIVATE_QA_CLI).toBe("1");
expect(manifestCall?.env?.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI).toBe("1");
expect(manifestCall?.env?.OPENCLAW_BUNDLED_PLUGINS_DIR).toBe(
path.join(sourceRoot, "extensions"),
);
@@ -229,7 +229,7 @@ describe("plugin-sdk qa-runner-runtime", () => {
const publicSurfaceCall = firstPublicSurfaceCall();
expect(publicSurfaceCall?.dirName).toBe("qa-matrix");
expect(publicSurfaceCall?.artifactBasename).toBe("runtime-api.js");
expect(publicSurfaceCall?.env?.OPENCLAW_ENABLE_PRIVATE_QA_CLI).toBe("1");
expect(publicSurfaceCall?.env?.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI).toBe("1");
expect(publicSurfaceCall?.env?.OPENCLAW_BUNDLED_PLUGINS_DIR).toBe(
path.join(sourceRoot, "extensions"),
);

24
src/plugin-sdk/qa-runtime.test-helpers.ts
View File

@@ -19,23 +19,23 @@ export function cleanupTempDirs(tempDirs: string[]): void {
}
}
/** Restores the private QA CLI env flag after a test mutates it. */
export function restorePrivateQaCliEnv(originalPrivateQaCli: string | undefined): void {
if (originalPrivateQaCli === undefined) {
delete process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI;
/** Restores the experimental QA CLI env flag after a test mutates it. */
export function restoreExperimentalQaCliEnv(originalExperimentalQaCli: string | undefined): void {
if (originalExperimentalQaCli === undefined) {
delete process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI;
} else {
process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI = originalPrivateQaCli;
process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI = originalExperimentalQaCli;
}
}
/** Creates a minimal source checkout shape that enables private QA runtime loading. */
export function makePrivateQaSourceRoot(tempDirs: string[], prefix: string): string {
/** Creates a minimal source checkout shape that enables experimental QA source runtime loading. */
export function makeExperimentalQaSourceRoot(tempDirs: string[], prefix: string): string {
const sourceRoot = fs.mkdtempSync(path.join(os.tmpdir(), prefix));
tempDirs.push(sourceRoot);
fs.mkdirSync(path.join(sourceRoot, "src"), { recursive: true });
fs.mkdirSync(path.join(sourceRoot, "extensions"), { recursive: true });
fs.writeFileSync(path.join(sourceRoot, ".git"), "gitdir: /tmp/mock\n", "utf8");
process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI = "1";
process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI = "1";
return sourceRoot;
}
@@ -63,14 +63,14 @@ export async function expectQaLabRuntimeSurfaceLoad(params: {
});
}
/** Asserts private QA loading rewrites bundled plugin lookup to the source extensions root. */
export async function expectPrivateQaLabRuntimeSurfaceLoad(params: {
/** Asserts experimental QA loading rewrites bundled plugin lookup to the source extensions root. */
export async function expectExperimentalQaLabRuntimeSurfaceLoad(params: {
tempDirs: string[];
importRuntime: () => Promise<QaRuntimeModule>;
loadBundledPluginPublicSurfaceModuleSync: SurfaceLoaderMock;
resolveOpenClawPackageRootSync: SurfaceLoaderMock;
}) {
const sourceRoot = makePrivateQaSourceRoot(params.tempDirs, "openclaw-qa-runtime-root-");
const sourceRoot = makeExperimentalQaSourceRoot(params.tempDirs, "openclaw-qa-runtime-root-");
params.resolveOpenClawPackageRootSync.mockReturnValue(sourceRoot);
const runtimeSurface = makeQaRuntimeSurface();
@@ -83,7 +83,7 @@ export async function expectPrivateQaLabRuntimeSurfaceLoad(params: {
dirName: "qa-lab",
artifactBasename: "runtime-api.js",
env: expect.objectContaining({
OPENCLAW_ENABLE_PRIVATE_QA_CLI: "1",
OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: "1",
OPENCLAW_BUNDLED_PLUGINS_DIR: path.join(sourceRoot, "extensions"),
}),
});

115
src/plugin-sdk/qa-runtime.test.ts
View File

@@ -5,9 +5,9 @@ import { Command } from "commander";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import {
cleanupTempDirs,
expectPrivateQaLabRuntimeSurfaceLoad,
expectExperimentalQaLabRuntimeSurfaceLoad,
expectQaLabRuntimeSurfaceLoad,
restorePrivateQaCliEnv,
restoreExperimentalQaCliEnv,
} from "./qa-runtime.test-helpers.js";
const loadBundledPluginPublicSurfaceModuleSync = vi.hoisted(() => vi.fn());
@@ -23,17 +23,17 @@ vi.mock("../infra/openclaw-root.js", () => ({
describe("plugin-sdk qa-runtime", () => {
const tempDirs: string[] = [];
const originalPrivateQaCli = process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI;
const originalExperimentalQaCli = process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI;
beforeEach(() => {
loadBundledPluginPublicSurfaceModuleSync.mockReset();
resolveOpenClawPackageRootSync.mockReset().mockReturnValue(null);
delete process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI;
delete process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI;
});
afterEach(() => {
cleanupTempDirs(tempDirs);
restorePrivateQaCliEnv(originalPrivateQaCli);
restoreExperimentalQaCliEnv(originalExperimentalQaCli);
});
it("stays cold until the runtime seam is used", async () => {
@@ -51,8 +51,8 @@ describe("plugin-sdk qa-runtime", () => {
});
});
it("uses the source bundled tree for qa-lab runtime loading in private qa mode", async () => {
await expectPrivateQaLabRuntimeSurfaceLoad({
it("uses the source bundled tree for qa-lab runtime loading with the experimental QA source override", async () => {
await expectExperimentalQaLabRuntimeSurfaceLoad({
tempDirs,
importRuntime: () => import("./qa-runtime.js"),
loadBundledPluginPublicSurfaceModuleSync,
@@ -149,6 +149,107 @@ describe("plugin-sdk qa-runtime", () => {
expectedStandardScenarioIds: module.LIVE_TRANSPORT_BASELINE_STANDARD_SCENARIO_IDS,
}),
).toEqual(["allowlist-block", "top-level-reply-shape"]);
expect(
module.LIVE_TRANSPORT_STANDARD_SCENARIOS.find((scenario) => scenario.id === "canary"),
).toMatchObject({
requiredCapabilities: ["inbound-message", "outbound-final-reply"],
userFlowId: "messaging.direct-reply",
});
});
it("re-exports the shared QA user-flow planner surface for QA Lab callers", async () => {
const module = await import("./qa-runtime.js");
expect(module.QA_USER_FLOW_SURFACES).toContain("messaging");
expect(
module.QA_USER_FLOW_STANDARD_FLOWS.find((flow) => flow.id === "tool.call-followthrough"),
).toMatchObject({
requiredCapabilities: ["tool.call", "model.final-response"],
});
expect(
module
.planQaStandardUserFlows({
availableCapabilities: ["tool.call", "model.final-response"],
requestedFlowIds: ["tool.call-followthrough"],
})
.selected.map((flow) => flow.id),
).toEqual(["tool.call-followthrough"]);
});
it("plans standard live transport scenarios from capabilities and driver support", async () => {
const module = await import("./qa-runtime.js");
const plan = module.planLiveTransportStandardScenarios({
availableCapabilities: [
"inbound-message",
"mention-gating",
"outbound-final-reply",
"sender-allowlist",
"top-level-reply",
],
driverSupportedScenarioIds: [
"canary",
"mention-gating",
"allowlist-block",
"top-level-reply-shape",
"thread-follow-up",
],
requestedScenarioIds: [
"canary",
"allowlist-block",
"top-level-reply-shape",
"thread-follow-up",
],
});
expect(plan.selected.map((scenario) => scenario.id)).toEqual([
"canary",
"allowlist-block",
"top-level-reply-shape",
]);
expect(
plan.skipped.map((scenario) => ({
id: scenario.id,
reason: scenario.reason,
missingCapabilities: scenario.missingCapabilities,
})),
).toEqual([
{ id: "mention-gating", reason: "not-requested", missingCapabilities: [] },
{ id: "restart-resume", reason: "not-requested", missingCapabilities: ["runtime-restart"] },
{
id: "thread-follow-up",
reason: "missing-capability",
missingCapabilities: ["inbound-thread", "outbound-thread-reply"],
},
{
id: "thread-isolation",
reason: "not-requested",
missingCapabilities: ["inbound-thread"],
},
{
id: "reaction-observation",
reason: "not-requested",
missingCapabilities: ["inbound-reaction"],
},
{ id: "help-command", reason: "not-requested", missingCapabilities: ["native-help-command"] },
]);
});
it("reports driver gaps before capability gaps in live transport plans", async () => {
const module = await import("./qa-runtime.js");
const plan = module.planLiveTransportStandardScenarios({
availableCapabilities: [],
driverSupportedScenarioIds: [],
requestedScenarioIds: ["thread-follow-up"],
});
expect(plan.selected).toEqual([]);
expect(plan.skipped.find((scenario) => scenario.id === "thread-follow-up")).toMatchObject({
reason: "driver-not-implemented",
missingCapabilities: ["inbound-message", "inbound-thread", "outbound-thread-reply"],
});
});
it("registers shared live transport QA CLI options", async () => {

32
src/plugin-sdk/qa-runtime.ts
View File

@@ -234,13 +234,45 @@ export type QaReportScenario = {
export {
LIVE_TRANSPORT_BASELINE_STANDARD_SCENARIO_IDS,
LIVE_TRANSPORT_STANDARD_SCENARIOS,
collectLiveTransportStandardScenarioCoverage,
findMissingLiveTransportStandardScenarios,
planLiveTransportStandardScenarios,
selectLiveTransportScenarios,
type LiveTransportScenarioCapabilityId,
type LiveTransportScenarioDefinition,
type LiveTransportStandardScenarioDefinition,
type LiveTransportStandardScenarioId,
type LiveTransportStandardScenarioPlan,
type LiveTransportStandardScenarioPlanEntry,
type LiveTransportStandardScenarioSkipReason,
} from "./qa-live-transport-scenarios.js";
export {
QA_USER_FLOW_STANDARD_CAPABILITIES,
QA_USER_FLOW_STANDARD_FLOWS,
QA_USER_FLOW_SURFACES,
assertKnownQaStandardUserFlowIds,
collectQaUserFlowCapabilities,
collectQaUserFlowSupportedFlowIds,
planQaStandardUserFlows,
planQaUserFlows,
type QaStandardUserFlowCapabilityId,
type QaStandardUserFlowId,
type QaUserFlowActionDescriptor,
type QaUserFlowCapabilityDefinition,
type QaUserFlowCapabilityMapping,
type QaUserFlowContractFamily,
type QaUserFlowContractRef,
type QaUserFlowDefinition,
type QaUserFlowExecutionDescriptor,
type QaUserFlowExecutionTarget,
type QaUserFlowPlan,
type QaUserFlowPlanEntry,
type QaUserFlowSkipReason,
type QaUserFlowSurfaceId,
} from "./qa-user-flows.js";
/** Docker command runner abstraction used by QA Docker helpers and tests. */
export type QaDockerRunCommand = (
command: string,

123
src/plugin-sdk/qa-user-flows.test.ts Normal file
View File

@@ -0,0 +1,123 @@
import { describe, expect, it } from "vitest";
import {
collectQaUserFlowCapabilities,
collectQaUserFlowSupportedFlowIds,
planQaStandardUserFlows,
planQaUserFlows,
} from "./qa-user-flows.js";
describe("plugin-sdk qa-user-flows", () => {
it("plans flows from optional owner mappings without requiring every plugin to map everything", () => {
const mappings = [
{
ownerId: "qa-channel",
provides: ["messaging.inbound-message", "messaging.outbound-final-reply"],
supportedFlowIds: ["messaging.direct-reply"],
},
{
ownerId: "openai",
provides: ["model.final-response"],
},
] as const;
expect(collectQaUserFlowCapabilities(mappings)).toEqual([
"messaging.inbound-message",
"messaging.outbound-final-reply",
"model.final-response",
]);
expect(collectQaUserFlowSupportedFlowIds(mappings)).toEqual(["messaging.direct-reply"]);
const plan = planQaStandardUserFlows({
mappings,
requestedFlowIds: ["messaging.direct-reply", "tool.call-followthrough"],
});
expect(plan.selected.map((flow) => flow.id)).toEqual(["messaging.direct-reply"]);
expect(
plan.skipped.map((flow) => ({
id: flow.id,
reason: flow.reason,
missingCapabilities: flow.missingCapabilities,
})),
).toContainEqual({
id: "tool.call-followthrough",
reason: "driver-not-implemented",
missingCapabilities: ["tool.call"],
});
expect(plan.selected[0]).toMatchObject({
execution: { runner: "qa-lab-flow", target: "running-gateway" },
requiredCapabilities: ["messaging.inbound-message", "messaging.outbound-final-reply"],
});
});
it("uses live-transport mappings for channel-native user flows that do not have QA Lab markdown yet", () => {
const plan = planQaStandardUserFlows({
availableCapabilities: [
"messaging.inbound-message",
"messaging.mention-gating",
"messaging.outbound-final-reply",
"setup.native-help-command",
],
requestedFlowIds: ["messaging.mention-gating", "setup.native-help-command"],
});
expect(
plan.selected.map((flow) => ({
execution: flow.execution,
id: flow.id,
})),
).toEqual([
{
id: "messaging.mention-gating",
execution: { runner: "live-transport", target: "running-gateway" },
},
{
id: "setup.native-help-command",
execution: { runner: "live-transport", target: "running-gateway" },
},
]);
});
it("keeps generic planner id validation and skip ordering shared", () => {
const flows = [
{
id: "alpha",
title: "Alpha",
description: "Alpha flow",
execution: { runner: "qa-lab-flow", target: "running-gateway" },
surface: "messaging",
action: { actor: "user", verb: "send", object: "message" },
requiredCapabilities: ["cap.alpha"],
},
{
id: "beta",
title: "Beta",
description: "Beta flow",
execution: { runner: "qa-lab-flow", target: "running-gateway" },
surface: "tool",
action: { actor: "user", verb: "request", object: "tool" },
requiredCapabilities: ["cap.beta"],
},
] as const;
expect(() =>
planQaUserFlows({
flows,
availableCapabilities: [],
requestedFlowIds: ["missing"],
}),
).toThrow("unknown QA user flow id: missing");
const plan = planQaUserFlows({
flows,
availableCapabilities: ["cap.beta"],
driverSupportedFlowIds: ["beta"],
requestedFlowIds: ["beta"],
});
expect(plan.selected.map((flow) => flow.id)).toEqual(["beta"]);
expect(plan.skipped.map((flow) => [flow.id, flow.reason, flow.missingCapabilities])).toEqual([
["alpha", "not-requested", ["cap.alpha"]],
]);
});
});

586
src/plugin-sdk/qa-user-flows.ts Normal file
View File

@@ -0,0 +1,586 @@
/** Generic user-flow planning contracts used by QA Lab and plugin-owned QA drivers. */
export const QA_USER_FLOW_SURFACES = [
"messaging",
"setup",
"model",
"tool",
"approval",
"memory",
"media",
"plugin",
"recovery",
"scheduling",
"security",
"workspace",
] as const;
export type QaUserFlowSurfaceId = (typeof QA_USER_FLOW_SURFACES)[number] | (string & {});
export type QaUserFlowContractFamily =
| "agent-runtime"
| "approval"
| "channel"
| "gateway"
| "media"
| "memory"
| "plugin"
| "provider"
| "scheduler"
| "security"
| "setup"
| "tool"
| "workspace"
| (string & {});
export type QaUserFlowActionDescriptor = {
actor: "user" | "operator" | "system";
object?: string;
verb: string;
};
export type QaUserFlowContractRef = {
family: QaUserFlowContractFamily;
name: string;
};
export type QaUserFlowExecutionTarget =
| "running-gateway"
| "gateway-lab"
| "contract-fixture"
| "static-catalog"
| (string & {});
export type QaUserFlowExecutionDescriptor = {
runner: "qa-lab-flow" | "live-transport" | "contract-test" | "manual" | (string & {});
target: QaUserFlowExecutionTarget;
};
export type QaUserFlowCapabilityDefinition<TCapabilityId extends string = string> = {
contracts?: readonly QaUserFlowContractRef[];
description: string;
id: TCapabilityId;
surface: QaUserFlowSurfaceId;
title: string;
};
export type QaUserFlowDefinition<
TId extends string = string,
TCapabilityId extends string = string,
> = {
action: QaUserFlowActionDescriptor;
contracts?: readonly QaUserFlowContractRef[];
description: string;
execution: QaUserFlowExecutionDescriptor;
id: TId;
qaScenarioIds?: readonly string[];
requiredCapabilities: readonly TCapabilityId[];
surface: QaUserFlowSurfaceId;
title: string;
};
export type QaUserFlowCapabilityMapping<TCapabilityId extends string = string> = {
/** Plugin, runtime, or driver id that owns this optional flow mapping. */
ownerId: string;
/** Capability atoms this owner can prove for user-flow planning. */
provides: readonly TCapabilityId[];
/** Flow ids this owner has concrete driver coverage for. Omit when inferred by capabilities. */
supportedFlowIds?: readonly string[];
};
export type QaUserFlowPlanEntry<TDefinition extends QaUserFlowDefinition> = TDefinition & {
missingCapabilities: readonly TDefinition["requiredCapabilities"][number][];
};
export type QaUserFlowSkipReason =
| "driver-not-implemented"
| "missing-capability"
| "not-requested";
export type QaUserFlowPlan<TDefinition extends QaUserFlowDefinition = QaUserFlowDefinition> = {
selected: readonly TDefinition[];
skipped: readonly (QaUserFlowPlanEntry<TDefinition> & {
reason: QaUserFlowSkipReason;
})[];
};
export const QA_USER_FLOW_STANDARD_CAPABILITIES = [
{
id: "messaging.inbound-message",
title: "Receive message",
description: "A user can send text into OpenClaw through a message surface.",
surface: "messaging",
contracts: [{ family: "channel", name: "base message ingress contract" }],
},
{
id: "messaging.outbound-final-reply",
title: "Send final reply",
description: "OpenClaw can deliver the assistant's final answer back to the user.",
surface: "messaging",
contracts: [{ family: "channel", name: "outbound reply contract" }],
},
{
id: "messaging.thread-reply",
title: "Reply in thread",
description: "Threaded user prompts preserve their reply context.",
surface: "messaging",
contracts: [{ family: "channel", name: "thread binding contract" }],
},
{
id: "messaging.reaction-events",
title: "Observe reactions",
description: "Native reaction/edit/delete style events can be observed and normalized.",
surface: "messaging",
contracts: [{ family: "channel", name: "message actions contract" }],
},
{
id: "messaging.mention-gating",
title: "Gate unmentioned message",
description: "A group message that does not address OpenClaw does not trigger a reply.",
surface: "messaging",
contracts: [{ family: "channel", name: "mention gating contract" }],
},
{
id: "security.sender-allowlist",
title: "Block unauthorized sender",
description: "A sender outside the configured allowlist does not trigger a reply.",
surface: "security",
contracts: [{ family: "channel", name: "sender allowlist contract" }],
},
{
id: "setup.native-help-command",
title: "Handle native help command",
description: "A user can ask the transport-specific command surface for help.",
surface: "setup",
contracts: [{ family: "channel", name: "native command contract" }],
},
{
id: "setup.provider-auth",
title: "Authenticate provider",
description: "A user can connect provider credentials through setup or auth repair.",
surface: "setup",
contracts: [{ family: "provider", name: "provider auth contract" }],
},
{
id: "setup.config-apply",
title: "Apply configuration",
description: "A config/setup change becomes active without corrupting runtime state.",
surface: "setup",
contracts: [{ family: "setup", name: "config mutation contract" }],
},
{
id: "model.final-response",
title: "Return model response",
description: "A selected model can produce a final response for a user turn.",
surface: "model",
contracts: [{ family: "provider", name: "provider runtime contract" }],
},
{
id: "model.switch",
title: "Switch models",
description: "A user can switch models and continue the same task coherently.",
surface: "model",
contracts: [{ family: "provider", name: "provider selection contract" }],
},
{
id: "tool.call",
title: "Call tool",
description: "A user request can drive an OpenClaw tool call and continue to completion.",
surface: "tool",
contracts: [{ family: "tool", name: "tool runtime contract" }],
},
{
id: "approval.native-roundtrip",
title: "Approve action",
description: "A user can approve or deny a pending action through a supported surface.",
surface: "approval",
contracts: [{ family: "approval", name: "approval runtime contract" }],
},
{
id: "memory.store",
title: "Store memory",
description: "A user-visible fact can be committed to the configured memory surface.",
surface: "memory",
contracts: [{ family: "memory", name: "memory host contract" }],
},
{
id: "memory.recall",
title: "Recall memory",
description: "A later user turn can recall relevant scoped memory.",
surface: "memory",
contracts: [{ family: "memory", name: "memory query contract" }],
},
{
id: "media.image-input",
title: "Understand image",
description: "A user can attach an image and receive a grounded answer about it.",
surface: "media",
contracts: [{ family: "media", name: "media understanding contract" }],
},
{
id: "plugin.lifecycle",
title: "Load plugin",
description: "A plugin can be installed, discovered, and made available to a user task.",
surface: "plugin",
contracts: [{ family: "plugin", name: "plugin registration contract" }],
},
{
id: "recovery.restart-resume",
title: "Resume after restart",
description: "A user flow can continue after a runtime or gateway restart.",
surface: "recovery",
contracts: [{ family: "gateway", name: "restart recovery contract" }],
},
{
id: "scheduling.reminder",
title: "Deliver reminder",
description: "A scheduled user reminder can be created and delivered once.",
surface: "scheduling",
contracts: [{ family: "scheduler", name: "scheduled task contract" }],
},
{
id: "security.redaction",
title: "Protect secret",
description: "Secrets stay redacted in user-visible logs, traces, and tool output.",
surface: "security",
contracts: [{ family: "security", name: "secret redaction contract" }],
},
{
id: "workspace.edit-loop",
title: "Edit workspace",
description: "A user can ask for workspace edits and receive a coherent completion report.",
surface: "workspace",
contracts: [{ family: "workspace", name: "workspace tool contract" }],
},
] as const satisfies readonly QaUserFlowCapabilityDefinition[];
export type QaStandardUserFlowCapabilityId =
(typeof QA_USER_FLOW_STANDARD_CAPABILITIES)[number]["id"];
export const QA_USER_FLOW_STANDARD_FLOWS = [
{
id: "messaging.direct-reply",
title: "Direct message reply",
description: "A user sends a message and receives a final answer on the same surface.",
execution: { runner: "qa-lab-flow", target: "running-gateway" },
surface: "messaging",
action: { actor: "user", verb: "send", object: "message" },
requiredCapabilities: ["messaging.inbound-message", "messaging.outbound-final-reply"],
contracts: [{ family: "channel", name: "base channel plugin contract" }],
qaScenarioIds: ["channel-chat-baseline", "dm-chat-baseline"],
},
{
id: "messaging.thread-follow-up",
title: "Thread follow-up",
description: "A user follows up inside a thread and receives the answer in that context.",
execution: { runner: "qa-lab-flow", target: "running-gateway" },
surface: "messaging",
action: { actor: "user", verb: "reply", object: "thread" },
requiredCapabilities: ["messaging.inbound-message", "messaging.thread-reply"],
contracts: [{ family: "channel", name: "thread binding contract" }],
qaScenarioIds: ["thread-follow-up"],
},
{
id: "messaging.reaction-edit-delete",
title: "Message action observation",
description: "A user edits, deletes, or reacts to a message and the transport observes it.",
execution: { runner: "qa-lab-flow", target: "running-gateway" },
surface: "messaging",
action: { actor: "user", verb: "react", object: "message" },
requiredCapabilities: ["messaging.reaction-events"],
contracts: [{ family: "channel", name: "message actions contract" }],
qaScenarioIds: ["reaction-edit-delete"],
},
{
id: "messaging.mention-gating",
title: "Mention gating",
description: "A group user message that does not mention OpenClaw does not receive a reply.",
execution: { runner: "live-transport", target: "running-gateway" },
surface: "messaging",
action: { actor: "user", verb: "send", object: "unmentioned group message" },
requiredCapabilities: ["messaging.inbound-message", "messaging.mention-gating"],
contracts: [{ family: "channel", name: "mention gating contract" }],
},
{
id: "security.sender-allowlist-block",
title: "Sender allowlist block",
description: "A blocked user sends a message and receives no OpenClaw reply.",
execution: { runner: "live-transport", target: "running-gateway" },
surface: "security",
action: { actor: "user", verb: "send", object: "blocked sender message" },
requiredCapabilities: ["messaging.inbound-message", "security.sender-allowlist"],
contracts: [{ family: "channel", name: "sender allowlist contract" }],
},
{
id: "setup.native-help-command",
title: "Native help command",
description: "A user asks for help through the transport command surface and gets a reply.",
execution: { runner: "live-transport", target: "running-gateway" },
surface: "setup",
action: { actor: "user", verb: "request", object: "native help command" },
requiredCapabilities: [
"messaging.inbound-message",
"setup.native-help-command",
"messaging.outbound-final-reply",
],
contracts: [{ family: "channel", name: "native command contract" }],
},
{
id: "setup.provider-auth",
title: "Provider auth setup",
description: "A user connects provider credentials and can use the provider afterward.",
execution: { runner: "qa-lab-flow", target: "running-gateway" },
surface: "setup",
action: { actor: "user", verb: "connect", object: "provider" },
requiredCapabilities: ["setup.provider-auth", "model.final-response"],
contracts: [{ family: "provider", name: "provider auth contract" }],
qaScenarioIds: ["anthropic-opus-api-key-smoke", "auth-profile-doctor-migration-safety"],
},
{
id: "setup.config-hot-apply",
title: "Config hot apply",
description: "A user changes configuration and the runtime observes the new behavior.",
execution: { runner: "qa-lab-flow", target: "running-gateway" },
surface: "setup",
action: { actor: "user", verb: "change", object: "configuration" },
requiredCapabilities: ["setup.config-apply"],
contracts: [{ family: "setup", name: "config mutation contract" }],
qaScenarioIds: ["config-patch-hot-apply", "config-apply-restart-wakeup"],
},
{
id: "model.switch-follow-up",
title: "Model switch follow-up",
description: "A user switches models and continues the task without losing context.",
execution: { runner: "qa-lab-flow", target: "running-gateway" },
surface: "model",
action: { actor: "user", verb: "switch", object: "model" },
requiredCapabilities: ["model.final-response", "model.switch"],
contracts: [{ family: "provider", name: "provider selection contract" }],
qaScenarioIds: ["model-switch-follow-up", "model-switch-tool-continuity"],
},
{
id: "tool.call-followthrough",
title: "Tool call follow-through",
description: "A user asks for a tool-backed task and receives a final answer after the tool.",
execution: { runner: "qa-lab-flow", target: "running-gateway" },
surface: "tool",
action: { actor: "user", verb: "request", object: "tool-backed task" },
requiredCapabilities: ["tool.call", "model.final-response"],
contracts: [{ family: "agent-runtime", name: "OpenClaw-owned tool runtime contract" }],
qaScenarioIds: ["approval-turn-tool-followthrough", "message-tool"],
},
{
id: "approval.native-roundtrip",
title: "Approval roundtrip",
description: "A user approves or denies a pending tool action and the run honors it.",
execution: { runner: "qa-lab-flow", target: "running-gateway" },
surface: "approval",
action: { actor: "user", verb: "decide", object: "approval request" },
requiredCapabilities: ["approval.native-roundtrip", "tool.call"],
contracts: [{ family: "approval", name: "approval runtime contract" }],
qaScenarioIds: ["approval-turn-tool-followthrough", "approval-denial-stop"],
},
{
id: "memory.scoped-recall",
title: "Scoped memory recall",
description: "A user stores a fact and receives a later answer that recalls the right fact.",
execution: { runner: "qa-lab-flow", target: "running-gateway" },
surface: "memory",
action: { actor: "user", verb: "ask", object: "remembered fact" },
requiredCapabilities: ["memory.store", "memory.recall"],
contracts: [{ family: "memory", name: "memory host/query contract" }],
qaScenarioIds: ["memory-recall", "thread-memory-isolation"],
},
{
id: "media.image-understanding",
title: "Image understanding",
description: "A user attaches an image and receives an answer grounded in the image.",
execution: { runner: "qa-lab-flow", target: "running-gateway" },
surface: "media",
action: { actor: "user", verb: "attach", object: "image" },
requiredCapabilities: ["media.image-input", "model.final-response"],
contracts: [{ family: "media", name: "media understanding contract" }],
qaScenarioIds: ["image-understanding-attachment"],
},
{
id: "plugin.lifecycle-hot-reload",
title: "Plugin lifecycle hot reload",
description: "A user installs or updates a plugin and can use its newly available surface.",
execution: { runner: "qa-lab-flow", target: "running-gateway" },
surface: "plugin",
action: { actor: "user", verb: "install", object: "plugin" },
requiredCapabilities: ["plugin.lifecycle"],
contracts: [{ family: "plugin", name: "plugin registration contract" }],
qaScenarioIds: ["plugin-lifecycle-hot-reload", "mcp-plugin-tools-call"],
},
{
id: "recovery.restart-resume",
title: "Restart resume",
description: "A user-visible run survives a gateway/runtime restart and completes.",
execution: { runner: "qa-lab-flow", target: "running-gateway" },
surface: "recovery",
action: { actor: "system", verb: "restart", object: "gateway" },
requiredCapabilities: ["recovery.restart-resume", "model.final-response"],
contracts: [{ family: "gateway", name: "restart recovery contract" }],
qaScenarioIds: ["gateway-restart-inflight-run"],
},
{
id: "scheduling.reminder-roundtrip",
title: "Reminder roundtrip",
description: "A user creates a reminder and receives it exactly once at the expected time.",
execution: { runner: "qa-lab-flow", target: "running-gateway" },
surface: "scheduling",
action: { actor: "user", verb: "schedule", object: "reminder" },
requiredCapabilities: ["scheduling.reminder", "messaging.outbound-final-reply"],
contracts: [{ family: "scheduler", name: "scheduled task contract" }],
qaScenarioIds: ["reminder-roundtrip", "cron-single-run-no-duplicate"],
},
{
id: "security.secret-redaction",
title: "Secret redaction",
description: "A user action that touches secrets does not leak them into visible evidence.",
execution: { runner: "qa-lab-flow", target: "running-gateway" },
surface: "security",
action: { actor: "user", verb: "inspect", object: "diagnostics" },
requiredCapabilities: ["security.redaction"],
contracts: [{ family: "security", name: "secret redaction contract" }],
qaScenarioIds: ["redaction-no-secret-leak", "secret-redaction-tool-logs"],
},
{
id: "workspace.edit-loop",
title: "Workspace edit loop",
description: "A user asks for workspace changes and receives a coherent completion report.",
execution: { runner: "jsonl-replay", target: "running-gateway" },
surface: "workspace",
action: { actor: "user", verb: "edit", object: "workspace" },
requiredCapabilities: ["workspace.edit-loop", "tool.call"],
contracts: [{ family: "workspace", name: "workspace tool contract" }],
},
] as const satisfies readonly QaUserFlowDefinition<string, QaStandardUserFlowCapabilityId>[];
export type QaStandardUserFlowId = (typeof QA_USER_FLOW_STANDARD_FLOWS)[number]["id"];
const QA_USER_FLOW_STANDARD_ID_SET = new Set(QA_USER_FLOW_STANDARD_FLOWS.map((flow) => flow.id));
function assertKnownQaUserFlowIds<TDefinition extends QaUserFlowDefinition>(
flows: readonly TDefinition[],
ids: readonly string[],
) {
const knownIds = new Set(flows.map((flow) => flow.id));
for (const id of ids) {
if (!knownIds.has(id)) {
throw new Error(`unknown QA user flow id: ${id}`);
}
}
}
/** Collects unique capability atoms from optional plugin, runtime, or driver mappings. */
export function collectQaUserFlowCapabilities<TCapabilityId extends string>(
mappings: readonly QaUserFlowCapabilityMapping<TCapabilityId>[],
) {
const capabilities: TCapabilityId[] = [];
const seen = new Set<TCapabilityId>();
for (const mapping of mappings) {
for (const capability of mapping.provides) {
if (seen.has(capability)) {
continue;
}
seen.add(capability);
capabilities.push(capability);
}
}
return capabilities;
}
/** Collects concrete flow ids from mappings that declare driver-backed support. */
export function collectQaUserFlowSupportedFlowIds(
mappings: readonly QaUserFlowCapabilityMapping[],
) {
const flowIds: string[] = [];
const seen = new Set<string>();
for (const mapping of mappings) {
for (const flowId of mapping.supportedFlowIds ?? []) {
if (seen.has(flowId)) {
continue;
}
seen.add(flowId);
flowIds.push(flowId);
}
}
return flowIds;
}
/** Plans user-facing flows from capability mappings and optional driver support. */
export function planQaUserFlows<TDefinition extends QaUserFlowDefinition>(params: {
availableCapabilities: readonly string[];
driverSupportedFlowIds?: readonly string[];
flows: readonly TDefinition[];
requestedFlowIds?: readonly string[];
}): QaUserFlowPlan<TDefinition> {
assertKnownQaUserFlowIds(params.flows, params.driverSupportedFlowIds ?? []);
assertKnownQaUserFlowIds(params.flows, params.requestedFlowIds ?? []);
const availableCapabilities = new Set(params.availableCapabilities);
const driverSupportedFlowIds = params.driverSupportedFlowIds
? new Set(params.driverSupportedFlowIds)
: null;
const requestedFlowIds = params.requestedFlowIds ? new Set(params.requestedFlowIds) : null;
const selected: TDefinition[] = [];
const skipped: QaUserFlowPlan<TDefinition>["skipped"][number][] = [];
for (const flow of params.flows) {
const missingCapabilities = flow.requiredCapabilities.filter(
(capability) => !availableCapabilities.has(capability),
);
const baseEntry = {
...flow,
missingCapabilities,
};
if (requestedFlowIds && !requestedFlowIds.has(flow.id)) {
skipped.push({ ...baseEntry, reason: "not-requested" });
continue;
}
if (driverSupportedFlowIds && !driverSupportedFlowIds.has(flow.id)) {
skipped.push({ ...baseEntry, reason: "driver-not-implemented" });
continue;
}
if (missingCapabilities.length > 0) {
skipped.push({ ...baseEntry, reason: "missing-capability" });
continue;
}
selected.push(flow);
}
return { selected, skipped };
}
/** Plans the built-in standard user-flow catalog from optional owner mappings. */
export function planQaStandardUserFlows(params: {
availableCapabilities?: readonly QaStandardUserFlowCapabilityId[];
mappings?: readonly QaUserFlowCapabilityMapping<QaStandardUserFlowCapabilityId>[];
requestedFlowIds?: readonly QaStandardUserFlowId[];
}) {
const mappedCapabilities = params.mappings ? collectQaUserFlowCapabilities(params.mappings) : [];
const mappedFlowIds = params.mappings ? collectQaUserFlowSupportedFlowIds(params.mappings) : [];
const driverSupportedFlowIds = mappedFlowIds.length > 0 ? mappedFlowIds : undefined;
return planQaUserFlows({
flows: QA_USER_FLOW_STANDARD_FLOWS,
availableCapabilities: [...mappedCapabilities, ...(params.availableCapabilities ?? [])],
...(driverSupportedFlowIds ? { driverSupportedFlowIds } : {}),
...(params.requestedFlowIds ? { requestedFlowIds: params.requestedFlowIds } : {}),
});
}
/** Fails fast when caller-provided standard flow ids drift from the shared catalog. */
export function assertKnownQaStandardUserFlowIds(ids: readonly QaStandardUserFlowId[]) {
for (const id of ids) {
if (!QA_USER_FLOW_STANDARD_ID_SET.has(id)) {
throw new Error(`unknown QA standard user flow id: ${id}`);
}
}
}

8
src/plugin-sdk/root-alias.cjs
View File

@@ -265,9 +265,13 @@ function findDistChunkByPrefix(prefix) {
}
}
function isExperimentalQaCliEnabled() {
return process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI === "1";
}
function listPluginSdkExportedSubpaths() {
const packageRoot = getPackageRoot();
const cacheKey = `${packageRoot}::privateQa=${process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI === "1" ? "1" : "0"}`;
const cacheKey = `${packageRoot}::experimentalQa=${isExperimentalQaCliEnabled() ? "1" : "0"}`;
if (pluginSdkSubpathsCache.has(cacheKey)) {
return pluginSdkSubpathsCache.get(cacheKey);
}
@@ -290,7 +294,7 @@ function listPluginSdkExportedSubpaths() {
}
function listPrivateLocalOnlyPluginSdkSubpaths() {
if (process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI !== "1") {
if (!isExperimentalQaCliEnabled()) {
return [];
}
try {

8
src/plugins/bundled-plugin-metadata.test.ts
View File

@@ -372,11 +372,9 @@ describe("bundled plugin metadata", () => {
},
);
it("excludes non-packaged QA sidecars from the packaged runtime sidecar baseline", () => {
expect(BUNDLED_RUNTIME_SIDECAR_PATHS).not.toContain(
"dist/extensions/qa-channel/runtime-api.js",
);
expect(BUNDLED_RUNTIME_SIDECAR_PATHS).not.toContain("dist/extensions/qa-lab/runtime-api.js");
it("includes packaged QA sidecars and excludes non-packaged QA Matrix sidecars", () => {
expect(BUNDLED_RUNTIME_SIDECAR_PATHS).toContain("dist/extensions/qa-channel/runtime-api.js");
expect(BUNDLED_RUNTIME_SIDECAR_PATHS).toContain("dist/extensions/qa-lab/runtime-api.js");
expect(BUNDLED_RUNTIME_SIDECAR_PATHS).not.toContain("dist/extensions/qa-matrix/runtime-api.js");
});

2
src/plugins/bundled-plugin-naming.test.ts
View File

@@ -35,7 +35,7 @@ const DIR_ID_EXCEPTIONS = new Map<string, string>([
// Historical directory name kept until a wider repo cleanup is worth the churn.
["kimi-coding", "kimi"],
]);
const NON_PACKAGED_BUNDLED_PLUGIN_DIRS = new Set(["qa-channel", "qa-lab", "qa-matrix"]);
const NON_PACKAGED_BUNDLED_PLUGIN_DIRS = new Set(["qa-matrix"]);
const ALLOWED_PACKAGE_SUFFIXES = [
"",
"-provider",

2
src/plugins/contracts/plugin-sdk-root-alias.test.ts
View File

@@ -548,7 +548,7 @@ describe("plugin-sdk root alias", () => {
"ssrf-runtime-internal.ts",
);
const lazyModule = loadRootAliasWithStubs({
env: { OPENCLAW_ENABLE_PRIVATE_QA_CLI: "1" },
env: { OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: "1" },
privateLocalOnlySubpaths: ["qa-lab", "../escape", "nested/path", "ssrf-runtime-internal"],
existingPaths: [qaLabPath, ssrfRuntimeInternalPath],
monolithicExports: {

2
src/plugins/runtime-sidecar-paths-baseline.ts
View File

@@ -4,7 +4,7 @@ import path from "node:path";
import { tryReadJsonSync } from "../infra/json-files.js";
import { listBundledPluginMetadata } from "./bundled-plugin-metadata.js";
const NON_PACKAGED_RUNTIME_SIDECAR_PLUGIN_DIRS = new Set(["qa-channel", "qa-lab", "qa-matrix"]);
const NON_PACKAGED_RUNTIME_SIDECAR_PLUGIN_DIRS = new Set(["qa-matrix"]);
function buildBundledDistArtifactPath(dirName: string, artifact: string): string {
return ["dist", "extensions", dirName, artifact].join("/");

58
src/plugins/sdk-alias.test.ts
View File

@@ -709,7 +709,7 @@ describe("plugin sdk alias helpers", () => {
"utf-8",
);
const subpaths = withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: "1" }, () =>
const subpaths = withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: "1" }, () =>
listPluginSdkExportedSubpaths({
modulePath: path.join(fixture.root, "src", "plugins", "loader.ts"),
}),
@@ -770,18 +770,18 @@ describe("plugin sdk alias helpers", () => {
);
fs.writeFileSync(shadowCodexEntry, 'export const plugin = "shadow";\n', "utf-8");
const codexSubpaths = withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined }, () =>
const codexSubpaths = withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined }, () =>
listPluginSdkExportedSubpaths({
modulePath: sourceCodexEntry,
}),
);
const otherSubpaths = withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined }, () =>
const otherSubpaths = withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined }, () =>
listPluginSdkExportedSubpaths({
modulePath: sourceOtherEntry,
}),
);
const installedCodexSubpaths = withCwd(installedCodexRoot, () =>
withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined }, () =>
withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined }, () =>
listPluginSdkExportedSubpaths({
modulePath: installedCodexEntry,
argv1: path.join(fixture.root, "openclaw.mjs"),
@@ -789,7 +789,7 @@ describe("plugin sdk alias helpers", () => {
),
);
const installedOtherSubpaths = withCwd(installedOtherRoot, () =>
withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined }, () =>
withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined }, () =>
listPluginSdkExportedSubpaths({
modulePath: installedOtherEntry,
argv1: path.join(fixture.root, "openclaw.mjs"),
@@ -797,7 +797,7 @@ describe("plugin sdk alias helpers", () => {
),
);
const shadowCodexSubpaths = withCwd(shadowCodexRoot, () =>
withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined }, () =>
withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined }, () =>
listPluginSdkExportedSubpaths({
modulePath: shadowCodexEntry,
argv1: path.join(fixture.root, "openclaw.mjs"),
@@ -849,7 +849,7 @@ describe("plugin sdk alias helpers", () => {
}),
).toEqual(["core"]);
const privateSubpaths = withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: "1" }, () =>
const privateSubpaths = withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: "1" }, () =>
listPluginSdkExportedSubpaths({
modulePath: path.join(fixture.root, "src", "plugins", "loader.ts"),
}),
@@ -963,7 +963,7 @@ describe("plugin sdk alias helpers", () => {
bundledPluginFile("qa-matrix", "src/index.ts"),
);
const aliases = withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: "1", NODE_ENV: undefined }, () =>
const aliases = withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: "1", NODE_ENV: undefined }, () =>
buildPluginLoaderAliasMap(sourcePluginEntry),
);
@@ -1107,15 +1107,15 @@ describe("plugin sdk alias helpers", () => {
fs.writeFileSync(shadowCodexEntry, 'export const plugin = "shadow";\n', "utf-8");
const aliases = withEnv(
{ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined, NODE_ENV: undefined },
{ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined, NODE_ENV: undefined },
() => buildPluginLoaderAliasMap(sourcePluginEntry),
);
const otherAliases = withEnv(
{ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined, NODE_ENV: undefined },
{ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined, NODE_ENV: undefined },
() => buildPluginLoaderAliasMap(sourceOtherPluginEntry),
);
const devRootAliases = withEnv(
{ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined, NODE_ENV: undefined },
{ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined, NODE_ENV: undefined },
() =>
buildPluginLoaderAliasMap(
distCodexEntry,
@@ -1126,7 +1126,7 @@ describe("plugin sdk alias helpers", () => {
),
);
const installedAliases = withCwd(installedCodexRoot, () =>
withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined, NODE_ENV: undefined }, () =>
withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined, NODE_ENV: undefined }, () =>
buildPluginLoaderAliasMap(
installedCodexEntry,
path.join(fixture.root, "openclaw.mjs"),
@@ -1136,7 +1136,7 @@ describe("plugin sdk alias helpers", () => {
),
);
const shadowCodexAliases = withCwd(shadowCodexRoot, () =>
withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined, NODE_ENV: undefined }, () =>
withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined, NODE_ENV: undefined }, () =>
buildPluginLoaderAliasMap(
shadowCodexEntry,
path.join(fixture.root, "openclaw.mjs"),
@@ -1146,7 +1146,7 @@ describe("plugin sdk alias helpers", () => {
),
);
const installedOtherAliases = withCwd(installedOtherRoot, () =>
withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined, NODE_ENV: undefined }, () =>
withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined, NODE_ENV: undefined }, () =>
buildPluginLoaderAliasMap(
installedOtherEntry,
path.join(fixture.root, "openclaw.mjs"),
@@ -1260,55 +1260,55 @@ describe("plugin sdk alias helpers", () => {
packageName: "@openclaw/ollama",
});
const sourceSubpaths = withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined }, () =>
const sourceSubpaths = withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined }, () =>
listPluginSdkExportedSubpaths({
modulePath: sourceOllamaEntry,
}),
);
const sourceBrowserSubpaths = withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined }, () =>
const sourceBrowserSubpaths = withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined }, () =>
listPluginSdkExportedSubpaths({
modulePath: sourceBrowserEntry,
}),
);
const privateQaOtherSubpaths = withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: "1" }, () =>
const privateQaOtherSubpaths = withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: "1" }, () =>
listPluginSdkExportedSubpaths({
modulePath: sourceOtherPluginEntry,
}),
);
const sourceAliases = withEnv(
{ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined, NODE_ENV: undefined },
{ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined, NODE_ENV: undefined },
() => buildPluginLoaderAliasMap(sourceOllamaEntry),
);
const sourceBrowserAliases = withEnv(
{ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined, NODE_ENV: undefined },
{ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined, NODE_ENV: undefined },
() => buildPluginLoaderAliasMap(sourceBrowserEntry),
);
const distAliases = withEnv(
{ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined, NODE_ENV: undefined },
{ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined, NODE_ENV: undefined },
() => buildPluginLoaderAliasMap(distOllamaEntry, undefined, undefined, "dist"),
);
const distBrowserAliases = withEnv(
{ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined, NODE_ENV: undefined },
{ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined, NODE_ENV: undefined },
() => buildPluginLoaderAliasMap(distBrowserEntry, undefined, undefined, "dist"),
);
const distRuntimeAliases = withEnv(
{ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined, NODE_ENV: undefined },
{ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined, NODE_ENV: undefined },
() => buildPluginLoaderAliasMap(distRuntimeOllamaEntry, undefined, undefined, "dist"),
);
const distRuntimeBrowserAliases = withEnv(
{ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined, NODE_ENV: undefined },
{ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined, NODE_ENV: undefined },
() => buildPluginLoaderAliasMap(distRuntimeBrowserEntry, undefined, undefined, "dist"),
);
const otherAliases = withEnv(
{ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined, NODE_ENV: undefined },
{ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined, NODE_ENV: undefined },
() => buildPluginLoaderAliasMap(sourceOtherPluginEntry),
);
const privateQaOtherAliases = withEnv(
{ OPENCLAW_ENABLE_PRIVATE_QA_CLI: "1", NODE_ENV: undefined },
{ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: "1", NODE_ENV: undefined },
() => buildPluginLoaderAliasMap(sourceOtherPluginEntry),
);
const installedAliases = withCwd(installedOllamaRoot, () =>
withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined, NODE_ENV: undefined }, () =>
withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined, NODE_ENV: undefined }, () =>
buildPluginLoaderAliasMap(
installedOllamaEntry,
path.join(fixture.root, "openclaw.mjs"),
@@ -1805,7 +1805,7 @@ describe("plugin sdk alias helpers", () => {
bundledPluginFile("qa-lab", "src/live-transports/slack/slack-live.runtime.ts"),
);
const aliases = withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: "1", NODE_ENV: undefined }, () =>
const aliases = withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: "1", NODE_ENV: undefined }, () =>
buildPluginLoaderAliasMap(sourcePluginEntry),
);
@@ -2551,10 +2551,10 @@ describe("buildPluginLoaderAliasMap memoization", () => {
fs.writeFileSync(sourceQaRuntimePath, "export const qaRuntime = true;\n", "utf-8");
const entry = writePluginEntry(fixture.root, bundledPluginFile("private-qa", "src/index.ts"));
const publicAliases = withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: undefined }, () =>
const publicAliases = withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: undefined }, () =>
buildPluginLoaderAliasMap(entry),
);
const privateAliases = withEnv({ OPENCLAW_ENABLE_PRIVATE_QA_CLI: "1" }, () =>
const privateAliases = withEnv({ OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI: "1" }, () =>
buildPluginLoaderAliasMap(entry),
);

2
src/plugins/sdk-alias.ts
View File

@@ -1368,7 +1368,7 @@ function resolveWorkspacePackageAliasMap(params: {
}
function shouldIncludePrivateLocalOnlyPluginSdkSubpaths() {
return process.env.OPENCLAW_ENABLE_PRIVATE_QA_CLI === "1";
return process.env.OPENCLAW_ENABLE_EXPERIMENTAL_QA_CLI === "1";
}
function isBundledPluginModulePath(params: {