Compare commits

..

62 Commits

Author SHA1 Message Date
Vincent Koc
d49a014424 fix(plugins): stabilize registry package paths 2026-04-25 12:33:17 -07:00
Vincent Koc
0c3a0a3682 fix(plugins): satisfy registry repair lint 2026-04-25 12:33:17 -07:00
Vincent Koc
ee19c91591 fix(plugins): add doctor registry repair 2026-04-25 12:33:16 -07:00
Vincent Koc
64582bb3a7 docs(diagnostics-otel): clarify genai semconv exports 2026-04-25 12:30:14 -07:00
Peter Steinberger
d4971aad2c docs: require feasible live verification 2026-04-25 20:27:21 +01:00
Peter Steinberger
30325f567c fix: use prompt snapshots for live context diagnostics 2026-04-25 20:25:44 +01:00
Peter Steinberger
b732f21a86 fix: clarify voice-call setup diagnostics 2026-04-25 20:24:36 +01:00
Vincent Koc
44648440a5 fix(diagnostics-otel): stabilize genai token metric model attr 2026-04-25 12:22:55 -07:00
Peter Steinberger
75d64cd4b8 feat: expose generic image background option 2026-04-25 20:21:46 +01:00
Peter Steinberger
03fd7df929 fix: remove duplicate diagnostic stability case 2026-04-25 20:21:39 +01:00
Peter Steinberger
d757396785 test(ui): consolidate chat jsdom suites 2026-04-25 20:17:23 +01:00
Peter Steinberger
7436e395d5 test(node-host): cache native binary fixture lookup 2026-04-25 20:17:23 +01:00
Peter Steinberger
f34513ac66 perf(memory): avoid duplicate session store reads 2026-04-25 20:17:22 +01:00
Vincent Koc
5815ca93d9 fix(diagnostics-otel): honor genai usage semconv opt-in 2026-04-25 12:13:50 -07:00
Peter Steinberger
86d897cfaa feat(android): expose talk mode
Co-authored-by: alex-latitude <213670856+alex-latitude@users.noreply.github.com>
2026-04-25 20:12:38 +01:00
Peter Steinberger
791ad0864a fix: strip invalid thinking replay signatures
Fixes #45010.
Supersedes #70054.

Co-authored-by: Chris Staples <chris.staples@sophos.com>
Co-authored-by: Fourier <yang.fourier@gmail.com>
2026-04-25 20:12:30 +01:00
Peter Steinberger
47a63f7acf fix(logging): merge duplicate context diagnostic case 2026-04-25 20:11:08 +01:00
Peter Steinberger
e6ab61762a fix(check): pass lock env to changed lint lanes 2026-04-25 20:11:08 +01:00
Peter Steinberger
1e7ae07772 fix(cli): dedupe onboard auth flags for completion cache 2026-04-25 20:11:08 +01:00
Peter Steinberger
d9486c683b fix: stabilize macos npm update smoke 2026-04-25 20:09:32 +01:00
Peter Steinberger
17401e31de fix: avoid changed gate lint self-lock 2026-04-25 20:09:00 +01:00
mushuiyu_xydt
0e1ef93e84 fix(minimax): use dedicated image generation endpoint (#61155)
* fix(minimax): use dedicated image generation endpoint

MiniMax image generation uses a dedicated API endpoint
(api.minimax.io/v1/image_generation) that is separate from the
text/chat API endpoint (api.minimax.io/anthropic).

Previously, the resolveMinimaxImageBaseUrl function would extract
the origin from the provider's configured baseUrl. If a user had
configured their baseUrl to the chat endpoint (e.g.,
api.minimax.chat/anthropic), the image generation would incorrectly
use that endpoint, resulting in "invalid api key" errors.

This fix always uses the dedicated image generation endpoint,
ignoring the provider's baseUrl configuration for image generation.

Fixes #61149

* fix(minimax): support CN endpoint for image generation

Respect MINIMAX_API_HOST environment variable to determine whether
to use the global (api.minimax.io) or CN (api.minimaxi.com) endpoint
for image generation.

This ensures that CN users who configure MINIMAX_API_HOST to use
api.minimaxi.com will continue to use the CN endpoint for image
generation, while global users continue to use api.minimax.io.

The original bug was caused by the code extracting the origin from
the provider's configured baseUrl, which could be set to incorrect
endpoints like api.minimax.chat. This fix uses the dedicated image
generation endpoints instead.

Fixes #61149

* fix(minimax): infer CN endpoint from provider config when env is unset

When MINIMAX_API_HOST is not set, fall back to checking the provider's
configured baseUrl to determine whether to use the CN or global image
endpoint. This ensures CN users who went through onboarding (which sets
models.providers.minimax.baseUrl to https://api.minimaxi.com/anthropic)
are correctly routed to the CN image endpoint.

The isMinimaxCnHost check ensures we only use the baseUrl origin for
CN detection - invalid endpoints like api.minimax.chat would not match
minimaxi.com and would correctly fall through to the global default.

Fixes #61149

* test(minimax): cover dedicated image endpoints

* fix(logging): handle context assembly diagnostics

* Revert "fix(logging): handle context assembly diagnostics"

This reverts commit f51d2f7d67f8193268dd37553ac77e80a0423390.

* test(minimax): isolate image endpoint env

* docs(changelog): credit minimax image fix

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-04-25 20:07:52 +01:00
Quratulain-bilal
7d58362f3f docs(browser): note tilde expansion also covers per-profile paths (#71601)
* docs(browser): note tilde expansion also covers per-profile paths

The 95a2c9b fix expanded "~" for both `browser.executablePath` and
per-profile `profiles.<name>.executablePath` (config.ts:382 calls
`normalizeExecutablePath` for profile overrides). Per-profile
`userDataDir` on existing-session profiles is also tilde-expanded
(config.ts:391 via `resolveUserPath`). The configuration reference
only mentioned the top-level `browser.executablePath` case.

* docs(browser): align tilde path config help

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-04-25 20:05:03 +01:00
Vincent Koc
5671fdca87 feat(diagnostics-otel): add genai usage span identity 2026-04-25 12:03:10 -07:00
Peter Steinberger
5eab16e086 fix: improve google meet setup diagnostics 2026-04-25 20:01:24 +01:00
Peter Steinberger
e36b77c13e docs(changelog): drop self-thanks 2026-04-25 20:01:00 +01:00
Peter Steinberger
d68574653e docs(changelog): split 2026.4.24 and 2026.4.25 notes 2026-04-25 19:59:54 +01:00
Quratulain-bilal
8170df9127 docs(browser): document local startup timeout bounds (#71672)
* docs(browser): document local startup timeout bounds

The new browser.localLaunchTimeoutMs and browser.localCdpReadyTimeoutMs
options are clamped to MAX_BROWSER_STARTUP_TIMEOUT_MS (120000 ms) by
normalizeStartupTimeoutMs in extensions/browser/src/browser/config.ts,
and zero/negative/non-finite values fall back to the defaults. Without
this in the configuration reference, users setting a higher value see
no error and silently get the 120 s ceiling, or set 0 expecting 'no
timeout' and silently get the default.

* docs(browser): clarify startup timeout validation

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-04-25 19:59:53 +01:00
Peter Steinberger
b66f01bdca fix: expose transparent image infer options 2026-04-25 19:58:41 +01:00
Vincent Koc
cd7a8f870b feat(diagnostics-otel): add genai usage span attrs 2026-04-25 11:56:13 -07:00
91wan
bb2b68b34e fix(acp): pass Codex ACP model thinking overrides
Fix ACP Codex model/thinking override propagation.\n\nThanks @91wan.
2026-04-25 19:56:03 +01:00
Peter Steinberger
e9d9726f2d fix: handle context assembled diagnostics 2026-04-25 19:54:28 +01:00
Peter Steinberger
a018db771d fix: preserve omitted thinking replay turns 2026-04-25 19:54:28 +01:00
Peter Steinberger
690c98ad99 test(plugins): align install ledger mocks 2026-04-25 19:54:12 +01:00
Vincent Koc
c410e48382 fix(plugins): keep onboarding install records out of config 2026-04-25 11:52:19 -07:00
Peter Steinberger
bbc0884e23 docs(changelog): restore 2026.4.24 release notes 2026-04-25 19:51:11 +01:00
Vincent Koc
9bd348fdec fix(plugins): harden install ledger path handling 2026-04-25 11:48:17 -07:00
Vincent Koc
dc19069d71 feat(diagnostics-otel): add genai operation duration metric 2026-04-25 11:48:10 -07:00
Peter Steinberger
81307fc11d test: hoist backup archive mocks 2026-04-25 19:48:03 +01:00
Peter Steinberger
599ae7fed8 docs: clarify tool result details persistence 2026-04-25 19:47:19 +01:00
Peter Steinberger
fecf1e9b8f fix: align plugin install tests with ledger store 2026-04-25 19:44:11 +01:00
Peter Steinberger
4c0e9a4b2e fix(plugins): honor inferred agent model defaults 2026-04-25 19:40:32 +01:00
Peter Steinberger
cd8cb8254a fix(logging): remove duplicate context diagnostic case 2026-04-25 19:39:20 +01:00
Peter Steinberger
2055e6ceba fix(logging): include context assembly diagnostics in stability log 2026-04-25 19:39:20 +01:00
Peter Steinberger
8ea3099cd3 test(codex): accept visible session model reply 2026-04-25 19:39:20 +01:00
Peter Steinberger
e4f544790c test: isolate gateway live model sessions 2026-04-25 19:39:20 +01:00
Peter Steinberger
02639d3ec8 fix(plugins): alias wildcard runtime dependency exports 2026-04-25 19:39:20 +01:00
Peter Steinberger
14c9cfb637 fix(plugins): alias runtime dependency export subpaths 2026-04-25 19:39:20 +01:00
Peter Steinberger
9e9aa4722a fix(plugins): load mirrored runtime deps through ESM-safe aliases 2026-04-25 19:39:20 +01:00
Peter Steinberger
d2ab6b4fd5 fix(plugins): preserve package deps for runtime mirrors 2026-04-25 19:39:19 +01:00
Troy Hitch
63241bf1e0 fix(bonjour): suppress ciao cancellation across plugin runtime copies
Fix the bundled Bonjour gateway discovery crash-loop caused by ciao probe cancellation rejections after the Bonjour plugin migration.

The plugin entry now wires the existing rejection handler into the advertiser, and the unhandled-rejection handler registry is anchored on globalThis so staged plugin SDK module copies register into the same process-level handler set used by the host.

Verification:
- pnpm test:serial extensions/bonjour/src/advertiser.test.ts src/infra/unhandled-rejections.fatal-detection.test.ts
- OPENCLAW_LOCAL_CHECK_MODE=throttled pnpm check:changed partially completed: conflict markers plus core/core-test/extensions/extension-test typecheck passed; local lint lane hit a self-lock and was stopped.
2026-04-25 11:38:30 -07:00
Vincent Koc
888448facc feat(plugins): move install records to managed ledger 2026-04-25 11:37:10 -07:00
Peter Steinberger
e473577eaa test(voice): harden live STT transcript checks 2026-04-25 19:36:01 +01:00
Vincent Koc
f204f0c999 docs(logging): document new OTEL metrics and spans from recent diagnostics-otel feats
Five recent diagnostics-otel feat commits added user-facing OpenTelemetry
surfaces but did not update docs/logging.md, so the listed metrics and
spans drifted out of sync with what the plugin actually exports:

- 7bbd47349e adds gen_ai.client.token.usage histogram (GenAI semconv)
- b8a41739d5 adds memory heap/rss histograms, pressure counter and span
- d6ef1fcf24 adds openclaw.tool.loop counters and span
- ff172f46a5 adds openclaw.context.assembled span
- 44114328b4 adds openclaw.provider.request_id_hash attr on
  openclaw.model.call spans

Append the new metrics under existing model-usage and exec sections,
add a 'Diagnostics internals' subsection for memory + tool-loop
metrics, and add the three new spans (context.assembled, tool.loop,
memory.pressure) plus the request-id-hash attribute to the spans
listing.
2026-04-25 11:35:20 -07:00
Vincent Koc
7bbd47349e feat(diagnostics-otel): add genai token usage metric 2026-04-25 11:31:45 -07:00
Peter Steinberger
73706ca244 test: stabilize QA session memory ranking 2026-04-25 19:30:28 +01:00
Peter Steinberger
de0097a23c fix: support transparent OpenAI image generation 2026-04-25 19:28:56 +01:00
Peter Steinberger
0bf4876add fix: sanitize assembled diagnostic context 2026-04-25 19:23:51 +01:00
Peter Steinberger
a00c225899 test: split pure tool-card coverage 2026-04-25 19:23:51 +01:00
Peter Steinberger
e1495c3372 test: streamline memory and tts suites 2026-04-25 19:23:51 +01:00
Peter Steinberger
75fcb8c56d perf: lazy-load heavy test imports 2026-04-25 19:23:51 +01:00
Peter Steinberger
31456e3326 fix(providers): handle proxied DeepSeek V4 replay 2026-04-25 19:23:15 +01:00
162 changed files with 7723 additions and 3289 deletions

View File

@@ -9,6 +9,7 @@ Telegraph style. Root rules only. Read scoped `AGENTS.md` before subtree work.
- Run docs list first: `pnpm docs:list` if available; read relevant docs only.
- High-confidence answers only when fixing/triaging: verify source, tests, shipped/current behavior, and dependency contracts before deciding.
- Dependency-backed behavior: read upstream dependency docs/source/types first. Do not assume APIs, defaults, errors, timing, or runtime behavior.
- Live-verify when feasible. Check env/`~/.profile` for keys before assuming live tests are blocked; keep secret output redacted.
- Missing deps: `pnpm install`, retry once, then report first actionable error.
- CODEOWNERS: maint/refactor/tests ok. Larger behavior/product/security/ownership: owner ask/review.
- Wording: product/docs/UI/changelog say "plugin/plugins"; `extensions/` is internal.

File diff suppressed because it is too large Load Diff

View File

@@ -3,6 +3,7 @@
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_DATA_SYNC" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MICROPHONE" />
<uses-permission android:name="android.permission.POST_NOTIFICATIONS" />
<uses-permission
android:name="android.permission.NEARBY_WIFI_DEVICES"
@@ -52,7 +53,7 @@
<service
android:name=".NodeForegroundService"
android:exported="false"
android:foregroundServiceType="dataSync" />
android:foregroundServiceType="dataSync|microphone" />
<service
android:name=".node.DeviceNotificationListenerService"
android:label="@string/app_name"

View File

@@ -101,7 +101,8 @@ class MainViewModel(app: Application) : AndroidViewModel(app) {
val onboardingCompleted: StateFlow<Boolean> = prefs.onboardingCompleted
val canvasDebugStatusEnabled: StateFlow<Boolean> = prefs.canvasDebugStatusEnabled
val speakerEnabled: StateFlow<Boolean> = prefs.speakerEnabled
val micEnabled: StateFlow<Boolean> = prefs.talkEnabled
val voiceCaptureMode: StateFlow<VoiceCaptureMode> = runtimeState(initial = VoiceCaptureMode.Off) { it.voiceCaptureMode }
val micEnabled: StateFlow<Boolean> = runtimeState(initial = false) { it.micEnabled }
val micCooldown: StateFlow<Boolean> = runtimeState(initial = false) { it.micCooldown }
val micStatusText: StateFlow<String> = runtimeState(initial = "Mic off") { it.micStatusText }
@@ -111,6 +112,10 @@ class MainViewModel(app: Application) : AndroidViewModel(app) {
val micConversation: StateFlow<List<VoiceConversationEntry>> = runtimeState(initial = emptyList()) { it.micConversation }
val micInputLevel: StateFlow<Float> = runtimeState(initial = 0f) { it.micInputLevel }
val micIsSending: StateFlow<Boolean> = runtimeState(initial = false) { it.micIsSending }
val talkModeEnabled: StateFlow<Boolean> = runtimeState(initial = false) { it.talkModeEnabled }
val talkModeListening: StateFlow<Boolean> = runtimeState(initial = false) { it.talkModeListening }
val talkModeSpeaking: StateFlow<Boolean> = runtimeState(initial = false) { it.talkModeSpeaking }
val talkModeStatusText: StateFlow<String> = runtimeState(initial = "Off") { it.talkModeStatusText }
val chatSessionKey: StateFlow<String> = runtimeState(initial = "main") { it.chatSessionKey }
val chatSessionId: StateFlow<String?> = runtimeState(initial = null) { it.chatSessionId }
@@ -283,6 +288,10 @@ class MainViewModel(app: Application) : AndroidViewModel(app) {
ensureRuntime().setMicEnabled(enabled)
}
fun setTalkModeEnabled(enabled: Boolean) {
ensureRuntime().setTalkModeEnabled(enabled)
}
fun setSpeakerEnabled(enabled: Boolean) {
ensureRuntime().setSpeakerEnabled(enabled)
}

View File

@@ -3,12 +3,14 @@ package ai.openclaw.app
import android.app.Notification
import android.app.NotificationChannel
import android.app.NotificationManager
import android.app.Service
import android.app.PendingIntent
import android.app.Service
import android.content.Context
import android.content.Intent
import android.content.pm.ServiceInfo
import androidx.core.app.NotificationCompat
import androidx.core.app.ServiceCompat
import androidx.core.content.ContextCompat
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.Job
@@ -21,6 +23,7 @@ class NodeForegroundService : Service() {
private val scope: CoroutineScope = CoroutineScope(SupervisorJob() + Dispatchers.Main)
private var notificationJob: Job? = null
private var didStartForeground = false
private var voiceCaptureMode = VoiceCaptureMode.Off
override fun onCreate() {
super.onCreate()
@@ -36,22 +39,51 @@ class NodeForegroundService : Service() {
notificationJob =
scope.launch {
combine(
runtime.statusText,
runtime.serverName,
runtime.isConnected,
runtime.micEnabled,
runtime.micIsListening,
) { status, server, connected, micEnabled, micListening ->
Quint(status, server, connected, micEnabled, micListening)
}.collect { (status, server, connected, micEnabled, micListening) ->
val title = if (connected) "OpenClaw Node · Connected" else "OpenClaw Node"
val micSuffix =
if (micEnabled) {
if (micListening) " · Mic: Listening" else " · Mic: Pending"
} else {
""
combine(
runtime.statusText,
runtime.serverName,
runtime.isConnected,
runtime.voiceCaptureMode,
) { status, server, connected, mode ->
VoiceNotificationBase(
status = status,
server = server,
connected = connected,
mode = mode,
)
},
combine(
runtime.micEnabled,
runtime.micIsListening,
runtime.talkModeListening,
runtime.talkModeSpeaking,
) { micEnabled, micListening, talkListening, talkSpeaking ->
VoiceNotificationCapture(
micEnabled = micEnabled,
micListening = micListening,
talkListening = talkListening,
talkSpeaking = talkSpeaking,
)
},
) { base, capture ->
VoiceNotificationState(base = base, capture = capture)
}.collect { state ->
voiceCaptureMode = state.mode
val title =
when {
state.connected && state.mode == VoiceCaptureMode.TalkMode -> "OpenClaw Node · Talk"
state.connected -> "OpenClaw Node · Connected"
else -> "OpenClaw Node"
}
val text = (server?.let { "$status · $it" } ?: status) + micSuffix
val text =
(state.server?.let { "${state.status} · $it" } ?: state.status) +
voiceNotificationSuffix(
mode = state.mode,
manualMicEnabled = state.capture.micEnabled,
manualMicListening = state.capture.micListening,
talkListening = state.capture.talkListening,
talkSpeaking = state.capture.talkSpeaking,
)
startForegroundWithTypes(
notification = buildNotification(title = title, text = text),
@@ -60,13 +92,27 @@ class NodeForegroundService : Service() {
}
}
override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
override fun onStartCommand(
intent: Intent?,
flags: Int,
startId: Int,
): Int {
when (intent?.action) {
ACTION_STOP -> {
(application as NodeApp).peekRuntime()?.disconnect()
stopSelf()
return START_NOT_STICKY
}
ACTION_SET_VOICE_CAPTURE_MODE -> {
voiceCaptureMode = intent.getStringExtra(EXTRA_VOICE_CAPTURE_MODE).toVoiceCaptureMode()
startForegroundWithTypes(
notification =
buildNotification(
title = "OpenClaw Node",
text = if (voiceCaptureMode == VoiceCaptureMode.TalkMode) "Talk mode active" else "Connected",
),
)
}
}
// Keep running; connection is managed by NodeRuntime (auto-reconnect + manual).
return START_STICKY
@@ -127,17 +173,13 @@ class NodeForegroundService : Service() {
.build()
}
private fun updateNotification(notification: Notification) {
val mgr = getSystemService(Context.NOTIFICATION_SERVICE) as NotificationManager
mgr.notify(NOTIFICATION_ID, notification)
}
private fun startForegroundWithTypes(notification: Notification) {
val serviceTypes = foregroundServiceTypesForVoiceMode(voiceCaptureMode)
if (didStartForeground) {
updateNotification(notification)
ServiceCompat.startForeground(this, NOTIFICATION_ID, notification, serviceTypes)
return
}
startForeground(NOTIFICATION_ID, notification, ServiceInfo.FOREGROUND_SERVICE_TYPE_DATA_SYNC)
ServiceCompat.startForeground(this, NOTIFICATION_ID, notification, serviceTypes)
didStartForeground = true
}
@@ -146,6 +188,8 @@ class NodeForegroundService : Service() {
private const val NOTIFICATION_ID = 1
private const val ACTION_STOP = "ai.openclaw.app.action.STOP"
private const val ACTION_SET_VOICE_CAPTURE_MODE = "ai.openclaw.app.action.SET_VOICE_CAPTURE_MODE"
private const val EXTRA_VOICE_CAPTURE_MODE = "ai.openclaw.app.extra.VOICE_CAPTURE_MODE"
fun start(context: Context) {
val intent = Intent(context, NodeForegroundService::class.java)
@@ -156,7 +200,85 @@ class NodeForegroundService : Service() {
val intent = Intent(context, NodeForegroundService::class.java).setAction(ACTION_STOP)
context.startService(intent)
}
fun setVoiceCaptureMode(
context: Context,
mode: VoiceCaptureMode,
) {
val intent =
Intent(context, NodeForegroundService::class.java)
.setAction(ACTION_SET_VOICE_CAPTURE_MODE)
.putExtra(EXTRA_VOICE_CAPTURE_MODE, mode.name)
if (mode == VoiceCaptureMode.TalkMode) {
ContextCompat.startForegroundService(context, intent)
} else {
context.startService(intent)
}
}
}
}
private data class Quint<A, B, C, D, E>(val first: A, val second: B, val third: C, val fourth: D, val fifth: E)
internal fun foregroundServiceTypesForVoiceMode(mode: VoiceCaptureMode): Int {
val base = ServiceInfo.FOREGROUND_SERVICE_TYPE_DATA_SYNC
return if (mode == VoiceCaptureMode.TalkMode) {
base or ServiceInfo.FOREGROUND_SERVICE_TYPE_MICROPHONE
} else {
base
}
}
internal fun voiceNotificationSuffix(
mode: VoiceCaptureMode,
manualMicEnabled: Boolean,
manualMicListening: Boolean,
talkListening: Boolean,
talkSpeaking: Boolean,
): String {
return when (mode) {
VoiceCaptureMode.TalkMode ->
when {
talkSpeaking -> " · Talk: Speaking"
talkListening -> " · Talk: Listening"
else -> " · Talk: On"
}
VoiceCaptureMode.ManualMic ->
if (manualMicEnabled) {
if (manualMicListening) " · Mic: Listening" else " · Mic: Pending"
} else {
""
}
VoiceCaptureMode.Off -> ""
}
}
private fun String?.toVoiceCaptureMode(): VoiceCaptureMode {
return VoiceCaptureMode.entries.firstOrNull { it.name == this } ?: VoiceCaptureMode.Off
}
private data class VoiceNotificationBase(
val status: String,
val server: String?,
val connected: Boolean,
val mode: VoiceCaptureMode,
)
private data class VoiceNotificationCapture(
val micEnabled: Boolean,
val micListening: Boolean,
val talkListening: Boolean,
val talkSpeaking: Boolean,
)
private data class VoiceNotificationState(
val base: VoiceNotificationBase,
val capture: VoiceNotificationCapture,
) {
val status: String
get() = base.status
val server: String?
get() = base.server
val connected: Boolean
get() = base.connected
val mode: VoiceCaptureMode
get() = base.mode
}

View File

@@ -64,6 +64,8 @@ class NodeRuntime(
private val json = Json { ignoreUnknownKeys = true }
private val externalAudioCaptureActive = MutableStateFlow(false)
private val _voiceCaptureMode = MutableStateFlow(VoiceCaptureMode.Off)
val voiceCaptureMode: StateFlow<VoiceCaptureMode> = _voiceCaptureMode.asStateFlow()
private val discovery = GatewayDiscovery(appContext, scope = scope)
val gateways: StateFlow<List<GatewayEndpoint>> = discovery.gateways
@@ -428,6 +430,18 @@ class NodeRuntime(
)
}
val talkModeEnabled: StateFlow<Boolean>
get() = talkMode.isEnabled
val talkModeListening: StateFlow<Boolean>
get() = talkMode.isListening
val talkModeSpeaking: StateFlow<Boolean>
get() = talkMode.isSpeaking
val talkModeStatusText: StateFlow<String>
get() = talkMode.statusText
private fun syncMainSessionKey(agentId: String?) {
val resolvedKey = resolveNodeMainSessionKey(agentId)
// Always push the resolved session key into TalkMode, even when the
@@ -599,17 +613,8 @@ class NodeRuntime(
prefs.loadGatewayToken()
}
scope.launch {
prefs.talkEnabled.collect { enabled ->
// MicCaptureManager handles STT + send to gateway, while the dedicated
// reply speaker handles TTS for assistant replies in the voice tab.
micCapture.setMicEnabled(enabled)
if (enabled) {
talkMode.ttsOnAllResponses = false
scope.launch { talkMode.ensureChatSubscribed() }
}
externalAudioCaptureActive.value = enabled
}
if (prefs.voiceMicEnabled.value) {
setVoiceCaptureMode(VoiceCaptureMode.ManualMic, persistManualMic = false)
}
scope.launch(Dispatchers.Default) {
@@ -643,7 +648,7 @@ class NodeRuntime(
if (value) {
reconnectPreferredGatewayOnForeground()
} else {
stopActiveVoiceSession()
stopManualVoiceSession()
}
}
@@ -757,21 +762,17 @@ class NodeRuntime(
fun setVoiceScreenActive(active: Boolean) {
if (!active) {
stopActiveVoiceSession()
stopManualVoiceSession()
}
// Don't re-enable on active=true; mic toggle drives that
}
fun setMicEnabled(value: Boolean) {
prefs.setTalkEnabled(value)
if (value) {
// Tapping mic on interrupts any active TTS (barge-in)
stopVoicePlayback()
talkMode.ttsOnAllResponses = false
scope.launch { talkMode.ensureChatSubscribed() }
}
micCapture.setMicEnabled(value)
externalAudioCaptureActive.value = value
setVoiceCaptureMode(if (value) VoiceCaptureMode.ManualMic else VoiceCaptureMode.Off)
}
fun setTalkModeEnabled(value: Boolean) {
setVoiceCaptureMode(if (value) VoiceCaptureMode.TalkMode else VoiceCaptureMode.Off)
}
val speakerEnabled: StateFlow<Boolean>
@@ -786,11 +787,72 @@ class NodeRuntime(
talkMode.setPlaybackEnabled(value)
}
private fun setVoiceCaptureMode(
mode: VoiceCaptureMode,
persistManualMic: Boolean = true,
) {
if (mode == VoiceCaptureMode.TalkMode && !hasRecordAudioPermission()) {
_voiceCaptureMode.value = VoiceCaptureMode.Off
externalAudioCaptureActive.value = false
return
}
if (_voiceCaptureMode.value == mode) return
_voiceCaptureMode.value = mode
when (mode) {
VoiceCaptureMode.Off -> {
talkMode.ttsOnAllResponses = false
talkMode.setEnabled(false)
stopVoicePlayback()
micCapture.setMicEnabled(false)
if (persistManualMic) {
prefs.setVoiceMicEnabled(false)
}
NodeForegroundService.setVoiceCaptureMode(appContext, VoiceCaptureMode.Off)
externalAudioCaptureActive.value = false
}
VoiceCaptureMode.ManualMic -> {
talkMode.ttsOnAllResponses = false
talkMode.setEnabled(false)
NodeForegroundService.setVoiceCaptureMode(appContext, VoiceCaptureMode.ManualMic)
if (persistManualMic) {
prefs.setVoiceMicEnabled(true)
}
// Tapping mic on interrupts any active TTS (barge-in).
stopVoicePlayback()
scope.launch { talkMode.ensureChatSubscribed() }
micCapture.setMicEnabled(true)
externalAudioCaptureActive.value = true
}
VoiceCaptureMode.TalkMode -> {
if (persistManualMic) {
prefs.setVoiceMicEnabled(false)
}
micCapture.setMicEnabled(false)
NodeForegroundService.setVoiceCaptureMode(appContext, VoiceCaptureMode.TalkMode)
talkMode.ttsOnAllResponses = true
talkMode.setPlaybackEnabled(speakerEnabled.value)
scope.launch { talkMode.ensureChatSubscribed() }
talkMode.setEnabled(true)
externalAudioCaptureActive.value = true
}
}
}
private fun stopManualVoiceSession() {
if (_voiceCaptureMode.value != VoiceCaptureMode.ManualMic) return
setVoiceCaptureMode(VoiceCaptureMode.Off)
}
private fun stopActiveVoiceSession() {
talkMode.ttsOnAllResponses = false
talkMode.setEnabled(false)
stopVoicePlayback()
micCapture.setMicEnabled(false)
prefs.setTalkEnabled(false)
prefs.setVoiceMicEnabled(false)
NodeForegroundService.setVoiceCaptureMode(appContext, VoiceCaptureMode.Off)
_voiceCaptureMode.value = VoiceCaptureMode.Off
externalAudioCaptureActive.value = false
}
@@ -970,6 +1032,7 @@ class NodeRuntime(
}
fun disconnect() {
stopActiveVoiceSession()
connectedEndpoint = null
activeGatewayAuth = null
_pendingGatewayTrust.value = null

View File

@@ -37,6 +37,7 @@ class SecurePrefs(
private const val notificationsForwardingMaxEventsPerMinuteKey =
"notifications.forwarding.maxEventsPerMinute"
private const val notificationsForwardingSessionKeyKey = "notifications.forwarding.sessionKey"
private const val voiceMicEnabledKey = "voice.micEnabled"
}
private val appContext = context.applicationContext
@@ -162,8 +163,8 @@ class SecurePrefs(
private val _voiceWakeMode = MutableStateFlow(loadVoiceWakeMode())
val voiceWakeMode: StateFlow<VoiceWakeMode> = _voiceWakeMode
private val _talkEnabled = MutableStateFlow(plainPrefs.getBoolean("talk.enabled", false))
val talkEnabled: StateFlow<Boolean> = _talkEnabled
private val _voiceMicEnabled = MutableStateFlow(plainPrefs.getBoolean(voiceMicEnabledKey, false))
val voiceMicEnabled: StateFlow<Boolean> = _voiceMicEnabled
private val _speakerEnabled = MutableStateFlow(plainPrefs.getBoolean("voice.speakerEnabled", true))
val speakerEnabled: StateFlow<Boolean> = _speakerEnabled
@@ -478,9 +479,9 @@ class SecurePrefs(
_voiceWakeMode.value = mode
}
fun setTalkEnabled(value: Boolean) {
plainPrefs.edit { putBoolean("talk.enabled", value) }
_talkEnabled.value = value
fun setVoiceMicEnabled(value: Boolean) {
plainPrefs.edit { putBoolean(voiceMicEnabledKey, value) }
_voiceMicEnabled.value = value
}
fun setSpeakerEnabled(value: Boolean) {

View File

@@ -0,0 +1,7 @@
package ai.openclaw.app
enum class VoiceCaptureMode {
Off,
ManualMic,
TalkMode,
}

View File

@@ -35,10 +35,11 @@ import androidx.compose.foundation.lazy.rememberLazyListState
import androidx.compose.foundation.shape.CircleShape
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.Mic
import androidx.compose.material.icons.filled.MicOff
import androidx.compose.material.icons.automirrored.filled.VolumeOff
import androidx.compose.material.icons.automirrored.filled.VolumeUp
import androidx.compose.material.icons.filled.Mic
import androidx.compose.material.icons.filled.MicOff
import androidx.compose.material.icons.filled.RecordVoiceOver
import androidx.compose.material3.Button
import androidx.compose.material3.ButtonDefaults
import androidx.compose.material3.Icon
@@ -69,6 +70,7 @@ import androidx.lifecycle.Lifecycle
import androidx.lifecycle.LifecycleEventObserver
import androidx.lifecycle.compose.LocalLifecycleOwner
import ai.openclaw.app.MainViewModel
import ai.openclaw.app.VoiceCaptureMode
import ai.openclaw.app.voice.VoiceConversationEntry
import ai.openclaw.app.voice.VoiceConversationRole
import kotlin.math.max
@@ -81,6 +83,7 @@ fun VoiceTabScreen(viewModel: MainViewModel) {
val listState = rememberLazyListState()
val gatewayStatus by viewModel.statusText.collectAsState()
val voiceCaptureMode by viewModel.voiceCaptureMode.collectAsState()
val micEnabled by viewModel.micEnabled.collectAsState()
val micCooldown by viewModel.micCooldown.collectAsState()
val speakerEnabled by viewModel.speakerEnabled.collectAsState()
@@ -90,12 +93,15 @@ fun VoiceTabScreen(viewModel: MainViewModel) {
val micConversation by viewModel.micConversation.collectAsState()
val micInputLevel by viewModel.micInputLevel.collectAsState()
val micIsSending by viewModel.micIsSending.collectAsState()
val talkModeEnabled by viewModel.talkModeEnabled.collectAsState()
val talkModeListening by viewModel.talkModeListening.collectAsState()
val talkModeSpeaking by viewModel.talkModeSpeaking.collectAsState()
val hasStreamingAssistant = micConversation.any { it.role == VoiceConversationRole.Assistant && it.isStreaming }
val showThinkingBubble = micIsSending && !hasStreamingAssistant
var hasMicPermission by remember { mutableStateOf(context.hasRecordAudioPermission()) }
var pendingMicEnable by remember { mutableStateOf(false) }
var pendingVoicePermissionAction by remember { mutableStateOf<PendingVoicePermissionAction?>(null) }
DisposableEffect(lifecycleOwner, context) {
val observer =
@@ -107,7 +113,7 @@ fun VoiceTabScreen(viewModel: MainViewModel) {
lifecycleOwner.lifecycle.addObserver(observer)
onDispose {
lifecycleOwner.lifecycle.removeObserver(observer)
// Stop TTS when leaving the voice screen
// Manual mic is tied to the Voice tab; Talk Mode is explicit and can continue.
viewModel.setVoiceScreenActive(false)
}
}
@@ -115,10 +121,14 @@ fun VoiceTabScreen(viewModel: MainViewModel) {
val requestMicPermission =
rememberLauncherForActivityResult(ActivityResultContracts.RequestPermission()) { granted ->
hasMicPermission = granted
if (granted && pendingMicEnable) {
viewModel.setMicEnabled(true)
if (granted) {
when (pendingVoicePermissionAction) {
PendingVoicePermissionAction.ManualMic -> viewModel.setMicEnabled(true)
PendingVoicePermissionAction.TalkMode -> viewModel.setTalkModeEnabled(true)
null -> Unit
}
}
pendingMicEnable = false
pendingVoicePermissionAction = null
}
LaunchedEffect(micConversation.size, showThinkingBubble) {
@@ -161,12 +171,12 @@ fun VoiceTabScreen(viewModel: MainViewModel) {
tint = mobileTextTertiary,
)
Text(
"Tap the mic to start",
"Tap mic or Talk",
style = mobileHeadline,
color = mobileTextSecondary,
)
Text(
"Each pause sends a turn automatically.",
"Mic sends turns; Talk keeps the conversation open.",
style = mobileCallout,
color = mobileTextTertiary,
)
@@ -263,7 +273,7 @@ fun VoiceTabScreen(viewModel: MainViewModel) {
if (hasMicPermission) {
viewModel.setMicEnabled(true)
} else {
pendingMicEnable = true
pendingVoicePermissionAction = PendingVoicePermissionAction.ManualMic
requestMicPermission.launch(Manifest.permission.RECORD_AUDIO)
}
},
@@ -287,11 +297,39 @@ fun VoiceTabScreen(viewModel: MainViewModel) {
}
}
// Invisible spacer to balance the row (matches speaker column width)
Column(horizontalAlignment = Alignment.CenterHorizontally) {
Box(modifier = Modifier.size(48.dp))
Column(horizontalAlignment = Alignment.CenterHorizontally, verticalArrangement = Arrangement.spacedBy(4.dp)) {
IconButton(
onClick = {
if (talkModeEnabled) {
viewModel.setTalkModeEnabled(false)
return@IconButton
}
if (hasMicPermission) {
viewModel.setTalkModeEnabled(true)
} else {
pendingVoicePermissionAction = PendingVoicePermissionAction.TalkMode
requestMicPermission.launch(Manifest.permission.RECORD_AUDIO)
}
},
modifier = Modifier.size(48.dp),
colors =
IconButtonDefaults.iconButtonColors(
containerColor = if (talkModeEnabled) mobileSuccessSoft else mobileSurface,
),
) {
Icon(
imageVector = Icons.Default.RecordVoiceOver,
contentDescription = if (talkModeEnabled) "Turn Talk Mode off" else "Turn Talk Mode on",
modifier = Modifier.size(22.dp),
tint = if (talkModeEnabled) mobileSuccess else mobileTextSecondary,
)
}
Spacer(modifier = Modifier.height(4.dp))
Text("", style = mobileCaption2)
Text(
if (talkModeEnabled) "Talk on" else "Talk",
style = mobileCaption2,
color = if (talkModeEnabled) mobileSuccess else mobileTextTertiary,
)
}
}
@@ -299,6 +337,9 @@ fun VoiceTabScreen(viewModel: MainViewModel) {
val queueCount = micQueuedMessages.size
val stateText =
when {
voiceCaptureMode == VoiceCaptureMode.TalkMode && talkModeSpeaking -> "Talk speaking"
voiceCaptureMode == VoiceCaptureMode.TalkMode && talkModeListening -> "Talk listening"
voiceCaptureMode == VoiceCaptureMode.TalkMode -> "Talk on"
queueCount > 0 -> "$queueCount queued"
micIsSending -> "Sending"
micCooldown -> "Cooldown"
@@ -307,14 +348,15 @@ fun VoiceTabScreen(viewModel: MainViewModel) {
}
val stateColor =
when {
voiceCaptureMode == VoiceCaptureMode.TalkMode -> mobileSuccess
micEnabled -> mobileSuccess
micIsSending -> mobileAccent
else -> mobileTextSecondary
}
Surface(
shape = RoundedCornerShape(999.dp),
color = if (micEnabled) mobileSuccessSoft else mobileSurface,
border = BorderStroke(1.dp, if (micEnabled) mobileSuccess.copy(alpha = 0.3f) else mobileBorder),
color = if (micEnabled || talkModeEnabled) mobileSuccessSoft else mobileSurface,
border = BorderStroke(1.dp, if (micEnabled || talkModeEnabled) mobileSuccess.copy(alpha = 0.3f) else mobileBorder),
) {
Text(
"$gatewayStatus · $stateText",
@@ -353,6 +395,11 @@ fun VoiceTabScreen(viewModel: MainViewModel) {
}
}
private enum class PendingVoicePermissionAction {
ManualMic,
TalkMode,
}
@Composable
private fun VoiceTurnBubble(entry: VoiceConversationEntry) {
val isUser = entry.role == VoiceConversationRole.User

View File

@@ -2,6 +2,7 @@ package ai.openclaw.app
import android.app.Notification
import android.content.Intent
import android.content.pm.ServiceInfo
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNotNull
import org.junit.Test
@@ -30,6 +31,35 @@ class NodeForegroundServiceTest {
assertEquals(expectedFlags, savedIntent.flags and expectedFlags)
}
@Test
fun foregroundServiceTypesForVoiceMode_addsMicrophoneOnlyForTalkMode() {
assertEquals(
ServiceInfo.FOREGROUND_SERVICE_TYPE_DATA_SYNC,
foregroundServiceTypesForVoiceMode(VoiceCaptureMode.Off),
)
assertEquals(
ServiceInfo.FOREGROUND_SERVICE_TYPE_DATA_SYNC,
foregroundServiceTypesForVoiceMode(VoiceCaptureMode.ManualMic),
)
assertEquals(
ServiceInfo.FOREGROUND_SERVICE_TYPE_DATA_SYNC or ServiceInfo.FOREGROUND_SERVICE_TYPE_MICROPHONE,
foregroundServiceTypesForVoiceMode(VoiceCaptureMode.TalkMode),
)
}
@Test
fun voiceNotificationSuffixReflectsActiveCaptureMode() {
assertEquals("", voiceNotificationSuffix(VoiceCaptureMode.Off, false, false, false, false))
assertEquals(
" · Mic: Listening",
voiceNotificationSuffix(VoiceCaptureMode.ManualMic, true, true, false, false),
)
assertEquals(
" · Talk: Speaking",
voiceNotificationSuffix(VoiceCaptureMode.TalkMode, false, false, true, true),
)
}
private fun buildNotification(service: NodeForegroundService): Notification {
val method =
NodeForegroundService::class.java.getDeclaredMethod(

View File

@@ -2,7 +2,9 @@ package ai.openclaw.app
import android.content.Context
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertNull
import org.junit.Assert.assertTrue
import org.junit.Test
import org.junit.runner.RunWith
import org.robolectric.RobolectricTestRunner
@@ -22,6 +24,32 @@ class SecurePrefsTest {
assertEquals("whileUsing", plainPrefs.getString("location.enabledMode", null))
}
@Test
fun voiceMicEnabled_ignoresOldTalkEnabledKey() {
val context = RuntimeEnvironment.getApplication()
val plainPrefs = context.getSharedPreferences("openclaw.node", Context.MODE_PRIVATE)
plainPrefs.edit().clear().putBoolean("talk.enabled", true).commit()
val prefs = SecurePrefs(context)
assertFalse(prefs.voiceMicEnabled.value)
assertFalse(plainPrefs.contains("voice.micEnabled"))
}
@Test
fun setVoiceMicEnabled_persistsNewKeyOnly() {
val context = RuntimeEnvironment.getApplication()
val plainPrefs = context.getSharedPreferences("openclaw.node", Context.MODE_PRIVATE)
plainPrefs.edit().clear().putBoolean("talk.enabled", false).commit()
val prefs = SecurePrefs(context)
prefs.setVoiceMicEnabled(true)
assertTrue(prefs.voiceMicEnabled.value)
assertTrue(plainPrefs.getBoolean("voice.micEnabled", false))
assertFalse(plainPrefs.getBoolean("talk.enabled", false))
}
@Test
fun saveGatewayBootstrapToken_persistsSeparatelyFromSharedToken() {
val context = RuntimeEnvironment.getApplication()

View File

@@ -1,4 +1,4 @@
c8d24c55df89a76f44cd6ab5fdb7c28b0b3a8adadcd2c94a1d81263512075c0f config-baseline.json
97c37380e03c167ee710adb0ee297573146e78434635780226b744841628370b config-baseline.core.json
6ed33ef102e7c92816243bfabc3626222a679c3270c12ec5ea47b28b66204b3b config-baseline.json
f86cb4d57ec1f5fd75008be0ab86151194945eb013a47ab4bdeaddafd3780da7 config-baseline.core.json
7cd9c908f066c143eab2a201efbc9640f483ab28bba92ddeca1d18cc2b528bc3 config-baseline.channel.json
7825b56a5b3fcdbe2e09ef8fe5d9f12ac3598435afebe20413051e45b0d1968e config-baseline.plugin.json

View File

@@ -156,7 +156,9 @@ Use `image` for generation, edit, and description.
```bash
openclaw infer image generate --prompt "friendly lobster illustration" --json
openclaw infer image generate --prompt "cinematic product photo of headphones" --json
openclaw infer image generate --model openai/gpt-image-1.5 --output-format png --background transparent --prompt "simple red circle sticker on a transparent background" --json
openclaw infer image generate --prompt "slow image backend" --timeout-ms 180000 --json
openclaw infer image edit --file ./logo.png --model openai/gpt-image-1.5 --output-format png --background transparent --prompt "keep the logo, remove the background" --json
openclaw infer image describe --file ./photo.jpg --json
openclaw infer image describe --file ./ui-screenshot.png --model openai/gpt-4.1-mini --json
openclaw infer image describe --file ./photo.jpg --model ollama/qwen2.5vl:7b --json
@@ -165,6 +167,10 @@ openclaw infer image describe --file ./photo.jpg --model ollama/qwen2.5vl:7b --j
Notes:
- Use `image edit` when starting from existing input files.
- Use `--output-format png --background transparent` with
`--model openai/gpt-image-1.5` for transparent-background OpenAI PNG output;
`--openai-background` remains available as an OpenAI-specific alias. Providers
that do not declare background support report the hint as an ignored override.
- Use `image providers --json` to verify which bundled image providers are
discoverable, configured, selected, and which generation/edit capabilities
each provider exposes.

View File

@@ -77,6 +77,19 @@ gateway-backed session transcript, so they are the source of truth.
Details: [Session management](/concepts/session).
## Tool result metadata
Tool result `content` is the model-visible result. Tool result `details` is
runtime metadata for UI rendering, diagnostics, media delivery, and plugins.
OpenClaw keeps that boundary explicit:
- `toolResult.details` is stripped before provider replay and compaction input.
- Persisted session transcripts keep only bounded `details`; oversized metadata
is replaced with a compact summary marked `persistedDetailsTruncated: true`.
- Plugins and tools should put text the model must read in `content`, not only
in `details`.
## Inbound bodies and history context
OpenClaw separates the **prompt body** from the **command body**:

View File

@@ -342,8 +342,8 @@ Time format in system prompt. Default: `auto` (OS preference).
- Also used as fallback routing when the selected/default model cannot accept image input.
- `imageGenerationModel`: accepts either a string (`"provider/model"`) or an object (`{ primary, fallbacks }`).
- Used by the shared image-generation capability and any future tool/plugin surface that generates images.
- Typical values: `google/gemini-3.1-flash-image-preview` for native Gemini image generation, `fal/fal-ai/flux/dev` for fal, or `openai/gpt-image-2` for OpenAI Images.
- If you select a provider/model directly, configure matching provider auth too (for example `GEMINI_API_KEY` or `GOOGLE_API_KEY` for `google/*`, `OPENAI_API_KEY` or OpenAI Codex OAuth for `openai/gpt-image-2`, `FAL_KEY` for `fal/*`).
- Typical values: `google/gemini-3.1-flash-image-preview` for native Gemini image generation, `fal/fal-ai/flux/dev` for fal, `openai/gpt-image-2` for OpenAI Images, or `openai/gpt-image-1.5` for transparent-background OpenAI PNG/WebP output.
- If you select a provider/model directly, configure matching provider auth too (for example `GEMINI_API_KEY` or `GOOGLE_API_KEY` for `google/*`, `OPENAI_API_KEY` or OpenAI Codex OAuth for `openai/gpt-image-2` / `openai/gpt-image-1.5`, `FAL_KEY` for `fal/*`).
- If omitted, `image_generate` can still infer an auth-backed provider default. It tries the current default provider first, then the remaining registered image-generation providers in provider-id order.
- `musicGenerationModel`: accepts either a string (`"provider/model"`) or an object (`{ primary, fallbacks }`).
- Used by the shared music-generation capability and the built-in `music_generate` tool.

View File

@@ -280,9 +280,12 @@ See [Plugins](/tools/plugin).
- Local managed profiles use `browser.localLaunchTimeoutMs` for Chrome CDP HTTP
discovery after process start and `browser.localCdpReadyTimeoutMs` for
post-launch CDP websocket readiness. Raise them on slower hosts where Chrome
starts successfully but readiness checks race startup.
starts successfully but readiness checks race startup. Both values must be
positive integers up to `120000` ms; invalid config values are rejected.
- Auto-detect order: default browser if Chromium-based → Chrome → Brave → Edge → Chromium → Chrome Canary.
- `browser.executablePath` accepts `~` for your OS home directory.
- `browser.executablePath` and `browser.profiles.<name>.executablePath` both
accept `~` and `~/...` for your OS home directory before Chromium launch.
Per-profile `userDataDir` on `existing-session` profiles is also tilde-expanded.
- Control service: loopback only (port derived from `gateway.port`, default `18791`).
- `extraArgs` appends extra launch flags to local Chromium startup (for example
`--disable-gpu`, window sizing, or debug flags).
@@ -914,6 +917,7 @@ Notes:
- `otel.sampleRate`: trace sampling rate `0``1`.
- `otel.flushIntervalMs`: periodic telemetry flush interval in ms.
- `otel.captureContent`: opt-in raw content capture for OTEL span attributes. Defaults to off. Boolean `true` captures non-system message/tool content; the object form lets you enable `inputMessages`, `outputMessages`, `toolInputs`, `toolOutputs`, and `systemPrompt` explicitly.
- `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental`: environment toggle for latest experimental GenAI span provider attributes. By default spans keep the legacy `gen_ai.system` attribute for compatibility; GenAI metrics use bounded semantic attributes.
- `OPENCLAW_OTEL_PRELOADED=1`: environment toggle for hosts that already registered a global OpenTelemetry SDK. OpenClaw then skips plugin-owned SDK startup/shutdown while keeping diagnostic listeners active.
- `cacheTrace.enabled`: log cache trace snapshots for embedded runs (default: `false`).
- `cacheTrace.filePath`: output path for cache trace JSONL (default: `$OPENCLAW_STATE_DIR/logs/cache-trace.jsonl`).

View File

@@ -198,6 +198,9 @@ diagnostics + the exporter plugin are enabled.
Model usage:
- `model.usage`: tokens, cost, duration, context, provider/model/channel, session ids.
`usage` is provider/turn accounting for cost and telemetry; `context.used`
is the current prompt/context snapshot and can be lower than provider
`usage.total` when cached input or tool-loop calls are involved.
Message flow:
@@ -307,7 +310,8 @@ Notes:
- You can also enable the plugin with `openclaw plugins enable diagnostics-otel`.
- `protocol` currently supports `http/protobuf` only. `grpc` is ignored.
- Metrics include token usage, cost, context size, run duration, and message-flow
counters/histograms (webhooks, queueing, session state, queue depth/wait).
counters/histograms (webhooks, queueing, session state, queue depth/wait),
plus GenAI token usage and model-call duration histograms.
- Traces/metrics can be toggled with `traces` / `metrics` (default: on). Traces
include model usage spans plus webhook/message processing spans when enabled.
- Raw model/tool content is not exported by default. Use
@@ -316,6 +320,10 @@ Notes:
- Set `headers` when your collector requires auth.
- Environment variables supported: `OTEL_EXPORTER_OTLP_ENDPOINT`,
`OTEL_SERVICE_NAME`, `OTEL_EXPORTER_OTLP_PROTOCOL`.
- Set `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` to emit the
latest experimental GenAI provider span attribute (`gen_ai.provider.name`)
instead of the legacy span attribute (`gen_ai.system`). GenAI metrics always
use bounded, low-cardinality semantic attributes.
- Set `OPENCLAW_OTEL_PRELOADED=1` when another preload or host process already
registered the global OpenTelemetry SDK. In that mode the plugin does not start
or shut down its own SDK, but it still wires OpenClaw diagnostic listeners and
@@ -333,6 +341,12 @@ Model usage:
`openclaw.provider`, `openclaw.model`)
- `openclaw.context.tokens` (histogram, attrs: `openclaw.context`,
`openclaw.channel`, `openclaw.provider`, `openclaw.model`)
- `gen_ai.client.token.usage` (histogram, GenAI semantic-conventions metric,
attrs: `gen_ai.token.type` = `input`/`output`, `gen_ai.provider.name`,
`gen_ai.operation.name`, `gen_ai.request.model`)
- `gen_ai.client.operation.duration` (histogram, seconds, GenAI
semantic-conventions metric, attrs: `gen_ai.provider.name`,
`gen_ai.operation.name`, `gen_ai.request.model`, optional `error.type`)
Message flow:
@@ -371,18 +385,35 @@ Exec:
- `openclaw.exec.duration_ms` (histogram, attrs: `openclaw.exec.target`,
`openclaw.exec.mode`, `openclaw.outcome`, `openclaw.failureKind`)
Diagnostics internals (memory + tool loop):
- `openclaw.memory.heap_used_bytes` (histogram, attrs: `openclaw.memory.kind`)
- `openclaw.memory.rss_bytes` (histogram)
- `openclaw.memory.pressure` (counter, attrs: `openclaw.memory.level`)
- `openclaw.tool.loop.iterations` (counter, attrs: `openclaw.toolName`,
`openclaw.outcome`)
- `openclaw.tool.loop.duration_ms` (histogram, attrs: `openclaw.toolName`,
`openclaw.outcome`)
### Exported spans (names + key attributes)
- `openclaw.model.usage`
- `openclaw.channel`, `openclaw.provider`, `openclaw.model`
- `openclaw.tokens.*` (input/output/cache_read/cache_write/total)
- `gen_ai.system` by default, or `gen_ai.provider.name` when latest GenAI
semantic conventions are opted in
- `gen_ai.request.model`, `gen_ai.operation.name`, `gen_ai.usage.*`
- `openclaw.run`
- `openclaw.outcome`, `openclaw.channel`, `openclaw.provider`,
`openclaw.model`, `openclaw.errorCategory`
- `openclaw.model.call`
- `gen_ai.system`, `gen_ai.request.model`, `gen_ai.operation.name`,
- `gen_ai.system` by default, or `gen_ai.provider.name` when latest GenAI
semantic conventions are opted in
- `gen_ai.request.model`, `gen_ai.operation.name`,
`openclaw.provider`, `openclaw.model`, `openclaw.api`,
`openclaw.transport`
`openclaw.transport`, `openclaw.provider.request_id_hash` (bounded
SHA-based hash of the upstream provider request id; raw ids are not
exported)
- `openclaw.tool.execution`
- `gen_ai.tool.name`, `openclaw.toolName`, `openclaw.errorCategory`,
`openclaw.tool.params.*`
@@ -403,6 +434,16 @@ Exec:
`openclaw.errorCategory`, `openclaw.delivery.result_count`
- `openclaw.session.stuck`
- `openclaw.state`, `openclaw.ageMs`, `openclaw.queueDepth`
- `openclaw.context.assembled`
- `openclaw.prompt.size`, `openclaw.history.size`,
`openclaw.context.tokens`, `openclaw.errorCategory` (no prompt,
history, response, or session-key content)
- `openclaw.tool.loop`
- `openclaw.toolName`, `openclaw.outcome`, `openclaw.iterations`,
`openclaw.errorCategory` (no loop messages, params, or tool output)
- `openclaw.memory.pressure`
- `openclaw.memory.level`, `openclaw.memory.heap_used_bytes`,
`openclaw.memory.rss_bytes`
When content capture is explicitly enabled, model/tool spans can also include
bounded, redacted `openclaw.content.*` attributes for the specific content
@@ -419,6 +460,9 @@ classes you opted into.
`OTEL_EXPORTER_OTLP_ENDPOINT`.
- If the endpoint already contains `/v1/traces` or `/v1/metrics`, it is used as-is.
- If the endpoint already contains `/v1/logs`, it is used as-is for logs.
- `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` controls only the
GenAI span provider attribute shape. Existing dashboards that read
`gen_ai.system` can keep the default until they migrate.
- `OPENCLAW_OTEL_PRELOADED=1` reuses an externally registered OpenTelemetry SDK
for traces/metrics instead of starting a plugin-owned NodeSDK.
- `diagnostics.otel.logs` enables OTLP log export for the main logger output.

View File

@@ -91,6 +91,13 @@ Defaults:
- Click cloud: stop speaking
- Click X: exit Talk mode
## Android UI
- Voice tab toggle: **Talk**
- Manual **Mic** and **Talk** are mutually exclusive runtime capture modes.
- Manual Mic stops when the app leaves the foreground or the user leaves the Voice tab.
- Talk Mode keeps running until toggled off or the Android node disconnects, and uses Android's microphone foreground-service type while active.
## Notes
- Requires Speech + Microphone permissions.

View File

@@ -199,8 +199,10 @@ See [Camera node](/nodes/camera) for parameters and CLI helpers.
### 8) Voice + expanded Android command surface
- Voice: Android uses a single mic on/off flow in the Voice tab with transcript capture and `talk.speak` playback. Local system TTS is used only when `talk.speak` is unavailable. Voice stops when the app leaves the foreground.
- Voice wake/talk-mode toggles are currently removed from Android UX/runtime.
- Voice tab: Android has two explicit capture modes. **Mic** is a manual Voice-tab session that sends each pause as a chat turn and stops when the app leaves the foreground or the user leaves the Voice tab. **Talk** is continuous Talk Mode and keeps listening until toggled off or the node disconnects.
- Talk Mode promotes the existing foreground service from `dataSync` to `dataSync|microphone` before capture starts, then demotes it when Talk Mode stops. Android 14+ requires the `FOREGROUND_SERVICE_MICROPHONE` declaration, the `RECORD_AUDIO` runtime grant, and the microphone service type at runtime.
- Spoken replies use `talk.speak` through the configured gateway Talk provider. Local system TTS is used only when `talk.speak` is unavailable.
- Voice wake remains disabled in the Android UX/runtime.
- Additional Android command families (availability depends on device + permissions):
- `device.status`, `device.info`, `device.permissions`, `device.health`
- `notifications.list`, `notifications.actions` (see [Notification forwarding](#notification-forwarding) below)

View File

@@ -920,6 +920,9 @@ source-plane diagnostics without adding a second raw filesystem-path disclosure
surface. Legacy `plugins.installs` config entries are still read as a
compatibility fallback while the state-managed `plugins/installs.json` ledger
becomes the install source of truth.
`openclaw doctor --fix` migrates those legacy config entries into the managed
ledger and refreshes the cold registry index without loading plugin runtime
modules.
## Context engine plugins

View File

@@ -79,6 +79,8 @@ audio bridge, node pinning, delayed realtime intro, and, when Twilio delegation
is configured, whether the `voice-call` plugin and Twilio credentials are ready.
Treat any `ok: false` check as a blocker before asking an agent to join.
Use `openclaw googlemeet setup --json` for scripts or machine-readable output.
Use `--transport chrome`, `--transport chrome-node`, or `--transport twilio`
to preflight a specific transport before an agent tries it.
Join a meeting:
@@ -303,11 +305,17 @@ display name, or remote IP.
Common failure checks:
- `Configured Google Meet node ... is not usable: offline`: the pinned node is
known to the Gateway but unavailable. Agents should treat that node as
diagnostic state, not as a usable Chrome host, and report the setup blocker
instead of falling back to another transport unless the user asked for that.
- `No connected Google Meet-capable node`: start `openclaw node run` in the VM,
approve pairing, and make sure `openclaw plugins enable google-meet` and
`openclaw plugins enable browser` were run in the VM. Also confirm the
Gateway host allows both node commands with
`gateway.nodes.allowCommands: ["googlemeet.chrome", "browser.proxy"]`.
- `BlackHole 2ch audio device not found`: install `blackhole-2ch` on the host
being checked and reboot before using local Chrome audio.
- `BlackHole 2ch audio device not found on the node`: install `blackhole-2ch`
in the VM and reboot the VM.
- Chrome opens but cannot join: sign in to the browser profile inside the VM, or

View File

@@ -147,6 +147,21 @@ Rules:
- `onResolution` receives the resolved approval decision — `allow-once`,
`allow-always`, `deny`, `timeout`, or `cancelled`.
### Tool result persistence
Tool results can include structured `details` for UI rendering, diagnostics,
media routing, or plugin-owned metadata. Treat `details` as runtime metadata,
not prompt content:
- OpenClaw strips `toolResult.details` before provider replay and compaction
input so metadata does not become model context.
- Persisted session entries keep only bounded `details`. Oversized details are
replaced with a compact summary and `persistedDetailsTruncated: true`.
- `tool_result_persist` and `before_message_write` run before the final
persistence cap. Hooks should still keep returned `details` small and avoid
placing prompt-relevant text only in `details`; put model-visible tool output
in `content`.
## Prompt and model hooks
Use the phase-specific hooks for new plugins:

View File

@@ -53,6 +53,12 @@ Restart the Gateway afterwards.
Set config under `plugins.entries.voice-call.config`:
If `enabled` is true but the selected provider is missing credentials, Gateway
startup logs a setup-incomplete warning with the missing keys and skips starting
the runtime. Run `openclaw voicecall setup` to see the same readiness details.
Commands, RPC calls, and agent tools still return the exact missing provider
configuration when used.
```json5
{
plugins: {

View File

@@ -50,11 +50,16 @@ The bundled `fal` image-generation provider defaults to
| Size overrides | Supported |
| Aspect ratio | Supported |
| Resolution | Supported |
| Output format | `png` or `jpeg` |
<Warning>
The fal image edit endpoint does **not** support `aspectRatio` overrides.
</Warning>
Use `outputFormat: "png"` when you want PNG output. fal does not declare an
explicit transparent-background control in OpenClaw, so `background:
"transparent"` is reported as an ignored override for fal models.
To use fal as the default image provider:
```json5

View File

@@ -27,6 +27,7 @@ changing config.
| GPT-5.5 with ChatGPT/Codex subscription auth | `openai-codex/gpt-5.5` | Default PI route for Codex OAuth. Best first choice for subscription setups. |
| GPT-5.5 with native Codex app-server behavior | `openai/gpt-5.5` plus `embeddedHarness.runtime: "codex"` | Forces the Codex app-server harness for that model ref. |
| Image generation or editing | `openai/gpt-image-2` | Works with either `OPENAI_API_KEY` or OpenAI Codex OAuth. |
| Transparent-background images | `openai/gpt-image-1.5` | Use `outputFormat=png` or `webp` and `openai.background=transparent`. |
<Note>
GPT-5.5 is available through both direct OpenAI Platform API-key access and
@@ -254,8 +255,33 @@ See [Image Generation](/tools/image-generation) for shared tool parameters, prov
</Note>
`gpt-image-2` is the default for both OpenAI text-to-image generation and image
editing. `gpt-image-1` remains usable as an explicit model override, but new
OpenAI image workflows should use `openai/gpt-image-2`.
editing. `gpt-image-1.5`, `gpt-image-1`, and `gpt-image-1-mini` remain usable as
explicit model overrides. Use `openai/gpt-image-1.5` for transparent-background
PNG/WebP output; the current `gpt-image-2` API rejects
`background: "transparent"`.
For a transparent-background request, agents should call `image_generate` with
`model: "openai/gpt-image-1.5"`, `outputFormat: "png"` or `"webp"`, and
`background: "transparent"`; the older `openai.background` provider option is
still accepted. OpenClaw also protects the public OpenAI and
OpenAI Codex OAuth routes by rewriting default `openai/gpt-image-2` transparent
requests to `gpt-image-1.5`; Azure and custom OpenAI-compatible endpoints keep
their configured deployment/model names.
The same setting is exposed for headless CLI runs:
```bash
openclaw infer image generate \
--model openai/gpt-image-1.5 \
--output-format png \
--background transparent \
--prompt "A simple red circle sticker on a transparent background" \
--json
```
Use the same `--output-format` and `--background` flags with
`openclaw infer image edit` when starting from an input file.
`--openai-background` remains available as an OpenAI-specific alias.
For Codex OAuth installs, keep the same `openai/gpt-image-2` ref. When an
`openai-codex` OAuth profile is configured, OpenClaw resolves that stored OAuth
@@ -275,6 +301,12 @@ Generate:
/tool image_generate model=openai/gpt-image-2 prompt="A polished launch poster for OpenClaw on macOS" size=3840x2160 count=1
```
Generate a transparent PNG:
```
/tool image_generate model=openai/gpt-image-1.5 prompt="A simple red circle sticker on a transparent background" outputFormat=png background=transparent
```
Edit:
```

View File

@@ -123,6 +123,15 @@ Use the table below to pick the right model for your use case.
</Tip>
## DeepSeek V4 replay behavior
If Venice exposes DeepSeek V4 models such as `venice/deepseek-v4-pro` or
`venice/deepseek-v4-flash`, OpenClaw fills the required DeepSeek V4
`reasoning_content` replay placeholder on assistant tool-call turns when the
proxy omits it. Venice rejects DeepSeek's native top-level `thinking` control,
so OpenClaw keeps that provider-specific replay fix separate from the native
DeepSeek provider's thinking controls.
## Built-in catalog (41 total)
<AccordionGroup>

View File

@@ -101,6 +101,13 @@ Assistant transcript entries persist the same normalized usage shape, including
returns usage metadata. This gives `/usage cost` and transcript-backed session
status a stable source even after the live runtime state is gone.
OpenClaw keeps provider usage accounting separate from the current context
snapshot. Provider `usage.total` can include cached input, output, and multiple
tool-loop model calls, so it is useful for cost and telemetry but can overstate
the live context window. Context displays and diagnostics use the latest prompt
snapshot (`promptTokens`, or the last model call when no prompt snapshot is
available) for `context.used`.
## Cost estimation (when shown)
Costs are estimated from your model pricing config:

View File

@@ -23,6 +23,7 @@ Scope includes:
- Tool result pairing repair
- Turn validation / ordering
- Thought signature cleanup
- Thinking signature cleanup
- Image payload sanitization
- User-input provenance tagging (for inter-session routed prompts)
- Empty assistant error-turn repair for Bedrock Converse replay
@@ -133,6 +134,12 @@ external end-user instructions.
- Tool result pairing repair and synthetic tool results.
- Turn validation (merge consecutive user turns to satisfy strict alternation).
- Thinking blocks with missing, empty, or blank replay signatures are stripped
before provider conversion. If that empties an assistant turn, OpenClaw keeps
turn shape with non-empty omitted-reasoning text.
- Older thinking-only assistant turns that must be stripped are replaced with
non-empty omitted-reasoning text so provider adapters do not drop the replay
turn.
**Amazon Bedrock (Converse API)**
@@ -140,6 +147,11 @@ external end-user instructions.
before replay. Bedrock Converse rejects assistant messages with `content: []`, so
persisted assistant turns with `stopReason: "error"` and empty content are also
repaired on disk before load.
- Claude thinking blocks with missing, empty, or blank replay signatures are
stripped before Converse replay. If that empties an assistant turn, OpenClaw
keeps turn shape with non-empty omitted-reasoning text.
- Older thinking-only assistant turns that must be stripped are replaced with
non-empty omitted-reasoning text so the Converse replay keeps strict turn shape.
- Replay filters OpenClaw delivery-mirror and gateway-injected assistant turns.
- Image sanitization applies through the global rule.

View File

@@ -329,7 +329,8 @@ Interface details:
- `resumeSessionId` (optional): resume an existing ACP session instead of creating a new one. The agent replays its conversation history via `session/load`. Requires `runtime: "acp"`.
- `streamTo` (optional): `"parent"` streams initial ACP run progress summaries back to the requester session as system events.
- When available, accepted responses include `streamLogPath` pointing to a session-scoped JSONL log (`<sessionId>.acp-stream.jsonl`) you can tail for full relay history.
- `model` (optional): explicit model override for the ACP child session. Honored for `runtime: "acp"` so the child uses the requested model instead of silently falling back to the target agent default.
- `model` (optional): explicit model override for the ACP child session. Honored for `runtime: "acp"` so the child uses the requested model instead of silently falling back to the target agent default. Codex ACP spawns normalize OpenClaw Codex refs such as `openai-codex/gpt-5.4` to Codex ACP startup config before `session/new`; slash forms such as `openai-codex/gpt-5.4/high` also set Codex ACP reasoning effort.
- `thinking` (optional): explicit thinking/reasoning effort for the ACP child session. For Codex ACP, `minimal` maps to low effort, `low`/`medium`/`high`/`xhigh` map directly, and `off` omits the reasoning-effort startup override.
## Delivery model
@@ -522,7 +523,8 @@ Notes:
Equivalent operations:
- `/acp model <id>` maps to runtime config key `model`.
- `/acp model <id>` maps to runtime config key `model`. For Codex ACP, OpenClaw normalizes `openai-codex/<model>` to the adapter model id and maps slash reasoning suffixes such as `openai-codex/gpt-5.4/high` to Codex ACP `reasoning_effort`.
- `/acp set thinking <level>` maps to runtime config key `thinking`. For Codex ACP, OpenClaw sends the corresponding `reasoning_effort` where the adapter supports one.
- `/acp permissions <profile>` maps to runtime config key `approval_policy`.
- `/acp timeout <seconds>` maps to runtime config key `timeout`.
- `/acp cwd <path>` updates runtime cwd override directly.

View File

@@ -140,8 +140,8 @@ curl -s http://127.0.0.1:18791/tabs
On Raspberry Pi, older VPS hosts, or slow storage, raise
`browser.localLaunchTimeoutMs` when Chrome needs more time to expose its CDP HTTP
endpoint. Raise `browser.localCdpReadyTimeoutMs` when launch succeeds but
`openclaw browser start` still reports `not reachable after start`. Values are
capped at 120000 ms.
`openclaw browser start` still reports `not reachable after start`. Values must
be positive integers up to `120000` ms; invalid config values are rejected.
### Problem: "No Chrome tabs found for profile=\"user\""

View File

@@ -200,7 +200,8 @@ Browser settings live in `~/.openclaw/openclaw.json`.
process to expose its CDP HTTP endpoint. `localCdpReadyTimeoutMs` is the
follow-up budget for CDP websocket readiness after the process is discovered.
Raise these on Raspberry Pi, low-end VPS, or older hardware where Chromium
starts slowly. Values are capped at 120000 ms.
starts slowly. Values must be positive integers up to `120000` ms; invalid
config values are rejected.
- `actionTimeoutMs` is the default budget for browser `act` requests when the caller does not pass `timeoutMs`. The client transport adds a small slack window so long waits can finish instead of timing out at the HTTP boundary.
- `tabCleanup` is best-effort cleanup for tabs opened by primary-agent browser sessions. Subagent, cron, and ACP lifecycle cleanup still closes their explicit tracked tabs at session end; primary sessions keep active tabs reusable, then close idle or excess tracked tabs in the background.
@@ -235,12 +236,12 @@ Browser settings live in `~/.openclaw/openclaw.json`.
current process. `OPENCLAW_BROWSER_HEADLESS=0` forces headed mode for ordinary
starts and returns an actionable error on Linux hosts without a display server;
an explicit `start --headless` request still wins for that one launch.
- `executablePath` can be set globally or per local managed profile. Per-profile values override `browser.executablePath`, so different managed profiles can launch different Chromium-based browsers.
- `executablePath` can be set globally or per local managed profile. Per-profile values override `browser.executablePath`, so different managed profiles can launch different Chromium-based browsers. Both forms accept `~` for your OS home directory.
- `color` (top-level and per-profile) tints the browser UI so you can see which profile is active.
- Default profile is `openclaw` (managed standalone). Use `defaultProfile: "user"` to opt into the signed-in user browser.
- Auto-detect order: system default browser if Chromium-based; otherwise Chrome → Brave → Edge → Chromium → Chrome Canary.
- `driver: "existing-session"` uses Chrome DevTools MCP instead of raw CDP. Do not set `cdpUrl` for that driver.
- Set `browser.profiles.<name>.userDataDir` when an existing-session profile should attach to a non-default Chromium user profile (Brave, Edge, etc.).
- Set `browser.profiles.<name>.userDataDir` when an existing-session profile should attach to a non-default Chromium user profile (Brave, Edge, etc.). This path also accepts `~` for your OS home directory.
</Accordion>
@@ -250,7 +251,8 @@ Browser settings live in `~/.openclaw/openclaw.json`.
If your **system default** browser is Chromium-based (Chrome/Brave/Edge/etc),
OpenClaw uses it automatically. Set `browser.executablePath` to override
auto-detection. `~` expands to your OS home directory:
auto-detection. Top-level and per-profile `executablePath` values accept `~`
for your OS home directory:
```bash
openclaw config set browser.executablePath "/usr/bin/google-chrome"
@@ -532,7 +534,8 @@ Default behavior:
- The built-in `user` profile uses Chrome MCP auto-connect, which targets the
default local Google Chrome profile.
Use `userDataDir` for Brave, Edge, Chromium, or a non-default Chrome profile:
Use `userDataDir` for Brave, Edge, Chromium, or a non-default Chrome profile.
`~` expands to your OS home directory:
```json5
{

View File

@@ -48,19 +48,22 @@ The agent calls `image_generate` automatically. No tool allow-listing needed —
## Common routes
| Goal | Model ref | Auth |
| ---------------------------------------------------- | -------------------------------------------------- | ------------------------------------ |
| OpenAI image generation with API billing | `openai/gpt-image-2` | `OPENAI_API_KEY` |
| OpenAI image generation with Codex subscription auth | `openai/gpt-image-2` | OpenAI Codex OAuth |
| OpenRouter image generation | `openrouter/google/gemini-3.1-flash-image-preview` | `OPENROUTER_API_KEY` |
| LiteLLM image generation | `litellm/gpt-image-2` | `LITELLM_API_KEY` |
| Google Gemini image generation | `google/gemini-3.1-flash-image-preview` | `GEMINI_API_KEY` or `GOOGLE_API_KEY` |
| Goal | Model ref | Auth |
| ---------------------------------------------------- | -------------------------------------------------- | -------------------------------------- |
| OpenAI image generation with API billing | `openai/gpt-image-2` | `OPENAI_API_KEY` |
| OpenAI image generation with Codex subscription auth | `openai/gpt-image-2` | OpenAI Codex OAuth |
| OpenAI transparent-background PNG/WebP | `openai/gpt-image-1.5` | `OPENAI_API_KEY` or OpenAI Codex OAuth |
| OpenRouter image generation | `openrouter/google/gemini-3.1-flash-image-preview` | `OPENROUTER_API_KEY` |
| LiteLLM image generation | `litellm/gpt-image-2` | `LITELLM_API_KEY` |
| Google Gemini image generation | `google/gemini-3.1-flash-image-preview` | `GEMINI_API_KEY` or `GOOGLE_API_KEY` |
The same `image_generate` tool handles text-to-image and reference-image
editing. Use `image` for one reference or `images` for multiple references.
Provider-supported output hints such as `quality`, `outputFormat`, and
OpenAI-specific `background` are forwarded when available and reported as
ignored when a provider does not support them.
`background` are forwarded when available and reported as ignored when a
provider does not support them. Current bundled transparent-background support
is OpenAI-specific; other providers may still preserve PNG alpha if their
backend emits it.
## Supported providers
@@ -93,7 +96,8 @@ Use `"list"` to inspect available providers and models at runtime.
</ParamField>
<ParamField path="model" type="string">
Provider/model override, e.g. `openai/gpt-image-2`.
Provider/model override, e.g. `openai/gpt-image-2`; use
`openai/gpt-image-1.5` for transparent OpenAI backgrounds.
</ParamField>
<ParamField path="image" type="string">
@@ -124,6 +128,11 @@ Quality hint when the provider supports it.
Output format hint when the provider supports it.
</ParamField>
<ParamField path="background" type="'transparent' | 'opaque' | 'auto'">
Background hint when the provider supports it. Use `transparent` with
`outputFormat: "png"` or `"webp"` for transparency-capable providers.
</ParamField>
<ParamField path="count" type="number">
Number of images to generate (14).
</ParamField>
@@ -233,9 +242,10 @@ through the Codex Responses backend. Legacy Codex base URLs such as
`https://chatgpt.com/backend-api/codex` for image requests. It does not
silently fall back to `OPENAI_API_KEY` for that request. To force direct OpenAI
Images API routing, configure `models.providers.openai` explicitly with an API
key, custom base URL, or Azure endpoint. The older
`openai/gpt-image-1` model can still be selected explicitly, but new OpenAI
image-generation and image-editing requests should use `gpt-image-2`.
key, custom base URL, or Azure endpoint. The `openai/gpt-image-1.5`,
`openai/gpt-image-1`, and `openai/gpt-image-1-mini` models can still be
selected explicitly. Use `gpt-image-1.5` for transparent-background PNG/WebP
output; the current `gpt-image-2` API rejects `background: "transparent"`.
`gpt-image-2` supports both text-to-image generation and reference-image
editing through the same `image_generate` tool. OpenClaw forwards `prompt`,
@@ -260,8 +270,51 @@ OpenAI-specific options live under the `openai` object:
```
`openai.background` accepts `transparent`, `opaque`, or `auto`; transparent
outputs require `outputFormat` `png` or `webp`. `openai.outputCompression`
applies to JPEG/WebP outputs.
outputs require `outputFormat` `png` or `webp` and a transparency-capable OpenAI
image model. OpenClaw routes default `gpt-image-2` transparent-background
requests to `gpt-image-1.5`. `openai.outputCompression` applies to JPEG/WebP
outputs.
The top-level `background` hint is provider-neutral and currently maps to the
same OpenAI `background` request field when the OpenAI provider is selected.
Providers that do not declare background support return it in `ignoredOverrides`
instead of receiving the unsupported parameter.
When asking an agent for a transparent-background OpenAI image, the expected
tool call is:
```json
{
"model": "openai/gpt-image-1.5",
"prompt": "A simple red circle sticker on a transparent background",
"outputFormat": "png",
"background": "transparent"
}
```
The explicit `openai/gpt-image-1.5` model keeps the request portable across
tool summaries and harnesses. If the agent instead uses the default
`openai/gpt-image-2` with `openai.background: "transparent"` on the public
OpenAI or OpenAI Codex OAuth route, OpenClaw rewrites the provider request to
`gpt-image-1.5`. Azure and custom OpenAI-compatible endpoints keep their
configured deployment/model names.
For headless CLI generation, use the equivalent `openclaw infer` flags:
```bash
openclaw infer image generate \
--model openai/gpt-image-1.5 \
--output-format png \
--background transparent \
--prompt "A simple red circle sticker on a transparent background" \
--json
```
The same `--output-format` and `--background` flags are available on
`openclaw infer image edit`; `--openai-background` remains available as an
OpenAI-specific alias. Current bundled providers other than OpenAI do not
declare explicit background control, so `background: "transparent"` is reported
as ignored for them.
Generate one 4K landscape image:
@@ -269,6 +322,12 @@ Generate one 4K landscape image:
/tool image_generate action=generate model=openai/gpt-image-2 prompt="A clean editorial poster for OpenClaw image generation" size=3840x2160 count=1
```
Generate a transparent PNG:
```
/tool image_generate action=generate model=openai/gpt-image-1.5 prompt="A simple red circle sticker on a transparent background" outputFormat=png background=transparent
```
Generate two square images:
```

View File

@@ -266,6 +266,7 @@ openclaw plugins info <id> # inspect alias
openclaw plugins doctor # diagnostics
openclaw plugins registry # inspect persisted registry state
openclaw plugins registry --refresh # rebuild persisted registry
openclaw doctor --fix # repair registry/ledger migration state
openclaw plugins install <package> # install (ClawHub first, then npm)
openclaw plugins install clawhub:<pkg> # install from ClawHub only
@@ -279,7 +280,7 @@ openclaw plugins install <spec> --dangerously-force-unsafe-install
openclaw plugins update <id-or-npm-spec> # update one plugin
openclaw plugins update <id-or-npm-spec> --dangerously-force-unsafe-install
openclaw plugins update --all # update all
openclaw plugins uninstall <id> # remove config/install records
openclaw plugins uninstall <id> # remove config and install ledger records
openclaw plugins uninstall <id> --keep-files
openclaw plugins marketplace list <source>
openclaw plugins marketplace list <source> --json
@@ -307,6 +308,9 @@ uninstall, enable, and disable flows refresh that registry after changing plugin
state. If the registry is missing, stale, or invalid, `openclaw plugins registry
--refresh` rebuilds it from the durable install ledger, config policy, and
manifest/package metadata without loading plugin runtime modules.
If a machine still has legacy `plugins.installs` records in config, run
`openclaw doctor --fix` to move them into the managed
`plugins/installs.json` ledger and remove the config copy.
`openclaw plugins update <id-or-npm-spec>` applies to tracked installs. Passing
an npm package spec with a dist-tag or exact version resolves the package name

View File

@@ -1,6 +1,6 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import type { AcpRuntime } from "../runtime-api.js";
import { AcpxRuntime } from "./runtime.js";
import { AcpxRuntime, __testing } from "./runtime.js";
type TestSessionStore = {
load(sessionId: string): Promise<Record<string, unknown> | undefined>;
@@ -9,6 +9,8 @@ type TestSessionStore = {
const DOCUMENTED_OPENCLAW_BRIDGE_COMMAND =
"env OPENCLAW_HIDE_BANNER=1 OPENCLAW_SUPPRESS_NOTES=1 openclaw acp --url ws://127.0.0.1:18789 --token-file ~/.openclaw/gateway.token --session agent:main:main";
const CODEX_ACP_COMMAND = "npx @zed-industries/codex-acp@^0.11.1";
const CODEX_ACP_WRAPPER_COMMAND = `node "/tmp/openclaw/acpx/codex-acp-wrapper.mjs"`;
function makeRuntime(
baseStore: TestSessionStore,
@@ -20,6 +22,7 @@ function makeRuntime(
close: AcpRuntime["close"];
ensureSession: AcpRuntime["ensureSession"];
getStatus: NonNullable<AcpRuntime["getStatus"]>;
setConfigOption: NonNullable<AcpRuntime["setConfigOption"]>;
isHealthy(): boolean;
probeAvailability(): Promise<void>;
};
@@ -27,6 +30,7 @@ function makeRuntime(
close: AcpRuntime["close"];
ensureSession: AcpRuntime["ensureSession"];
getStatus: NonNullable<AcpRuntime["getStatus"]>;
setConfigOption: NonNullable<AcpRuntime["setConfigOption"]>;
isHealthy(): boolean;
probeAvailability(): Promise<void>;
};
@@ -55,6 +59,7 @@ function makeRuntime(
close: AcpRuntime["close"];
ensureSession: AcpRuntime["ensureSession"];
getStatus: NonNullable<AcpRuntime["getStatus"]>;
setConfigOption: NonNullable<AcpRuntime["setConfigOption"]>;
isHealthy(): boolean;
probeAvailability(): Promise<void>;
};
@@ -66,6 +71,7 @@ function makeRuntime(
close: AcpRuntime["close"];
ensureSession: AcpRuntime["ensureSession"];
getStatus: NonNullable<AcpRuntime["getStatus"]>;
setConfigOption: NonNullable<AcpRuntime["setConfigOption"]>;
isHealthy(): boolean;
probeAvailability(): Promise<void>;
};
@@ -79,6 +85,274 @@ describe("AcpxRuntime fresh reset wrapper", () => {
vi.restoreAllMocks();
});
it("normalizes OpenClaw Codex model ids for ACP startup", async () => {
const baseStore: TestSessionStore = {
load: vi.fn(async () => undefined),
save: vi.fn(async () => {}),
};
const { runtime, delegate } = makeRuntime(baseStore, {
agentRegistry: {
resolve: (agentName: string) => (agentName === "codex" ? CODEX_ACP_COMMAND : agentName),
list: () => ["codex", "openclaw"],
},
});
const ensure = vi.spyOn(delegate, "ensureSession").mockResolvedValue({
sessionKey: "agent:codex:acp:test",
backend: "acpx",
runtimeSessionName: "codex",
});
await runtime.ensureSession({
sessionKey: "agent:codex:acp:test",
agent: "codex",
mode: "persistent",
model: "openai-codex/gpt-5.4",
});
expect(ensure).toHaveBeenCalledWith(
expect.objectContaining({
model: "gpt-5.4",
}),
);
});
it("leaves Codex ACP startup defaults alone when no model or thinking is provided", async () => {
const baseStore: TestSessionStore = {
load: vi.fn(async () => undefined),
save: vi.fn(async () => {}),
};
const { runtime, delegate } = makeRuntime(baseStore, {
agentRegistry: {
resolve: (agentName: string) => (agentName === "codex" ? CODEX_ACP_COMMAND : agentName),
list: () => ["codex", "openclaw"],
},
});
const ensure = vi.spyOn(delegate, "ensureSession").mockResolvedValue({
sessionKey: "agent:codex:acp:test",
backend: "acpx",
runtimeSessionName: "codex",
});
await runtime.ensureSession({
sessionKey: "agent:codex:acp:test",
agent: "codex",
mode: "persistent",
});
expect(ensure).toHaveBeenCalledWith(
expect.objectContaining({
agent: "codex",
}),
);
expect(ensure.mock.calls[0]?.[0]).not.toHaveProperty("model");
expect(ensure.mock.calls[0]?.[0]).not.toHaveProperty("thinking");
});
it("does not normalize model startup for non-Codex ACP agents", async () => {
const baseStore: TestSessionStore = {
load: vi.fn(async () => undefined),
save: vi.fn(async () => {}),
};
const { runtime, delegate } = makeRuntime(baseStore, {
agentRegistry: {
resolve: (agentName: string) => (agentName === "main" ? CODEX_ACP_COMMAND : agentName),
list: () => ["main", "codex", "openclaw"],
},
});
const ensure = vi.spyOn(delegate, "ensureSession").mockResolvedValue({
sessionKey: "agent:main:acp:test",
backend: "acpx",
runtimeSessionName: "main",
});
await runtime.ensureSession({
sessionKey: "agent:main:acp:test",
agent: "main",
mode: "persistent",
model: "openai-codex/gpt-5.5",
});
expect(ensure).toHaveBeenCalledWith(
expect.objectContaining({
agent: "main",
model: "openai-codex/gpt-5.5",
}),
);
});
it("injects Codex ACP startup config into the scoped registry", () => {
expect(__testing.isCodexAcpCommand(CODEX_ACP_COMMAND)).toBe(true);
expect(__testing.isCodexAcpCommand(CODEX_ACP_WRAPPER_COMMAND)).toBe(true);
expect(
__testing.appendCodexAcpConfigOverrides(CODEX_ACP_COMMAND, {
model: "gpt-5.4",
reasoningEffort: "medium",
}),
).toBe(
"npx @zed-industries/codex-acp@^0.11.1 -c model=gpt-5.4 -c model_reasoning_effort=medium",
);
expect(__testing.isCodexAcpCommand("openclaw acp")).toBe(false);
});
it("passes gpt-5.5 Codex ACP startup through instead of blocking it", async () => {
const baseStore: TestSessionStore = {
load: vi.fn(async () => undefined),
save: vi.fn(async () => {}),
};
const { runtime, delegate } = makeRuntime(baseStore, {
agentRegistry: {
resolve: (agentName: string) => (agentName === "codex" ? CODEX_ACP_COMMAND : agentName),
list: () => ["codex", "openclaw"],
},
});
const ensure = vi.spyOn(delegate, "ensureSession").mockResolvedValue({
sessionKey: "agent:codex:acp:test",
backend: "acpx",
runtimeSessionName: "codex",
});
await runtime.ensureSession({
sessionKey: "agent:codex:acp:test",
agent: "codex",
mode: "persistent",
model: "openai-codex/gpt-5.5",
});
expect(ensure).toHaveBeenCalledWith(
expect.objectContaining({
model: "gpt-5.5",
}),
);
});
it("maps explicit Codex ACP thinking to startup reasoning effort", async () => {
const baseStore: TestSessionStore = {
load: vi.fn(async () => undefined),
save: vi.fn(async () => {}),
};
const { runtime, delegate } = makeRuntime(baseStore, {
agentRegistry: {
resolve: (agentName: string) => (agentName === "codex" ? CODEX_ACP_COMMAND : agentName),
list: () => ["codex", "openclaw"],
},
});
const ensure = vi.spyOn(delegate, "ensureSession").mockResolvedValue({
sessionKey: "agent:codex:acp:test",
backend: "acpx",
runtimeSessionName: "codex",
});
await runtime.ensureSession({
sessionKey: "agent:codex:acp:test",
agent: "codex",
mode: "persistent",
model: "openai-codex/gpt-5.4",
thinking: "x-high",
});
expect(ensure).toHaveBeenCalledWith(
expect.objectContaining({
model: "gpt-5.4/xhigh",
}),
);
});
it("normalizes Codex ACP model config controls to adapter ids", async () => {
const baseStore: TestSessionStore = {
load: vi.fn(async () => ({
acpxRecordId: "agent:codex:acp:test",
agentCommand: CODEX_ACP_COMMAND,
})),
save: vi.fn(async () => {}),
};
const { runtime, delegate } = makeRuntime(baseStore);
const setConfigOption = vi.spyOn(delegate, "setConfigOption").mockResolvedValue(undefined);
const handle: Parameters<NonNullable<AcpRuntime["setConfigOption"]>>[0]["handle"] = {
sessionKey: "agent:codex:acp:test",
backend: "acpx",
runtimeSessionName: "agent:codex:acp:test",
acpxRecordId: "agent:codex:acp:test",
};
await runtime.setConfigOption({
handle,
key: "model",
value: "openai-codex/gpt-5.4",
});
expect(setConfigOption).toHaveBeenNthCalledWith(1, {
handle,
key: "model",
value: "gpt-5.4",
});
expect(setConfigOption).toHaveBeenCalledOnce();
});
it("normalizes Codex ACP slash reasoning suffixes to config controls", async () => {
const baseStore: TestSessionStore = {
load: vi.fn(async () => ({
acpxRecordId: "agent:codex:acp:test",
agentCommand: CODEX_ACP_COMMAND,
})),
save: vi.fn(async () => {}),
};
const { runtime, delegate } = makeRuntime(baseStore);
const setConfigOption = vi.spyOn(delegate, "setConfigOption").mockResolvedValue(undefined);
const handle: Parameters<NonNullable<AcpRuntime["setConfigOption"]>>[0]["handle"] = {
sessionKey: "agent:codex:acp:test",
backend: "acpx",
runtimeSessionName: "agent:codex:acp:test",
acpxRecordId: "agent:codex:acp:test",
};
await runtime.setConfigOption({
handle,
key: "model",
value: "openai-codex/gpt-5.4/high",
});
expect(setConfigOption).toHaveBeenNthCalledWith(1, {
handle,
key: "model",
value: "gpt-5.4",
});
expect(setConfigOption).toHaveBeenNthCalledWith(2, {
handle,
key: "reasoning_effort",
value: "high",
});
});
it("normalizes Codex ACP thinking config controls to reasoning effort", async () => {
const baseStore: TestSessionStore = {
load: vi.fn(async () => ({
acpxRecordId: "agent:codex:acp:test",
agentCommand: CODEX_ACP_COMMAND,
})),
save: vi.fn(async () => {}),
};
const { runtime, delegate } = makeRuntime(baseStore);
const setConfigOption = vi.spyOn(delegate, "setConfigOption").mockResolvedValue(undefined);
const handle: Parameters<NonNullable<AcpRuntime["setConfigOption"]>>[0]["handle"] = {
sessionKey: "agent:codex:acp:test",
backend: "acpx",
runtimeSessionName: "agent:codex:acp:test",
acpxRecordId: "agent:codex:acp:test",
};
await runtime.setConfigOption({
handle,
key: "thinking",
value: "minimal",
});
expect(setConfigOption).toHaveBeenCalledWith({
handle,
key: "reasoning_effort",
value: "low",
});
});
it("keeps stale persistent loads hidden until a fresh record is saved", async () => {
const baseStore: TestSessionStore = {
load: vi.fn(async () => ({ acpxRecordId: "stale" }) as never),

View File

@@ -1,3 +1,4 @@
import { AsyncLocalStorage } from "node:async_hooks";
import {
ACPX_BACKEND_ID,
AcpxRuntime as BaseAcpxRuntime,
@@ -13,7 +14,7 @@ import {
type AcpRuntimeOptions,
type AcpRuntimeStatus,
} from "acpx/runtime";
import type { AcpRuntime } from "../runtime-api.js";
import { AcpRuntimeError, type AcpRuntime } from "../runtime-api.js";
type AcpSessionStore = AcpRuntimeOptions["sessionStore"];
type AcpSessionRecord = Parameters<AcpSessionStore["save"]>[0];
@@ -60,6 +61,27 @@ function createResetAwareSessionStore(baseStore: AcpSessionStore): ResetAwareSes
const OPENCLAW_BRIDGE_EXECUTABLE = "openclaw";
const OPENCLAW_BRIDGE_SUBCOMMAND = "acp";
const CODEX_ACP_AGENT_ID = "codex";
const CODEX_ACP_OPENCLAW_PREFIX = "openai-codex/";
const CODEX_ACP_REASONING_EFFORTS = new Set(["low", "medium", "high", "xhigh"]);
const CODEX_ACP_THINKING_ALIASES = new Map<string, string | undefined>([
["off", undefined],
["minimal", "low"],
["low", "low"],
["medium", "medium"],
["high", "high"],
["x-high", "xhigh"],
["x_high", "xhigh"],
["extra-high", "xhigh"],
["extra_high", "xhigh"],
["extra high", "xhigh"],
["xhigh", "xhigh"],
]);
type CodexAcpModelOverride = {
model?: string;
reasoningEffort?: string;
};
function normalizeAgentName(value: string | undefined): string | undefined {
const normalized = value?.trim().toLowerCase();
@@ -175,6 +197,149 @@ function isOpenClawBridgeCommand(command: string | undefined): boolean {
return /^openclaw(?:\.[cm]?js)?$/i.test(scriptName) && parts[2] === OPENCLAW_BRIDGE_SUBCOMMAND;
}
function isCodexAcpPackageSpec(value: string): boolean {
return /^@zed-industries\/codex-acp(?:@.+)?$/i.test(value.trim());
}
function isCodexAcpCommand(command: string | undefined): boolean {
if (!command) {
return false;
}
const parts = unwrapEnvCommand(splitCommandParts(command.trim()));
if (!parts.length) {
return false;
}
if (parts.some(isCodexAcpPackageSpec)) {
return true;
}
const commandName = basename(parts[0] ?? "");
if (/^codex-acp(?:\.exe)?$/i.test(commandName)) {
return true;
}
if (commandName !== "node") {
return false;
}
const scriptName = basename(parts[1] ?? "");
return /^codex-acp(?:-wrapper)?(?:\.[cm]?js)?$/i.test(scriptName);
}
function failUnsupportedCodexAcpModel(rawModel: string, detail?: string): never {
throw new AcpRuntimeError(
"ACP_INVALID_RUNTIME_OPTION",
detail ??
`Codex ACP model "${rawModel}" is not supported. Use openai-codex/<model> or <model>/<reasoning-effort>.`,
);
}
function failUnsupportedCodexAcpThinking(rawThinking: string): never {
throw new AcpRuntimeError(
"ACP_INVALID_RUNTIME_OPTION",
`Codex ACP thinking level "${rawThinking}" is not supported. Use off, minimal, low, medium, high, or xhigh.`,
);
}
function normalizeCodexAcpReasoningEffort(rawThinking: string | undefined): string | undefined {
const normalized = rawThinking?.trim().toLowerCase();
if (!normalized) {
return undefined;
}
if (!CODEX_ACP_THINKING_ALIASES.has(normalized)) {
failUnsupportedCodexAcpThinking(rawThinking ?? "");
}
return CODEX_ACP_THINKING_ALIASES.get(normalized);
}
function normalizeCodexAcpModelOverride(
rawModel: string | undefined,
rawThinking?: string,
): CodexAcpModelOverride | undefined {
const raw = rawModel?.trim();
const thinkingReasoningEffort = normalizeCodexAcpReasoningEffort(rawThinking);
if (!raw) {
return thinkingReasoningEffort ? { reasoningEffort: thinkingReasoningEffort } : undefined;
}
let value = raw;
if (value.toLowerCase().startsWith(CODEX_ACP_OPENCLAW_PREFIX)) {
value = value.slice(CODEX_ACP_OPENCLAW_PREFIX.length);
}
const parts = value.split("/");
if (parts.length > 2) {
failUnsupportedCodexAcpModel(
raw,
`Codex ACP model "${raw}" is not supported. Use openai-codex/<model> or <model>/<reasoning-effort>.`,
);
}
const model = (parts[0] ?? "").trim();
const modelReasoningEffort = normalizeCodexAcpReasoningEffort(parts[1]);
if (!model) {
failUnsupportedCodexAcpModel(
raw,
`Codex ACP model "${raw}" is not supported. Use openai-codex/<model> or <model>/<reasoning-effort>.`,
);
}
const reasoningEffort = thinkingReasoningEffort ?? modelReasoningEffort;
if (reasoningEffort && !CODEX_ACP_REASONING_EFFORTS.has(reasoningEffort)) {
failUnsupportedCodexAcpThinking(reasoningEffort);
}
return {
model,
...(reasoningEffort ? { reasoningEffort } : {}),
};
}
function codexAcpSessionModelId(override: CodexAcpModelOverride): string {
if (!override.model) {
return "";
}
return override.reasoningEffort
? `${override.model}/${override.reasoningEffort}`
: override.model;
}
function quoteShellArg(value: string): string {
if (/^[A-Za-z0-9_./:=@+-]+$/.test(value)) {
return value;
}
return `'${value.replace(/'/g, "'\\''")}'`;
}
function appendCodexAcpConfigOverrides(command: string, override: CodexAcpModelOverride): string {
const configArgs = override.model ? [`model=${override.model}`] : [];
if (override.reasoningEffort) {
configArgs.push(`model_reasoning_effort=${override.reasoningEffort}`);
}
if (configArgs.length === 0) {
return command;
}
return `${command} ${configArgs.map((arg) => `-c ${quoteShellArg(arg)}`).join(" ")}`;
}
function createModelScopedAgentRegistry(params: {
agentRegistry: AcpAgentRegistry;
scope: AsyncLocalStorage<CodexAcpModelOverride | undefined>;
}): AcpAgentRegistry {
return {
resolve(agentName: string): string | undefined {
const command = params.agentRegistry.resolve(agentName);
const override = params.scope.getStore();
if (
!override ||
normalizeAgentName(agentName) !== CODEX_ACP_AGENT_ID ||
typeof command !== "string" ||
!isCodexAcpCommand(command)
) {
return command;
}
return appendCodexAcpConfigOverrides(command, override);
},
list(): string[] {
return params.agentRegistry.list();
},
};
}
function resolveAgentCommand(params: {
agentName: string | undefined;
agentRegistry: AcpAgentRegistry;
@@ -211,6 +376,10 @@ function shouldUseDistinctBridgeDelegate(options: AcpRuntimeOptions): boolean {
export class AcpxRuntime implements AcpRuntime {
private readonly sessionStore: ResetAwareSessionStore;
private readonly agentRegistry: AcpAgentRegistry;
private readonly scopedAgentRegistry: AcpAgentRegistry;
private readonly codexAcpModelOverrideScope = new AsyncLocalStorage<
CodexAcpModelOverride | undefined
>();
private readonly delegate: BaseAcpxRuntime;
private readonly bridgeSafeDelegate: BaseAcpxRuntime;
private readonly probeDelegate: BaseAcpxRuntime;
@@ -221,9 +390,14 @@ export class AcpxRuntime implements AcpRuntime {
) {
this.sessionStore = createResetAwareSessionStore(options.sessionStore);
this.agentRegistry = options.agentRegistry;
this.scopedAgentRegistry = createModelScopedAgentRegistry({
agentRegistry: this.agentRegistry,
scope: this.codexAcpModelOverrideScope,
});
const sharedOptions = {
...options,
sessionStore: this.sessionStore,
agentRegistry: this.scopedAgentRegistry,
};
this.delegate = new BaseAcpxRuntime(sharedOptions, testOptions);
this.bridgeSafeDelegate = shouldUseDistinctBridgeDelegate(options)
@@ -259,6 +433,18 @@ export class AcpxRuntime implements AcpRuntime {
return this.resolveDelegateForAgent(readAgentFromHandle(handle));
}
private async resolveCommandForHandle(handle: AcpRuntimeHandle): Promise<string | undefined> {
const record = await this.sessionStore.load(handle.acpxRecordId ?? handle.sessionKey);
const recordCommand = readAgentCommandFromRecord(record);
if (recordCommand) {
return recordCommand;
}
return resolveAgentCommandForName({
agentName: readAgentFromHandle(handle),
agentRegistry: this.agentRegistry,
});
}
isHealthy(): boolean {
return this.probeDelegate.isHealthy();
}
@@ -271,8 +457,32 @@ export class AcpxRuntime implements AcpRuntime {
return this.probeDelegate.doctor();
}
ensureSession(input: Parameters<AcpRuntime["ensureSession"]>[0]): Promise<AcpRuntimeHandle> {
return this.resolveDelegateForAgent(input.agent).ensureSession(input);
async ensureSession(
input: Parameters<AcpRuntime["ensureSession"]>[0],
): Promise<AcpRuntimeHandle> {
const command = resolveAgentCommandForName({
agentName: input.agent,
agentRegistry: this.agentRegistry,
});
const delegate = this.resolveDelegateForCommand(command);
const codexModelOverride =
normalizeAgentName(input.agent) === CODEX_ACP_AGENT_ID && isCodexAcpCommand(command)
? normalizeCodexAcpModelOverride(input.model, input.thinking)
: undefined;
if (!codexModelOverride) {
return delegate.ensureSession(input);
}
const normalizedInput = {
...input,
...(codexAcpSessionModelId(codexModelOverride)
? { model: codexAcpSessionModelId(codexModelOverride) }
: {}),
};
return this.codexAcpModelOverrideScope.run(codexModelOverride, () =>
delegate.ensureSession(normalizedInput),
);
}
async *runTurn(input: Parameters<AcpRuntime["runTurn"]>[0]): AsyncIterable<AcpRuntimeEvent> {
@@ -299,6 +509,39 @@ export class AcpxRuntime implements AcpRuntime {
input: Parameters<NonNullable<AcpRuntime["setConfigOption"]>>[0],
): Promise<void> {
const delegate = await this.resolveDelegateForHandle(input.handle);
const command = await this.resolveCommandForHandle(input.handle);
if (
(input.key === "model" ||
input.key === "thinking" ||
input.key === "thought_level" ||
input.key === "reasoning_effort") &&
isCodexAcpCommand(command)
) {
const override =
input.key === "model"
? normalizeCodexAcpModelOverride(input.value)
: normalizeCodexAcpModelOverride(undefined, input.value);
if (!override && input.key !== "model") {
return;
}
if (override) {
if (override.model) {
await delegate.setConfigOption({
...input,
key: "model",
value: override.model,
});
}
if (override.reasoningEffort) {
await delegate.setConfigOption({
...input,
key: "reasoning_effort",
value: override.reasoningEffort,
});
}
return;
}
}
await delegate.setConfigOption(input);
}
@@ -334,4 +577,11 @@ export {
encodeAcpxRuntimeHandleState,
};
export const __testing = {
appendCodexAcpConfigOverrides,
codexAcpSessionModelId,
isCodexAcpCommand,
normalizeCodexAcpModelOverride,
};
export type { AcpAgentRegistry, AcpRuntimeOptions, AcpSessionRecord, AcpSessionStore };

View File

@@ -922,6 +922,53 @@ describe("active-memory plugin", () => {
});
});
it("infers the configured provider for bare active-memory default models", async () => {
api.config = {
agents: {
defaults: {
model: { primary: "gpt-5.5" },
},
},
models: {
providers: {
"openai-codex": {
baseUrl: "https://chatgpt.com/backend-api/codex",
models: [
{
id: "gpt-5.5",
name: "GPT 5.5",
reasoning: true,
input: ["text"],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 200_000,
maxTokens: 128_000,
},
],
},
},
},
};
api.pluginConfig = {
agents: ["main"],
};
plugin.register(api as unknown as OpenClawPluginApi);
await hooks.before_prompt_build(
{ prompt: "what wings should i order? bare model default", messages: [] },
{
agentId: "main",
trigger: "user",
sessionKey: "agent:main:main",
messageProvider: "webchat",
},
);
expect(runEmbeddedPiAgent.mock.calls.at(-1)?.[0]).toMatchObject({
provider: "openai-codex",
model: "gpt-5.5",
});
});
it("skips recall when no model or explicit fallback resolves", async () => {
api.config = {};
api.pluginConfig = {

View File

@@ -7,6 +7,7 @@ import {
resolveAgentDir,
resolveAgentEffectiveModelPrimary,
resolveAgentWorkspaceDir,
resolveDefaultModelForAgent,
} from "openclaw/plugin-sdk/agent-runtime";
import {
resolveLivePluginConfigObject,
@@ -1550,13 +1551,11 @@ function extractRecentTurns(messages: unknown[]): ActiveRecallRecentTurn[] {
return turns;
}
function parseModelCandidate(modelRef: string | undefined) {
function parseModelCandidate(modelRef: string | undefined, defaultProvider = DEFAULT_PROVIDER) {
if (!modelRef) {
return undefined;
}
return (
parseModelRef(modelRef, DEFAULT_PROVIDER) ?? { provider: DEFAULT_PROVIDER, model: modelRef }
);
return parseModelRef(modelRef, defaultProvider) ?? { provider: defaultProvider, model: modelRef };
}
function getModelRef(
@@ -1570,14 +1569,20 @@ function getModelRef(
): { provider: string; model: string } | undefined {
const currentRunModel =
ctx?.modelProviderId && ctx?.modelId ? `${ctx.modelProviderId}/${ctx.modelId}` : undefined;
const configuredDefaultModel = resolveAgentEffectiveModelPrimary(api.config, agentId)
? resolveDefaultModelForAgent({ cfg: api.config, agentId })
: undefined;
const defaultProvider = configuredDefaultModel?.provider ?? DEFAULT_PROVIDER;
const candidates = [
config.model,
currentRunModel,
resolveAgentEffectiveModelPrimary(api.config, agentId),
configuredDefaultModel
? `${configuredDefaultModel.provider}/${configuredDefaultModel.model}`
: undefined,
config.modelFallback,
];
for (const candidate of candidates) {
const parsed = parseModelCandidate(candidate);
const parsed = parseModelCandidate(candidate, defaultProvider);
if (parsed) {
return parsed;
}

View File

@@ -1,4 +1,5 @@
import { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry";
import { registerUnhandledRejectionHandler } from "openclaw/plugin-sdk/runtime";
import { startGatewayBonjourAdvertiser } from "./src/advertiser.js";
function formatBonjourInstanceName(displayName: string) {
@@ -32,7 +33,7 @@ export default definePluginEntry({
cliPath: ctx.cliPath,
minimal: ctx.minimal,
},
{ logger: api.logger },
{ logger: api.logger, registerUnhandledRejectionHandler },
);
return { stop: advertiser.stop };
},

View File

@@ -484,12 +484,12 @@ describe("gateway bonjour advertiser", () => {
expect(createService).toHaveBeenCalledTimes(2);
expect(advertise).toHaveBeenCalledTimes(2);
expect(destroy).toHaveBeenCalledTimes(1);
expect(shutdown).toHaveBeenCalledTimes(1);
expect(shutdown).not.toHaveBeenCalled();
expect(events).toEqual(["advertise:1", "destroy", "advertise:2"]);
await started.stop();
expect(destroy).toHaveBeenCalledTimes(2);
expect(shutdown).toHaveBeenCalledTimes(2);
expect(shutdown).toHaveBeenCalledTimes(1);
});
it("treats probing-to-announcing churn as one unhealthy window", async () => {
@@ -527,9 +527,10 @@ describe("gateway bonjour advertiser", () => {
expect(createService).toHaveBeenCalledTimes(2);
expect(advertise).toHaveBeenCalledTimes(3);
expect(destroy).toHaveBeenCalledTimes(1);
expect(shutdown).toHaveBeenCalledTimes(1);
expect(shutdown).not.toHaveBeenCalled();
await started.stop();
expect(shutdown).toHaveBeenCalledTimes(1);
});
it("normalizes hostnames with domains for service names", async () => {

View File

@@ -233,8 +233,9 @@ export async function startGatewayBonjourAdvertiser(
gatewayTxt.sshPort = String(opts.sshPort ?? 22);
}
const responder = getResponder();
function createCycle(): BonjourCycle {
const responder = getResponder();
const services: Array<{ label: string; svc: BonjourService }> = [];
const gateway = responder.createService({
@@ -259,7 +260,7 @@ export async function startGatewayBonjourAdvertiser(
return { responder, services, cleanupUnhandledRejection };
}
async function stopCycle(cycle: BonjourCycle | null) {
async function stopCycle(cycle: BonjourCycle | null, opts?: { shutdownResponder?: boolean }) {
if (!cycle) {
return;
}
@@ -271,7 +272,9 @@ export async function startGatewayBonjourAdvertiser(
}
}
try {
await cycle.responder.shutdown();
if (opts?.shutdownResponder) {
await cycle.responder.shutdown();
}
} catch {
/* ignore */
} finally {
@@ -442,7 +445,7 @@ export async function startGatewayBonjourAdvertiser(
} catch {
// ignore
}
await stopCycle(cycle);
await stopCycle(cycle, { shutdownResponder: true });
restoreConsoleLog();
},
};

View File

@@ -691,6 +691,187 @@ describe("diagnostics-otel service", () => {
await service.stop?.(ctx);
});
test("exports GenAI client token usage histogram for input and output only", async () => {
const service = createDiagnosticsOtelService();
const ctx = createOtelContext(OTEL_TEST_ENDPOINT, { metrics: true });
await service.start(ctx);
emitDiagnosticEvent({
type: "model.usage",
sessionKey: "session-key",
channel: "webchat",
provider: "openai",
model: "gpt-5.4",
usage: {
input: 12,
output: 7,
cacheRead: 3,
cacheWrite: 2,
promptTokens: 17,
total: 24,
},
});
await flushDiagnosticEvents();
expect(telemetryState.meter.createHistogram).toHaveBeenCalledWith(
"gen_ai.client.token.usage",
expect.objectContaining({
unit: "{token}",
advice: {
explicitBucketBoundaries: expect.arrayContaining([1, 4, 16, 1024, 67108864]),
},
}),
);
const genAiTokenUsage = telemetryState.histograms.get("gen_ai.client.token.usage");
expect(genAiTokenUsage?.record).toHaveBeenCalledTimes(2);
expect(genAiTokenUsage?.record).toHaveBeenCalledWith(12, {
"gen_ai.operation.name": "chat",
"gen_ai.provider.name": "openai",
"gen_ai.request.model": "gpt-5.4",
"gen_ai.token.type": "input",
});
expect(genAiTokenUsage?.record).toHaveBeenCalledWith(7, {
"gen_ai.operation.name": "chat",
"gen_ai.provider.name": "openai",
"gen_ai.request.model": "gpt-5.4",
"gen_ai.token.type": "output",
});
expect(JSON.stringify(genAiTokenUsage?.record.mock.calls)).not.toContain("session-key");
await service.stop?.(ctx);
});
test("keeps GenAI token usage metric model attribute present when model is unavailable", async () => {
const service = createDiagnosticsOtelService();
const ctx = createOtelContext(OTEL_TEST_ENDPOINT, { metrics: true });
await service.start(ctx);
emitDiagnosticEvent({
type: "model.usage",
provider: "openai",
usage: { input: 2 },
});
await flushDiagnosticEvents();
expect(telemetryState.histograms.get("gen_ai.client.token.usage")?.record).toHaveBeenCalledWith(
2,
{
"gen_ai.operation.name": "chat",
"gen_ai.provider.name": "openai",
"gen_ai.request.model": "unknown",
"gen_ai.token.type": "input",
},
);
await service.stop?.(ctx);
});
test("exports GenAI usage attributes on model usage spans without diagnostic identifiers", async () => {
const service = createDiagnosticsOtelService();
const ctx = createOtelContext(OTEL_TEST_ENDPOINT, { traces: true });
await service.start(ctx);
emitDiagnosticEvent({
type: "model.usage",
sessionKey: "session-key",
sessionId: "session-id",
provider: "anthropic",
model: "claude-sonnet-4.6",
usage: {
input: 100,
output: 40,
cacheRead: 30,
cacheWrite: 20,
promptTokens: 150,
total: 190,
},
durationMs: 25,
});
await flushDiagnosticEvents();
const modelUsageCall = telemetryState.tracer.startSpan.mock.calls.find(
(call) => call[0] === "openclaw.model.usage",
);
expect(modelUsageCall?.[1]).toMatchObject({
attributes: {
"gen_ai.operation.name": "chat",
"gen_ai.system": "anthropic",
"gen_ai.request.model": "claude-sonnet-4.6",
"gen_ai.usage.input_tokens": 150,
"gen_ai.usage.output_tokens": 40,
"gen_ai.usage.cache_read.input_tokens": 30,
"gen_ai.usage.cache_creation.input_tokens": 20,
},
});
expect(modelUsageCall?.[1]).toEqual({
attributes: expect.not.objectContaining({
"openclaw.sessionKey": expect.anything(),
"openclaw.sessionId": expect.anything(),
"gen_ai.provider.name": expect.anything(),
"gen_ai.input.messages": expect.anything(),
"gen_ai.output.messages": expect.anything(),
}),
startTime: expect.any(Number),
});
expect(JSON.stringify(modelUsageCall)).not.toContain("session-key");
await service.stop?.(ctx);
});
test("exports GenAI client operation duration histogram without diagnostic identifiers", async () => {
const service = createDiagnosticsOtelService();
const ctx = createOtelContext(OTEL_TEST_ENDPOINT, { metrics: true });
await service.start(ctx);
emitDiagnosticEvent({
type: "model.call.completed",
runId: "run-1",
callId: "call-1",
sessionKey: "session-key",
provider: "openai",
model: "gpt-5.4",
api: "openai-completions",
durationMs: 250,
});
emitDiagnosticEvent({
type: "model.call.error",
runId: "run-1",
callId: "call-2",
sessionKey: "session-key",
provider: "google",
model: "gemini-2.5-flash",
api: "google-generative-ai",
durationMs: 1250,
errorCategory: "TimeoutError",
});
await flushDiagnosticEvents();
expect(telemetryState.meter.createHistogram).toHaveBeenCalledWith(
"gen_ai.client.operation.duration",
expect.objectContaining({
unit: "s",
advice: {
explicitBucketBoundaries: expect.arrayContaining([0.01, 0.32, 2.56, 81.92]),
},
}),
);
const genAiOperationDuration = telemetryState.histograms.get(
"gen_ai.client.operation.duration",
);
expect(genAiOperationDuration?.record).toHaveBeenCalledTimes(2);
expect(genAiOperationDuration?.record).toHaveBeenCalledWith(0.25, {
"gen_ai.operation.name": "text_completion",
"gen_ai.provider.name": "openai",
"gen_ai.request.model": "gpt-5.4",
});
expect(genAiOperationDuration?.record).toHaveBeenCalledWith(1.25, {
"gen_ai.operation.name": "generate_content",
"gen_ai.provider.name": "google",
"gen_ai.request.model": "gemini-2.5-flash",
"error.type": "TimeoutError",
});
expect(JSON.stringify(genAiOperationDuration?.record.mock.calls)).not.toContain("session-key");
expect(JSON.stringify(genAiOperationDuration?.record.mock.calls)).not.toContain("run-1");
await service.stop?.(ctx);
});
test("exports run, model call, and tool execution lifecycle spans", async () => {
const service = createDiagnosticsOtelService();
const ctx = createOtelContext(OTEL_TEST_ENDPOINT, { traces: true, metrics: true });
@@ -926,6 +1107,13 @@ describe("diagnostics-otel service", () => {
api: "openai-completions",
durationMs: 80,
});
emitDiagnosticEvent({
type: "model.usage",
provider: "openai",
model: "gpt-5.4",
usage: { input: 3, output: 2 },
durationMs: 10,
});
await flushDiagnosticEvents();
const modelCall = telemetryState.tracer.startSpan.mock.calls.find(
@@ -944,6 +1132,22 @@ describe("diagnostics-otel service", () => {
}),
startTime: expect.any(Number),
});
const modelUsage = telemetryState.tracer.startSpan.mock.calls.find(
(call) => call[0] === "openclaw.model.usage",
);
expect(modelUsage?.[1]).toMatchObject({
attributes: {
"gen_ai.provider.name": "openai",
"gen_ai.request.model": "gpt-5.4",
"gen_ai.operation.name": "chat",
},
});
expect(modelUsage?.[1]).toEqual({
attributes: expect.not.objectContaining({
"gen_ai.system": expect.anything(),
}),
startTime: expect.any(Number),
});
await service.stop?.(ctx);
});

View File

@@ -52,6 +52,12 @@ const BLOCKED_OTEL_LOG_ATTRIBUTE_KEYS = new Set(["__proto__", "prototype", "cons
const PRELOADED_OTEL_SDK_ENV = "OPENCLAW_OTEL_PRELOADED";
const OTEL_SEMCONV_STABILITY_OPT_IN_ENV = "OTEL_SEMCONV_STABILITY_OPT_IN";
const GEN_AI_LATEST_EXPERIMENTAL_OPT_IN = "gen_ai_latest_experimental";
const GEN_AI_TOKEN_USAGE_BUCKETS = [
1, 4, 16, 64, 256, 1024, 4096, 16384, 65536, 262144, 1048576, 4194304, 16777216, 67108864,
];
const GEN_AI_OPERATION_DURATION_BUCKETS = [
0.01, 0.02, 0.04, 0.08, 0.16, 0.32, 0.64, 1.28, 2.56, 5.12, 10.24, 20.48, 40.96, 81.92,
];
type OtelContentCapturePolicy = {
inputMessages: boolean;
@@ -171,17 +177,41 @@ function genAiOperationName(
return "chat";
}
function positiveFiniteNumber(value: number | undefined): number | undefined {
return typeof value === "number" && Number.isFinite(value) && value > 0 ? value : undefined;
}
function assignPositiveNumberAttr(
attrs: Record<string, string | number>,
key: string,
value: number | undefined,
): void {
const normalized = positiveFiniteNumber(value);
if (normalized !== undefined) {
attrs[key] = normalized;
}
}
function assignGenAiSpanIdentityAttrs(
attrs: Record<string, string | number | boolean>,
input: { api?: string; model?: string; provider?: string },
): void {
if (emitLatestGenAiSemconv()) {
attrs["gen_ai.provider.name"] = lowCardinalityAttr(input.provider);
} else {
attrs["gen_ai.system"] = lowCardinalityAttr(input.provider);
}
if (input.model) {
attrs["gen_ai.request.model"] = lowCardinalityAttr(input.model);
}
attrs["gen_ai.operation.name"] = genAiOperationName(input.api);
}
function assignGenAiModelCallAttrs(
attrs: Record<string, string | number | boolean>,
evt: ModelCallLifecycleDiagnosticEvent,
): void {
if (emitLatestGenAiSemconv()) {
attrs["gen_ai.provider.name"] = evt.provider;
} else {
attrs["gen_ai.system"] = evt.provider;
}
attrs["gen_ai.request.model"] = evt.model;
attrs["gen_ai.operation.name"] = genAiOperationName(evt.api);
assignGenAiSpanIdentityAttrs(attrs, evt);
}
function addUpstreamRequestIdSpanEvent(
@@ -575,6 +605,23 @@ export function createDiagnosticsOtelService(): OpenClawPluginService {
unit: "1",
description: "Token usage by type",
});
const genAiTokenUsageHistogram = meter.createHistogram("gen_ai.client.token.usage", {
unit: "{token}",
description: "Number of input and output tokens used by GenAI client operations",
advice: {
explicitBucketBoundaries: GEN_AI_TOKEN_USAGE_BUCKETS,
},
});
const genAiOperationDurationHistogram = meter.createHistogram(
"gen_ai.client.operation.duration",
{
unit: "s",
description: "GenAI client operation duration",
advice: {
explicitBucketBoundaries: GEN_AI_OPERATION_DURATION_BUCKETS,
},
},
);
const costCounter = meter.createCounter("openclaw.cost.usd", {
unit: "1",
description: "Estimated model cost (USD)",
@@ -854,13 +901,26 @@ export function createDiagnosticsOtelService(): OpenClawPluginService {
"openclaw.provider": evt.provider ?? "unknown",
"openclaw.model": evt.model ?? "unknown",
};
const genAiAttrs: Record<string, string> = {
"gen_ai.operation.name": "chat",
"gen_ai.provider.name": lowCardinalityAttr(evt.provider),
"gen_ai.request.model": lowCardinalityAttr(evt.model),
};
const usage = evt.usage;
if (usage.input) {
tokensCounter.add(usage.input, { ...attrs, "openclaw.token": "input" });
genAiTokenUsageHistogram.record(usage.input, {
...genAiAttrs,
"gen_ai.token.type": "input",
});
}
if (usage.output) {
tokensCounter.add(usage.output, { ...attrs, "openclaw.token": "output" });
genAiTokenUsageHistogram.record(usage.output, {
...genAiAttrs,
"gen_ai.token.type": "output",
});
}
if (usage.cacheRead) {
tokensCounter.add(usage.cacheRead, { ...attrs, "openclaw.token": "cache_read" });
@@ -897,6 +957,9 @@ export function createDiagnosticsOtelService(): OpenClawPluginService {
if (!tracesEnabled) {
return;
}
const genAiInputTokens =
usage.promptTokens ??
(usage.input ?? 0) + (usage.cacheRead ?? 0) + (usage.cacheWrite ?? 0);
const spanAttrs: Record<string, string | number> = {
...attrs,
"openclaw.tokens.input": usage.input ?? 0,
@@ -905,6 +968,19 @@ export function createDiagnosticsOtelService(): OpenClawPluginService {
"openclaw.tokens.cache_write": usage.cacheWrite ?? 0,
"openclaw.tokens.total": usage.total ?? 0,
};
assignGenAiSpanIdentityAttrs(spanAttrs, evt);
assignPositiveNumberAttr(spanAttrs, "gen_ai.usage.input_tokens", genAiInputTokens);
assignPositiveNumberAttr(spanAttrs, "gen_ai.usage.output_tokens", usage.output);
assignPositiveNumberAttr(
spanAttrs,
"gen_ai.usage.cache_read.input_tokens",
usage.cacheRead,
);
assignPositiveNumberAttr(
spanAttrs,
"gen_ai.usage.cache_creation.input_tokens",
usage.cacheWrite,
);
const span = spanWithDuration("openclaw.model.usage", spanAttrs, evt.durationMs, {
parentContext: contextForTrustedDiagnosticSpanParent(evt, metadata),
@@ -1284,12 +1360,25 @@ export function createDiagnosticsOtelService(): OpenClawPluginService {
"openclaw.api": lowCardinalityAttr(evt.api),
"openclaw.transport": lowCardinalityAttr(evt.transport),
});
const genAiModelCallMetricAttrs = (
evt: ModelCallLifecycleDiagnosticEvent,
errorType?: string,
) => ({
"gen_ai.operation.name": genAiOperationName(evt.api),
"gen_ai.provider.name": lowCardinalityAttr(evt.provider),
"gen_ai.request.model": lowCardinalityAttr(evt.model),
...(errorType ? { "error.type": errorType } : {}),
});
const recordModelCallCompleted = (
evt: Extract<DiagnosticEventPayload, { type: "model.call.completed" }>,
metadata: DiagnosticEventMetadata,
) => {
modelCallDurationHistogram.record(evt.durationMs, modelCallMetricAttrs(evt));
genAiOperationDurationHistogram.record(
evt.durationMs / 1000,
genAiModelCallMetricAttrs(evt),
);
if (!tracesEnabled) {
return;
}
@@ -1321,18 +1410,23 @@ export function createDiagnosticsOtelService(): OpenClawPluginService {
evt: Extract<DiagnosticEventPayload, { type: "model.call.error" }>,
metadata: DiagnosticEventMetadata,
) => {
const errorType = lowCardinalityAttr(evt.errorCategory, "other");
modelCallDurationHistogram.record(evt.durationMs, {
...modelCallMetricAttrs(evt),
"openclaw.errorCategory": lowCardinalityAttr(evt.errorCategory, "other"),
"openclaw.errorCategory": errorType,
});
genAiOperationDurationHistogram.record(
evt.durationMs / 1000,
genAiModelCallMetricAttrs(evt, errorType),
);
if (!tracesEnabled) {
return;
}
const spanAttrs: Record<string, string | number | boolean> = {
"openclaw.provider": evt.provider,
"openclaw.model": evt.model,
"openclaw.errorCategory": lowCardinalityAttr(evt.errorCategory, "other"),
"error.type": lowCardinalityAttr(evt.errorCategory, "other"),
"openclaw.errorCategory": errorType,
"error.type": errorType,
};
assignGenAiModelCallAttrs(spanAttrs, evt);
if (evt.api) {

View File

@@ -61,7 +61,7 @@ describeLive("elevenlabs plugin live", () => {
const normalized = normalizeTranscriptForMatch(transcript?.text ?? "");
expect(normalized).toContain("openclaw");
expect(normalized).toContain("elevenlabs");
expect(normalized).toMatch(/(?:elevenlabs|11labs)/);
}, 90_000);
it("streams realtime STT through the registered transcription provider", async () => {

View File

@@ -76,6 +76,7 @@ describe("fal image-generation provider", () => {
cfg: {},
count: 2,
size: "1536x1024",
outputFormat: "jpeg",
});
expectFalJsonPost({
@@ -85,7 +86,7 @@ describe("fal image-generation provider", () => {
prompt: "draw a cat",
image_size: { width: 1536, height: 1024 },
num_images: 2,
output_format: "png",
output_format: "jpeg",
},
});
expect(fetchWithSsrFGuardMock).toHaveBeenNthCalledWith(

View File

@@ -25,6 +25,7 @@ const DEFAULT_FAL_BASE_URL = "https://fal.run";
const DEFAULT_FAL_IMAGE_MODEL = "fal-ai/flux/dev";
const DEFAULT_FAL_EDIT_SUBPATH = "image-to-image";
const DEFAULT_OUTPUT_FORMAT = "png";
const FAL_OUTPUT_FORMATS = ["png", "jpeg"] as const;
const FAL_SUPPORTED_SIZES = [
"1024x1024",
"1024x1536",
@@ -292,6 +293,9 @@ export function buildFalImageGenerationProvider(): ImageGenerationProvider {
aspectRatios: [...FAL_SUPPORTED_ASPECT_RATIOS],
resolutions: ["1K", "2K", "4K"],
},
output: {
formats: [...FAL_OUTPUT_FORMATS],
},
},
async generateImage(req) {
const auth = await resolveApiKeyForProvider({
@@ -333,7 +337,7 @@ export function buildFalImageGenerationProvider(): ImageGenerationProvider {
const requestBody: Record<string, unknown> = {
prompt: req.prompt,
num_images: req.count ?? 1,
output_format: DEFAULT_OUTPUT_FORMAT,
output_format: req.outputFormat ?? DEFAULT_OUTPUT_FORMAT,
};
if (imageSize !== undefined) {
requestBody.image_size = imageSize;

View File

@@ -858,19 +858,25 @@ describe("google-meet plugin", () => {
});
it("reports setup status through the tool", async () => {
const { tools } = setup({
chrome: {
audioInputCommand: ["openclaw-audio-bridge", "capture"],
audioOutputCommand: ["openclaw-audio-bridge", "play"],
},
});
const tool = tools[0] as {
execute: (id: string, params: unknown) => Promise<{ details: { ok?: boolean } }>;
};
const originalPlatform = process.platform;
Object.defineProperty(process, "platform", { value: "darwin" });
try {
const { tools } = setup({
chrome: {
audioInputCommand: ["openclaw-audio-bridge", "capture"],
audioOutputCommand: ["openclaw-audio-bridge", "play"],
},
});
const tool = tools[0] as {
execute: (id: string, params: unknown) => Promise<{ details: { ok?: boolean } }>;
};
const result = await tool.execute("id", { action: "setup_status" });
const result = await tool.execute("id", { action: "setup_status" });
expect(result.details.ok).toBe(true);
expect(result.details.ok).toBe(true);
} finally {
Object.defineProperty(process, "platform", { value: originalPlatform });
}
});
it("reports attendance through the tool", async () => {
@@ -1045,7 +1051,20 @@ describe("google-meet plugin", () => {
defaultTransport: "chrome-node",
chromeNode: { node: "parallels-macos" },
},
{ nodesListResult: { nodes: [] } },
{
nodesListResult: {
nodes: [
{
nodeId: "node-1",
displayName: "parallels-macos",
connected: false,
caps: [],
commands: [],
remoteIp: "192.168.0.25",
},
],
},
},
);
const tool = tools[0] as {
execute: (
@@ -1062,10 +1081,97 @@ describe("google-meet plugin", () => {
expect.objectContaining({
id: "chrome-node-connected",
ok: false,
message: expect.stringContaining("No connected Google Meet-capable node"),
message: expect.stringContaining("parallels-macos"),
}),
]),
);
const check = result.details.checks?.find(
(item) => (item as { id?: unknown }).id === "chrome-node-connected",
) as { message?: string } | undefined;
expect(check?.message).toContain("offline");
expect(check?.message).toContain("missing googlemeet.chrome");
expect(check?.message).toContain("missing browser.proxy/browser capability");
});
it("reports missing local Chrome audio prerequisites in setup status", async () => {
const originalPlatform = process.platform;
Object.defineProperty(process, "platform", { value: "darwin" });
try {
const { tools } = setup(
{ defaultTransport: "chrome" },
{
runCommandWithTimeoutHandler: async (argv) => {
if (argv[0] === "/usr/sbin/system_profiler") {
return { code: 0, stdout: "Built-in Output", stderr: "" };
}
return { code: 0, stdout: "", stderr: "" };
},
},
);
const tool = tools[0] as {
execute: (
id: string,
params: unknown,
) => Promise<{ details: { ok?: boolean; checks?: unknown[] } }>;
};
const result = await tool.execute("id", { action: "setup_status", transport: "chrome" });
expect(result.details.ok).toBe(false);
expect(result.details.checks).toEqual(
expect.arrayContaining([
expect.objectContaining({
id: "chrome-local-audio-device",
ok: false,
message: expect.stringContaining("BlackHole 2ch audio device not found"),
}),
]),
);
} finally {
Object.defineProperty(process, "platform", { value: originalPlatform });
}
});
it("reports missing local Chrome audio commands in setup status", async () => {
const originalPlatform = process.platform;
Object.defineProperty(process, "platform", { value: "darwin" });
try {
const { tools } = setup(
{ defaultTransport: "chrome" },
{
runCommandWithTimeoutHandler: async (argv) => {
if (argv[0] === "/usr/sbin/system_profiler") {
return { code: 0, stdout: "BlackHole 2ch", stderr: "" };
}
if (argv[0] === "/bin/sh" && argv.at(-1) === "play") {
return { code: 1, stdout: "", stderr: "" };
}
return { code: 0, stdout: "", stderr: "" };
},
},
);
const tool = tools[0] as {
execute: (
id: string,
params: unknown,
) => Promise<{ details: { ok?: boolean; checks?: unknown[] } }>;
};
const result = await tool.execute("id", { action: "setup_status", transport: "chrome" });
expect(result.details.ok).toBe(false);
expect(result.details.checks).toEqual(
expect.arrayContaining([
expect.objectContaining({
id: "chrome-local-audio-commands",
ok: false,
message: "Chrome audio command missing: play",
}),
]),
);
} finally {
Object.defineProperty(process, "platform", { value: originalPlatform });
}
});
it("reports Twilio delegation readiness when voice-call is enabled", async () => {
@@ -1217,7 +1323,7 @@ describe("google-meet plugin", () => {
});
expect(respond.mock.calls[0]?.[0]).toBe(true);
expect(nodesList).toHaveBeenCalledWith({ connected: true });
expect(nodesList.mock.calls[0]).toEqual([]);
expect(nodesInvoke).toHaveBeenCalledWith(
expect.objectContaining({
nodeId: "node-1",

View File

@@ -566,10 +566,10 @@ export default definePluginEntry({
api.registerGatewayMethod(
"googlemeet.setup",
async ({ respond }: GatewayRequestHandlerOptions) => {
async ({ params, respond }: GatewayRequestHandlerOptions) => {
try {
const rt = await ensureRuntime();
respond(true, await rt.setupStatus());
respond(true, await rt.setupStatus({ transport: normalizeTransport(params?.transport) }));
} catch (err) {
sendError(respond, err);
}
@@ -741,7 +741,7 @@ export default definePluginEntry({
name: "google_meet",
label: "Google Meet",
description:
"Join and track Google Meet sessions through Chrome or Twilio. If a Meet tab is already open after a timeout, call recover_current_tab before retrying join to report login, permission, or admission blockers without opening another tab.",
"Join and track Google Meet sessions through Chrome or Twilio. Call setup_status before join/create/test_speech; if it reports a Chrome node offline or local audio missing, surface that blocker instead of retrying or switching transports. Offline nodes are diagnostics only, not usable candidates. If a Meet tab is already open after a timeout, call recover_current_tab before retrying join to report login, permission, or admission blockers without opening another tab.",
parameters: GoogleMeetToolSchema,
async execute(_toolCallId, params) {
const raw = asParamRecord(params);
@@ -797,7 +797,7 @@ export default definePluginEntry({
}
case "setup_status": {
const rt = await ensureRuntime();
return json(await rt.setupStatus());
return json(await rt.setupStatus({ transport: normalizeTransport(raw.transport) }));
}
case "resolve_space": {
const { token: _token, ...result } = await resolveSpaceFromParams(config, raw);

View File

@@ -129,6 +129,7 @@ export type GoogleMeetExportManifest = {
type SetupOptions = {
json?: boolean;
transport?: GoogleMeetTransport;
};
type DoctorOptions = {
@@ -1975,10 +1976,11 @@ export function registerGoogleMeetCli(params: {
root
.command("setup")
.description("Show Google Meet transport setup status")
.option("--transport <transport>", "Transport to check: chrome, chrome-node, or twilio")
.option("--json", "Print JSON output", false)
.action(async (options: SetupOptions) => {
const rt = await params.ensureRuntime();
const status = await rt.setupStatus();
const status = await rt.setupStatus({ transport: options.transport });
if (options.json) {
writeStdoutJson(status);
return;

View File

@@ -8,6 +8,7 @@ import { addGoogleMeetSetupCheck, getGoogleMeetSetupStatus } from "./setup.js";
import { isSameMeetUrlForReuse, resolveChromeNodeInfo } from "./transports/chrome-browser-proxy.js";
import { createMeetWithBrowserProxyOnNode } from "./transports/chrome-create.js";
import {
assertBlackHole2chAvailable,
launchChromeMeet,
launchChromeMeetOnNode,
recoverCurrentMeetTabOnNode,
@@ -53,6 +54,21 @@ function resolveMode(input: GoogleMeetMode | undefined, config: GoogleMeetConfig
return input ?? config.defaultMode;
}
function collectChromeAudioCommands(config: GoogleMeetConfig): string[] {
const commands = config.chrome.audioBridgeCommand
? [config.chrome.audioBridgeCommand[0]]
: [config.chrome.audioInputCommand?.[0], config.chrome.audioOutputCommand?.[0]];
return [...new Set(commands.filter((value): value is string => Boolean(value?.trim())))];
}
async function commandExists(runtime: PluginRuntime, command: string): Promise<boolean> {
const result = await runtime.system.runCommandWithTimeout(
["/bin/sh", "-lc", 'command -v "$1" >/dev/null 2>&1', "sh", command],
{ timeoutMs: 5_000 },
);
return result.code === 0;
}
export class GoogleMeetRuntime {
readonly #sessions = new Map<string, GoogleMeetSession>();
readonly #sessionStops = new Map<string, () => Promise<void>>();
@@ -86,14 +102,15 @@ export class GoogleMeetRuntime {
return session ? { found: true, session } : { found: false };
}
async setupStatus() {
async setupStatus(options: { transport?: GoogleMeetTransport } = {}) {
const transport = resolveTransport(options.transport, this.params.config);
const shouldCheckChromeNode =
transport === "chrome-node" ||
(!options.transport && Boolean(this.params.config.chromeNode.node));
let status = getGoogleMeetSetupStatus(this.params.config, {
fullConfig: this.params.fullConfig,
});
if (
this.params.config.defaultTransport === "chrome-node" ||
Boolean(this.params.config.chromeNode.node)
) {
if (shouldCheckChromeNode) {
try {
const node = await resolveChromeNodeInfo({
runtime: this.params.runtime,
@@ -113,6 +130,47 @@ export class GoogleMeetRuntime {
});
}
}
if (transport === "chrome") {
try {
await assertBlackHole2chAvailable({
runtime: this.params.runtime,
timeoutMs: Math.min(this.params.config.chrome.joinTimeoutMs, 10_000),
});
status = addGoogleMeetSetupCheck(status, {
id: "chrome-local-audio-device",
ok: true,
message: "BlackHole 2ch audio device found",
});
} catch (error) {
status = addGoogleMeetSetupCheck(status, {
id: "chrome-local-audio-device",
ok: false,
message: formatErrorMessage(error),
});
}
const commands = collectChromeAudioCommands(this.params.config);
const missingCommands: string[] = [];
for (const command of commands) {
try {
if (!(await commandExists(this.params.runtime, command))) {
missingCommands.push(command);
}
} catch {
missingCommands.push(command);
}
}
status = addGoogleMeetSetupCheck(status, {
id: "chrome-local-audio-commands",
ok: commands.length > 0 && missingCommands.length === 0,
message:
commands.length === 0
? "Chrome realtime audio commands are not configured"
: missingCommands.length === 0
? `Chrome audio command${commands.length === 1 ? "" : "s"} available: ${commands.join(", ")}`
: `Chrome audio command${missingCommands.length === 1 ? "" : "s"} missing: ${missingCommands.join(", ")}`,
});
}
return status;
}

View File

@@ -24,6 +24,12 @@ export type GoogleMeetTestNodeListResult = {
}>;
};
type CommandResult = {
code: number;
stdout?: string;
stderr?: string;
};
export function captureStdout() {
let output = "";
const writeSpy = vi.spyOn(process.stdout, "write").mockImplementation(((chunk: unknown) => {
@@ -50,6 +56,10 @@ export function setupGoogleMeetPlugin(
params?: unknown;
timeoutMs?: number;
}) => Promise<unknown>;
runCommandWithTimeoutHandler?: (
argv: string[],
options?: { timeoutMs?: number },
) => Promise<CommandResult>;
} = {},
) {
const methods = new Map<string, unknown>();
@@ -112,12 +122,17 @@ export function setupGoogleMeetPlugin(
}
return options.nodesInvokeResult ?? { launched: true };
});
const runCommandWithTimeout = vi.fn(async (argv: string[]) => {
if (argv[0] === "/usr/sbin/system_profiler") {
return { code: 0, stdout: "BlackHole 2ch", stderr: "" };
}
return { code: 0, stdout: "", stderr: "" };
});
const runCommandWithTimeout = vi.fn(
async (argv: string[], runOptions?: { timeoutMs?: number }) => {
if (options.runCommandWithTimeoutHandler) {
return options.runCommandWithTimeoutHandler(argv, runOptions);
}
if (argv[0] === "/usr/sbin/system_profiler") {
return { code: 0, stdout: "BlackHole 2ch", stderr: "" };
}
return { code: 0, stdout: "", stderr: "" };
},
);
const api = createTestPluginApi({
id: "google-meet",
name: "Google Meet",

View File

@@ -54,27 +54,78 @@ function isGoogleMeetNode(node: GoogleMeetNodeInfo) {
);
}
function matchesRequestedNode(node: GoogleMeetNodeInfo, requested: string): boolean {
return [node.nodeId, node.displayName, node.remoteIp].some((value) => value === requested);
}
function formatNodeLabel(node: GoogleMeetNodeInfo): string {
const parts = [node.displayName, node.nodeId, node.remoteIp].filter(Boolean);
return parts.length > 0 ? parts.join(" / ") : "unknown node";
}
function describeNodeUsabilityIssues(node: GoogleMeetNodeInfo): string[] {
const commands = Array.isArray(node.commands) ? node.commands : [];
const caps = Array.isArray(node.caps) ? node.caps : [];
const issues: string[] = [];
if (node.connected !== true) {
issues.push("offline");
}
if (!commands.includes("googlemeet.chrome")) {
issues.push("missing googlemeet.chrome");
}
if (!commands.includes("browser.proxy") && !caps.includes("browser")) {
issues.push("missing browser.proxy/browser capability");
}
return issues;
}
async function listGoogleMeetNodes(
runtime: PluginRuntime,
params?: { connected?: boolean },
): Promise<{ nodes: GoogleMeetNodeInfo[] }> {
try {
return params ? await runtime.nodes.list(params) : await runtime.nodes.list();
} catch (error) {
throw new Error("Google Meet node inventory unavailable", {
cause: error,
});
}
}
export async function resolveChromeNodeInfo(params: {
runtime: PluginRuntime;
requestedNode?: string;
}): Promise<GoogleMeetNodeInfo> {
const list = await params.runtime.nodes.list({ connected: true });
const requested = params.requestedNode?.trim();
if (requested) {
const list = await listGoogleMeetNodes(params.runtime);
const matches = list.nodes.filter((node) => matchesRequestedNode(node, requested));
if (matches.length === 1) {
const [node] = matches;
if (isGoogleMeetNode(node)) {
return node;
}
throw new Error(
`Configured Google Meet node ${requested} is not usable (${formatNodeLabel(node)}): ${describeNodeUsabilityIssues(node).join("; ")}. Start or reinstall \`openclaw node run\` on that Chrome host, approve pairing, and allow googlemeet.chrome plus browser.proxy.`,
);
}
if (matches.length > 1) {
throw new Error(
`Configured Google Meet node ${requested} is ambiguous (${matches.length} matches). Pin chromeNode.node to a unique node id, display name, or remote IP.`,
);
}
throw new Error(
`Configured Google Meet node ${requested} was not found. Run \`openclaw nodes status\` and start or approve the Chrome node.`,
);
}
const list = await listGoogleMeetNodes(params.runtime, { connected: true });
const nodes = list.nodes.filter(isGoogleMeetNode);
if (nodes.length === 0) {
throw new Error(
"No connected Google Meet-capable node with browser proxy. Run `openclaw node run` on the Chrome host with browser proxy enabled, approve pairing, and allow googlemeet.chrome plus browser.proxy.",
);
}
const requested = params.requestedNode?.trim();
if (requested) {
const matches = nodes.filter((node) =>
[node.nodeId, node.displayName, node.remoteIp].some((value) => value === requested),
);
if (matches.length === 1) {
return matches[0];
}
throw new Error(`Google Meet node not found or ambiguous: ${requested}`);
}
if (nodes.length === 1) {
return nodes[0];
}

View File

@@ -1,17 +1,22 @@
import * as providerAuth from "openclaw/plugin-sdk/provider-auth-runtime";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { installPinnedHostnameTestHooks } from "../../src/media-understanding/audio.test-helpers.js";
import { buildMinimaxImageGenerationProvider } from "./image-generation-provider.js";
import {
buildMinimaxImageGenerationProvider,
buildMinimaxPortalImageGenerationProvider,
} from "./image-generation-provider.js";
installPinnedHostnameTestHooks();
describe("minimax image-generation provider", () => {
beforeEach(() => {
vi.clearAllMocks();
vi.stubEnv("MINIMAX_API_HOST", "");
});
afterEach(() => {
vi.restoreAllMocks();
vi.unstubAllEnvs();
});
function mockMinimaxApiKey() {
@@ -41,6 +46,10 @@ describe("minimax image-generation provider", () => {
return fetchMock;
}
function expectImageGenerationUrl(fetchMock: ReturnType<typeof vi.fn>, url: string) {
expect(fetchMock).toHaveBeenCalledWith(url, expect.any(Object));
}
it("generates PNG buffers through the shared provider HTTP path", async () => {
mockMinimaxApiKey();
const fetchMock = mockSuccessfulMinimaxImageResponse();
@@ -81,7 +90,7 @@ describe("minimax image-generation provider", () => {
});
});
it("uses the configured provider base URL origin", async () => {
it("keeps the dedicated global image endpoint when text config uses the global API host", async () => {
mockMinimaxApiKey();
const fetchMock = mockSuccessfulMinimaxImageResponse();
@@ -102,36 +111,118 @@ describe("minimax image-generation provider", () => {
},
});
expect(fetchMock).toHaveBeenCalledWith(
"https://api.minimax.io/v1/image_generation",
expect.any(Object),
);
expectImageGenerationUrl(fetchMock, "https://api.minimax.io/v1/image_generation");
});
it("does not allow private-network routing just because a custom base URL is configured", async () => {
it("does not inherit unrelated MiniMax text endpoint hosts for image generation", async () => {
mockMinimaxApiKey();
const fetchMock = vi.fn();
vi.stubGlobal("fetch", fetchMock);
const fetchMock = mockSuccessfulMinimaxImageResponse();
const provider = buildMinimaxImageGenerationProvider();
await expect(
provider.generateImage({
provider: "minimax",
model: "image-01",
prompt: "draw a cat",
cfg: {
models: {
providers: {
minimax: {
baseUrl: "http://127.0.0.1:8080/anthropic",
models: [],
},
await provider.generateImage({
provider: "minimax",
model: "image-01",
prompt: "draw a cat",
cfg: {
models: {
providers: {
minimax: {
baseUrl: "https://api.minimax.chat/anthropic",
models: [],
},
},
},
}),
).rejects.toThrow("Blocked hostname or private/internal/special-use IP address");
},
});
expect(fetchMock).not.toHaveBeenCalled();
expectImageGenerationUrl(fetchMock, "https://api.minimax.io/v1/image_generation");
});
it("uses the dedicated CN image endpoint when CN API host is configured", async () => {
vi.stubEnv("MINIMAX_API_HOST", "https://api.minimaxi.com/anthropic");
mockMinimaxApiKey();
const fetchMock = mockSuccessfulMinimaxImageResponse();
const provider = buildMinimaxImageGenerationProvider();
await provider.generateImage({
provider: "minimax",
model: "image-01",
prompt: "draw a cat",
cfg: {},
});
expectImageGenerationUrl(fetchMock, "https://api.minimaxi.com/v1/image_generation");
});
it("infers the dedicated CN image endpoint from MiniMax provider config", async () => {
mockMinimaxApiKey();
const fetchMock = mockSuccessfulMinimaxImageResponse();
const provider = buildMinimaxImageGenerationProvider();
await provider.generateImage({
provider: "minimax",
model: "image-01",
prompt: "draw a cat",
cfg: {
models: {
providers: {
minimax: {
baseUrl: "https://api.minimaxi.com/anthropic",
models: [],
},
},
},
},
});
expectImageGenerationUrl(fetchMock, "https://api.minimaxi.com/v1/image_generation");
});
it("infers the dedicated CN image endpoint from MiniMax Portal provider config", async () => {
mockMinimaxApiKey();
const fetchMock = mockSuccessfulMinimaxImageResponse();
const provider = buildMinimaxPortalImageGenerationProvider();
await provider.generateImage({
provider: "minimax-portal",
model: "image-01",
prompt: "draw a cat",
cfg: {
models: {
providers: {
"minimax-portal": {
baseUrl: "api.minimaxi.com/anthropic",
models: [],
},
},
},
},
});
expectImageGenerationUrl(fetchMock, "https://api.minimaxi.com/v1/image_generation");
});
it("ignores private custom text endpoints for image generation", async () => {
mockMinimaxApiKey();
const fetchMock = mockSuccessfulMinimaxImageResponse();
const provider = buildMinimaxImageGenerationProvider();
await provider.generateImage({
provider: "minimax",
model: "image-01",
prompt: "draw a cat",
cfg: {
models: {
providers: {
minimax: {
baseUrl: "http://127.0.0.1:8080/anthropic",
models: [],
},
},
},
},
});
expectImageGenerationUrl(fetchMock, "https://api.minimax.io/v1/image_generation");
});
});

View File

@@ -8,6 +8,7 @@ import {
} from "openclaw/plugin-sdk/provider-http";
const DEFAULT_MINIMAX_IMAGE_BASE_URL = "https://api.minimax.io";
const CN_MINIMAX_IMAGE_BASE_URL = "https://api.minimaxi.com";
const DEFAULT_MODEL = "image-01";
const DEFAULT_OUTPUT_MIME = "image/png";
const MINIMAX_SUPPORTED_ASPECT_RATIOS = [
@@ -36,20 +37,37 @@ type MinimaxImageApiResponse = {
};
};
function isMinimaxCnHost(value: string | undefined): boolean {
const trimmed = value?.trim();
if (!trimmed) {
return false;
}
const candidate = /^[a-z][a-z\d+.-]*:\/\//iu.test(trimmed) ? trimmed : `https://${trimmed}`;
try {
const hostname = new URL(candidate).hostname.toLowerCase();
return hostname === "minimaxi.com" || hostname.endsWith(".minimaxi.com");
} catch {
return false;
}
}
function resolveMinimaxImageBaseUrl(
cfg: Parameters<typeof resolveApiKeyForProvider>[0]["cfg"],
providerId: string,
): string {
const direct = cfg?.models?.providers?.[providerId]?.baseUrl?.trim();
if (!direct) {
return DEFAULT_MINIMAX_IMAGE_BASE_URL;
// MiniMax image generation uses dedicated endpoints that are separate from
// the text/chat API endpoints. First check MINIMAX_API_HOST env var,
// then fall back to the provider's configured baseUrl to determine region.
const apiHost = process.env.MINIMAX_API_HOST;
if (isMinimaxCnHost(apiHost)) {
return CN_MINIMAX_IMAGE_BASE_URL;
}
// Extract origin from the configured base URL (which may include path like /anthropic)
try {
return new URL(direct).origin;
} catch {
return DEFAULT_MINIMAX_IMAGE_BASE_URL;
// CN onboarding stores region in provider config without requiring env var
const providerBaseUrl = cfg?.models?.providers?.[providerId]?.baseUrl;
if (isMinimaxCnHost(providerBaseUrl)) {
return CN_MINIMAX_IMAGE_BASE_URL;
}
return DEFAULT_MINIMAX_IMAGE_BASE_URL;
}
function buildMinimaxImageProvider(providerId: string): ImageGenerationProvider {

View File

@@ -194,13 +194,19 @@ describe("openai image generation provider", () => {
const provider = buildOpenAIImageGenerationProvider();
expect(provider.defaultModel).toBe("gpt-image-2");
expect(provider.models).toEqual(["gpt-image-2"]);
expect(provider.models).toEqual([
"gpt-image-2",
"gpt-image-1.5",
"gpt-image-1",
"gpt-image-1-mini",
]);
expect(provider.capabilities.geometry?.sizes).toEqual(
expect.arrayContaining(["2048x2048", "3840x2160", "2160x3840"]),
);
expect(provider.capabilities.output).toEqual({
formats: ["png", "jpeg", "webp"],
qualities: ["low", "medium", "high", "auto"],
backgrounds: ["transparent", "opaque", "auto"],
});
});
@@ -428,6 +434,70 @@ describe("openai image generation provider", () => {
});
});
it("routes transparent default-model requests to the OpenAI image model that supports alpha", async () => {
mockGeneratedPngResponse();
const provider = buildOpenAIImageGenerationProvider();
const result = await provider.generateImage({
provider: "openai",
model: "gpt-image-2",
prompt: "Transparent sticker",
cfg: {},
outputFormat: "png",
background: "transparent",
});
expect(postJsonRequestMock).toHaveBeenCalledWith(
expect.objectContaining({
url: "https://api.openai.com/v1/images/generations",
body: expect.objectContaining({
model: "gpt-image-1.5",
output_format: "png",
background: "transparent",
}),
}),
);
expect(result.model).toBe("gpt-image-1.5");
});
it("does not reroute transparent requests for custom OpenAI-compatible endpoints", async () => {
mockGeneratedPngResponse();
const provider = buildOpenAIImageGenerationProvider();
await provider.generateImage({
provider: "openai",
model: "gpt-image-2",
prompt: "Transparent custom endpoint sticker",
cfg: {
models: {
providers: {
openai: {
baseUrl: "https://openai-compatible.example.com/v1",
models: [],
},
},
},
},
outputFormat: "png",
providerOptions: {
openai: {
background: "transparent",
},
},
});
expect(postJsonRequestMock).toHaveBeenCalledWith(
expect.objectContaining({
url: "https://openai-compatible.example.com/v1/images/generations",
body: expect.objectContaining({
model: "gpt-image-2",
output_format: "png",
background: "transparent",
}),
}),
);
});
it("allows loopback image requests for the synthetic mock-openai provider", async () => {
mockGeneratedPngResponse();
@@ -684,6 +754,43 @@ describe("openai image generation provider", () => {
});
});
it("routes transparent default-model Codex OAuth requests to the alpha-capable image model", async () => {
mockCodexAuthOnly();
mockCodexImageStream({ imageData: "codex-transparent-image" });
const provider = buildOpenAIImageGenerationProvider();
const result = await provider.generateImage({
provider: "openai",
model: "gpt-image-2",
prompt: "Draw a transparent Codex sticker",
cfg: {},
authStore: { version: 1, profiles: {} },
outputFormat: "png",
providerOptions: {
openai: {
background: "transparent",
},
},
});
expect(postJsonRequestMock).toHaveBeenCalledWith(
expect.objectContaining({
url: "https://chatgpt.com/backend-api/codex/responses",
body: expect.objectContaining({
tools: [
expect.objectContaining({
type: "image_generation",
model: "gpt-image-1.5",
output_format: "png",
background: "transparent",
}),
],
}),
}),
);
expect(result.model).toBe("gpt-image-1.5");
});
it("uses configured Codex OAuth directly instead of probing an available OpenAI API key", async () => {
resolveApiKeyForProviderMock.mockImplementation(async (params?: { provider?: string }) => {
if (params?.provider === "openai") {
@@ -1213,6 +1320,46 @@ describe("openai image generation provider", () => {
);
});
it("does not reroute transparent background requests for Azure deployment names", async () => {
mockGeneratedPngResponse();
const provider = buildOpenAIImageGenerationProvider();
await provider.generateImage({
provider: "openai",
model: "gpt-image-2",
prompt: "Transparent Azure sticker",
cfg: {
models: {
providers: {
openai: {
baseUrl: "https://myresource.openai.azure.com",
models: [],
},
},
},
},
outputFormat: "png",
providerOptions: {
openai: {
background: "transparent",
},
},
});
expect(postJsonRequestMock).toHaveBeenCalledWith(
expect.objectContaining({
url: "https://myresource.openai.azure.com/openai/deployments/gpt-image-2/images/generations?api-version=2024-12-01-preview",
body: {
prompt: "Transparent Azure sticker",
n: 1,
size: "1024x1024",
output_format: "png",
background: "transparent",
},
}),
);
});
it("uses api-key header and deployment-scoped URL for .cognitiveservices.azure.com hosts", async () => {
mockGeneratedPngResponse();

View File

@@ -30,6 +30,7 @@ const DEFAULT_OPENAI_IMAGE_BASE_URL = "https://api.openai.com/v1";
const DEFAULT_OPENAI_CODEX_IMAGE_BASE_URL = OPENAI_CODEX_RESPONSES_BASE_URL;
const DEFAULT_OPENAI_CODEX_IMAGE_RESPONSES_MODEL = "gpt-5.5";
const OPENAI_CODEX_IMAGE_INSTRUCTIONS = "You are an image generation assistant.";
const OPENAI_TRANSPARENT_BACKGROUND_IMAGE_MODEL = "gpt-image-1.5";
const DEFAULT_OPENAI_IMAGE_TIMEOUT_MS = 180_000;
const DEFAULT_OUTPUT_MIME = "image/png";
const DEFAULT_OUTPUT_EXTENSION = "png";
@@ -51,7 +52,14 @@ const MAX_CODEX_IMAGE_BASE64_CHARS = 64 * 1024 * 1024;
const LOG_VALUE_MAX_CHARS = 256;
const MOCK_OPENAI_PROVIDER_ID = "mock-openai";
const OPENAI_OUTPUT_FORMATS = ["png", "jpeg", "webp"] as const;
const OPENAI_BACKGROUNDS = ["transparent", "opaque", "auto"] as const;
const OPENAI_QUALITIES = ["low", "medium", "high", "auto"] as const;
const OPENAI_IMAGE_MODELS = [
DEFAULT_OPENAI_IMAGE_MODEL,
OPENAI_TRANSPARENT_BACKGROUND_IMAGE_MODEL,
"gpt-image-1",
"gpt-image-1-mini",
] as const;
const log = createSubsystemLogger("image-generation/openai");
const AZURE_HOSTNAME_SUFFIXES = [
@@ -167,10 +175,11 @@ function appendOpenAIImageOptions(
req: Parameters<ImageGenerationProvider["generateImage"]>[0],
): void {
const openai = req.providerOptions?.openai;
const background = openai?.background ?? req.background;
const entries: Record<string, unknown> = {
...(req.quality !== undefined ? { quality: req.quality } : {}),
...(req.outputFormat !== undefined ? { output_format: req.outputFormat } : {}),
...(openai?.background !== undefined ? { background: openai.background } : {}),
...(background !== undefined ? { background } : {}),
...(openai?.moderation !== undefined ? { moderation: openai.moderation } : {}),
...(openai?.outputCompression !== undefined
? { output_compression: openai.outputCompression }
@@ -186,6 +195,21 @@ function appendOpenAIImageOptions(
}
}
function resolveOpenAIImageRequestModel(
req: Parameters<ImageGenerationProvider["generateImage"]>[0],
options?: { allowTransparentDefaultReroute?: boolean },
): string {
const model = req.model || DEFAULT_OPENAI_IMAGE_MODEL;
if (
options?.allowTransparentDefaultReroute === true &&
model === DEFAULT_OPENAI_IMAGE_MODEL &&
(req.providerOptions?.openai?.background ?? req.background) === "transparent"
) {
return OPENAI_TRANSPARENT_BACKGROUND_IMAGE_MODEL;
}
return model;
}
function shouldAllowPrivateImageEndpoint(req: {
provider: string;
cfg: OpenClawConfig | undefined;
@@ -468,7 +492,7 @@ function createOpenAIImageGenerationProviderBase(params: {
id: params.id,
label: params.label,
defaultModel: DEFAULT_OPENAI_IMAGE_MODEL,
models: [DEFAULT_OPENAI_IMAGE_MODEL],
models: [...OPENAI_IMAGE_MODELS],
isConfigured: params.isConfigured,
capabilities: {
generate: {
@@ -491,6 +515,7 @@ function createOpenAIImageGenerationProviderBase(params: {
output: {
formats: [...OPENAI_OUTPUT_FORMATS],
qualities: [...OPENAI_QUALITIES],
backgrounds: [...OPENAI_BACKGROUNDS],
},
},
generateImage: params.generateImage,
@@ -517,7 +542,9 @@ function logCodexImageAuthSelected(params: {
authMode?: unknown;
timeoutMs: number;
}) {
const model = params.req.model || DEFAULT_OPENAI_IMAGE_MODEL;
const model = resolveOpenAIImageRequestModel(params.req, {
allowTransparentDefaultReroute: true,
});
log.info(
`image auth selected: provider=openai-codex mode=${sanitizeLogValue(
params.authMode,
@@ -549,11 +576,14 @@ async function generateOpenAICodexImage(params: {
transport: "http",
});
const model = req.model || DEFAULT_OPENAI_IMAGE_MODEL;
const model = resolveOpenAIImageRequestModel(req, {
allowTransparentDefaultReroute: true,
});
const count = resolveOpenAIImageCount(req.count);
const size = req.size ?? DEFAULT_SIZE;
const timeoutMs = resolveOpenAIImageTimeoutMs(req.timeoutMs);
const openai = req.providerOptions?.openai;
const background = openai?.background ?? req.background;
headers.set("Content-Type", "application/json");
const content: Array<Record<string, unknown>> = [
{ type: "input_text", text: req.prompt },
@@ -584,7 +614,7 @@ async function generateOpenAICodexImage(params: {
size,
...(req.quality !== undefined ? { quality: req.quality } : {}),
...(req.outputFormat !== undefined ? { output_format: req.outputFormat } : {}),
...(openai?.background !== undefined ? { background: openai.background } : {}),
...(background !== undefined ? { background } : {}),
...(openai?.outputCompression !== undefined
? { output_compression: openai.outputCompression }
: {}),
@@ -711,7 +741,9 @@ export function buildOpenAIImageGenerationProvider(): ImageGenerationProvider {
transport: "http",
});
const model = req.model || DEFAULT_OPENAI_IMAGE_MODEL;
const model = resolveOpenAIImageRequestModel(req, {
allowTransparentDefaultReroute: publicOpenAIBaseUrl,
});
const count = resolveOpenAIImageCount(req.count);
const size = req.size ?? DEFAULT_SIZE;
const timeoutMs = resolveOpenAIImageTimeoutMs(req.timeoutMs);

View File

@@ -834,7 +834,9 @@ describe("qa mock openai server", () => {
}),
});
expect(memory.status).toBe(200);
expect(await memory.text()).toContain('"name":"memory_search"');
const memoryText = await memory.text();
expect(memoryText).toContain('"name":"memory_search"');
expect(memoryText).toContain('\\"corpus\\":\\"sessions\\"');
const memoryFollowup = await fetch(`${server.baseUrl}/v1/responses`, {
method: "POST",

View File

@@ -1373,6 +1373,7 @@ async function buildResponsesPayload(
return buildToolCallEventsWithArgs("memory_search", {
query: "current Project Nebula codename ORBIT-10",
maxResults: 3,
corpus: "sessions",
});
}
const results = Array.isArray(toolJson?.results)

View File

@@ -686,6 +686,128 @@ describe("skill-workshop", () => {
);
});
it("uses the configured agent default for reviewer fallback", async () => {
const workspaceDir = await makeTempDir();
const stateDir = await makeTempDir();
const runEmbeddedPiAgent = vi.fn(async () => ({
payloads: [{ text: JSON.stringify({ action: "none" }) }],
meta: {},
}));
const api = createTestPluginApi({
config: {
agents: {
defaults: {
model: { primary: "openai-codex/gpt-5.5" },
},
},
},
runtime: {
agent: {
defaults: { provider: "openai", model: "gpt-5.4" },
resolveAgentDir: () => path.join(workspaceDir, ".agent"),
runEmbeddedPiAgent,
},
state: {
resolveStateDir: () => stateDir,
},
} as never,
});
await reviewTranscriptForProposal({
api,
config: {
enabled: true,
autoCapture: true,
approvalPolicy: "pending",
reviewMode: "llm",
reviewInterval: 1,
reviewMinToolCalls: 1,
reviewTimeoutMs: 5_000,
maxPending: 50,
maxSkillBytes: 40_000,
},
ctx: { agentId: "main", workspaceDir },
messages: [{ role: "user", content: "Remember this repeatable fix." }],
});
expect(runEmbeddedPiAgent).toHaveBeenCalledWith(
expect.objectContaining({
provider: "openai-codex",
model: "gpt-5.5",
}),
);
});
it("infers reviewer fallback provider for a bare configured model", async () => {
const workspaceDir = await makeTempDir();
const stateDir = await makeTempDir();
const runEmbeddedPiAgent = vi.fn(async () => ({
payloads: [{ text: JSON.stringify({ action: "none" }) }],
meta: {},
}));
const api = createTestPluginApi({
config: {
agents: {
defaults: {
model: { primary: "gpt-5.5" },
},
},
models: {
providers: {
"openai-codex": {
baseUrl: "https://chatgpt.com/backend-api/codex",
models: [
{
id: "gpt-5.5",
name: "GPT 5.5",
reasoning: true,
input: ["text"],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 200_000,
maxTokens: 128_000,
},
],
},
},
},
},
runtime: {
agent: {
defaults: { provider: "openai", model: "gpt-5.4" },
resolveAgentDir: () => path.join(workspaceDir, ".agent"),
runEmbeddedPiAgent,
},
state: {
resolveStateDir: () => stateDir,
},
} as never,
});
await reviewTranscriptForProposal({
api,
config: {
enabled: true,
autoCapture: true,
approvalPolicy: "pending",
reviewMode: "llm",
reviewInterval: 1,
reviewMinToolCalls: 1,
reviewTimeoutMs: 5_000,
maxPending: 50,
maxSkillBytes: 40_000,
},
ctx: { agentId: "main", workspaceDir },
messages: [{ role: "user", content: "Remember this bare-model default." }],
});
expect(runEmbeddedPiAgent).toHaveBeenCalledWith(
expect.objectContaining({
provider: "openai-codex",
model: "gpt-5.5",
}),
);
});
it("runs reviewer after threshold and queues the proposal", async () => {
const workspaceDir = await makeTempDir();
const stateDir = await makeTempDir();

View File

@@ -1,6 +1,10 @@
import { randomUUID } from "node:crypto";
import fs from "node:fs/promises";
import path from "node:path";
import {
resolveAgentEffectiveModelPrimary,
resolveDefaultModelForAgent,
} from "openclaw/plugin-sdk/agent-runtime";
import type { OpenClawPluginApi } from "../api.js";
import type { SkillWorkshopConfig } from "./config.js";
import { normalizeSkillName } from "./skills.js";
@@ -34,6 +38,22 @@ type ReviewerJson = {
newText?: string;
};
function resolveReviewerFallbackModel(params: { api: OpenClawPluginApi; agentId: string }): {
provider: string;
model: string;
} {
if (resolveAgentEffectiveModelPrimary(params.api.config, params.agentId)) {
return resolveDefaultModelForAgent({
cfg: params.api.config,
agentId: params.agentId,
});
}
return {
provider: params.api.runtime.agent.defaults.provider,
model: params.api.runtime.agent.defaults.model,
};
}
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value);
}
@@ -224,6 +244,10 @@ export async function reviewTranscriptForProposal(params: {
});
const sessionId = `skill-workshop-review-${randomUUID()}`;
const stateDir = params.api.runtime.state.resolveStateDir();
const fallbackModel = resolveReviewerFallbackModel({
api: params.api,
agentId: params.ctx.agentId,
});
const result = await params.api.runtime.agent.runEmbeddedPiAgent({
sessionId,
sessionKey: params.ctx.sessionKey,
@@ -235,8 +259,8 @@ export async function reviewTranscriptForProposal(params: {
agentDir: params.api.runtime.agent.resolveAgentDir(params.api.config, params.ctx.agentId),
config: params.api.config,
prompt,
provider: params.ctx.modelProviderId ?? params.api.runtime.agent.defaults.provider,
model: params.ctx.modelId ?? params.api.runtime.agent.defaults.model,
provider: params.ctx.modelProviderId ?? fallbackModel.provider,
model: params.ctx.modelId ?? fallbackModel.model,
timeoutMs: params.config.reviewTimeoutMs,
runId: sessionId,
trigger: "manual",

View File

@@ -35,4 +35,60 @@ describe("venice provider plugin", () => {
} as never),
).toBeUndefined();
});
it("fills missing DeepSeek V4 reasoning_content on Venice replay turns", async () => {
const provider = await registerSingleProviderPlugin(plugin);
const capturedPayloads: Record<string, unknown>[] = [];
const baseStreamFn = (_model: unknown, _context: unknown, options: unknown) => {
const payload = {
model: "deepseek-v4-pro",
thinking: { type: "enabled" },
reasoning_effort: "high",
messages: [
{
role: "assistant",
tool_calls: [
{
id: "call_1",
type: "function",
function: { name: "read", arguments: "{}" },
},
],
},
],
};
(options as { onPayload?: (payload: Record<string, unknown>) => void })?.onPayload?.(payload);
capturedPayloads.push(payload);
return {} as never;
};
const streamFn = provider.wrapStreamFn?.({
streamFn: baseStreamFn as never,
providerId: "venice",
modelId: "deepseek-v4-pro",
thinkingLevel: "high",
} as never);
expect(streamFn).toBeTypeOf("function");
await streamFn?.({ provider: "venice", id: "deepseek-v4-pro" } as never, {} as never, {});
expect(capturedPayloads).toEqual([
{
model: "deepseek-v4-pro",
messages: [
{
role: "assistant",
tool_calls: [
{
id: "call_1",
type: "function",
function: { name: "read", arguments: "{}" },
},
],
reasoning_content: "",
},
],
},
]);
});
});

View File

@@ -3,6 +3,7 @@ import { applyXaiModelCompat } from "openclaw/plugin-sdk/provider-tools";
import { normalizeLowercaseStringOrEmpty } from "openclaw/plugin-sdk/text-runtime";
import { applyVeniceConfig, VENICE_DEFAULT_MODEL_REF } from "./onboard.js";
import { buildVeniceProvider } from "./provider-catalog.js";
import { createVeniceDeepSeekV4Wrapper } from "./stream.js";
const PROVIDER_ID = "venice";
@@ -44,5 +45,6 @@ export default defineSingleProviderPluginEntry({
},
normalizeResolvedModel: ({ modelId, model }) =>
isXaiBackedVeniceModel(modelId) ? applyXaiModelCompat(model) : undefined,
wrapStreamFn: (ctx) => createVeniceDeepSeekV4Wrapper(ctx.streamFn, ctx.thinkingLevel),
},
});

View File

@@ -0,0 +1,37 @@
import type { ProviderWrapStreamFnContext } from "openclaw/plugin-sdk/plugin-entry";
import { createPayloadPatchStreamWrapper } from "openclaw/plugin-sdk/provider-stream-shared";
function isVeniceDeepSeekV4ModelId(modelId: unknown): boolean {
return modelId === "deepseek-v4-flash" || modelId === "deepseek-v4-pro";
}
function ensureVeniceDeepSeekV4Replay(payload: Record<string, unknown>): void {
delete payload.thinking;
delete payload.reasoning;
delete payload.reasoning_effort;
if (!Array.isArray(payload.messages)) {
return;
}
for (const message of payload.messages) {
if (!message || typeof message !== "object") {
continue;
}
const record = message as Record<string, unknown>;
if (record.role === "assistant" && Array.isArray(record.tool_calls)) {
record.reasoning_content ??= "";
}
}
}
export function createVeniceDeepSeekV4Wrapper(
baseStreamFn: ProviderWrapStreamFnContext["streamFn"],
thinkingLevel: ProviderWrapStreamFnContext["thinkingLevel"],
): ProviderWrapStreamFnContext["streamFn"] {
void thinkingLevel;
return createPayloadPatchStreamWrapper(baseStreamFn, ({ payload, model }) => {
if (model.provider === "venice" && isVeniceDeepSeekV4ModelId(model.id)) {
ensureVeniceDeepSeekV4Replay(payload);
}
});
}

View File

@@ -149,6 +149,7 @@ describe("voice-call plugin", () => {
afterEach(() => {
vi.restoreAllMocks();
vi.unstubAllEnvs();
delete (globalThis as Record<PropertyKey, unknown>)[Symbol.for("openclaw.voice-call.runtime")];
delete (globalThis as Record<PropertyKey, unknown>)[
Symbol.for("openclaw.voice-call.runtimePromise")
@@ -197,6 +198,50 @@ describe("voice-call plugin", () => {
expect(respond).toHaveBeenCalledWith(true, { callId: "call-2", initiated: true });
});
it("does not log a startup error when provider setup is incomplete", async () => {
vi.stubEnv("TWILIO_ACCOUNT_SID", "");
vi.stubEnv("TWILIO_AUTH_TOKEN", "");
vi.stubEnv("TWILIO_FROM_NUMBER", "");
const { service } = setup({ provider: "twilio" });
await service?.start(createServiceContext());
expect(createVoiceCallRuntime).not.toHaveBeenCalled();
expect(
noopLogger.error.mock.calls.some(([message]) =>
String(message).includes("Failed to start runtime"),
),
).toBe(false);
expect(noopLogger.warn).toHaveBeenCalledWith(
expect.stringContaining("Runtime not started; setup incomplete"),
);
expect(noopLogger.warn).toHaveBeenCalledWith(expect.stringContaining("TWILIO_ACCOUNT_SID"));
});
it("still reports missing provider setup when a command needs the runtime", async () => {
vi.stubEnv("TWILIO_ACCOUNT_SID", "");
vi.stubEnv("TWILIO_AUTH_TOKEN", "");
vi.stubEnv("TWILIO_FROM_NUMBER", "");
const { methods } = setup({ provider: "twilio" });
const handler = methods.get("voicecall.initiate") as
| ((ctx: {
params: Record<string, unknown>;
respond: ReturnType<typeof vi.fn>;
}) => Promise<void>)
| undefined;
const respond = vi.fn();
await handler?.({ params: { message: "Hi", to: "+15550001234" }, respond });
expect(createVoiceCallRuntime).not.toHaveBeenCalled();
expect(respond).toHaveBeenCalledWith(
false,
expect.objectContaining({
error: expect.stringContaining("TWILIO_ACCOUNT_SID"),
}),
);
});
it("initiates a call via voicecall.initiate", async () => {
const { methods } = setup({ provider: "mock" });
const handler = methods.get("voicecall.initiate") as

View File

@@ -611,6 +611,12 @@ export default definePluginEntry({
if (!config.enabled) {
return;
}
if (!validation.valid) {
api.logger.warn(
`[voice-call] Runtime not started; setup incomplete: ${validation.errors.join("; ")}`,
);
return;
}
try {
await ensureRuntime();
} catch (err) {

View File

@@ -1,10 +1,23 @@
import fsSync from "node:fs";
import os from "node:os";
import path from "node:path";
import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest";
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
vi.mock("../../../../src/media/mime.js", () => ({
detectMime: async (opts: { filePath?: string }) => {
if (opts.filePath?.endsWith(".png")) {
return "image/png";
}
if (opts.filePath?.endsWith(".wav")) {
return "audio/wav";
}
return undefined;
},
}));
import {
buildMultimodalChunkForIndexing,
buildFileEntry,
buildMultimodalChunkForIndexing,
chunkMarkdown,
isMemoryPath,
listMemoryFiles,
@@ -20,7 +33,7 @@ let sharedTempRoot = "";
let sharedTempId = 0;
beforeAll(() => {
sharedTempRoot = fsSync.mkdtempSync(path.join(os.tmpdir(), "memory-host-sdk-tests-"));
sharedTempRoot = fsSync.mkdtempSync(path.join(os.tmpdir(), "memory-host-sdk-package-tests-"));
});
afterAll(() => {
@@ -38,210 +51,63 @@ function setupTempDirLifecycle(prefix: string): () => string {
return () => tmpDir;
}
describe("normalizeExtraMemoryPaths", () => {
it("trims, resolves, and dedupes paths", () => {
const multimodal: MemoryMultimodalSettings = {
enabled: true,
modalities: ["image", "audio"],
maxFileBytes: DEFAULT_MEMORY_MULTIMODAL_MAX_FILE_BYTES,
};
describe("memory host SDK package internals", () => {
const getTmpDir = setupTempDirLifecycle("memory-package-");
it("normalizes additional memory paths", () => {
const workspaceDir = path.join(os.tmpdir(), "memory-test-workspace");
const absPath = path.resolve(path.sep, "shared-notes");
const result = normalizeExtraMemoryPaths(workspaceDir, [
" notes ",
"./notes",
absPath,
absPath,
"",
]);
expect(result).toEqual([path.resolve(workspaceDir, "notes"), absPath]);
});
});
describe("listMemoryFiles", () => {
const getTmpDir = setupTempDirLifecycle("memory-test-");
const multimodal: MemoryMultimodalSettings = {
enabled: true,
modalities: ["image", "audio"],
maxFileBytes: DEFAULT_MEMORY_MULTIMODAL_MAX_FILE_BYTES,
};
it("includes files from additional paths (directory)", async () => {
const tmpDir = getTmpDir();
fsSync.writeFileSync(path.join(tmpDir, "MEMORY.md"), "# Default memory");
const extraDir = path.join(tmpDir, "extra-notes");
fsSync.mkdirSync(extraDir, { recursive: true });
fsSync.writeFileSync(path.join(extraDir, "note1.md"), "# Note 1");
fsSync.writeFileSync(path.join(extraDir, "note2.md"), "# Note 2");
fsSync.writeFileSync(path.join(extraDir, "ignore.txt"), "Not a markdown file");
const files = await listMemoryFiles(tmpDir, [extraDir]);
expect(files).toHaveLength(3);
expect(files.some((file) => file.endsWith("MEMORY.md"))).toBe(true);
expect(files.some((file) => file.endsWith("note1.md"))).toBe(true);
expect(files.some((file) => file.endsWith("note2.md"))).toBe(true);
expect(files.some((file) => file.endsWith("ignore.txt"))).toBe(false);
expect(
normalizeExtraMemoryPaths(workspaceDir, [" notes ", "./notes", absPath, absPath, ""]),
).toEqual([path.resolve(workspaceDir, "notes"), absPath]);
});
it("includes files from additional paths (single file)", async () => {
const tmpDir = getTmpDir();
fsSync.writeFileSync(path.join(tmpDir, "MEMORY.md"), "# Default memory");
const singleFile = path.join(tmpDir, "standalone.md");
fsSync.writeFileSync(singleFile, "# Standalone");
const files = await listMemoryFiles(tmpDir, [singleFile]);
expect(files).toHaveLength(2);
expect(files.some((file) => file.endsWith("standalone.md"))).toBe(true);
});
it("ignores lowercase root memory.md when canonical MEMORY.md is absent", async () => {
const tmpDir = getTmpDir();
fsSync.writeFileSync(path.join(tmpDir, "memory.md"), "# Legacy memory");
const files = await listMemoryFiles(tmpDir, [path.join(tmpDir, "memory.md")]);
expect(files).toEqual([]);
});
it("prefers MEMORY.md when both root files exist", async () => {
it("lists canonical markdown and enabled multimodal files", async () => {
const tmpDir = getTmpDir();
fsSync.writeFileSync(path.join(tmpDir, "MEMORY.md"), "# Default memory");
fsSync.writeFileSync(path.join(tmpDir, "memory.md"), "# Legacy memory");
const files = await listMemoryFiles(tmpDir, [path.join(tmpDir, "memory.md"), tmpDir]);
expect(files).toEqual([path.join(tmpDir, "MEMORY.md")]);
});
it("skips root-memory repair backups from extra workspace paths", async () => {
const tmpDir = getTmpDir();
fsSync.writeFileSync(path.join(tmpDir, "MEMORY.md"), "# Default memory");
const repairDir = path.join(tmpDir, ".openclaw-repair", "root-memory", "2026-04-23");
fsSync.mkdirSync(repairDir, { recursive: true });
fsSync.writeFileSync(path.join(repairDir, "memory.md"), "# Archived legacy memory");
const files = await listMemoryFiles(tmpDir, [tmpDir]);
expect(files).toHaveLength(1);
expect(files[0]).toBe(path.join(tmpDir, "MEMORY.md"));
});
it("handles relative paths in additional paths", async () => {
const tmpDir = getTmpDir();
fsSync.writeFileSync(path.join(tmpDir, "MEMORY.md"), "# Default memory");
const extraDir = path.join(tmpDir, "subdir");
fsSync.mkdirSync(extraDir, { recursive: true });
fsSync.writeFileSync(path.join(extraDir, "nested.md"), "# Nested");
const files = await listMemoryFiles(tmpDir, ["subdir"]);
expect(files).toHaveLength(2);
expect(files.some((file) => file.endsWith("nested.md"))).toBe(true);
});
it("ignores non-existent additional paths", async () => {
const tmpDir = getTmpDir();
fsSync.writeFileSync(path.join(tmpDir, "MEMORY.md"), "# Default memory");
const files = await listMemoryFiles(tmpDir, ["/does/not/exist"]);
expect(files).toHaveLength(1);
});
it("ignores symlinked files and directories", async () => {
const tmpDir = getTmpDir();
fsSync.writeFileSync(path.join(tmpDir, "MEMORY.md"), "# Default memory");
const extraDir = path.join(tmpDir, "extra");
fsSync.mkdirSync(extraDir, { recursive: true });
fsSync.writeFileSync(path.join(extraDir, "note.md"), "# Note");
const targetFile = path.join(tmpDir, "target.md");
fsSync.writeFileSync(targetFile, "# Target");
const linkFile = path.join(extraDir, "linked.md");
const targetDir = path.join(tmpDir, "target-dir");
fsSync.mkdirSync(targetDir, { recursive: true });
fsSync.writeFileSync(path.join(targetDir, "nested.md"), "# Nested");
const linkDir = path.join(tmpDir, "linked-dir");
let symlinksOk = true;
try {
fsSync.symlinkSync(targetFile, linkFile, "file");
fsSync.symlinkSync(targetDir, linkDir, "dir");
} catch (err) {
const code = (err as NodeJS.ErrnoException).code;
if (code === "EPERM" || code === "EACCES") {
symlinksOk = false;
} else {
throw err;
}
}
const files = await listMemoryFiles(tmpDir, [extraDir, linkDir]);
expect(files.some((file) => file.endsWith("note.md"))).toBe(true);
if (symlinksOk) {
expect(files.some((file) => file.endsWith("linked.md"))).toBe(false);
expect(files.some((file) => file.endsWith("nested.md"))).toBe(false);
}
});
it("dedupes overlapping extra paths that resolve to the same file", async () => {
const tmpDir = getTmpDir();
fsSync.writeFileSync(path.join(tmpDir, "MEMORY.md"), "# Default memory");
const files = await listMemoryFiles(tmpDir, [tmpDir, ".", path.join(tmpDir, "MEMORY.md")]);
const memoryMatches = files.filter((file) => file.endsWith("MEMORY.md"));
expect(memoryMatches).toHaveLength(1);
});
it("includes image and audio files from extra paths when multimodal is enabled", async () => {
const tmpDir = getTmpDir();
const extraDir = path.join(tmpDir, "media");
fsSync.mkdirSync(extraDir, { recursive: true });
fsSync.writeFileSync(path.join(extraDir, "diagram.png"), Buffer.from("png"));
fsSync.writeFileSync(path.join(extraDir, "note.wav"), Buffer.from("wav"));
fsSync.writeFileSync(path.join(extraDir, "ignore.bin"), Buffer.from("bin"));
fsSync.writeFileSync(path.join(extraDir, "ignore.txt"), "ignored");
const files = await listMemoryFiles(tmpDir, [extraDir], multimodal);
expect(files.some((file) => file.endsWith("diagram.png"))).toBe(true);
expect(files.some((file) => file.endsWith("note.wav"))).toBe(true);
expect(files.some((file) => file.endsWith("ignore.bin"))).toBe(false);
const files = await listMemoryFiles(
tmpDir,
[path.join(tmpDir, "memory.md"), extraDir],
multimodal,
);
expect(files.map((file) => path.relative(tmpDir, file)).toSorted()).toEqual([
"MEMORY.md",
path.join("extra", "diagram.png"),
path.join("extra", "note.md"),
]);
});
});
describe("isMemoryPath", () => {
it("allows explicit access to top-level dreams.md", () => {
it("keeps package-specific dreams path casing", () => {
expect(isMemoryPath("dreams.md")).toBe(true);
});
});
describe("buildFileEntry", () => {
const getTmpDir = setupTempDirLifecycle("memory-build-entry-");
const multimodal: MemoryMultimodalSettings = {
enabled: true,
modalities: ["image", "audio"],
maxFileBytes: DEFAULT_MEMORY_MULTIMODAL_MAX_FILE_BYTES,
};
it("returns null when the file disappears before reading", async () => {
const tmpDir = getTmpDir();
const target = path.join(tmpDir, "ghost.md");
fsSync.writeFileSync(target, "ghost", "utf-8");
fsSync.rmSync(target);
const entry = await buildFileEntry(target, tmpDir);
expect(entry).toBeNull();
expect(isMemoryPath("DREAMS.md")).toBe(false);
});
it("returns metadata when the file exists", async () => {
it("builds markdown and multimodal file entries", async () => {
const tmpDir = getTmpDir();
const target = path.join(tmpDir, "note.md");
fsSync.writeFileSync(target, "hello", "utf-8");
const entry = await buildFileEntry(target, tmpDir);
expect(entry).not.toBeNull();
expect(entry?.path).toBe("note.md");
expect(entry?.size).toBeGreaterThan(0);
});
const notePath = path.join(tmpDir, "note.md");
const imagePath = path.join(tmpDir, "diagram.png");
fsSync.writeFileSync(notePath, "hello", "utf-8");
fsSync.writeFileSync(imagePath, Buffer.from("png"));
it("returns multimodal metadata for eligible image files", async () => {
const tmpDir = getTmpDir();
const target = path.join(tmpDir, "diagram.png");
fsSync.writeFileSync(target, Buffer.from("png"));
const note = await buildFileEntry(notePath, tmpDir);
const image = await buildFileEntry(imagePath, tmpDir, multimodal);
const entry = await buildFileEntry(target, tmpDir, multimodal);
expect(entry).toMatchObject({
expect(note).toMatchObject({ path: "note.md", kind: "markdown" });
expect(image).toMatchObject({
path: "diagram.png",
kind: "multimodal",
modality: "image",
@@ -250,228 +116,51 @@ describe("buildFileEntry", () => {
});
});
it("builds a multimodal chunk lazily for indexing", async () => {
it("builds multimodal chunks lazily and rejects changed files", async () => {
const tmpDir = getTmpDir();
const target = path.join(tmpDir, "diagram.png");
fsSync.writeFileSync(target, Buffer.from("png"));
const imagePath = path.join(tmpDir, "diagram.png");
fsSync.writeFileSync(imagePath, Buffer.from("png"));
const entry = await buildFileEntry(target, tmpDir, multimodal);
const entry = await buildFileEntry(imagePath, tmpDir, multimodal);
const built = await buildMultimodalChunkForIndexing(entry!);
expect(built?.chunk.embeddingInput?.parts).toEqual([
{ type: "text", text: "Image file: diagram.png" },
expect.objectContaining({ type: "inline-data", mimeType: "image/png" }),
]);
expect(built?.structuredInputBytes).toBeGreaterThan(0);
fsSync.writeFileSync(imagePath, Buffer.alloc(entry!.size + 32, 1));
await expect(buildMultimodalChunkForIndexing(entry!)).resolves.toBeNull();
});
it("skips lazy multimodal indexing when file state changes after discovery", async () => {
for (const testCase of [
{
name: "grows",
mutate: (target: string, entrySize: number) => {
fsSync.writeFileSync(target, Buffer.alloc(entrySize + 32, 1));
},
},
{
name: "bytes change",
mutate: (target: string) => {
fsSync.writeFileSync(target, Buffer.from("gif"));
},
},
] as const) {
const tmpDir = getTmpDir();
const target = path.join(tmpDir, `${testCase.name}.png`);
fsSync.writeFileSync(target, Buffer.from("png"));
it("chunks mixed text and preserves surrogate pairs", () => {
const mixed = Array.from(
{ length: 30 },
(_, index) => `Line ${index}: 这是中英文混合的测试内容 with English`,
).join("\n");
const mixedChunks = chunkMarkdown(mixed, { tokens: 50, overlap: 0 });
expect(mixedChunks.length).toBeGreaterThan(1);
expect(mixedChunks.map((chunk) => chunk.text).join("\n")).toContain("Line 29");
const entry = await buildFileEntry(target, tmpDir, multimodal);
expect(entry, testCase.name).not.toBeNull();
testCase.mutate(target, entry!.size);
await expect(buildMultimodalChunkForIndexing(entry!), testCase.name).resolves.toBeNull();
}
});
});
describe("chunkMarkdown", () => {
it("splits overly long lines into max-sized chunks", () => {
const chunkTokens = 400;
const maxChars = chunkTokens * 4;
const content = "a".repeat(maxChars * 3 + 25);
const chunks = chunkMarkdown(content, { tokens: chunkTokens, overlap: 0 });
expect(chunks.length).toBeGreaterThan(1);
for (const chunk of chunks) {
expect(chunk.text.length).toBeLessThanOrEqual(maxChars);
}
});
it("produces more chunks for CJK text than for equal-length ASCII text", () => {
// CJK chars ≈ 1 token each; ASCII chars ≈ 0.25 tokens each.
// For the same raw character count, CJK content should produce more chunks
// because each character "weighs" ~4× more in token estimation.
const chunkTokens = 50;
// 400 ASCII chars → ~100 tokens → fits in ~2 chunks
const asciiLines = Array.from({ length: 20 }, () => "a".repeat(20)).join("\n");
const asciiChunks = chunkMarkdown(asciiLines, { tokens: chunkTokens, overlap: 0 });
// 400 CJK chars → ~400 tokens → needs ~8 chunks
const cjkLines = Array.from({ length: 20 }, () => "你".repeat(20)).join("\n");
const cjkChunks = chunkMarkdown(cjkLines, { tokens: chunkTokens, overlap: 0 });
expect(cjkChunks.length).toBeGreaterThan(asciiChunks.length);
});
it("respects token budget for Chinese text", () => {
// With tokens=100, each CJK char ≈ 1 token, so chunks should hold ~100 CJK chars.
const chunkTokens = 100;
const lines: string[] = [];
for (let i = 0; i < 50; i++) {
lines.push("这是一个测试句子用来验证分块逻辑是否正确处理中文文本内容");
}
const content = lines.join("\n");
const chunks = chunkMarkdown(content, { tokens: chunkTokens, overlap: 0 });
expect(chunks.length).toBeGreaterThan(1);
// Each chunk's CJK content should not vastly exceed the token budget.
// With CJK-aware estimation, each char ≈ 1 token, so chunk text length
// (in CJK chars) should be roughly <= tokens budget (with some tolerance
// for line boundaries).
for (const chunk of chunks) {
// Count actual CJK characters in the chunk
const cjkCount = (chunk.text.match(/[\u4e00-\u9fff]/g) ?? []).length;
// Allow 2× tolerance for line-boundary rounding
expect(cjkCount).toBeLessThanOrEqual(chunkTokens * 2);
}
});
it("keeps English chunking behavior unchanged", () => {
const chunkTokens = 100;
const maxChars = chunkTokens * 4; // 400 chars
const content = "hello world this is a test. ".repeat(50);
const chunks = chunkMarkdown(content, { tokens: chunkTokens, overlap: 0 });
expect(chunks.length).toBeGreaterThan(1);
for (const chunk of chunks) {
expect(chunk.text.length).toBeLessThanOrEqual(maxChars);
}
});
it("handles mixed CJK and ASCII content correctly", () => {
const chunkTokens = 50;
const lines: string[] = [];
for (let i = 0; i < 30; i++) {
lines.push(`Line ${i}: 这是中英文混合的测试内容 with some English text`);
}
const content = lines.join("\n");
const chunks = chunkMarkdown(content, { tokens: chunkTokens, overlap: 0 });
// Should produce multiple chunks and not crash
expect(chunks.length).toBeGreaterThan(1);
// Verify all content is preserved
const reconstructed = chunks.map((c) => c.text).join("\n");
// Due to overlap=0, the concatenated chunks should cover all lines
expect(reconstructed).toContain("Line 0");
expect(reconstructed).toContain("Line 29");
});
it("splits very long CJK lines into budget-sized segments", () => {
// A single line of 2000 CJK characters (no newlines).
// With tokens=200, each CJK char ≈ 1 token.
const longCjkLine = "中".repeat(2000);
const chunks = chunkMarkdown(longCjkLine, { tokens: 200, overlap: 0 });
expect(chunks.length).toBeGreaterThanOrEqual(8);
for (const chunk of chunks) {
const cjkCount = (chunk.text.match(/[\u4E00-\u9FFF]/g) ?? []).length;
expect(cjkCount).toBeLessThanOrEqual(200 * 2);
}
});
it("does not break surrogate pairs when splitting long CJK lines", () => {
// "𠀀" (U+20000) is a surrogate pair: 2 UTF-16 code units per character.
// With an odd token budget, the fine-split must not cut inside a pair.
const surrogateChar = "\u{20000}"; // 𠀀
const longLine = surrogateChar.repeat(120);
const chunks = chunkMarkdown(longLine, { tokens: 31, overlap: 0 });
for (const chunk of chunks) {
// No chunk should contain the Unicode replacement character U+FFFD,
// which would indicate a broken surrogate pair.
const surrogateChar = "\u{20000}";
const surrogateChunks = chunkMarkdown(surrogateChar.repeat(120), {
tokens: 31,
overlap: 0,
});
for (const chunk of surrogateChunks) {
expect(chunk.text).not.toContain("\uFFFD");
// Every character in the chunk should be a valid string (no lone surrogates).
for (let i = 0; i < chunk.text.length; i += 1) {
const code = chunk.text.charCodeAt(i);
if (code >= 0xd800 && code <= 0xdbff) {
// High surrogate must be followed by a low surrogate
const next = chunk.text.charCodeAt(i + 1);
expect(next).toBeGreaterThanOrEqual(0xdc00);
expect(next).toBeLessThanOrEqual(0xdfff);
}
}
}
});
it("does not over-split long Latin lines (backward compat)", () => {
// 2000 ASCII chars / 800 maxChars -> about 3 segments, not 10 tiny ones.
const longLatinLine = "a".repeat(2000);
const chunks = chunkMarkdown(longLatinLine, { tokens: 200, overlap: 0 });
expect(chunks.length).toBeLessThanOrEqual(5);
});
});
describe("remapChunkLines", () => {
it("remaps chunk line numbers using a lineMap", () => {
// Simulate 5 content lines that came from JSONL lines [4, 6, 7, 10, 13] (1-indexed)
it("remaps chunk lines using JSONL source line maps", () => {
const lineMap = [4, 6, 7, 10, 13];
// Create chunks from content that has 5 lines
const content = "User: Hello\nAssistant: Hi\nUser: Question\nAssistant: Answer\nUser: Thanks";
const chunks = chunkMarkdown(content, { tokens: 400, overlap: 0 });
expect(chunks.length).toBeGreaterThan(0);
// Before remapping, startLine/endLine reference content line numbers (1-indexed)
expect(chunks[0].startLine).toBe(1);
// Remap
remapChunkLines(chunks, lineMap);
// After remapping, line numbers should reference original JSONL lines
// Content line 1 → JSONL line 4, content line 5 → JSONL line 13
expect(chunks[0].startLine).toBe(4);
const lastChunk = chunks[chunks.length - 1];
expect(lastChunk.endLine).toBe(13);
});
it("preserves original line numbers when lineMap is undefined", () => {
const content = "Line one\nLine two\nLine three";
const chunks = chunkMarkdown(content, { tokens: 400, overlap: 0 });
const originalStart = chunks[0].startLine;
const originalEnd = chunks[chunks.length - 1].endLine;
remapChunkLines(chunks, undefined);
expect(chunks[0].startLine).toBe(originalStart);
expect(chunks[chunks.length - 1].endLine).toBe(originalEnd);
});
it("handles multi-chunk content with correct remapping", () => {
// Use small chunk size to force multiple chunks
// lineMap: 10 content lines from JSONL lines [2, 5, 8, 11, 14, 17, 20, 23, 26, 29]
const lineMap = [2, 5, 8, 11, 14, 17, 20, 23, 26, 29];
const contentLines = lineMap.map((_, i) =>
i % 2 === 0 ? `User: Message ${i}` : `Assistant: Reply ${i}`,
const chunks = chunkMarkdown(
"User: Hello\nAssistant: Hi\nUser: Question\nAssistant: Answer\nUser: Thanks",
{ tokens: 400, overlap: 0 },
);
const content = contentLines.join("\n");
// Use very small chunk size to force splitting
const chunks = chunkMarkdown(content, { tokens: 10, overlap: 0 });
expect(chunks.length).toBeGreaterThan(1);
remapChunkLines(chunks, lineMap);
// First chunk should start at JSONL line 2
expect(chunks[0].startLine).toBe(2);
// Last chunk should end at JSONL line 29
expect(chunks[chunks.length - 1].endLine).toBe(29);
// Each chunk's startLine should be ≤ its endLine
for (const chunk of chunks) {
expect(chunk.startLine).toBeLessThanOrEqual(chunk.endLine);
}
expect(chunks[0].startLine).toBe(4);
expect(chunks[chunks.length - 1].endLine).toBe(13);
});
});

View File

@@ -127,6 +127,18 @@ steps:
- ref: transcriptPath
- expr: "[JSON.stringify({ type: 'session', id: config.transcriptId, timestamp: new Date(now - 120000).toISOString() }), JSON.stringify({ type: 'message', message: { role: 'user', timestamp: new Date(now - 90000).toISOString(), content: [{ type: 'text', text: config.transcriptQuestion }] } }), JSON.stringify({ type: 'message', message: { role: 'assistant', timestamp: new Date(now - 60000).toISOString(), content: [{ type: 'text', text: config.transcriptAnswer }] } })].join('\\n') + '\\n'"
- utf8
- call: readRawQaSessionStore
saveAs: sessionStore
args:
- ref: env
- set: sessionStorePath
value:
expr: "path.join(env.gateway.tempRoot, 'state', 'agents', 'qa', 'sessions', 'sessions.json')"
- call: fs.writeFile
args:
- ref: sessionStorePath
- expr: "JSON.stringify({ ...sessionStore, ['agent:qa:seed-session-memory-ranking']: { sessionId: config.transcriptId, updatedAt: now, sessionFile: transcriptPath, origin: { label: 'QA seeded session memory ranking transcript' } } }, null, 2)"
- utf8
- call: forceMemoryIndex
args:
- env:

View File

@@ -68,6 +68,7 @@ export function createChangedCheckPlan(result, options = {}) {
}
};
const addTypecheck = (name, args) => add(name, args, createSparseTsgoSkipEnv(baseEnv));
const addLint = (name, args) => add(name, args, baseEnv);
add("conflict markers", ["check:no-conflict-markers"]);
@@ -109,7 +110,7 @@ export function createChangedCheckPlan(result, options = {}) {
if (runAll) {
addTypecheck("typecheck all", ["tsgo:all"]);
add("lint", ["lint"]);
addLint("lint", ["lint"]);
add("runtime import cycles", ["check:import-cycles"]);
return {
commands,
@@ -135,16 +136,16 @@ export function createChangedCheckPlan(result, options = {}) {
}
if (lanes.core || lanes.coreTests) {
add("lint core", ["lint:core"]);
addLint("lint core", ["lint:core"]);
}
if (lanes.extensions || lanes.extensionTests) {
add("lint extensions", ["lint:extensions"]);
addLint("lint extensions", ["lint:extensions"]);
}
if (lanes.tooling) {
add("lint scripts", ["lint:scripts"]);
addLint("lint scripts", ["lint:scripts"]);
}
if (lanes.apps) {
add("lint apps", ["lint:apps"]);
addLint("lint apps", ["lint:apps"]);
}
if (lanes.core || lanes.extensions) {

View File

@@ -718,7 +718,17 @@ function Invoke-OpenClawUpdateWithTimeout {
$updateJob = Start-Job -ScriptBlock {
param([string]$Path, [string]$Target)
$output = & $Path update --tag $Target --yes --json *>&1
$previousDisableBundledPlugins = $env:OPENCLAW_DISABLE_BUNDLED_PLUGINS
$env:OPENCLAW_DISABLE_BUNDLED_PLUGINS = '1'
try {
$output = & $Path update --tag $Target --yes --json *>&1
} finally {
if ($null -eq $previousDisableBundledPlugins) {
Remove-Item Env:OPENCLAW_DISABLE_BUNDLED_PLUGINS -ErrorAction SilentlyContinue
} else {
$env:OPENCLAW_DISABLE_BUNDLED_PLUGINS = $previousDisableBundledPlugins
}
}
[pscustomobject]@{
ExitCode = $LASTEXITCODE
Output = ($output | Out-String).Trim()
@@ -1649,7 +1659,11 @@ stop_openclaw_gateway_processes() {
# host can observe new plugin metadata mid-update and abort config validation.
scrub_future_plugin_entries
stop_openclaw_gateway_processes
/opt/homebrew/bin/openclaw update --tag "$update_target" --yes --json
# The baseline updater process may run its post-install doctor through the old
# host while new bundled plugin metadata is already on disk. Keep this
# same-guest update hop focused on core/package migration; post-update smoke
# below starts the fresh gateway with bundled plugins enabled.
OPENCLAW_DISABLE_BUNDLED_PLUGINS=1 /opt/homebrew/bin/openclaw update --tag "$update_target" --yes --json
# Same-guest npm upgrades can leave the old gateway process holding the old
# bundled plugin host version. Stop it before post-update config commands.
stop_openclaw_gateway_processes
@@ -1782,7 +1796,7 @@ stop_openclaw_gateway_processes() {
# the old host can observe new plugin metadata mid-update and abort validation.
scrub_future_plugin_entries
stop_openclaw_gateway_processes
openclaw update --tag "$update_target" --yes --json
OPENCLAW_DISABLE_BUNDLED_PLUGINS=1 openclaw update --tag "$update_target" --yes --json
# The fresh Linux lane starts a manual gateway; stop the old process before
# post-update config validation sees mixed old-host/new-plugin metadata.
stop_openclaw_gateway_processes

View File

@@ -29,8 +29,16 @@ cat > \"\$HOME/.openclaw/extensions/lossless-claw/package.json\" <<'JSON'
JSON
cat > \"\$HOME/.openclaw/openclaw.json\" <<'JSON'
{
\"plugins\": {
\"installs\": {
\"plugins\": {}
}
JSON
mkdir -p \"\$HOME/.openclaw/plugins\"
cat > \"\$HOME/.openclaw/plugins/installs.json\" <<'JSON'
{
\"version\": 1,
\"warning\": \"DO NOT EDIT. This file is generated by OpenClaw plugin install/update/uninstall commands. Use `openclaw plugins install/update/uninstall` instead.\",
\"updatedAtMs\": 1777118400000,
\"records\": {
\"lossless-claw\": {
\"source\": \"npm\",
\"spec\": \"@example/lossless-claw@0.9.0\",
@@ -41,7 +49,6 @@ cat > \"\$HOME/.openclaw/openclaw.json\" <<'JSON'
\"integrity\": \"sha512-same\",
\"shasum\": \"same\"
}
}
}
}
JSON

View File

@@ -74,16 +74,30 @@ const config = fs.existsSync(configPath)
const plugins = (config.plugins ??= {});
const entries = (plugins.entries ??= {});
entries[pluginId] = { ...(entries[pluginId] ?? {}), enabled };
const installs = (plugins.installs ??= {});
installs[pluginId] = {
...(installs[pluginId] ?? {}),
delete plugins.installs;
plugins.allow = Array.from(new Set([...(plugins.allow ?? []), pluginId])).sort();
fs.mkdirSync(path.dirname(configPath), { recursive: true });
fs.writeFileSync(configPath, `${JSON.stringify(config, null, 2)}\n`, "utf8");
const ledgerPath = path.join(process.env.HOME, ".openclaw", "plugins", "installs.json");
const ledger = fs.existsSync(ledgerPath)
? JSON.parse(fs.readFileSync(ledgerPath, "utf8"))
: {
version: 1,
warning:
"DO NOT EDIT. This file is generated by OpenClaw plugin install/update/uninstall commands. Use `openclaw plugins install/update/uninstall` instead.",
records: {},
};
ledger.updatedAtMs = Date.now();
ledger.records ??= {};
ledger.records[pluginId] = {
...(ledger.records[pluginId] ?? {}),
source: "path",
installPath: pluginRoot,
sourcePath: pluginRoot,
};
plugins.allow = Array.from(new Set([...(plugins.allow ?? []), pluginId])).sort();
fs.mkdirSync(path.dirname(configPath), { recursive: true });
fs.writeFileSync(configPath, `${JSON.stringify(config, null, 2)}\n`, "utf8");
fs.mkdirSync(path.dirname(ledgerPath), { recursive: true });
fs.writeFileSync(ledgerPath, `${JSON.stringify(ledger, null, 2)}\n`, "utf8");
NODE
}

View File

@@ -316,6 +316,8 @@ export class AcpSessionManager {
...(input.cwd !== undefined ? { cwd: input.cwd } : {}),
});
const requestedCwd = initialRuntimeOptions.cwd;
const requestedModel = initialRuntimeOptions.model;
const requestedThinking = initialRuntimeOptions.thinking;
this.enforceConcurrentSessionLimit({
cfg: input.cfg,
sessionKey,
@@ -327,6 +329,8 @@ export class AcpSessionManager {
agent,
mode: input.mode,
resumeSessionId: input.resumeSessionId,
...(requestedModel ? { model: requestedModel } : {}),
...(requestedThinking ? { thinking: requestedThinking } : {}),
cwd: requestedCwd,
}),
fallbackCode: "ACP_SESSION_INIT_FAILED",
@@ -1378,6 +1382,8 @@ export class AcpSessionManager {
const mode = params.meta.mode;
const runtimeOptions = resolveRuntimeOptionsFromMeta(params.meta);
const cwd = runtimeOptions.cwd ?? normalizeText(params.meta.cwd);
const model = normalizeText(runtimeOptions.model);
const thinking = normalizeText(runtimeOptions.thinking);
const configuredBackend = (params.meta.backend || params.cfg.acp?.backend || "").trim();
const cached = this.getCachedRuntimeState(params.sessionKey);
if (cached) {
@@ -1434,6 +1440,8 @@ export class AcpSessionManager {
agent,
mode,
...(resumeSessionId ? { resumeSessionId } : {}),
...(model ? { model } : {}),
...(thinking ? { thinking } : {}),
cwd,
}),
fallbackCode: "ACP_SESSION_INIT_FAILED",

View File

@@ -4,7 +4,6 @@ import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vite
import type { OpenClawConfig } from "../../config/config.js";
import type { AcpSessionRuntimeOptions, SessionAcpMeta } from "../../config/sessions/types.js";
import { resetHeartbeatWakeStateForTests } from "../../infra/heartbeat-wake.js";
import { resetSystemEventsForTest } from "../../infra/system-events.js";
import { withTempDir } from "../../test-helpers/temp-dir.js";
import type { AcpRuntime, AcpRuntimeCapabilities } from "../runtime/types.js";
@@ -105,7 +104,15 @@ function createRuntime(): {
setConfigOption: ReturnType<typeof vi.fn>;
} {
const ensureSession = vi.fn(
async (input: { sessionKey: string; agent: string; mode: "persistent" | "oneshot" }) => ({
async (input: {
sessionKey: string;
agent: string;
mode: "persistent" | "oneshot";
model?: string;
thinking?: string;
cwd?: string;
resumeSessionId?: string;
}) => ({
sessionKey: input.sessionKey,
backend: "acpx",
runtimeSessionName: `${input.sessionKey}:${input.mode}:runtime`,
@@ -237,7 +244,6 @@ describe("AcpSessionManager", () => {
} else {
process.env.OPENCLAW_STATE_DIR = ORIGINAL_STATE_DIR;
}
resetSystemEventsForTest();
resetHeartbeatWakeStateForTests();
resetTaskRegistryForTests({ persist: false });
resetTaskFlowRegistryForTests({ persist: false });
@@ -1066,6 +1072,82 @@ describe("AcpSessionManager", () => {
);
});
it("passes persisted model runtime options into ensureSession after restart", async () => {
const runtimeState = createRuntime();
hoisted.requireAcpRuntimeBackendMock.mockReturnValue({
id: "acpx",
runtime: runtimeState.runtime,
});
const sessionKey = "agent:codex:acp:binding:demo-binding:default:model-restart";
hoisted.readAcpSessionEntryMock.mockImplementation((paramsUnknown: unknown) => {
const key = (paramsUnknown as { sessionKey?: string }).sessionKey ?? sessionKey;
return {
sessionKey: key,
storeSessionKey: key,
acp: {
...readySessionMeta(),
runtimeOptions: {
model: "openai-codex/gpt-5.4",
},
},
};
});
const manager = new AcpSessionManager();
await manager.runTurn({
cfg: baseCfg,
sessionKey,
text: "after restart",
mode: "prompt",
requestId: "r-binding-restart-model",
});
expect(runtimeState.ensureSession).toHaveBeenCalledWith(
expect.objectContaining({
sessionKey,
model: "openai-codex/gpt-5.4",
}),
);
});
it("passes persisted thinking runtime options into ensureSession after restart", async () => {
const runtimeState = createRuntime();
hoisted.requireAcpRuntimeBackendMock.mockReturnValue({
id: "acpx",
runtime: runtimeState.runtime,
});
const sessionKey = "agent:codex:acp:binding:demo-binding:default:thinking-restart";
hoisted.readAcpSessionEntryMock.mockImplementation((paramsUnknown: unknown) => {
const key = (paramsUnknown as { sessionKey?: string }).sessionKey ?? sessionKey;
return {
sessionKey: key,
storeSessionKey: key,
acp: {
...readySessionMeta(),
runtimeOptions: {
thinking: "high",
},
},
};
});
const manager = new AcpSessionManager();
await manager.runTurn({
cfg: baseCfg,
sessionKey,
text: "after restart",
mode: "prompt",
requestId: "r-binding-restart-thinking",
});
expect(runtimeState.ensureSession).toHaveBeenCalledWith(
expect.objectContaining({
sessionKey,
thinking: "high",
}),
);
});
it("does not resume persisted ACP identity for oneshot sessions after restart", async () => {
const runtimeState = createRuntime();
hoisted.requireAcpRuntimeBackendMock.mockReturnValue({
@@ -1310,6 +1392,7 @@ describe("AcpSessionManager", () => {
acp: readySessionMeta({
runtimeOptions: {
model: "openai-codex/gpt-5.4",
thinking: "high",
},
}),
});
@@ -1322,12 +1405,21 @@ describe("AcpSessionManager", () => {
mode: "persistent",
runtimeOptions: {
model: "openai-codex/gpt-5.4",
thinking: "high",
},
});
expect(extractRuntimeOptionsFromUpserts()).toContainEqual({
model: "openai-codex/gpt-5.4",
thinking: "high",
});
expect(runtimeState.ensureSession).toHaveBeenCalledWith(
expect.objectContaining({
sessionKey: "agent:codex:acp:session-a",
model: "openai-codex/gpt-5.4",
thinking: "high",
}),
);
});
it("preserves runtimeOptions cwd when initializeSession cwd is omitted", async () => {
@@ -2603,6 +2695,7 @@ describe("AcpSessionManager", () => {
runtimeOptions: {
runtimeMode: "plan",
model: "openai-codex/gpt-5.4",
thinking: "high",
permissionProfile: "strict",
timeoutSeconds: 120,
},
@@ -2629,6 +2722,12 @@ describe("AcpSessionManager", () => {
value: "openai-codex/gpt-5.4",
}),
);
expect(runtimeState.setConfigOption).toHaveBeenCalledWith(
expect.objectContaining({
key: "thinking",
value: "high",
}),
);
expect(runtimeState.setConfigOption).toHaveBeenCalledWith(
expect.objectContaining({
key: "approval_policy",

View File

@@ -8,6 +8,7 @@ export { normalizeText } from "../normalize-text.js";
const MAX_RUNTIME_MODE_LENGTH = 64;
const MAX_MODEL_LENGTH = 200;
const MAX_THINKING_LENGTH = 32;
const MAX_PERMISSION_PROFILE_LENGTH = 80;
const MAX_CWD_LENGTH = 4096;
const MIN_TIMEOUT_SECONDS = 1;
@@ -81,6 +82,14 @@ export function validateRuntimeModelInput(rawModel: unknown): string {
});
}
export function validateRuntimeThinkingInput(rawThinking: unknown): string {
return validateBoundedText({
value: rawThinking,
field: "Thinking level",
maxLength: MAX_THINKING_LENGTH,
});
}
export function validateRuntimePermissionProfileInput(rawProfile: unknown): string {
return validateBoundedText({
value: rawProfile,
@@ -145,6 +154,7 @@ export function validateRuntimeOptionPatch(
const allowedKeys = new Set([
"runtimeMode",
"model",
"thinking",
"cwd",
"permissionProfile",
"timeoutSeconds",
@@ -171,6 +181,13 @@ export function validateRuntimeOptionPatch(
next.model = validateRuntimeModelInput(rawPatch.model);
}
}
if (Object.hasOwn(rawPatch, "thinking")) {
if (rawPatch.thinking === undefined) {
next.thinking = undefined;
} else {
next.thinking = validateRuntimeThinkingInput(rawPatch.thinking);
}
}
if (Object.hasOwn(rawPatch, "cwd")) {
if (rawPatch.cwd === undefined) {
next.cwd = undefined;
@@ -220,6 +237,7 @@ export function normalizeRuntimeOptions(
): AcpSessionRuntimeOptions {
const runtimeMode = normalizeText(options?.runtimeMode);
const model = normalizeText(options?.model);
const thinking = normalizeText(options?.thinking);
const cwd = normalizeText(options?.cwd);
const permissionProfile = normalizeText(options?.permissionProfile);
let timeoutSeconds: number | undefined;
@@ -237,6 +255,7 @@ export function normalizeRuntimeOptions(
return {
...(runtimeMode ? { runtimeMode } : {}),
...(model ? { model } : {}),
...(thinking ? { thinking } : {}),
...(cwd ? { cwd } : {}),
...(permissionProfile ? { permissionProfile } : {}),
...(typeof timeoutSeconds === "number" ? { timeoutSeconds } : {}),
@@ -287,6 +306,7 @@ export function buildRuntimeControlSignature(options: AcpSessionRuntimeOptions):
return JSON.stringify({
runtimeMode: normalized.runtimeMode ?? null,
model: normalized.model ?? null,
thinking: normalized.thinking ?? null,
permissionProfile: normalized.permissionProfile ?? null,
timeoutSeconds: normalized.timeoutSeconds ?? null,
backendExtras: extras,
@@ -301,6 +321,9 @@ export function buildRuntimeConfigOptionPairs(
if (normalized.model) {
pairs.set("model", normalized.model);
}
if (normalized.thinking) {
pairs.set("thinking", normalized.thinking);
}
if (normalized.permissionProfile) {
pairs.set("approval_policy", normalized.permissionProfile);
}
@@ -324,6 +347,13 @@ export function inferRuntimeOptionPatchFromConfigOption(
if (normalizedKey === "model") {
return { model: validateRuntimeModelInput(validated.value) };
}
if (
normalizedKey === "thinking" ||
normalizedKey === "thought_level" ||
normalizedKey === "reasoning_effort"
) {
return { thinking: validateRuntimeThinkingInput(validated.value) };
}
if (
normalizedKey === "approval_policy" ||
normalizedKey === "permission_profile" ||

View File

@@ -36,6 +36,10 @@ export type AcpRuntimeEnsureInput = {
agent: string;
mode: AcpRuntimeSessionMode;
resumeSessionId?: string;
/** Optional runtime model override that must be available during session creation. */
model?: string;
/** Optional runtime thinking/reasoning override that must be available during session creation. */
thinking?: string;
cwd?: string;
env?: Record<string, string>;
};

View File

@@ -26,7 +26,6 @@ import type {
ToolCallLocation,
ToolKind,
} from "@agentclientprotocol/sdk";
import { PROTOCOL_VERSION } from "@agentclientprotocol/sdk";
import { listThinkingLevels } from "../auto-reply/thinking.js";
import type { GatewayClient } from "../gateway/client.js";
import type { EventFrame } from "../gateway/protocol/index.js";
@@ -37,7 +36,6 @@ import {
} from "../infra/fixed-window-rate-limit.js";
import { normalizeOptionalString } from "../shared/string-coerce.js";
import { shortenHomePath } from "../utils.js";
import { getAvailableCommands } from "./commands.js";
import {
extractAttachmentsFromPrompt,
extractToolCallContent,
@@ -63,6 +61,21 @@ const ACP_ELEVATED_LEVEL_CONFIG_ID = "elevated_level";
const ACP_LOAD_SESSION_REPLAY_LIMIT = 1_000_000;
const ACP_GATEWAY_DISCONNECT_GRACE_MS = 5_000;
let acpCommandsModulePromise: Promise<typeof import("./commands.js")> | undefined;
let acpSdkModulePromise: Promise<typeof import("@agentclientprotocol/sdk")> | undefined;
async function getAvailableCommandsForAcp() {
acpCommandsModulePromise ??= import("./commands.js");
const { getAvailableCommands } = await acpCommandsModulePromise;
return getAvailableCommands();
}
async function getAcpProtocolVersion() {
acpSdkModulePromise ??= import("@agentclientprotocol/sdk");
const { PROTOCOL_VERSION } = await acpSdkModulePromise;
return PROTOCOL_VERSION;
}
type DisconnectContext = {
generation: number;
reason: string;
@@ -502,7 +515,7 @@ export class AcpGatewayAgent implements Agent {
async initialize(_params: InitializeRequest): Promise<InitializeResponse> {
return {
protocolVersion: PROTOCOL_VERSION,
protocolVersion: await getAcpProtocolVersion(),
agentCapabilities: {
loadSession: true,
promptCapabilities: {
@@ -1212,7 +1225,7 @@ export class AcpGatewayAgent implements Agent {
sessionId,
update: {
sessionUpdate: "available_commands_update",
availableCommands: getAvailableCommands(),
availableCommands: await getAvailableCommandsForAcp(),
},
});
}

View File

@@ -716,12 +716,13 @@ describe("spawnAcpDirect", () => {
expect(transcriptCalls[1]?.threadId).toBe("child-thread");
});
it("passes model override into ACP session initialization", async () => {
it("passes model and thinking overrides into ACP session initialization", async () => {
const result = await spawnAcpDirect(
{
task: "Investigate flaky tests",
agentId: "codex",
model: "openai-codex/gpt-5.4",
thinking: "high",
},
{
agentSessionKey: "agent:main:main",
@@ -735,6 +736,7 @@ describe("spawnAcpDirect", () => {
agent: "codex",
runtimeOptions: {
model: "openai-codex/gpt-5.4",
thinking: "high",
},
}),
);

View File

@@ -98,6 +98,7 @@ export type SpawnAcpParams = {
agentId?: string;
resumeSessionId?: string;
model?: string;
thinking?: string;
cwd?: string;
mode?: SpawnAcpMode;
thread?: boolean;
@@ -826,6 +827,7 @@ async function initializeAcpSpawnRuntime(params: {
runtimeMode: AcpRuntimeSessionMode;
resumeSessionId?: string;
model?: string;
thinking?: string;
cwd?: string;
}): Promise<AcpSpawnInitializedRuntime> {
const storePath = resolveStorePath(params.cfg.session?.store, { agentId: params.targetAgentId });
@@ -850,7 +852,13 @@ async function initializeAcpSpawnRuntime(params: {
agent: params.targetAgentId,
mode: params.runtimeMode,
resumeSessionId: params.resumeSessionId,
runtimeOptions: params.model ? { model: params.model } : undefined,
runtimeOptions:
params.model || params.thinking
? {
...(params.model ? { model: params.model } : {}),
...(params.thinking ? { thinking: params.thinking } : {}),
}
: undefined,
cwd: params.cwd,
backendId: params.cfg.acp?.backend,
});
@@ -1191,6 +1199,7 @@ export async function spawnAcpDirect(
runtimeMode,
resumeSessionId: params.resumeSessionId,
model: params.model,
thinking: params.thinking,
cwd: runtimeCwd,
});
initializedRuntime = initializedSession.runtimeCloseHandle;

View File

@@ -1,8 +1,45 @@
import { describe, expect, it } from "vitest";
import type { ResponseObject } from "./openai-ws-connection.js";
import { buildAssistantMessageFromResponse } from "./openai-ws-message-conversion.js";
import { buildAssistantMessageFromResponse, convertTools } from "./openai-ws-message-conversion.js";
describe("openai ws message conversion", () => {
it("preserves image_generate transparent-background guidance in OpenAI tool payloads", () => {
const [tool] = convertTools([
{
name: "image_generate",
description:
'Generate images. For transparent OpenAI backgrounds, use outputFormat="png" or "webp" and openai.background="transparent"; OpenClaw routes the default OpenAI image model to gpt-image-1.5 for that mode.',
parameters: {
type: "object",
properties: {
model: {
type: "string",
description:
"Optional provider/model override; use openai/gpt-image-1.5 for transparent OpenAI backgrounds.",
},
outputFormat: { type: "string", enum: ["png", "jpeg", "webp"] },
openai: {
type: "object",
properties: {
background: {
type: "string",
enum: ["transparent", "opaque", "auto"],
description:
"For transparent output use outputFormat png or webp; OpenClaw routes the default OpenAI image model to gpt-image-1.5 for this mode.",
},
},
},
},
},
},
]);
expect(tool?.description).toContain('openai.background="transparent"');
expect(tool?.description).toContain("gpt-image-1.5");
expect(JSON.stringify(tool?.parameters)).toContain("openai/gpt-image-1.5");
expect(JSON.stringify(tool?.parameters)).toContain("transparent");
});
it("preserves cached token usage from responses usage details", () => {
const response: ResponseObject = {
id: "resp_123",

View File

@@ -7,6 +7,7 @@ import {
} from "./live-cache-test-support.js";
import { isLiveTestEnabled } from "./live-test-helpers.js";
import { wrapStreamFnSanitizeMalformedToolCalls } from "./pi-embedded-runner/run/attempt.tool-call-normalization.js";
import { OMITTED_ASSISTANT_REASONING_TEXT } from "./pi-embedded-runner/thinking.js";
import { buildAssistantMessageWithZeroUsage } from "./stream-message-shared.js";
const ANTHROPIC_LIVE = isLiveTestEnabled(["ANTHROPIC_LIVE_TEST"]);
@@ -33,7 +34,7 @@ function buildLiveAnthropicModel(): {
name: modelId,
api: "anthropic-messages" as const,
provider: "anthropic",
baseUrl: "https://api.anthropic.com/v1",
baseUrl: "https://api.anthropic.com",
reasoning: true,
input: ["text"] as const,
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
@@ -44,6 +45,94 @@ function buildLiveAnthropicModel(): {
}
describeLive("pi embedded anthropic replay sanitization (live)", () => {
it(
"accepts regular text-only assistant replay history",
async () => {
const { apiKey, model } = buildLiveAnthropicModel();
const messages: Message[] = [
{
role: "user",
content: "Remember the marker REGULAR_ANTHROPIC_REPLAY_OK.",
timestamp: Date.now(),
},
buildAssistantMessageWithZeroUsage({
model: { api: model.api, provider: model.provider, id: model.id },
content: [{ type: "text", text: "I remember REGULAR_ANTHROPIC_REPLAY_OK." }],
stopReason: "stop",
}),
{
role: "user",
content: "Reply with a short confirmation if this replay history is valid.",
timestamp: Date.now(),
},
];
logLiveCache(`anthropic regular replay live model=${model.provider}/${model.id}`);
const response = await completeSimpleWithLiveTimeout(
model,
{ messages },
{
apiKey,
cacheRetention: "none",
sessionId: "anthropic-regular-replay-live",
maxTokens: 64,
temperature: 0,
},
"anthropic regular text replay live synthetic transcript",
ANTHROPIC_TIMEOUT_MS,
);
const text = extractAssistantText(response);
logLiveCache(`anthropic regular replay live result=${JSON.stringify(text)}`);
expect(text.trim().length).toBeGreaterThan(0);
},
6 * 60_000,
);
it(
"accepts omitted-reasoning placeholder assistant replay history",
async () => {
const { apiKey, model } = buildLiveAnthropicModel();
const messages: Message[] = [
{
role: "user",
content: "Remember that the previous assistant reasoning was omitted.",
timestamp: Date.now(),
},
buildAssistantMessageWithZeroUsage({
model: { api: model.api, provider: model.provider, id: model.id },
content: [{ type: "text", text: OMITTED_ASSISTANT_REASONING_TEXT }],
stopReason: "stop",
}),
{
role: "user",
content: "Reply with exactly OK if this placeholder replay history is valid.",
timestamp: Date.now(),
},
];
logLiveCache(`anthropic omitted-reasoning replay live model=${model.provider}/${model.id}`);
const response = await completeSimpleWithLiveTimeout(
model,
{ messages },
{
apiKey,
cacheRetention: "none",
sessionId: "anthropic-omitted-reasoning-replay-live",
maxTokens: 64,
temperature: 0,
},
"anthropic omitted reasoning replay live synthetic transcript",
ANTHROPIC_TIMEOUT_MS,
);
const text = extractAssistantText(response);
logLiveCache(`anthropic omitted-reasoning replay live result=${JSON.stringify(text)}`);
expect(text.trim().length).toBeGreaterThan(0);
},
6 * 60_000,
);
it(
"preserves toolCall replay history that Anthropic accepts end-to-end",
async () => {

View File

@@ -16,6 +16,7 @@ import {
TEST_SESSION_ID,
} from "./pi-embedded-runner.sanitize-session-history.test-harness.js";
import { validateReplayTurns } from "./pi-embedded-runner/replay-history.js";
import { OMITTED_ASSISTANT_REASONING_TEXT } from "./pi-embedded-runner/thinking.js";
import { castAgentMessage, castAgentMessages } from "./test-helpers/agent-message-fixtures.js";
import { extractToolCallsFromAssistant } from "./tool-call-id.js";
import type { TranscriptPolicy } from "./transcript-policy.js";
@@ -1176,6 +1177,163 @@ describe("sanitizeSessionHistory", () => {
]);
});
it("keeps regular latest Anthropic thinking replay while preserving older stripped turns", async () => {
setNonGoogleModelApi();
const messages = castAgentMessages([
makeUserMessage("first"),
makeAssistantMessage([
{
type: "thinking",
thinking: "old private reasoning",
thinkingSignature: "sig_old",
},
]),
makeUserMessage("second"),
makeAssistantMessage([
{
type: "thinking",
thinking: "latest private reasoning",
thinkingSignature: "sig_latest",
},
{ type: "text", text: "latest visible answer" },
]),
]);
const result = await sanitizeAnthropicHistory({
messages,
modelId: "claude-3-7-sonnet-20250219",
});
expect((result[1] as Extract<AgentMessage, { role: "assistant" }>).content).toEqual([
{ type: "text", text: OMITTED_ASSISTANT_REASONING_TEXT },
]);
expect((result[3] as Extract<AgentMessage, { role: "assistant" }>).content).toEqual([
{
type: "thinking",
thinking: "latest private reasoning",
thinkingSignature: "sig_latest",
},
{ type: "text", text: "latest visible answer" },
]);
});
it.each([
{
provider: "anthropic",
modelApi: "anthropic-messages",
label: "anthropic",
},
{
provider: "amazon-bedrock",
modelApi: "bedrock-converse-stream",
label: "bedrock",
},
])(
"preserves older stripped thinking-only assistant turns for $label replay",
async ({ provider, modelApi }) => {
setNonGoogleModelApi();
const messages = castAgentMessages([
makeUserMessage("first"),
makeAssistantMessage([
{
type: "thinking",
thinking: "old private reasoning",
thinkingSignature: "sig_old",
},
]),
makeUserMessage("second"),
makeAssistantMessage([{ type: "text", text: "latest visible answer" }]),
]);
const result = await sanitizeAnthropicHistory({
provider,
modelApi,
messages,
modelId: "claude-3-7-sonnet-20250219",
});
expect((result[1] as Extract<AgentMessage, { role: "assistant" }>).content).toEqual([
{ type: "text", text: OMITTED_ASSISTANT_REASONING_TEXT },
]);
expect((result[3] as Extract<AgentMessage, { role: "assistant" }>).content).toEqual([
{ type: "text", text: "latest visible answer" },
]);
},
);
it.each([
{
provider: "anthropic",
modelApi: "anthropic-messages",
label: "anthropic",
},
{
provider: "amazon-bedrock",
modelApi: "bedrock-converse-stream",
label: "bedrock",
},
])("strips invalid thinking signatures before $label replay", async ({ provider, modelApi }) => {
setNonGoogleModelApi();
const messages = castAgentMessages([
makeUserMessage("first"),
makeAssistantMessage([
{ type: "thinking", thinking: "missing signature" },
{ type: "thinking", thinking: "blank signature", thinkingSignature: " " },
{ type: "thinking", thinking: "signed", thinkingSignature: "sig_latest" },
{ type: "text", text: "latest visible answer" },
]),
]);
const result = await sanitizeAnthropicHistory({
provider,
modelApi,
messages,
modelId: "claude-sonnet-4-6",
});
expect((result[1] as Extract<AgentMessage, { role: "assistant" }>).content).toEqual([
{ type: "thinking", thinking: "signed", thinkingSignature: "sig_latest" },
{ type: "text", text: "latest visible answer" },
]);
});
it.each([
{
provider: "anthropic",
modelApi: "anthropic-messages",
label: "anthropic",
},
{
provider: "amazon-bedrock",
modelApi: "bedrock-converse-stream",
label: "bedrock",
},
])(
"uses non-empty omitted-reasoning fallback when all $label thinking signatures are invalid",
async ({ provider, modelApi }) => {
setNonGoogleModelApi();
const messages = castAgentMessages([
makeUserMessage("first"),
makeAssistantMessage([{ type: "thinking", thinking: "blank", thinkingSignature: "" }]),
]);
const result = await sanitizeAnthropicHistory({
provider,
modelApi,
messages,
modelId: "claude-sonnet-4-6",
});
expect((result[1] as Extract<AgentMessage, { role: "assistant" }>).content).toEqual([
{ type: "text", text: OMITTED_ASSISTANT_REASONING_TEXT },
]);
},
);
it("uses immutable thinking replay for anthropic-compatible providers when policy preserves signatures", async () => {
setNonGoogleModelApi();

View File

@@ -41,7 +41,7 @@ import {
type AssistantUsageSnapshot,
type UsageLike,
} from "../usage.js";
import { dropThinkingBlocks } from "./thinking.js";
import { dropThinkingBlocks, stripInvalidThinkingSignatures } from "./thinking.js";
const INTER_SESSION_PREFIX_BASE = "[Inter-session message]";
const MODEL_SNAPSHOT_CUSTOM_TYPE = "model-snapshot";
@@ -544,9 +544,12 @@ export async function sanitizeSessionHistory(params: {
...resolveImageSanitizationLimits(params.config),
},
);
const droppedThinking = policy.dropThinkingBlocks
? dropThinkingBlocks(sanitizedImages)
const validatedThinkingSignatures = policy.preserveSignatures
? stripInvalidThinkingSignatures(sanitizedImages)
: sanitizedImages;
const droppedThinking = policy.dropThinkingBlocks
? dropThinkingBlocks(validatedThinkingSignatures)
: validatedThinkingSignatures;
const sanitizedToolCalls = sanitizeToolCallInputs(droppedThinking, {
allowedToolNames: params.allowedToolNames,
allowProviderOwnedThinkingReplay,

View File

@@ -3,10 +3,12 @@ import { createAssistantMessageEventStream } from "@mariozechner/pi-ai";
import { describe, expect, it } from "vitest";
import { castAgentMessage, castAgentMessages } from "../test-helpers/agent-message-fixtures.js";
import {
OMITTED_ASSISTANT_REASONING_TEXT,
assessLastAssistantMessage,
dropThinkingBlocks,
isAssistantMessageWithContent,
sanitizeThinkingForRecovery,
stripInvalidThinkingSignatures,
wrapAnthropicStreamWithRecovery,
} from "./thinking.js";
@@ -103,6 +105,135 @@ describe("dropThinkingBlocks", () => {
{ type: "text", text: "latest text" },
]);
});
it("uses non-empty omitted-reasoning text when an older assistant turn is thinking-only", () => {
const messages: AgentMessage[] = [
castAgentMessage({ role: "user", content: "first" }),
castAgentMessage({
role: "assistant",
content: [{ type: "thinking", thinking: "old", thinkingSignature: "sig_old" }],
}),
castAgentMessage({ role: "user", content: "second" }),
castAgentMessage({
role: "assistant",
content: [
{ type: "thinking", thinking: "latest", thinkingSignature: "sig_latest" },
{ type: "text", text: "latest text" },
],
}),
];
const result = dropThinkingBlocks(messages);
const oldAssistant = result[1] as Extract<AgentMessage, { role: "assistant" }>;
const latestAssistant = result[3] as Extract<AgentMessage, { role: "assistant" }>;
const originalLatestAssistant = messages[3] as Extract<AgentMessage, { role: "assistant" }>;
expect(oldAssistant.content).toEqual([
{ type: "text", text: OMITTED_ASSISTANT_REASONING_TEXT },
]);
expect(latestAssistant.content).toEqual(originalLatestAssistant.content);
});
it("uses non-empty omitted-reasoning text when an older assistant turn is redacted-thinking-only", () => {
const messages: AgentMessage[] = [
castAgentMessage({ role: "user", content: "first" }),
castAgentMessage({
role: "assistant",
content: [{ type: "redacted_thinking", data: "opaque" }],
}),
castAgentMessage({ role: "user", content: "second" }),
castAgentMessage({
role: "assistant",
content: [{ type: "text", text: "latest text" }],
}),
];
const result = dropThinkingBlocks(messages);
const oldAssistant = result[1] as Extract<AgentMessage, { role: "assistant" }>;
expect(oldAssistant.content).toEqual([
{ type: "text", text: OMITTED_ASSISTANT_REASONING_TEXT },
]);
});
});
describe("stripInvalidThinkingSignatures", () => {
it("returns the original reference when no invalid thinking signatures are present", () => {
const messages: AgentMessage[] = [
castAgentMessage({ role: "user", content: "hello" }),
castAgentMessage({
role: "assistant",
content: [
{ type: "thinking", thinking: "internal", thinkingSignature: "sig" },
{ type: "text", text: "answer" },
],
}),
];
const result = stripInvalidThinkingSignatures(messages);
expect(result).toBe(messages);
});
it("strips thinking blocks with missing, empty, or blank signatures", () => {
const messages: AgentMessage[] = [
castAgentMessage({
role: "assistant",
content: [
{ type: "thinking", thinking: "missing" },
{ type: "thinking", thinking: "empty", thinkingSignature: "" },
{ type: "thinking", thinking: "blank", thinkingSignature: " " },
{ type: "thinking", thinking: "signed", thinkingSignature: "sig" },
{ type: "text", text: "answer" },
],
}),
];
const result = stripInvalidThinkingSignatures(messages);
const assistant = result[0] as Extract<AgentMessage, { role: "assistant" }>;
expect(result).not.toBe(messages);
expect(assistant.content).toEqual([
{ type: "thinking", thinking: "signed", thinkingSignature: "sig" },
{ type: "text", text: "answer" },
]);
});
it("uses non-empty omitted-reasoning text when all thinking signatures are invalid", () => {
const messages: AgentMessage[] = [
castAgentMessage({
role: "assistant",
content: [{ type: "thinking", thinking: "reasoning", thinkingSignature: "" }],
}),
];
const result = stripInvalidThinkingSignatures(messages);
const assistant = result[0] as Extract<AgentMessage, { role: "assistant" }>;
expect(assistant.content).toEqual([{ type: "text", text: OMITTED_ASSISTANT_REASONING_TEXT }]);
});
it("strips redacted thinking blocks with invalid opaque signatures", () => {
const messages: AgentMessage[] = [
castAgentMessage({
role: "assistant",
content: [
{ type: "redacted_thinking", data: "" },
{ type: "redacted_thinking", signature: " " },
{ type: "redacted_thinking", data: "opaque" },
{ type: "text", text: "answer" },
],
}),
];
const result = stripInvalidThinkingSignatures(messages);
const assistant = result[0] as Extract<AgentMessage, { role: "assistant" }>;
expect(assistant.content).toEqual([
{ type: "redacted_thinking", data: "opaque" },
{ type: "text", text: "answer" },
]);
});
});
describe("sanitizeThinkingForRecovery", () => {
@@ -191,11 +322,13 @@ describe("wrapAnthropicStreamWithRecovery", () => {
"thinking or redacted_thinking blocks in the latest assistant message cannot be modified",
);
it("retries once when the request is rejected before streaming", async () => {
it("retries once with omitted-reasoning text when the request is rejected before streaming", async () => {
let callCount = 0;
const contexts: Array<{ messages?: AgentMessage[] }> = [];
const wrapped = wrapAnthropicStreamWithRecovery(
(() => {
((_model, context) => {
callCount += 1;
contexts.push(context as { messages?: AgentMessage[] });
return Promise.reject(anthropicThinkingError);
}) as Parameters<typeof wrapAnthropicStreamWithRecovery>[0],
{ id: "test-session" },
@@ -216,6 +349,44 @@ describe("wrapAnthropicStreamWithRecovery", () => {
),
).rejects.toBe(anthropicThinkingError);
expect(callCount).toBe(2);
expect(contexts[1]?.messages?.[0]).toMatchObject({
role: "assistant",
content: [{ type: "text", text: OMITTED_ASSISTANT_REASONING_TEXT }],
});
});
it("retries with visible assistant text when stripping thinking leaves content", async () => {
const contexts: Array<{ messages?: AgentMessage[] }> = [];
const wrapped = wrapAnthropicStreamWithRecovery(
((_model, context) => {
contexts.push(context as { messages?: AgentMessage[] });
return Promise.reject(anthropicThinkingError);
}) as Parameters<typeof wrapAnthropicStreamWithRecovery>[0],
{ id: "test-session" },
);
await expect(
wrapped(
{} as never,
{
messages: castAgentMessages([
{
role: "assistant",
content: [
{ type: "thinking", thinking: "secret", thinkingSignature: "sig" },
{ type: "text", text: "visible answer" },
],
},
]),
} as never,
{} as never,
),
).rejects.toBe(anthropicThinkingError);
expect(contexts[1]?.messages?.[0]).toMatchObject({
role: "assistant",
content: [{ type: "text", text: "visible answer" }],
});
});
it("does not retry when the stream fails after yielding a chunk", async () => {

View File

@@ -9,6 +9,7 @@ type RecoveryAssessment = "valid" | "incomplete-thinking" | "incomplete-text";
type RecoverySessionMeta = { id: string; recoveredAnthropicThinking?: boolean };
const THINKING_BLOCK_ERROR_PATTERN = /thinking or redacted_thinking blocks?.* cannot be modified/i;
export const OMITTED_ASSISTANT_REASONING_TEXT = "[assistant reasoning omitted]";
export function isAssistantMessageWithContent(message: AgentMessage): message is AssistantMessage {
return (
@@ -55,6 +56,72 @@ function hasMeaningfulText(block: AssistantContentBlock): boolean {
: false;
}
function buildOmittedAssistantReasoningContent(): AssistantContentBlock[] {
// Provider converters drop blank text blocks; keep this neutral text non-empty so the assistant turn survives replay.
return [{ type: "text", text: OMITTED_ASSISTANT_REASONING_TEXT } as AssistantContentBlock];
}
function hasReplayableThinkingSignature(block: AssistantContentBlock): boolean {
if (!isThinkingBlock(block)) {
return false;
}
const record = block as {
data?: unknown;
signature?: unknown;
thinkingSignature?: unknown;
thought_signature?: unknown;
};
const candidates =
(block as { type?: unknown }).type === "redacted_thinking"
? [record.data, record.signature, record.thinkingSignature, record.thought_signature]
: [record.signature, record.thinkingSignature, record.thought_signature];
return candidates.some((signature) => {
return typeof signature === "string" && signature.trim().length > 0;
});
}
/**
* Strip thinking blocks with clearly invalid replay signatures.
*
* Anthropic and Bedrock reject persisted thinking blocks when the signature is
* absent, empty, or blank. They are also the authority for opaque signature
* validity, so this intentionally avoids local length or shape heuristics.
*/
export function stripInvalidThinkingSignatures(messages: AgentMessage[]): AgentMessage[] {
let touched = false;
const out: AgentMessage[] = [];
for (const message of messages) {
if (!isAssistantMessageWithContent(message)) {
out.push(message);
continue;
}
const nextContent: AssistantContentBlock[] = [];
let changed = false;
for (const block of message.content) {
if (!isThinkingBlock(block) || hasReplayableThinkingSignature(block)) {
nextContent.push(block);
continue;
}
changed = true;
touched = true;
}
if (!changed) {
out.push(message);
continue;
}
out.push({
...message,
content: nextContent.length > 0 ? nextContent : buildOmittedAssistantReasoningContent(),
});
}
return touched ? out : messages;
}
/**
* Strip `type: "thinking"` and `type: "redacted_thinking"` content blocks from
* all assistant messages except the latest one.
@@ -63,8 +130,8 @@ function hasMeaningfulText(block: AssistantContentBlock): boolean {
* providers that require replay signatures can continue the conversation.
*
* If a non-latest assistant message becomes empty after stripping, it is
* replaced with a synthetic `{ type: "text", text: "" }` block to preserve
* turn structure (some providers require strict user/assistant alternation).
* replaced with a synthetic non-empty text block to preserve turn structure
* through provider adapters that filter blank text blocks.
*
* Returns the original array reference when nothing was changed (callers can
* use reference equality to skip downstream work).
@@ -104,9 +171,7 @@ export function dropThinkingBlocks(messages: AgentMessage[]): AgentMessage[] {
out.push(msg);
continue;
}
// Preserve the assistant turn even if all blocks were thinking-only.
const content =
nextContent.length > 0 ? nextContent : [{ type: "text", text: "" } as AssistantContentBlock];
const content = nextContent.length > 0 ? nextContent : buildOmittedAssistantReasoningContent();
out.push({ ...msg, content });
}
return touched ? out : messages;
@@ -130,10 +195,7 @@ function stripAllThinkingBlocks(messages: AgentMessage[]): AgentMessage[] {
touched = true;
out.push({
...message,
content:
nextContent.length > 0
? nextContent
: ([{ type: "text", text: "" }] as AssistantContentBlock[]),
content: nextContent.length > 0 ? nextContent : buildOmittedAssistantReasoningContent(),
});
}
return touched ? out : messages;

View File

@@ -10,6 +10,11 @@ export type EmbeddedPiAgentMeta = {
agentHarnessId?: string;
cliSessionBinding?: CliSessionBinding;
compactionCount?: number;
/**
* Prompt/context snapshot from the latest model request. Prefer this for
* context-window utilization because provider usage totals can include cached
* and completion tokens that are useful for billing but noisy as live context.
*/
promptTokens?: number;
usage?: {
input?: number;

View File

@@ -44,6 +44,10 @@ function resolveMaxToolResultChars(opts?: { maxToolResultChars?: number }): numb
return Math.max(1, opts?.maxToolResultChars ?? DEFAULT_MAX_LIVE_TOOL_RESULT_CHARS);
}
// `details` is runtime/UI metadata, not model-visible tool output. Keep the
// session JSONL useful for debugging without letting metadata blobs dominate
// disk, replay repair, transcript broadcasts, or future tooling that reads raw
// sessions. Model-visible text belongs in tool result `content`.
const MAX_PERSISTED_TOOL_RESULT_DETAILS_BYTES = 8_192;
const MAX_PERSISTED_DETAIL_STRING_CHARS = 2_000;
const MAX_PERSISTED_DETAIL_SESSION_COUNT = 10;
@@ -102,6 +106,9 @@ function buildPersistedDetailsFallback(
originalSize: BoundedJsonUtf8Bytes,
sanitizedBytes?: number,
): Record<string, unknown> {
// If even the structured summary is too large, keep only shape and stable
// status fields. This preserves "what happened?" without persisting the raw
// diagnostics payload that caused the cap to trip.
const fallback: Record<string, unknown> = {
persistedDetailsTruncated: true,
finalDetailsTruncated: true,
@@ -150,6 +157,8 @@ function sanitizeToolResultDetailsForPersistence(details: unknown): unknown {
if (details === undefined || details === null) {
return details;
}
// Measure with an early-exit walker so hostile or enormous details do not
// need to be fully stringified just to learn they exceed the persistence cap.
const originalSize = boundedJsonUtf8Bytes(details, MAX_PERSISTED_TOOL_RESULT_DETAILS_BYTES);
if (originalSize.complete && originalSize.bytes <= MAX_PERSISTED_TOOL_RESULT_DETAILS_BYTES) {
return details;

View File

@@ -218,6 +218,19 @@ describe("createImageGenerateTool", () => {
expect(createImageGenerateTool({ config: {} })).toBeNull();
});
it("tells agents how to request transparent OpenAI backgrounds", () => {
vi.stubEnv("OPENAI_API_KEY", "openai-key");
stubImageGenerationProviders();
const tool = requireImageGenerateTool(createImageGenerateTool({ config: {} }));
expect(tool.description).toContain('outputFormat="png" or "webp"');
expect(tool.description).toContain('background="transparent"');
expect(tool.description).toContain("openai.background");
expect(tool.description).toContain("gpt-image-1.5");
expect(JSON.stringify(tool.parameters)).toContain("openai/gpt-image-1.5");
});
it("matches image-generation providers across canonical provider aliases", () => {
vi.spyOn(imageGenerationRuntime, "listRuntimeImageGenerationProviders").mockReturnValue([
{
@@ -595,6 +608,62 @@ describe("createImageGenerateTool", () => {
});
});
it("forwards transparent OpenAI background requests with a PNG output format", async () => {
const generateImage = vi.spyOn(imageGenerationRuntime, "generateImage").mockResolvedValue({
provider: "openai",
model: "gpt-image-1.5",
attempts: [],
ignoredOverrides: [],
images: [
{
buffer: Buffer.from("png-out"),
mimeType: "image/png",
fileName: "transparent.png",
},
],
});
vi.spyOn(mediaStore, "saveMediaBuffer").mockResolvedValue({
path: "/tmp/transparent.png",
id: "transparent.png",
size: 7,
contentType: "image/png",
});
const tool = createToolWithPrimaryImageModel("openai/gpt-image-1.5");
const result = await tool.execute("call-openai-transparent", {
prompt: "A transparent badge",
outputFormat: "png",
openai: {
background: "transparent",
},
});
expect(generateImage).toHaveBeenCalledWith(
expect.objectContaining({
cfg: expect.objectContaining({
agents: expect.objectContaining({
defaults: expect.objectContaining({
imageGenerationModel: { primary: "openai/gpt-image-1.5" },
}),
}),
}),
outputFormat: "png",
providerOptions: {
openai: {
background: "transparent",
},
},
}),
);
expect(result).toMatchObject({
details: {
provider: "openai",
model: "gpt-image-1.5",
outputFormat: "png",
},
});
});
it("includes MEDIA paths in content text so follow-up replies use the real saved file", async () => {
vi.spyOn(imageGenerationRuntime, "listRuntimeImageGenerationProviders").mockReturnValue([
{

View File

@@ -8,6 +8,7 @@ import {
} from "../../image-generation/runtime.js";
import type {
ImageGenerationIgnoredOverride,
ImageGenerationBackground,
ImageGenerationOpenAIBackground,
ImageGenerationOpenAIModeration,
ImageGenerationOpenAIOptions,
@@ -62,7 +63,7 @@ const MAX_INPUT_IMAGES = 5;
const DEFAULT_RESOLUTION: ImageGenerationResolution = "1K";
const SUPPORTED_QUALITIES = ["low", "medium", "high", "auto"] as const;
const SUPPORTED_OUTPUT_FORMATS = ["png", "jpeg", "webp"] as const;
const SUPPORTED_OPENAI_BACKGROUNDS = ["transparent", "opaque", "auto"] as const;
const SUPPORTED_BACKGROUNDS = ["transparent", "opaque", "auto"] as const;
const SUPPORTED_OPENAI_MODERATIONS = ["low", "auto"] as const;
const SUPPORTED_ASPECT_RATIOS = new Set([
"1:1",
@@ -96,7 +97,10 @@ const ImageGenerateToolSchema = Type.Object({
}),
),
model: Type.Optional(
Type.String({ description: "Optional provider/model override, e.g. openai/gpt-image-2." }),
Type.String({
description:
"Optional provider/model override, e.g. openai/gpt-image-2; use openai/gpt-image-1.5 for transparent OpenAI backgrounds.",
}),
),
filename: Type.Optional(
Type.String({
@@ -128,10 +132,15 @@ const ImageGenerateToolSchema = Type.Object({
outputFormat: optionalStringEnum(SUPPORTED_OUTPUT_FORMATS, {
description: "Optional output format hint: png, jpeg, or webp when the provider supports it.",
}),
background: optionalStringEnum(SUPPORTED_BACKGROUNDS, {
description:
"Optional background hint: transparent, opaque, or auto when the provider supports it. For transparent output use outputFormat png or webp.",
}),
openai: Type.Optional(
Type.Object({
background: optionalStringEnum(SUPPORTED_OPENAI_BACKGROUNDS, {
description: "OpenAI-only background hint: transparent, opaque, or auto.",
background: optionalStringEnum(SUPPORTED_BACKGROUNDS, {
description:
"OpenAI-only background hint: transparent, opaque, or auto. For transparent output use outputFormat png or webp; OpenClaw routes the default OpenAI image model to gpt-image-1.5 for this mode.",
}),
moderation: optionalStringEnum(SUPPORTED_OPENAI_MODERATIONS, {
description: "OpenAI-only moderation hint: low or auto.",
@@ -266,12 +275,23 @@ function normalizeOpenAIBackground(
if (!normalized) {
return undefined;
}
if ((SUPPORTED_OPENAI_BACKGROUNDS as readonly string[]).includes(normalized)) {
if ((SUPPORTED_BACKGROUNDS as readonly string[]).includes(normalized)) {
return normalized as ImageGenerationOpenAIBackground;
}
throw new ToolInputError("openai.background must be one of transparent, opaque, or auto");
}
function normalizeBackground(raw: string | undefined): ImageGenerationBackground | undefined {
const normalized = raw?.trim().toLowerCase();
if (!normalized) {
return undefined;
}
if ((SUPPORTED_BACKGROUNDS as readonly string[]).includes(normalized)) {
return normalized as ImageGenerationBackground;
}
throw new ToolInputError("background must be one of transparent, opaque, or auto");
}
function normalizeOpenAIModeration(
raw: string | undefined,
): ImageGenerationOpenAIModeration | undefined {
@@ -570,7 +590,7 @@ export function createImageGenerateTool(options?: {
label: "Image Generation",
name: "image_generate",
description:
'Generate new images or edit reference images with the configured or inferred image-generation model. Set agents.defaults.imageGenerationModel.primary to pick a provider/model. Providers declare their own auth/readiness; use action="list" to inspect registered providers, models, readiness, and auth hints. Generated images are delivered automatically from the tool result as MEDIA paths.',
'Generate new images or edit reference images with the configured or inferred image-generation model. For transparent backgrounds, use outputFormat="png" or "webp" and background="transparent"; OpenAI also accepts openai.background and OpenClaw routes the default OpenAI image model to gpt-image-1.5 for that mode. Set agents.defaults.imageGenerationModel.primary to pick a provider/model. Providers declare their own auth/readiness; use action="list" to inspect registered providers, models, readiness, and auth hints. Generated images are delivered automatically from the tool result as MEDIA paths.',
parameters: ImageGenerateToolSchema,
execute: async (_toolCallId, args) => {
const params = args as Record<string, unknown>;
@@ -612,6 +632,12 @@ export function createImageGenerateTool(options?: {
if ((provider.capabilities.geometry?.aspectRatios?.length ?? 0) > 0) {
caps.push(`aspect ratios ${provider.capabilities.geometry?.aspectRatios?.join(", ")}`);
}
if ((provider.capabilities.output?.formats?.length ?? 0) > 0) {
caps.push(`formats ${provider.capabilities.output?.formats?.join("/")}`);
}
if ((provider.capabilities.output?.backgrounds?.length ?? 0) > 0) {
caps.push(`backgrounds ${provider.capabilities.output?.backgrounds?.join("/")}`);
}
const modelLine =
provider.models.length > 0
? `models: ${provider.models.join(", ")}`
@@ -641,6 +667,7 @@ export function createImageGenerateTool(options?: {
const timeoutMs = readGenerationTimeoutMs(params) ?? imageGenerationModelConfig.timeoutMs;
const quality = normalizeQuality(readStringParam(params, "quality"));
const outputFormat = normalizeOutputFormat(readStringParam(params, "outputFormat"));
const background = normalizeBackground(readStringParam(params, "background"));
const providerOptions = normalizeProviderOptions(params);
const selectedProvider = resolveSelectedImageGenerationProvider({
config: effectiveCfg,
@@ -689,6 +716,7 @@ export function createImageGenerateTool(options?: {
resolution,
quality,
outputFormat,
background,
count,
inputImages,
timeoutMs,
@@ -776,6 +804,7 @@ export function createImageGenerateTool(options?: {
: {}),
...(quality ? { quality } : {}),
...(outputFormat ? { outputFormat } : {}),
...(background ? { background } : {}),
...(filename ? { filename } : {}),
...(timeoutMs !== undefined ? { timeoutMs } : {}),
attempts: result.attempts,

View File

@@ -259,6 +259,7 @@ export function createSessionsSpawnTool(
agentId: requestedAgentId,
resumeSessionId,
model: modelOverride,
thinking: thinkingOverrideRaw,
cwd,
mode: mode === "run" || mode === "session" ? mode : undefined,
thread,

View File

@@ -10,6 +10,11 @@ import {
import * as sessionTypesModule from "../../config/sessions.js";
import type { SessionEntry } from "../../config/sessions.js";
import { loadSessionStore, saveSessionStore } from "../../config/sessions.js";
import {
onInternalDiagnosticEvent,
resetDiagnosticEventsForTest,
type DiagnosticEventPayload,
} from "../../infra/diagnostic-events.js";
import {
clearMemoryPluginState,
registerMemoryFlushPlanResolver,
@@ -138,6 +143,7 @@ type RunWithModelFallbackParams = {
};
beforeEach(() => {
resetDiagnosticEventsForTest();
embeddedRunTesting.resetActiveEmbeddedRuns();
replyRunRegistryTesting.resetReplyRunRegistry();
runEmbeddedPiAgentMock.mockClear();
@@ -169,6 +175,7 @@ beforeEach(() => {
});
afterEach(() => {
resetDiagnosticEventsForTest();
vi.useRealTimers();
clearMemoryPluginState();
replyRunRegistryTesting.resetReplyRunRegistry();
@@ -289,6 +296,167 @@ describe("runReplyAgent auto-compaction token update", () => {
// totalTokens should use lastCallUsage (55k), not accumulated (75k)
expect(stored[sessionKey].totalTokens).toBe(55_000);
});
it("reports live diagnostic context from promptTokens, not provider usage totals", async () => {
const tmp = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-usage-diagnostic-"));
const storePath = path.join(tmp, "sessions.json");
const sessionKey = "main";
const sessionEntry = {
sessionId: "session",
updatedAt: Date.now(),
totalTokens: 50_000,
};
await seedSessionStore({ storePath, sessionKey, entry: sessionEntry });
runEmbeddedPiAgentMock.mockResolvedValue({
payloads: [{ text: "ok" }],
meta: {
agentMeta: {
usage: { input: 75_000, output: 5_000, cacheRead: 25_000, total: 105_000 },
lastCallUsage: { input: 55_000, output: 2_000, cacheRead: 25_000, total: 82_000 },
promptTokens: 44_000,
},
},
});
const diagnostics: DiagnosticEventPayload[] = [];
const unsubscribe = onInternalDiagnosticEvent((event) => {
diagnostics.push(event);
});
const { typing, sessionCtx, resolvedQueue, followupRun } = createBaseRun({
storePath,
sessionEntry,
});
try {
await runReplyAgent({
commandBody: "hello",
followupRun,
queueKey: "main",
resolvedQueue,
shouldSteer: false,
shouldFollowup: false,
isActive: false,
isStreaming: false,
typing,
sessionCtx,
sessionEntry,
sessionStore: { [sessionKey]: sessionEntry },
sessionKey,
storePath,
defaultModel: "anthropic/claude-opus-4-6",
agentCfgContextTokens: 200_000,
resolvedVerboseLevel: "off",
isNewSession: false,
blockStreamingEnabled: false,
resolvedBlockStreamingBreak: "message_end",
shouldInjectGroupIntro: false,
typingMode: "instant",
});
} finally {
unsubscribe();
}
const usageEvent = diagnostics.find((event) => event.type === "model.usage");
expect(usageEvent).toMatchObject({
type: "model.usage",
usage: {
input: 75_000,
output: 5_000,
cacheRead: 25_000,
promptTokens: 100_000,
total: 105_000,
},
context: {
limit: 200_000,
used: 44_000,
},
});
});
it("falls back to last-call prompt usage for live diagnostic context", async () => {
const tmp = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-usage-diagnostic-last-"));
const storePath = path.join(tmp, "sessions.json");
const sessionKey = "main";
const sessionEntry = {
sessionId: "session",
updatedAt: Date.now(),
totalTokens: 50_000,
};
await seedSessionStore({ storePath, sessionKey, entry: sessionEntry });
runEmbeddedPiAgentMock.mockResolvedValue({
payloads: [{ text: "ok" }],
meta: {
agentMeta: {
usage: { input: 75_000, output: 5_000, cacheRead: 25_000, total: 105_000 },
lastCallUsage: {
input: 55_000,
output: 2_000,
cacheRead: 25_000,
cacheWrite: 1_000,
total: 83_000,
},
},
},
});
const diagnostics: DiagnosticEventPayload[] = [];
const unsubscribe = onInternalDiagnosticEvent((event) => {
diagnostics.push(event);
});
const { typing, sessionCtx, resolvedQueue, followupRun } = createBaseRun({
storePath,
sessionEntry,
});
try {
await runReplyAgent({
commandBody: "hello",
followupRun,
queueKey: "main",
resolvedQueue,
shouldSteer: false,
shouldFollowup: false,
isActive: false,
isStreaming: false,
typing,
sessionCtx,
sessionEntry,
sessionStore: { [sessionKey]: sessionEntry },
sessionKey,
storePath,
defaultModel: "anthropic/claude-opus-4-6",
agentCfgContextTokens: 200_000,
resolvedVerboseLevel: "off",
isNewSession: false,
blockStreamingEnabled: false,
resolvedBlockStreamingBreak: "message_end",
shouldInjectGroupIntro: false,
typingMode: "instant",
});
} finally {
unsubscribe();
}
const usageEvent = diagnostics.find((event) => event.type === "model.usage");
expect(usageEvent).toMatchObject({
type: "model.usage",
usage: {
input: 75_000,
output: 5_000,
cacheRead: 25_000,
promptTokens: 100_000,
total: 105_000,
},
context: {
limit: 200_000,
used: 81_000,
},
});
});
});
describe("runReplyAgent block streaming", () => {
@@ -913,6 +1081,7 @@ describe("runReplyAgent Active Memory inline debug", () => {
model: "claude",
usage: { input: 1200, output: 45, cacheRead: 800, cacheWrite: 200, total: 2245 },
lastCallUsage: { input: 1000, output: 45, cacheRead: 750, cacheWrite: 150, total: 1945 },
promptTokens: 1250,
compactionCount: 1,
},
},
@@ -987,6 +1156,7 @@ describe("runReplyAgent Active Memory inline debug", () => {
expect(traceText).toContain("🔎 Usage (Session Total):");
expect(traceText).toContain("🔎 Usage (Last Turn Total):");
expect(traceText).toContain("🔎 Context Window (Last Model Request):");
expect(traceText).toContain("used=1,250 tok (1.3k)");
expect(traceText).toContain("🔎 Execution Result:");
expect(traceText).toContain("winner=anthropic/claude");
expect(traceText).toContain("fallbackUsed=yes");
@@ -1025,7 +1195,7 @@ describe("runReplyAgent Active Memory inline debug", () => {
expect(traceText).toContain("🔎 Model Input (User Role):");
expect(traceText).toContain("🔎 Model Output (Assistant Role):");
expect(traceText).toContain(
"Summary: winner=claude 🧠 low fallback=yes attempts=2 stop=end_turn prompt=1.9k/200k ⬇️ 1.2k ⬆️ 45 ♻️ 800 🆕 200 🔢 2.2k tools=2 compactions=1",
"Summary: winner=claude 🧠 low fallback=yes attempts=2 stop=end_turn prompt=1.3k/200k ⬇️ 1.2k ⬆️ 45 ♻️ 800 🆕 200 🔢 2.2k tools=2 compactions=1",
);
expect(traceText.indexOf("🔎 Execution Result:")).toBeGreaterThan(
traceText.indexOf("🔎 Context Window (Last Model Request):"),

View File

@@ -585,6 +585,13 @@ function resolveRequestPromptTokens(params: {
total?: number;
};
}): number | undefined {
if (
typeof params.promptTokens === "number" &&
Number.isFinite(params.promptTokens) &&
params.promptTokens > 0
) {
return params.promptTokens;
}
const lastCall = params.lastCallUsage;
if (lastCall) {
const input = lastCall.input ?? 0;
@@ -595,13 +602,6 @@ function resolveRequestPromptTokens(params: {
return sum;
}
}
if (
typeof params.promptTokens === "number" &&
Number.isFinite(params.promptTokens) &&
params.promptTokens > 0
) {
return params.promptTokens;
}
const usage = params.usage;
if (usage) {
const input = usage.input ?? 0;
@@ -1428,8 +1428,13 @@ export async function runReplyAgent(params: {
const output = usage.output ?? 0;
const cacheRead = usage.cacheRead ?? 0;
const cacheWrite = usage.cacheWrite ?? 0;
const promptTokens = input + cacheRead + cacheWrite;
const totalTokens = usage.total ?? promptTokens + output;
const usagePromptTokens = input + cacheRead + cacheWrite;
const totalTokens = usage.total ?? usagePromptTokens + output;
const contextUsedTokens = resolveRequestPromptTokens({
lastCallUsage: runResult.meta?.agentMeta?.lastCallUsage,
promptTokens,
usage,
});
const costConfig = resolveModelCostConfig({
provider: providerUsed,
model: modelUsed,
@@ -1455,13 +1460,13 @@ export async function runReplyAgent(params: {
output,
cacheRead,
cacheWrite,
promptTokens,
promptTokens: usagePromptTokens,
total: totalTokens,
},
lastCallUsage: runResult.meta?.agentMeta?.lastCallUsage,
context: {
limit: contextTokensUsed,
used: totalTokens,
...(contextUsedTokens !== undefined ? { used: contextUsedTokens } : {}),
},
costUsd,
durationMs: Date.now() - runStartedAt,

View File

@@ -92,7 +92,7 @@ describe("channel plugin module loader helpers", () => {
expect(createJiti).not.toHaveBeenCalled();
});
it("keeps Windows dist loads off Jiti native import", async () => {
it("uses native Jiti import for Windows dist loads", async () => {
const createJiti = vi.fn(() => vi.fn(() => ({ ok: true })));
vi.doMock("jiti", () => ({
createJiti,
@@ -119,7 +119,7 @@ describe("channel plugin module loader helpers", () => {
expect(createJiti).toHaveBeenCalledWith(
expect.any(String),
expect.objectContaining({
tryNative: false,
tryNative: true,
}),
);
} finally {

View File

@@ -553,6 +553,166 @@ describe("capability cli", () => {
);
});
it("passes image output format and generic background hints through to generation runtime", async () => {
mocks.generateImage.mockResolvedValue({
provider: "openai",
model: "gpt-image-1.5",
attempts: [],
images: [
{
buffer: Buffer.from("png-bytes"),
mimeType: "image/png",
fileName: "transparent.png",
},
],
});
await runRegisteredCli({
register: registerCapabilityCli as (program: Command) => void,
argv: [
"capability",
"image",
"generate",
"--prompt",
"transparent sticker",
"--model",
"openai/gpt-image-1.5",
"--output-format",
"png",
"--background",
"transparent",
"--json",
],
});
expect(mocks.generateImage).toHaveBeenCalledWith(
expect.objectContaining({
prompt: "transparent sticker",
modelOverride: "openai/gpt-image-1.5",
outputFormat: "png",
background: "transparent",
providerOptions: undefined,
}),
);
});
it("passes image output format and OpenAI background hints through to edit runtime", async () => {
mocks.generateImage.mockResolvedValue({
provider: "openai",
model: "gpt-image-1.5",
attempts: [],
images: [
{
buffer: Buffer.from("png-bytes"),
mimeType: "image/png",
fileName: "transparent-edit.png",
},
],
});
const inputPath = path.join(os.tmpdir(), `openclaw-image-edit-${Date.now()}.png`);
await fs.writeFile(inputPath, Buffer.from("png-input"));
await runRegisteredCli({
register: registerCapabilityCli as (program: Command) => void,
argv: [
"capability",
"image",
"edit",
"--file",
inputPath,
"--prompt",
"make background transparent",
"--model",
"openai/gpt-image-1.5",
"--output-format",
"png",
"--openai-background",
"transparent",
"--json",
],
});
expect(mocks.generateImage).toHaveBeenCalledWith(
expect.objectContaining({
prompt: "make background transparent",
modelOverride: "openai/gpt-image-1.5",
outputFormat: "png",
background: undefined,
providerOptions: {
openai: {
background: "transparent",
},
},
inputImages: [
expect.objectContaining({
fileName: path.basename(inputPath),
}),
],
}),
);
});
it("rejects unsupported image output format and background hints", async () => {
await expect(
runRegisteredCli({
register: registerCapabilityCli as (program: Command) => void,
argv: [
"capability",
"image",
"generate",
"--prompt",
"transparent sticker",
"--output-format",
"gif",
"--json",
],
}),
).rejects.toThrow("exit 1");
expect(mocks.runtime.error).toHaveBeenCalledWith(
"Error: --output-format must be one of png, jpeg, or webp",
);
mocks.runtime.error.mockClear();
await expect(
runRegisteredCli({
register: registerCapabilityCli as (program: Command) => void,
argv: [
"capability",
"image",
"generate",
"--prompt",
"transparent sticker",
"--openai-background",
"clear",
"--json",
],
}),
).rejects.toThrow("exit 1");
expect(mocks.runtime.error).toHaveBeenCalledWith(
"Error: --openai-background must be one of transparent, opaque, or auto",
);
mocks.runtime.error.mockClear();
await expect(
runRegisteredCli({
register: registerCapabilityCli as (program: Command) => void,
argv: [
"capability",
"image",
"generate",
"--prompt",
"transparent sticker",
"--background",
"clear",
"--json",
],
}),
).rejects.toThrow("exit 1");
expect(mocks.runtime.error).toHaveBeenCalledWith(
"Error: --background must be one of transparent, opaque, or auto",
);
});
it("streams url-only generated videos to --output paths", async () => {
mocks.generateVideo.mockResolvedValue({
provider: "vydra",

View File

@@ -22,6 +22,10 @@ import { buildGatewayConnectionDetailsWithResolvers } from "../gateway/connectio
import { isLoopbackHost } from "../gateway/net.js";
import { GATEWAY_CLIENT_MODES, GATEWAY_CLIENT_NAMES } from "../gateway/protocol/client-info.js";
import { generateImage, listRuntimeImageGenerationProviders } from "../image-generation/runtime.js";
import type {
ImageGenerationBackground,
ImageGenerationOutputFormat,
} from "../image-generation/types.js";
import { buildMediaUnderstandingRegistry } from "../media-understanding/provider-registry.js";
import {
describeImageFile,
@@ -78,6 +82,8 @@ import { removeCommandByName } from "./program/command-tree.js";
import { collectOption } from "./program/helpers.js";
type CapabilityTransport = "local" | "gateway";
const IMAGE_OUTPUT_FORMATS = ["png", "jpeg", "webp"] as const;
const IMAGE_BACKGROUNDS = ["transparent", "opaque", "auto"] as const;
type CapabilityMetadata = {
id: string;
@@ -95,6 +101,7 @@ type CapabilityEnvelope = {
model?: string;
attempts: Array<Record<string, unknown>>;
outputs: Array<Record<string, unknown>>;
ignoredOverrides?: Array<Record<string, unknown>>;
error?: string;
};
@@ -384,6 +391,9 @@ function formatEnvelopeForText(value: unknown): string {
`${envelope.capability} via ${envelope.transport}`,
...(envelope.provider ? [`provider: ${envelope.provider}`] : []),
...(envelope.model ? [`model: ${envelope.model}`] : []),
...(envelope.ignoredOverrides && envelope.ignoredOverrides.length > 0
? [`ignoredOverrides: ${JSON.stringify(envelope.ignoredOverrides)}`]
: []),
`outputs: ${String(envelope.outputs.length)}`,
];
for (const output of envelope.outputs) {
@@ -702,6 +712,9 @@ async function runImageGenerate(params: {
size?: string;
aspectRatio?: string;
resolution?: "1K" | "2K" | "4K";
outputFormat?: ImageGenerationOutputFormat;
background?: ImageGenerationBackground;
openaiBackground?: ImageGenerationBackground;
file?: string[];
output?: string;
timeoutMs?: number;
@@ -728,6 +741,11 @@ async function runImageGenerate(params: {
size: params.size,
aspectRatio: params.aspectRatio,
resolution: params.resolution,
outputFormat: params.outputFormat,
background: params.background,
providerOptions: params.openaiBackground
? { openai: { background: params.openaiBackground } }
: undefined,
timeoutMs: params.timeoutMs,
inputImages,
});
@@ -759,6 +777,7 @@ async function runImageGenerate(params: {
model: result.model,
attempts: result.attempts,
outputs,
ignoredOverrides: result.ignoredOverrides,
} satisfies CapabilityEnvelope;
}
@@ -851,6 +870,33 @@ function parseOptionalFiniteNumber(
return value;
}
function normalizeImageOutputFormat(
raw: string | undefined,
): ImageGenerationOutputFormat | undefined {
const normalized = normalizeLowercaseStringOrEmpty(raw);
if (!normalized) {
return undefined;
}
if ((IMAGE_OUTPUT_FORMATS as readonly string[]).includes(normalized)) {
return normalized as ImageGenerationOutputFormat;
}
throw new Error("--output-format must be one of png, jpeg, or webp");
}
function normalizeImageBackground(
raw: string | undefined,
label = "--background",
): ImageGenerationBackground | undefined {
const normalized = normalizeLowercaseStringOrEmpty(raw);
if (!normalized) {
return undefined;
}
if ((IMAGE_BACKGROUNDS as readonly string[]).includes(normalized)) {
return normalized as ImageGenerationBackground;
}
throw new Error(`${label} must be one of transparent, opaque, or auto`);
}
function normalizeVideoResolution(raw: string | undefined): VideoGenerationResolution | undefined {
const normalized = raw?.trim().toUpperCase();
if (!normalized) {
@@ -1438,6 +1484,9 @@ export function registerCapabilityCli(program: Command) {
.option("--size <size>", "Size hint like 1024x1024")
.option("--aspect-ratio <ratio>", "Aspect ratio hint like 16:9")
.option("--resolution <value>", "Resolution hint: 1K, 2K, or 4K")
.option("--output-format <format>", "Output format hint: png, jpeg, or webp")
.option("--background <value>", "Background hint: transparent, opaque, or auto")
.option("--openai-background <value>", "OpenAI background hint: transparent, opaque, or auto")
.option("--timeout-ms <ms>", "Provider request timeout in milliseconds")
.option("--output <path>", "Output path")
.option("--json", "Output JSON", false)
@@ -1451,6 +1500,12 @@ export function registerCapabilityCli(program: Command) {
size: opts.size as string | undefined,
aspectRatio: opts.aspectRatio as string | undefined,
resolution: opts.resolution as "1K" | "2K" | "4K" | undefined,
outputFormat: normalizeImageOutputFormat(opts.outputFormat as string | undefined),
background: normalizeImageBackground(opts.background as string | undefined),
openaiBackground: normalizeImageBackground(
opts.openaiBackground as string | undefined,
"--openai-background",
),
timeoutMs: parseOptionalFiniteNumber(opts.timeoutMs, "--timeout-ms"),
output: opts.output as string | undefined,
});
@@ -1464,6 +1519,9 @@ export function registerCapabilityCli(program: Command) {
.requiredOption("--file <path>", "Input file", collectOption, [])
.requiredOption("--prompt <text>", "Prompt text")
.option("--model <provider/model>", "Model override")
.option("--output-format <format>", "Output format hint: png, jpeg, or webp")
.option("--background <value>", "Background hint: transparent, opaque, or auto")
.option("--openai-background <value>", "OpenAI background hint: transparent, opaque, or auto")
.option("--timeout-ms <ms>", "Provider request timeout in milliseconds")
.option("--output <path>", "Output path")
.option("--json", "Output JSON", false)
@@ -1475,6 +1533,12 @@ export function registerCapabilityCli(program: Command) {
prompt: String(opts.prompt),
model: opts.model as string | undefined,
file: files,
outputFormat: normalizeImageOutputFormat(opts.outputFormat as string | undefined),
background: normalizeImageBackground(opts.background as string | undefined),
openaiBackground: normalizeImageBackground(
opts.openaiBackground as string | undefined,
"--openai-background",
),
timeoutMs: parseOptionalFiniteNumber(opts.timeoutMs, "--timeout-ms"),
output: opts.output as string | undefined,
});

View File

@@ -14,6 +14,7 @@ type ListMarketplacePluginsFn =
(typeof import("../plugins/marketplace.js"))["listMarketplacePlugins"];
type ResolveMarketplaceInstallShortcutFn =
(typeof import("../plugins/marketplace.js"))["resolveMarketplaceInstallShortcut"];
type LoadPluginInstallRecordsParams = { config?: OpenClawConfig };
// oxlint-disable-next-line typescript/no-unnecessary-type-parameters -- Test helper preserves mock call and result types.
function invokeMock<TArgs extends unknown[], TResult>(mock: unknown, ...args: TArgs): TResult {
@@ -32,9 +33,9 @@ export const listMarketplacePlugins: Mock<ListMarketplacePluginsFn> = vi.fn();
export const resolveMarketplaceInstallShortcut: Mock<ResolveMarketplaceInstallShortcutFn> = vi.fn();
export const enablePluginInConfig: UnknownMock = vi.fn();
export const recordPluginInstall: UnknownMock = vi.fn();
export const loadPluginInstallRecords: AsyncUnknownMock = vi.fn(async ({ config }) => {
const cfg = config as OpenClawConfig | undefined;
return structuredClone(cfg?.plugins?.installs ?? {});
export const loadPluginInstallRecords: AsyncUnknownMock = vi.fn(async (...args: unknown[]) => {
const params = args[0] as LoadPluginInstallRecordsParams | undefined;
return structuredClone(params?.config?.plugins?.installs ?? {});
});
export const writePersistedPluginInstallLedger: AsyncUnknownMock = vi.fn(async () => undefined);
export const clearPluginManifestRegistryCache: UnknownMock = vi.fn();
@@ -518,9 +519,9 @@ export function resetPluginsCliTestState() {
recordPluginInstall.mockImplementation(
((cfg: OpenClawConfig) => cfg) as (...args: unknown[]) => unknown,
);
loadPluginInstallRecords.mockImplementation(async ({ config }) => {
const cfg = config as OpenClawConfig | undefined;
return structuredClone(cfg?.plugins?.installs ?? {});
loadPluginInstallRecords.mockImplementation(async (...args: unknown[]) => {
const params = args[0] as LoadPluginInstallRecordsParams | undefined;
return structuredClone(params?.config?.plugins?.installs ?? {});
});
writePersistedPluginInstallLedger.mockResolvedValue(undefined);
loadPluginManifestRegistry.mockReturnValue({

View File

@@ -25,6 +25,7 @@ import {
runtimeErrors,
runtimeLogs,
writeConfigFile,
writePersistedPluginInstallLedger,
} from "./plugins-cli-test-helpers.js";
const CLI_STATE_ROOT = "/tmp/openclaw-state";
@@ -53,22 +54,6 @@ function createEmptyPluginConfig(): OpenClawConfig {
} as OpenClawConfig;
}
function createClawHubInstalledConfig(params: {
pluginId: string;
install: Record<string, unknown>;
}): OpenClawConfig {
const enabledCfg = createEnabledPluginConfig(params.pluginId);
return {
...enabledCfg,
plugins: {
...enabledCfg.plugins,
installs: {
[params.pluginId]: params.install,
},
},
} as OpenClawConfig;
}
function createClawHubInstallResult(params: {
pluginId: string;
packageName: string;
@@ -305,7 +290,7 @@ describe("plugins cli install", () => {
expect(writeConfigFile).not.toHaveBeenCalled();
});
it("installs marketplace plugins and persists config", async () => {
it("installs marketplace plugins and persists install ledger", async () => {
const cfg = {
plugins: {
entries: {},
@@ -320,19 +305,6 @@ describe("plugins cli install", () => {
},
},
} as OpenClawConfig;
const installedCfg = {
...enabledCfg,
plugins: {
...enabledCfg.plugins,
installs: {
alpha: {
source: "marketplace",
installPath: cliInstallPath("alpha"),
},
},
},
} as OpenClawConfig;
loadConfig.mockReturnValue(cfg);
installPluginFromMarketplace.mockResolvedValue({
ok: true,
@@ -345,20 +317,25 @@ describe("plugins cli install", () => {
marketplacePlugin: "alpha",
});
enablePluginInConfig.mockReturnValue({ config: enabledCfg });
recordPluginInstall.mockReturnValue(installedCfg);
buildPluginDiagnosticsReport.mockReturnValue({
plugins: [{ id: "alpha", kind: "provider" }],
diagnostics: [],
});
applyExclusiveSlotSelection.mockReturnValue({
config: installedCfg,
config: enabledCfg,
warnings: ["slot adjusted"],
});
await runPluginsCommand(["plugins", "install", "alpha", "--marketplace", "local/repo"]);
expect(clearPluginManifestRegistryCache).toHaveBeenCalledTimes(1);
expect(writeConfigFile).toHaveBeenCalledWith(installedCfg);
expect(writePersistedPluginInstallLedger).toHaveBeenCalledWith({
alpha: expect.objectContaining({
source: "marketplace",
installPath: cliInstallPath("alpha"),
}),
});
expect(writeConfigFile).toHaveBeenCalledWith(enabledCfg);
expect(runtimeLogs.some((line) => line.includes("slot adjusted"))).toBe(true);
expect(runtimeLogs.some((line) => line.includes("Installed plugin: alpha"))).toBe(true);
});
@@ -384,18 +361,6 @@ describe("plugins cli install", () => {
},
} as OpenClawConfig;
const enabledCfg = createEnabledPluginConfig("demo");
const installedCfg = createClawHubInstalledConfig({
pluginId: "demo",
install: {
source: "clawhub",
spec: "clawhub:demo@1.2.3",
installPath: cliInstallPath("demo"),
clawhubPackage: "demo",
clawhubFamily: "code-plugin",
clawhubChannel: "official",
},
});
loadConfig.mockReturnValue(cfg);
parseClawHubPluginSpec.mockReturnValue({ name: "demo" });
installPluginFromClawHub.mockResolvedValue(
@@ -407,9 +372,8 @@ describe("plugins cli install", () => {
}),
);
enablePluginInConfig.mockReturnValue({ config: enabledCfg });
recordPluginInstall.mockReturnValue(installedCfg);
applyExclusiveSlotSelection.mockReturnValue({
config: installedCfg,
config: enabledCfg,
warnings: [],
});
@@ -420,18 +384,17 @@ describe("plugins cli install", () => {
spec: "clawhub:demo",
}),
);
expect(recordPluginInstall).toHaveBeenCalledWith(
enabledCfg,
expect.objectContaining({
pluginId: "demo",
expect(writePersistedPluginInstallLedger).toHaveBeenCalledWith({
demo: expect.objectContaining({
source: "clawhub",
spec: "clawhub:demo@1.2.3",
installPath: cliInstallPath("demo"),
clawhubPackage: "demo",
clawhubFamily: "code-plugin",
clawhubChannel: "official",
}),
);
expect(writeConfigFile).toHaveBeenCalledWith(installedCfg);
});
expect(writeConfigFile).toHaveBeenCalledWith(enabledCfg);
expect(runtimeLogs.some((line) => line.includes("Installed plugin: demo"))).toBe(true);
expect(installPluginFromNpmSpec).not.toHaveBeenCalled();
});
@@ -478,16 +441,6 @@ describe("plugins cli install", () => {
},
} as OpenClawConfig;
const enabledCfg = createEnabledPluginConfig("demo");
const installedCfg = createClawHubInstalledConfig({
pluginId: "demo",
install: {
source: "clawhub",
spec: "clawhub:demo@1.2.3",
installPath: cliInstallPath("demo"),
clawhubPackage: "demo",
},
});
loadConfig.mockReturnValue(cfg);
installPluginFromClawHub.mockResolvedValue(
createClawHubInstallResult({
@@ -498,9 +451,8 @@ describe("plugins cli install", () => {
}),
);
enablePluginInConfig.mockReturnValue({ config: enabledCfg });
recordPluginInstall.mockReturnValue(installedCfg);
applyExclusiveSlotSelection.mockReturnValue({
config: installedCfg,
config: enabledCfg,
warnings: [],
});
@@ -512,7 +464,15 @@ describe("plugins cli install", () => {
}),
);
expect(installPluginFromNpmSpec).not.toHaveBeenCalled();
expect(writeConfigFile).toHaveBeenCalledWith(installedCfg);
expect(writePersistedPluginInstallLedger).toHaveBeenCalledWith({
demo: expect.objectContaining({
source: "clawhub",
spec: "clawhub:demo@1.2.3",
installPath: cliInstallPath("demo"),
clawhubPackage: "demo",
}),
});
expect(writeConfigFile).toHaveBeenCalledWith(enabledCfg);
});
it("falls back to npm when ClawHub does not have the package", async () => {

View File

@@ -26,6 +26,11 @@ vi.mock("../../commands/onboard-core-auth-flags.js", () => ({
description: "Mistral API key",
optionKey: "mistralApiKey",
},
{
cliOption: "--openai-api-key <key>",
description: "OpenAI API key (core fallback)",
optionKey: "openaiApiKey",
},
] as Array<{ cliOption: string; description: string; optionKey: string }>,
}));
@@ -156,6 +161,16 @@ describe("registerOnboardCommand", () => {
);
});
it("dedupes provider auth flags before registering command options", async () => {
await runCli(["onboard", "--openai-api-key", "sk-openai-test"]);
expect(setupWizardCommandMock).toHaveBeenCalledWith(
expect.objectContaining({
openaiApiKey: "sk-openai-test", // pragma: allowlist secret
}),
runtime,
);
});
it("forwards --gateway-token-ref-env", async () => {
await runCli(["onboard", "--gateway-token-ref-env", "OPENCLAW_GATEWAY_TOKEN"]);
expect(setupWizardCommandMock).toHaveBeenCalledWith(

View File

@@ -47,10 +47,39 @@ const AUTH_CHOICE_HELP = formatAuthChoiceChoicesForCli({
includeSkip: true,
});
const ONBOARD_AUTH_FLAGS = [
...CORE_ONBOARD_AUTH_FLAGS,
...resolveManifestProviderOnboardAuthFlags(),
] as const;
type OnboardAuthFlag = {
readonly cliOption: string;
readonly description: string;
readonly optionKey: string;
};
function extractCliFlags(cliOption: string): string[] {
return cliOption
.split(/[ ,|]+/)
.filter((part) => part.startsWith("-"))
.map((part) => {
const equalsIndex = part.indexOf("=");
return equalsIndex === -1 ? part : part.slice(0, equalsIndex);
});
}
function resolveOnboardAuthFlags(): OnboardAuthFlag[] {
const seenCliFlags = new Set<string>();
const flags: OnboardAuthFlag[] = [];
for (const flag of [...CORE_ONBOARD_AUTH_FLAGS, ...resolveManifestProviderOnboardAuthFlags()]) {
const cliFlags = extractCliFlags(flag.cliOption);
if (cliFlags.some((cliFlag) => seenCliFlags.has(cliFlag))) {
continue;
}
for (const cliFlag of cliFlags) {
seenCliFlags.add(cliFlag);
}
flags.push(flag);
}
return flags;
}
const ONBOARD_AUTH_FLAGS = resolveOnboardAuthFlags();
function pickOnboardProviderAuthOptionValues(
opts: Record<string, unknown>,

View File

@@ -5,15 +5,19 @@ import type { RuntimeEnv } from "../runtime.js";
import * as backupShared from "./backup-shared.js";
import { resolveBackupPlanFromPaths } from "./backup-shared.js";
export const tarCreateMock = vi.fn();
export const backupVerifyCommandMock = vi.fn();
const backupTestMocks = vi.hoisted(() => ({
backupVerifyCommandMock: vi.fn(),
tarCreateMock: vi.fn(),
}));
export const { backupVerifyCommandMock, tarCreateMock } = backupTestMocks;
vi.mock("tar", () => ({
c: tarCreateMock,
c: backupTestMocks.tarCreateMock,
}));
vi.mock("./backup-verify.js", () => ({
backupVerifyCommand: backupVerifyCommandMock,
backupVerifyCommand: backupTestMocks.backupVerifyCommandMock,
}));
export function createBackupTestRuntime(): RuntimeEnv {

View File

@@ -313,9 +313,7 @@ describe("ensureChannelSetupPluginInstalled", () => {
expect(result.installed).toBe(true);
expect(result.cfg.plugins?.entries?.["bundled-chat"]?.enabled).toBe(true);
expect(result.cfg.plugins?.allow).toContain("bundled-chat");
expect(result.cfg.plugins?.installs?.["bundled-chat"]?.source).toBe("npm");
expect(result.cfg.plugins?.installs?.["bundled-chat"]?.spec).toBe(bundledChatNpmSpec);
expect(result.cfg.plugins?.installs?.["bundled-chat"]?.installPath).toBe("/tmp/bundled-chat");
expect(result.cfg.plugins?.installs).toBeUndefined();
expect(installPluginFromNpmSpec).toHaveBeenCalledWith(
expect.objectContaining({
expectedIntegrity: bundledChatIntegrity,

View File

@@ -0,0 +1,149 @@
import fs from "node:fs";
import path from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import type { PluginCandidate } from "../plugins/discovery.js";
import { readPersistedPluginInstallLedger } from "../plugins/install-ledger-store.js";
import {
readPersistedInstalledPluginIndex,
writePersistedInstalledPluginIndex,
} from "../plugins/installed-plugin-index-store.js";
import type { InstalledPluginIndex } from "../plugins/installed-plugin-index.js";
import { cleanupTrackedTempDirs, makeTrackedTempDir } from "../plugins/test-helpers/fs-fixtures.js";
import { note } from "../terminal/note.js";
import { maybeRepairPluginRegistryState } from "./doctor-plugin-registry.js";
vi.mock("../terminal/note.js", () => ({
note: vi.fn(),
}));
const tempDirs: string[] = [];
afterEach(() => {
vi.mocked(note).mockReset();
cleanupTrackedTempDirs(tempDirs);
});
function makeTempDir() {
return makeTrackedTempDir("openclaw-doctor-plugin-registry", tempDirs);
}
function hermeticEnv(overrides: NodeJS.ProcessEnv = {}): NodeJS.ProcessEnv {
return {
OPENCLAW_BUNDLED_PLUGINS_DIR: undefined,
OPENCLAW_DISABLE_PLUGIN_DISCOVERY_CACHE: "1",
OPENCLAW_DISABLE_PLUGIN_MANIFEST_CACHE: "1",
OPENCLAW_VERSION: "2026.4.25",
VITEST: "true",
...overrides,
};
}
function createCandidate(rootDir: string, id = "demo"): PluginCandidate {
fs.writeFileSync(
path.join(rootDir, "index.ts"),
"throw new Error('runtime entry should not load during doctor registry repair');\n",
"utf8",
);
fs.writeFileSync(
path.join(rootDir, "openclaw.plugin.json"),
JSON.stringify({
id,
name: id,
configSchema: { type: "object" },
providers: [id],
}),
"utf8",
);
return {
idHint: id,
source: path.join(rootDir, "index.ts"),
rootDir,
origin: "global",
};
}
function createCurrentIndex(): InstalledPluginIndex {
return {
version: 1,
hostContractVersion: "2026.4.25",
compatRegistryVersion: "compat-v1",
migrationVersion: 2,
policyHash: "policy-v1",
generatedAtMs: 1777118400000,
plugins: [],
diagnostics: [],
};
}
describe("maybeRepairPluginRegistryState", () => {
it("reports legacy config install records without mutating state outside repair mode", async () => {
const stateDir = makeTempDir();
const nextConfig = await maybeRepairPluginRegistryState({
stateDir,
env: hermeticEnv(),
config: {
plugins: {
installs: {
demo: {
source: "npm",
resolvedName: "@vendor/demo",
},
},
},
},
prompter: { shouldRepair: false },
});
expect(nextConfig.plugins?.installs?.demo?.resolvedName).toBe("@vendor/demo");
await expect(readPersistedPluginInstallLedger({ stateDir })).resolves.toBeNull();
expect(vi.mocked(note).mock.calls.join("\n")).toContain("plugins.installs");
});
it("moves legacy config install records into the ledger and refreshes an existing registry", async () => {
const stateDir = makeTempDir();
const pluginDir = path.join(stateDir, "plugins", "demo");
fs.mkdirSync(pluginDir, { recursive: true });
await writePersistedInstalledPluginIndex(createCurrentIndex(), { stateDir });
const nextConfig = await maybeRepairPluginRegistryState({
stateDir,
candidates: [createCandidate(pluginDir)],
env: hermeticEnv(),
config: {
plugins: {
installs: {
demo: {
source: "npm",
resolvedName: "@vendor/demo",
resolvedVersion: "1.0.0",
},
},
},
},
prompter: { shouldRepair: true },
});
expect(nextConfig.plugins?.installs).toBeUndefined();
await expect(readPersistedPluginInstallLedger({ stateDir })).resolves.toMatchObject({
records: {
demo: {
source: "npm",
resolvedName: "@vendor/demo",
resolvedVersion: "1.0.0",
},
},
});
await expect(readPersistedInstalledPluginIndex({ stateDir })).resolves.toMatchObject({
refreshReason: "migration",
plugins: [
expect.objectContaining({
pluginId: "demo",
installRecord: expect.objectContaining({
source: "npm",
resolvedName: "@vendor/demo",
}),
}),
],
});
});
});

Some files were not shown because too many files have changed in this diff Show More