Replace Sharp image backend with Photon (#86437)

* refactor: replace sharp image backend with photon

* refactor: remove whatsapp jimp dependency

* chore: remove stale sharp install workarounds

* test: keep image fixtures off photon

* test: use valid prompt image fixtures

* test: account for optimized PNG fixtures

* test: use valid minimax image fixtures
This commit is contained in:
Peter Steinberger
2026-05-25 15:04:44 +01:00
committed by GitHub
parent 32ddfc22f5
commit b9f975b64e
48 changed files with 1424 additions and 3000 deletions

View File

@@ -80,7 +80,6 @@ Skills own workflows; root owns hard policy and routing.
- Runtime: Node 22.19+; Node 24 recommended. Keep Node + Bun paths working.
- Package manager/runtime: repo defaults only. No swaps without approval.
- Install: `pnpm install` (keep Bun lock/patches aligned if touched).
- Sharp/Homebrew libvips source-build fail: `SHARP_IGNORE_GLOBAL_LIBVIPS=1 pnpm install`.
- CLI: `pnpm openclaw ...` or `pnpm dev`; build: `pnpm build`.
- Tests in a normal source checkout: `pnpm test <path-or-filter> [vitest args...]`, `pnpm test:changed`, `pnpm test:serial`, `pnpm test:coverage`; never raw `vitest`.
- Tests in a Codex worktree or linked/sparse checkout: avoid direct local `pnpm test*`; use `node scripts/run-vitest.mjs <path-or-filter>` for tiny explicit-file proof, or Crabbox/Testbox for anything broader.

View File

@@ -11,6 +11,7 @@ Docs: https://docs.openclaw.ai
- Build: include `ui:build` in the `full` and `ciArtifacts` profiles of `scripts/build-all.mjs` so `pnpm build` always rebuilds `dist/control-ui` after `tsdown` cleans `dist`, removing the second-command requirement and the missing-asset failure mode for source/runtime installs and CI artifact uploads. (#85206)
- Migrate: import supported Hermes, OpenCode, and Codex auth credentials into OpenClaw auth profiles when credential migration is selected, with explicit opt-out and non-interactive controls. (#85667) Thanks @fuller-stack-dev.
- iOS: improve Talk mode with direct realtime voice sessions, compact toolbar status, and responsive voice waveform feedback. (#86355) Thanks @ngutman.
- Media: replace the Sharp image backend with Photon for metadata, resizing, EXIF orientation, and PNG alpha-preserving optimization so OpenClaw no longer installs Sharp or the WhatsApp Jimp fallback for image processing. (#86437)
### Fixes

View File

@@ -98,7 +98,7 @@ These are frequently reported but are typically closed with no code change:
- Reports that treat `POST /tools/invoke` under shared-secret bearer auth (`gateway.auth.mode="token"` or `"password"`) as a narrower per-request/per-scope authorization surface. That endpoint is designed as the same trusted-operator HTTP boundary: shared-secret bearer auth is full operator access there, narrower `x-openclaw-scopes` values do not reduce that path, and owner-only tool policy follows the shared-secret operator contract.
- Reports that only show differences in heuristic detection/parity (for example obfuscation-pattern detection on one exec path but not another, such as `node.invoke -> system.run` parity gaps) without demonstrating bypass of auth, approvals, allowlist enforcement, sandboxing, or other documented trust boundaries.
- Reports that only show an ACP tool can indirectly execute, mutate, orchestrate sessions, or reach another tool/runtime without demonstrating bypass of ACP prompt/approval, allowlist enforcement, sandboxing, or another documented trust boundary. ACP silent approval is intentionally limited to narrow readonly classes; parity-only indirect-command findings are hardening, not vulnerabilities.
- Reports that only show untrusted media bytes reaching a maintained native decoder dependency (for example Sharp/libvips/libheif) without proving the shipped dependency version is vulnerable and demonstrating crash, memory corruption, data exposure, or a boundary bypass through OpenClaw. JavaScript header sniffing and image dimension fast-paths are preflight/UX checks, not the security boundary for native decoder correctness.
- Reports that only show untrusted media bytes reaching a maintained native decoder dependency (for example image codec libraries such as libheif) without proving the shipped dependency version is vulnerable and demonstrating crash, memory corruption, data exposure, or a boundary bypass through OpenClaw. JavaScript header sniffing and image dimension fast-paths are preflight/UX checks, not the security boundary for native decoder correctness.
- Reports whose only impact is transient extra memory, CPU, or allocation work from decoding, base64 expansion, media transcoding, serialization, or other format conversion after the input was already accepted under OpenClaw's configured size/trust limits, including base64 decode-before-size-estimate findings. These are performance issues, not vulnerabilities, unless the report demonstrates unauthenticated amplification, bypass of configured limits, crash/process termination, persistent resource exhaustion, data exposure, or another documented boundary bypass.
- ReDoS/DoS claims that require trusted operator configuration input (for example catastrophic regex in `sessionFilter` or `logging.redactPatterns`) without a trust-boundary bypass.
- Archive/install extraction claims that require pre-existing local filesystem priming in trusted state (for example planting symlink/hardlink aliases under destination directories such as skills/tools paths) without showing an untrusted path that can create/control that primitive.

View File

@@ -64,6 +64,7 @@ const rootBundledPluginRuntimeDependencies = [
"@grammyjs/transformer-throttler",
"@homebridge/ciao",
"@mozilla/readability",
"@silvia-odwyer/photon-node",
"@slack/bolt",
"@slack/types",
"@slack/web-api",

View File

@@ -261,7 +261,7 @@ Defaults when omitted:
- `images.maxBytes`: 10MB
- `images.maxRedirects`: 3
- `images.timeoutMs`: 10s
- HEIC/HEIF `input_image` sources are accepted and normalized to JPEG before provider delivery.
- HEIC/HEIF `input_image` sources are accepted when a system converter is available and are normalized to JPEG before provider delivery. Supported converters are macOS `sips`, ImageMagick, GraphicsMagick, or ffmpeg.
Security note:

View File

@@ -108,15 +108,6 @@ If you already manage Node yourself:
</Tab>
</Tabs>
<Accordion title="Troubleshooting: sharp build errors (npm)">
If `sharp` fails due to a globally installed libvips:
```bash
SHARP_IGNORE_GLOBAL_LIBVIPS=1 npm install -g openclaw@latest
```
</Accordion>
### From source
For contributors or anyone who wants to run from a local checkout:

View File

@@ -85,7 +85,6 @@ Recommended for most interactive installs on macOS/Linux/WSL.
- Refreshes a loaded gateway service best-effort (`openclaw gateway install --force`, then restart)
- Runs `openclaw doctor --non-interactive` on upgrades and git installs (best effort)
- Attempts onboarding when appropriate (TTY available, onboarding not disabled, and bootstrap/config checks pass)
- Defaults `SHARP_IGNORE_GLOBAL_LIBVIPS=1`
</Step>
</Steps>
@@ -167,7 +166,6 @@ The script exits with code `2` for invalid method selection or invalid `--instal
| `OPENCLAW_DRY_RUN=1` | Dry run mode |
| `OPENCLAW_VERBOSE=1` | Debug mode |
| `OPENCLAW_NPM_LOGLEVEL=error\|warn\|notice` | npm log level |
| `SHARP_IGNORE_GLOBAL_LIBVIPS=0\|1` | Control sharp/libvips behavior (default: `1`) |
</Accordion>
</AccordionGroup>
@@ -269,7 +267,6 @@ by default, plus git-checkout installs under the same prefix flow.
| `OPENCLAW_GIT_UPDATE=0\|1` | Toggle git updates for existing checkouts |
| `OPENCLAW_NO_ONBOARD=1` | Skip onboarding |
| `OPENCLAW_NPM_LOGLEVEL=error\|warn\|notice` | npm log level |
| `SHARP_IGNORE_GLOBAL_LIBVIPS=0\|1` | Control sharp/libvips behavior (default: `1`) |
</Accordion>
</AccordionGroup>
@@ -417,15 +414,6 @@ Use non-interactive flags/env vars for predictable runs.
Some Linux setups point npm global prefix to root-owned paths. `install.sh` can switch prefix to `~/.npm-global` and append PATH exports to shell rc files (when those files exist).
</Accordion>
<Accordion title="sharp/libvips issues">
The scripts default `SHARP_IGNORE_GLOBAL_LIBVIPS=1` to avoid sharp building against system libvips. To override:
```bash
SHARP_IGNORE_GLOBAL_LIBVIPS=0 curl -fsSL --proto '=https' --tlsv1.2 https://openclaw.ai/install.sh | bash
```
</Accordion>
<Accordion title='Windows: "npm error spawn git / ENOENT"'>
Rerun the installer so it can bootstrap user-local MinGit, or install Git for Windows and reopen PowerShell.
</Accordion>

View File

@@ -1,4 +1,6 @@
import sharp from "sharp";
import fs from "node:fs/promises";
import { getImageMetadata } from "openclaw/plugin-sdk/media-runtime";
import { createSolidPngBuffer } from "openclaw/plugin-sdk/test-fixtures";
import { describe, expect, it } from "vitest";
import { normalizeBrowserScreenshot } from "./screenshot.js";
@@ -20,16 +22,7 @@ describe("browser screenshot normalization", () => {
}
it("shrinks oversized images to <=2000x2000 and <=5MB", async () => {
const bigPng = await sharp({
create: {
width: 2100,
height: 2100,
channels: 3,
background: { r: 12, g: 34, b: 56 },
},
})
.png({ compressionLevel: 0 })
.toBuffer();
const bigPng = createSolidPngBuffer(2100, 2100, { r: 12, g: 34, b: 56 });
const normalized = await normalizeBrowserScreenshot(bigPng, {
maxSide: 2000,
@@ -37,24 +30,15 @@ describe("browser screenshot normalization", () => {
});
expect(normalized.buffer.byteLength).toBeLessThanOrEqual(5 * 1024 * 1024);
const meta = await sharp(normalized.buffer).metadata();
expect(meta.width).toBeLessThanOrEqual(2000);
expect(meta.height).toBeLessThanOrEqual(2000);
const meta = await getImageMetadata(normalized.buffer);
expect(meta?.width).toBeLessThanOrEqual(2000);
expect(meta?.height).toBeLessThanOrEqual(2000);
expect(normalized.buffer[0]).toBe(0xff);
expect(normalized.buffer[1]).toBe(0xd8);
}, 120_000);
it("keeps already-small screenshots unchanged", async () => {
const jpeg = await sharp({
create: {
width: 800,
height: 600,
channels: 3,
background: { r: 255, g: 0, b: 0 },
},
})
.jpeg({ quality: 80 })
.toBuffer();
const jpeg = await fs.readFile("docs/assets/showcase/roof-camera-sky.jpg");
const normalized = await normalizeBrowserScreenshot(jpeg, {
maxSide: 2000,
@@ -65,16 +49,7 @@ describe("browser screenshot normalization", () => {
});
it("rejects screenshots above max side when no image processor is available", async () => {
const png = await sharp({
create: {
width: 420,
height: 120,
channels: 3,
background: { r: 12, g: 34, b: 56 },
},
})
.png({ compressionLevel: 9 })
.toBuffer();
const png = createSolidPngBuffer(420, 120, { r: 12, g: 34, b: 56 });
expect(png.byteLength).toBeLessThan(5 * 1024 * 1024);
await withUnavailableImageBackend(async () => {

View File

@@ -1,7 +1,8 @@
import { deflateSync, inflateSync } from "node:zlib";
import type { ImageMetadata } from "openclaw/plugin-sdk/media-runtime";
import type sharpImport from "sharp";
type SharpFactory = typeof sharpImport;
type PhotonModule = typeof import("@silvia-odwyer/photon-node");
type PhotonImage = InstanceType<PhotonModule["PhotonImage"]>;
type ResizeToJpegParams = {
buffer: Buffer;
@@ -21,45 +22,33 @@ type MediaUnderstandingImageOpsOptions = {
maxInputPixels: number;
};
const SHARP_MODULE = "sharp";
let photonPromise: Promise<PhotonModule> | null = null;
let sharpFactoryPromise: Promise<SharpFactory> | null = null;
function normalizeSharpFactory(mod: unknown): SharpFactory {
const candidates = [
(mod as { default?: unknown }).default,
((mod as { default?: { default?: unknown } }).default ?? {})?.default,
mod,
];
const sharp = candidates.find(
(candidate): candidate is SharpFactory => typeof candidate === "function",
);
if (!sharp) {
throw new Error("Optional dependency sharp did not expose an image processor");
const PNG_SIGNATURE = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
const CRC_TABLE = (() => {
const table = new Uint32Array(256);
for (let index = 0; index < table.length; index += 1) {
let value = index;
for (let bit = 0; bit < 8; bit += 1) {
value = value & 1 ? 0xedb88320 ^ (value >>> 1) : value >>> 1;
}
return sharp;
}
async function loadSharp(maxInputPixels: number): Promise<SharpFactory> {
if (!sharpFactoryPromise) {
sharpFactoryPromise = import(SHARP_MODULE)
.then((mod) => {
const sharp = normalizeSharpFactory(mod);
return ((buffer, options) =>
sharp(buffer, {
...options,
failOnError: false,
limitInputPixels: maxInputPixels,
})) as SharpFactory;
})
.catch((err) => {
sharpFactoryPromise = null;
throw new Error("Optional dependency sharp is required for image attachment processing", {
cause: err,
});
});
table[index] = value >>> 0;
}
return await sharpFactoryPromise;
return table;
})();
async function loadPhoton(): Promise<PhotonModule> {
photonPromise ??= import("@silvia-odwyer/photon-node").then((mod) => {
if (
typeof mod.PhotonImage?.new_from_byteslice !== "function" ||
typeof mod.resize !== "function" ||
mod.SamplingFilter?.Lanczos3 === undefined
) {
throw new Error("Photon did not expose the required image processor API");
}
return mod;
});
return await photonPromise;
}
function normalizeMaxInputPixels(value: number): number {
@@ -69,69 +58,604 @@ function normalizeMaxInputPixels(value: number): number {
return value;
}
function normalizeMetadata(meta: { width?: number; height?: number }): ImageMetadata | null {
const width = meta.width ?? 0;
const height = meta.height ?? 0;
if (!Number.isFinite(width) || !Number.isFinite(height)) {
return null;
}
if (width <= 0 || height <= 0) {
function normalizeMetadata(width: number, height: number): ImageMetadata | null {
if (!Number.isInteger(width) || !Number.isInteger(height) || width <= 0 || height <= 0) {
return null;
}
return { width, height };
}
function readPngMetadata(buffer: Buffer): ImageMetadata | null {
if (
buffer.length < 24 ||
!buffer.subarray(0, PNG_SIGNATURE.length).equals(PNG_SIGNATURE) ||
buffer.toString("ascii", 12, 16) !== "IHDR"
) {
return null;
}
return normalizeMetadata(buffer.readUInt32BE(16), buffer.readUInt32BE(20));
}
function readGifMetadata(buffer: Buffer): ImageMetadata | null {
if (buffer.length < 10) {
return null;
}
const signature = buffer.toString("ascii", 0, 6);
if (signature !== "GIF87a" && signature !== "GIF89a") {
return null;
}
return normalizeMetadata(buffer.readUInt16LE(6), buffer.readUInt16LE(8));
}
function readWebpMetadata(buffer: Buffer): ImageMetadata | null {
if (
buffer.length < 30 ||
buffer.toString("ascii", 0, 4) !== "RIFF" ||
buffer.toString("ascii", 8, 12) !== "WEBP"
) {
return null;
}
const chunkType = buffer.toString("ascii", 12, 16);
if (chunkType === "VP8X") {
return normalizeMetadata(1 + buffer.readUIntLE(24, 3), 1 + buffer.readUIntLE(27, 3));
}
if (chunkType === "VP8 ") {
return normalizeMetadata(buffer.readUInt16LE(26) & 0x3fff, buffer.readUInt16LE(28) & 0x3fff);
}
if (chunkType === "VP8L") {
if (buffer.length < 25 || buffer[20] !== 0x2f) {
return null;
}
const bits = buffer[21] | (buffer[22] << 8) | (buffer[23] << 16) | (buffer[24] << 24);
return normalizeMetadata((bits & 0x3fff) + 1, ((bits >> 14) & 0x3fff) + 1);
}
return null;
}
function readJpegMetadata(buffer: Buffer): ImageMetadata | null {
if (buffer.length < 4 || buffer[0] !== 0xff || buffer[1] !== 0xd8) {
return null;
}
let offset = 2;
while (offset + 8 < buffer.length) {
while (offset < buffer.length && buffer[offset] === 0xff) {
offset += 1;
}
if (offset >= buffer.length) {
return null;
}
const marker = buffer[offset];
offset += 1;
if (marker === 0xd8 || marker === 0xd9) {
continue;
}
if (marker === 0x01 || (marker >= 0xd0 && marker <= 0xd7)) {
continue;
}
if (offset + 1 >= buffer.length) {
return null;
}
const segmentLength = buffer.readUInt16BE(offset);
if (segmentLength < 2 || offset + segmentLength > buffer.length) {
return null;
}
const isStartOfFrame =
marker >= 0xc0 && marker <= 0xcf && marker !== 0xc4 && marker !== 0xc8 && marker !== 0xcc;
if (isStartOfFrame) {
if (segmentLength < 7 || offset + 6 >= buffer.length) {
return null;
}
return normalizeMetadata(buffer.readUInt16BE(offset + 5), buffer.readUInt16BE(offset + 3));
}
offset += segmentLength;
}
return null;
}
function readImageMetadataFromHeader(buffer: Buffer): ImageMetadata | null {
return (
readPngMetadata(buffer) ??
readGifMetadata(buffer) ??
readWebpMetadata(buffer) ??
readJpegMetadata(buffer)
);
}
function crc32(buffer: Buffer): number {
let crc = 0xffffffff;
for (const byte of buffer) {
crc = CRC_TABLE[(crc ^ byte) & 0xff] ^ (crc >>> 8);
}
return (crc ^ 0xffffffff) >>> 0;
}
function pngChunk(type: string, data: Buffer): Buffer {
const typeBuffer = Buffer.from(type, "ascii");
const length = Buffer.alloc(4);
length.writeUInt32BE(data.length, 0);
const crc = Buffer.alloc(4);
crc.writeUInt32BE(crc32(Buffer.concat([typeBuffer, data])), 0);
return Buffer.concat([length, typeBuffer, data, crc]);
}
function encodePngRgba(
pixels: Uint8Array,
width: number,
height: number,
compressionLevel = 6,
): Buffer {
const stride = width * 4;
const raw = Buffer.alloc((stride + 1) * height);
const source = Buffer.from(pixels.buffer, pixels.byteOffset, pixels.byteLength);
for (let row = 0; row < height; row += 1) {
const rawOffset = row * (stride + 1);
raw[rawOffset] = 0;
source.copy(raw, rawOffset + 1, row * stride, row * stride + stride);
}
const ihdr = Buffer.alloc(13);
ihdr.writeUInt32BE(width, 0);
ihdr.writeUInt32BE(height, 4);
ihdr[8] = 8;
ihdr[9] = 6;
ihdr[10] = 0;
ihdr[11] = 0;
ihdr[12] = 0;
return Buffer.concat([
PNG_SIGNATURE,
pngChunk("IHDR", ihdr),
pngChunk(
"IDAT",
deflateSync(raw, { level: Math.max(0, Math.min(9, Math.round(compressionLevel))) }),
),
pngChunk("IEND", Buffer.alloc(0)),
]);
}
function paethPredictor(left: number, up: number, upperLeft: number): number {
const prediction = left + up - upperLeft;
const distanceLeft = Math.abs(prediction - left);
const distanceUp = Math.abs(prediction - up);
const distanceUpperLeft = Math.abs(prediction - upperLeft);
if (distanceLeft <= distanceUp && distanceLeft <= distanceUpperLeft) {
return left;
}
return distanceUp <= distanceUpperLeft ? up : upperLeft;
}
function unfilterPngScanlines(
inflated: Buffer,
width: number,
height: number,
bytesPerPixel: number,
): Buffer | null {
const stride = width * bytesPerPixel;
if (inflated.length !== (stride + 1) * height) {
return null;
}
const out = Buffer.alloc(stride * height);
for (let row = 0; row < height; row += 1) {
const filter = inflated[row * (stride + 1)];
const sourceOffset = row * (stride + 1) + 1;
const targetOffset = row * stride;
for (let column = 0; column < stride; column += 1) {
const raw = inflated[sourceOffset + column] ?? 0;
const left = column >= bytesPerPixel ? (out[targetOffset + column - bytesPerPixel] ?? 0) : 0;
const up = row > 0 ? (out[targetOffset + column - stride] ?? 0) : 0;
const upperLeft =
row > 0 && column >= bytesPerPixel
? (out[targetOffset + column - stride - bytesPerPixel] ?? 0)
: 0;
let value: number;
switch (filter) {
case 0:
value = raw;
break;
case 1:
value = raw + left;
break;
case 2:
value = raw + up;
break;
case 3:
value = raw + Math.floor((left + up) / 2);
break;
case 4:
value = raw + paethPredictor(left, up, upperLeft);
break;
default:
return null;
}
out[targetOffset + column] = value & 0xff;
}
}
return out;
}
function decodeGrayscaleAlphaPng(buffer: Buffer): {
pixels: Uint8Array;
width: number;
height: number;
} | null {
if (buffer.length < 33 || !buffer.subarray(0, PNG_SIGNATURE.length).equals(PNG_SIGNATURE)) {
return null;
}
let width = 0;
let height = 0;
const idatChunks: Buffer[] = [];
for (let offset = 8; offset + 12 <= buffer.length; ) {
const length = buffer.readUInt32BE(offset);
const type = buffer.toString("ascii", offset + 4, offset + 8);
const dataStart = offset + 8;
const dataEnd = dataStart + length;
if (dataEnd + 4 > buffer.length) {
return null;
}
const data = buffer.subarray(dataStart, dataEnd);
if (type === "IHDR") {
if (
length !== 13 ||
data[8] !== 8 ||
data[9] !== 4 ||
data[10] !== 0 ||
data[11] !== 0 ||
data[12] !== 0
) {
return null;
}
width = data.readUInt32BE(0);
height = data.readUInt32BE(4);
} else if (type === "IDAT") {
idatChunks.push(data);
} else if (type === "IEND") {
break;
}
offset = dataEnd + 4;
}
const metadata = normalizeMetadata(width, height);
if (!metadata || idatChunks.length === 0) {
return null;
}
const expectedInflatedLength = (width * 2 + 1) * height;
const grayAlpha = unfilterPngScanlines(
inflateSync(Buffer.concat(idatChunks), { maxOutputLength: expectedInflatedLength }),
width,
height,
2,
);
if (!grayAlpha) {
return null;
}
const pixels = new Uint8Array(width * height * 4);
for (let source = 0, target = 0; source < grayAlpha.length; source += 2, target += 4) {
const gray = grayAlpha[source] ?? 0;
pixels[target] = gray;
pixels[target + 1] = gray;
pixels[target + 2] = gray;
pixels[target + 3] = grayAlpha[source + 1] ?? 255;
}
return { pixels, width, height };
}
function assertDecodedPixelBudget(image: PhotonImage, maxInputPixels: number): void {
const width = image.get_width();
const height = image.get_height();
if (width > Math.floor(maxInputPixels / height)) {
throw new Error(
`Image dimensions exceed the ${maxInputPixels.toLocaleString("en-US")} pixel input limit: ${width}x${height}`,
);
}
}
function assertHeaderPixelBudget(buffer: Buffer, maxInputPixels: number): void {
const meta = readImageMetadataFromHeader(buffer);
if (!meta) {
throw new Error("Unable to determine image dimensions; refusing to process");
}
if (meta.width > Math.floor(maxInputPixels / meta.height)) {
throw new Error(
`Image dimensions exceed the ${maxInputPixels.toLocaleString("en-US")} pixel input limit: ${meta.width}x${meta.height}`,
);
}
}
function readJpegExifOrientation(buffer: Buffer): number | null {
if (buffer.length < 4 || buffer[0] !== 0xff || buffer[1] !== 0xd8) {
return null;
}
let offset = 2;
while (offset + 4 < buffer.length) {
if (buffer[offset] !== 0xff) {
offset += 1;
continue;
}
const marker = buffer[offset + 1];
if (marker === 0xff) {
offset += 1;
continue;
}
if (marker === 0xda || marker === 0xd9) {
return null;
}
if (offset + 4 > buffer.length) {
return null;
}
const segmentLength = buffer.readUInt16BE(offset + 2);
if (segmentLength < 2 || offset + 2 + segmentLength > buffer.length) {
return null;
}
if (
marker === 0xe1 &&
segmentLength >= 14 &&
buffer.toString("ascii", offset + 4, offset + 8) === "Exif" &&
buffer[offset + 8] === 0 &&
buffer[offset + 9] === 0
) {
return readExifOrientationFromTiff(buffer, offset + 10, offset + 2 + segmentLength);
}
offset += 2 + segmentLength;
}
return null;
}
function readExifOrientationFromTiff(
buffer: Buffer,
tiffStart: number,
tiffEnd: number,
): number | null {
if (tiffStart + 8 > tiffEnd) {
return null;
}
const byteOrder = buffer.toString("ascii", tiffStart, tiffStart + 2);
const littleEndian = byteOrder === "II";
if (!littleEndian && byteOrder !== "MM") {
return null;
}
const readU16 = (offset: number) =>
littleEndian ? buffer.readUInt16LE(offset) : buffer.readUInt16BE(offset);
const readU32 = (offset: number) =>
littleEndian ? buffer.readUInt32LE(offset) : buffer.readUInt32BE(offset);
if (readU16(tiffStart + 2) !== 42) {
return null;
}
const ifd0Start = tiffStart + readU32(tiffStart + 4);
if (ifd0Start + 2 > tiffEnd) {
return null;
}
const entries = readU16(ifd0Start);
for (let index = 0; index < entries; index += 1) {
const entryOffset = ifd0Start + 2 + index * 12;
if (entryOffset + 12 > tiffEnd) {
return null;
}
if (readU16(entryOffset) === 0x0112) {
const orientation = readU16(entryOffset + 8);
return orientation >= 1 && orientation <= 8 ? orientation : null;
}
}
return null;
}
function transformOrientation(
rawPixels: Uint8Array,
width: number,
height: number,
orientation: number,
): { pixels: Uint8Array; width: number; height: number } {
if (orientation === 1) {
return { pixels: rawPixels, width, height };
}
const swapsAxes =
orientation === 5 || orientation === 6 || orientation === 7 || orientation === 8;
const outputWidth = swapsAxes ? height : width;
const outputHeight = swapsAxes ? width : height;
const out = new Uint8Array(outputWidth * outputHeight * 4);
for (let y = 0; y < height; y += 1) {
for (let x = 0; x < width; x += 1) {
let targetX = x;
let targetY = y;
switch (orientation) {
case 2:
targetX = width - 1 - x;
break;
case 3:
targetX = width - 1 - x;
targetY = height - 1 - y;
break;
case 4:
targetY = height - 1 - y;
break;
case 5:
targetX = y;
targetY = x;
break;
case 6:
targetX = height - 1 - y;
targetY = x;
break;
case 7:
targetX = height - 1 - y;
targetY = width - 1 - x;
break;
case 8:
targetX = y;
targetY = width - 1 - x;
break;
}
const sourceOffset = (y * width + x) * 4;
const targetOffset = (targetY * outputWidth + targetX) * 4;
out[targetOffset] = rawPixels[sourceOffset] ?? 0;
out[targetOffset + 1] = rawPixels[sourceOffset + 1] ?? 0;
out[targetOffset + 2] = rawPixels[sourceOffset + 2] ?? 0;
out[targetOffset + 3] = rawPixels[sourceOffset + 3] ?? 255;
}
}
return { pixels: out, width: outputWidth, height: outputHeight };
}
function applyExifOrientation(
photon: PhotonModule,
image: PhotonImage,
buffer: Buffer,
): PhotonImage {
const orientation = readJpegExifOrientation(buffer);
if (!orientation || orientation === 1) {
return image;
}
const transformed = transformOrientation(
image.get_raw_pixels(),
image.get_width(),
image.get_height(),
orientation,
);
image.free();
return new photon.PhotonImage(transformed.pixels, transformed.width, transformed.height);
}
function targetSize(
image: PhotonImage,
maxSide: number,
withoutEnlargement: boolean,
): { width: number; height: number } {
const width = image.get_width();
const height = image.get_height();
const maxDimension = Math.max(width, height);
if (maxDimension <= 0) {
throw new Error("Invalid image dimensions");
}
const requestedScale = maxSide / maxDimension;
const scale = withoutEnlargement ? Math.min(1, requestedScale) : requestedScale;
return {
width: Math.max(1, Math.round(width * scale)),
height: Math.max(1, Math.round(height * scale)),
};
}
function resizeImage(
photon: PhotonModule,
image: PhotonImage,
params: ResizeToJpegParams | ResizeToPngParams,
): PhotonImage {
const size = targetSize(image, params.maxSide, params.withoutEnlargement !== false);
if (size.width === image.get_width() && size.height === image.get_height()) {
return image;
}
const resized = photon.resize(image, size.width, size.height, photon.SamplingFilter.Lanczos3);
image.free();
return resized;
}
async function loadOrientedPhotonImage(
buffer: Buffer,
maxInputPixels: number,
): Promise<{ photon: PhotonModule; image: PhotonImage }> {
assertHeaderPixelBudget(buffer, maxInputPixels);
const photon = await loadPhoton();
let decoded: PhotonImage;
try {
decoded = photon.PhotonImage.new_from_byteslice(buffer);
} catch (err) {
const grayscaleAlpha = decodeGrayscaleAlphaPng(buffer);
if (!grayscaleAlpha) {
throw err;
}
decoded = new photon.PhotonImage(
grayscaleAlpha.pixels,
grayscaleAlpha.width,
grayscaleAlpha.height,
);
}
assertDecodedPixelBudget(decoded, maxInputPixels);
return { photon, image: applyExifOrientation(photon, decoded, buffer) };
}
export function createMediaAttachmentImageOps(options: MediaUnderstandingImageOpsOptions) {
const maxInputPixels = normalizeMaxInputPixels(options.maxInputPixels);
return {
async getImageMetadata(buffer: Buffer): Promise<ImageMetadata | null> {
const sharp = await loadSharp(maxInputPixels);
return normalizeMetadata(await sharp(buffer).metadata());
const { image } = await loadOrientedPhotonImage(buffer, maxInputPixels);
try {
return normalizeMetadata(image.get_width(), image.get_height());
} finally {
image.free();
}
},
async normalizeExifOrientation(buffer: Buffer): Promise<Buffer> {
const sharp = await loadSharp(maxInputPixels);
return await sharp(buffer).rotate().toBuffer();
const orientation = readJpegExifOrientation(buffer);
if (!orientation || orientation === 1) {
return buffer;
}
const { image } = await loadOrientedPhotonImage(buffer, maxInputPixels);
try {
return Buffer.from(image.get_bytes_jpeg(90));
} finally {
image.free();
}
},
async resizeToJpeg(params: ResizeToJpegParams): Promise<Buffer> {
const sharp = await loadSharp(maxInputPixels);
return await sharp(params.buffer)
.rotate()
.resize({
width: params.maxSide,
height: params.maxSide,
fit: "inside",
withoutEnlargement: params.withoutEnlargement !== false,
})
.jpeg({ quality: params.quality, mozjpeg: true })
.toBuffer();
const { photon, image } = await loadOrientedPhotonImage(params.buffer, maxInputPixels);
const resized = resizeImage(photon, image, params);
try {
return Buffer.from(resized.get_bytes_jpeg(params.quality));
} finally {
resized.free();
}
},
async convertHeicToJpeg(buffer: Buffer): Promise<Buffer> {
const sharp = await loadSharp(maxInputPixels);
return await sharp(buffer).jpeg({ quality: 90, mozjpeg: true }).toBuffer();
async convertHeicToJpeg(_buffer: Buffer): Promise<Buffer> {
throw new Error("Photon does not support HEIC/AVIF conversion");
},
async hasAlphaChannel(buffer: Buffer): Promise<boolean> {
const sharp = await loadSharp(maxInputPixels);
const meta = await sharp(buffer).metadata();
return meta.hasAlpha || meta.channels === 4;
const { image } = await loadOrientedPhotonImage(buffer, maxInputPixels);
try {
const pixels = image.get_raw_pixels();
for (let offset = 3; offset < pixels.length; offset += 4) {
if ((pixels[offset] ?? 255) < 255) {
return true;
}
}
return false;
} finally {
image.free();
}
},
async resizeToPng(params: ResizeToPngParams): Promise<Buffer> {
const sharp = await loadSharp(maxInputPixels);
const compressionLevel = params.compressionLevel ?? 6;
return await sharp(params.buffer)
.rotate()
.resize({
width: params.maxSide,
height: params.maxSide,
fit: "inside",
withoutEnlargement: params.withoutEnlargement !== false,
})
.png({ compressionLevel })
.toBuffer();
const { photon, image } = await loadOrientedPhotonImage(params.buffer, maxInputPixels);
const resized = resizeImage(photon, image, params);
try {
return encodePngRgba(
resized.get_raw_pixels(),
resized.get_width(),
resized.get_height(),
params.compressionLevel,
);
} finally {
resized.free();
}
},
};
}

View File

@@ -5,7 +5,7 @@
"description": "OpenClaw media understanding runtime package",
"type": "module",
"dependencies": {
"sharp": "0.34.5"
"@silvia-odwyer/photon-node": "0.3.4"
},
"devDependencies": {
"@openclaw/plugin-sdk": "workspace:*"

File diff suppressed because it is too large Load Diff

View File

@@ -11,7 +11,6 @@
"audio-decode": "2.2.3",
"baileys": "7.0.0-rc13",
"https-proxy-agent": "9.0.0",
"jimp": "1.6.1",
"typebox": "1.1.38"
},
"devDependencies": {

Binary file not shown.

After

Width:  |  Height:  |  Size: 634 KiB

View File

@@ -1,5 +1,5 @@
import crypto from "node:crypto";
import sharp from "sharp";
import fs from "node:fs/promises";
import { createNoisyPngBuffer, createSolidPngBuffer } from "openclaw/plugin-sdk/test-fixtures";
import { beforeAll, describe, expect, it, vi } from "vitest";
import {
createMockWebListener,
@@ -151,47 +151,33 @@ describe("web auto-reply", () => {
}
it("compresses common formats to jpeg under the cap", async () => {
const jpeg = await fs.readFile("docs/assets/showcase/roof-camera-sky.jpg");
const webp = await fs.readFile("extensions/whatsapp/src/__fixtures__/large-noisy.webp");
const formats = [
{
name: "png",
mime: "image/png",
make: (buf: Buffer, opts: { width: number; height: number }) =>
sharp(buf, {
raw: { width: opts.width, height: opts.height, channels: 3 },
})
.png({ compressionLevel: 0 })
.toBuffer(),
make: (opts: { width: number; height: number }) =>
Promise.resolve(createNoisyPngBuffer(opts.width, opts.height)),
},
{
name: "jpeg",
mime: "image/jpeg",
make: (buf: Buffer, opts: { width: number; height: number }) =>
sharp(buf, {
raw: { width: opts.width, height: opts.height, channels: 3 },
})
// Keep source > cap with fewer pixels so the test runs faster.
.jpeg({ quality: 100, chromaSubsampling: "4:4:4" })
.toBuffer(),
make: () => Promise.resolve(jpeg),
},
{
name: "webp",
mime: "image/webp",
make: (buf: Buffer, opts: { width: number; height: number }) =>
sharp(buf, {
raw: { width: opts.width, height: opts.height, channels: 3 },
})
.webp({ quality: 100 })
.toBuffer(),
make: () => Promise.resolve(webp),
},
] as const;
const width = 320;
const height = 320;
const sharedRaw = crypto.randomBytes(width * height * 3);
const width = 800;
const height = 800;
const renderedFormats = await Promise.all(
formats.map(async (fmt) =>
Object.assign({}, fmt, { image: await fmt.make(sharedRaw, { width, height }) }),
Object.assign({}, fmt, { image: await fmt.make({ width, height }) }),
),
);
@@ -244,16 +230,7 @@ describe("web auto-reply", () => {
});
it("honors channels.whatsapp.mediaMaxMb for outbound auto-replies", async () => {
const bigPng = await sharp({
create: {
width: 256,
height: 256,
channels: 3,
background: { r: 0, g: 0, b: 255 },
},
})
.png({ compressionLevel: 0 })
.toBuffer();
const bigPng = createNoisyPngBuffer(256, 256);
expect(bigPng.length).toBeGreaterThan(SMALL_MEDIA_CAP_BYTES);
await expectCompressedImageWithinCap({
mediaUrl: "https://example.com/big.png",
@@ -265,16 +242,7 @@ describe("web auto-reply", () => {
});
it("prefers per-account WhatsApp media caps for outbound auto-replies", async () => {
const bigPng = await sharp({
create: {
width: 256,
height: 256,
channels: 3,
background: { r: 255, g: 0, b: 0 },
},
})
.png({ compressionLevel: 0 })
.toBuffer();
const bigPng = createNoisyPngBuffer(256, 256);
expect(bigPng.length).toBeGreaterThan(SMALL_MEDIA_CAP_BYTES);
setLoadConfigMock(() => ({
@@ -345,16 +313,7 @@ describe("web auto-reply", () => {
sendMedia,
});
const smallPng = await sharp({
create: {
width: 64,
height: 64,
channels: 3,
background: { r: 0, g: 255, b: 0 },
},
})
.png()
.toBuffer();
const smallPng = createSolidPngBuffer(64, 64, { r: 0, g: 255, b: 0 });
const fetchMock = vi.spyOn(globalThis, "fetch").mockResolvedValue({
ok: true,
body: true,
@@ -410,16 +369,7 @@ describe("web auto-reply", () => {
sendMedia,
});
const png = await sharp({
create: {
width: 64,
height: 64,
channels: 3,
background: { r: 0, g: 0, b: 255 },
},
})
.png()
.toBuffer();
const png = createSolidPngBuffer(64, 64, { r: 0, g: 0, b: 255 });
const fetchMock = vi.spyOn(globalThis, "fetch").mockResolvedValue({
ok: true,

View File

@@ -0,0 +1,48 @@
import type { AnyMessageContent } from "baileys";
import { getImageMetadata, resizeToJpeg } from "openclaw/plugin-sdk/media-runtime";
const WHATSAPP_IMAGE_THUMBNAIL_SIDE = 32;
const WHATSAPP_IMAGE_THUMBNAIL_QUALITY = 50;
type ImagePreviewContent = AnyMessageContent & {
image?: unknown;
jpegThumbnail?: unknown;
width?: unknown;
height?: unknown;
};
export async function addWhatsAppImagePreviewFields<T extends AnyMessageContent>(
content: T,
): Promise<T> {
const image = (content as ImagePreviewContent).image;
if (!Buffer.isBuffer(image)) {
return content;
}
const current = content as ImagePreviewContent;
const hasDimensions = typeof current.width === "number" && typeof current.height === "number";
const hasThumbnail = typeof current.jpegThumbnail === "string";
if (hasDimensions && hasThumbnail) {
return content;
}
const metadata = hasDimensions ? null : await getImageMetadata(image).catch(() => null);
if (!hasDimensions && !metadata) {
return content;
}
const thumbnail = hasThumbnail
? null
: await resizeToJpeg({
buffer: image,
maxSide: WHATSAPP_IMAGE_THUMBNAIL_SIDE,
quality: WHATSAPP_IMAGE_THUMBNAIL_QUALITY,
withoutEnlargement: true,
}).catch(() => null);
return {
...content,
...(metadata ? { width: metadata.width, height: metadata.height } : {}),
...(thumbnail ? { jpegThumbnail: thumbnail.toString("base64") } : {}),
};
}

View File

@@ -15,6 +15,7 @@ import { createSubsystemLogger } from "openclaw/plugin-sdk/runtime-env";
import { maybeResolveWhatsAppApprovalReaction } from "../approval-reactions.js";
import { readWebSelfIdentityForDecision, WhatsAppAuthUnstableError } from "../auth-store.js";
import { getPrimaryIdentityId, resolveComparableIdentity } from "../identity.js";
import { addWhatsAppImagePreviewFields } from "../image-preview.js";
import { cacheInboundMessageMeta } from "../quoted-message.js";
import { DEFAULT_RECONNECT_POLICY, computeBackoff, sleepWithAbort } from "../reconnect.js";
import type { OpenClawConfig } from "../runtime-api.js";
@@ -942,9 +943,10 @@ export async function attachWebInboxToSocket(
payload: AnyMessageContent,
options?: MiscMessageGenerationOptions,
) => {
const previewPayload = await addWhatsAppImagePreviewFields(payload);
const result = await sendTrackedMessage(
chatJid,
await applyOutboundMentionsToContent(chatJid, payload),
await applyOutboundMentionsToContent(chatJid, previewPayload),
options,
);
return normalizeWhatsAppSendResult(result, "media");

View File

@@ -8,6 +8,10 @@ import { resolveWhatsAppOutboundMentions } from "./outbound-mentions.js";
import { createWebSendApi } from "./send-api.js";
const recordChannelActivity = vi.hoisted(() => vi.fn());
const imageOps = vi.hoisted(() => ({
getImageMetadata: vi.fn(),
resizeToJpeg: vi.fn(),
}));
vi.mock("openclaw/plugin-sdk/channel-activity-runtime", async () => {
const actual = await vi.importActual<
@@ -19,6 +23,17 @@ vi.mock("openclaw/plugin-sdk/channel-activity-runtime", async () => {
};
});
vi.mock("openclaw/plugin-sdk/media-runtime", async () => {
const actual = await vi.importActual<typeof import("openclaw/plugin-sdk/media-runtime")>(
"openclaw/plugin-sdk/media-runtime",
);
return {
...actual,
getImageMetadata: imageOps.getImageMetadata,
resizeToJpeg: imageOps.resizeToJpeg,
};
});
function requireRecord(value: unknown, label: string): Record<string, unknown> {
if (typeof value !== "object" || value === null) {
throw new Error(`${label} was not an object`);
@@ -53,6 +68,8 @@ describe("createWebSendApi", () => {
beforeEach(() => {
vi.clearAllMocks();
imageOps.getImageMetadata.mockResolvedValue(null);
imageOps.resizeToJpeg.mockRejectedValue(new Error("unexpected thumbnail generation"));
api = createWebSendApi({
sock: { sendMessage, sendPresenceUpdate },
defaultAccountId: "main",
@@ -247,6 +264,30 @@ describe("createWebSendApi", () => {
});
});
it("prepopulates image thumbnails and dimensions before Baileys media upload", async () => {
const payload = Buffer.from("img");
const thumbnail = Buffer.from("thumb");
imageOps.getImageMetadata.mockResolvedValueOnce({ width: 640, height: 480 });
imageOps.resizeToJpeg.mockResolvedValueOnce(thumbnail);
await api.sendMessage("+1555", "cap", payload, "image/png");
expect(imageOps.resizeToJpeg).toHaveBeenCalledWith({
buffer: payload,
maxSide: 32,
quality: 50,
withoutEnlargement: true,
});
expectSendContentFields(0, {
image: payload,
caption: "cap",
mimetype: "image/png",
jpegThumbnail: thumbnail.toString("base64"),
width: 640,
height: 480,
});
});
it("adds native mention metadata to group media captions", async () => {
api = createWebSendApi({
sock: { sendMessage, sendPresenceUpdate },

View File

@@ -6,6 +6,7 @@ import type {
} from "baileys";
import { recordChannelActivity } from "openclaw/plugin-sdk/channel-activity-runtime";
import { resolveWhatsAppDocumentFileName } from "../document-filename.js";
import { addWhatsAppImagePreviewFields } from "../image-preview.js";
import { isWhatsAppNewsletterJid } from "../normalize.js";
import { buildQuotedMessageOptions } from "../quoted-message.js";
import { toWhatsappJid, toWhatsappJidWithLid } from "../text-runtime.js";
@@ -96,11 +97,11 @@ export function createWebSendApi(params: {
mimetype: mediaType,
};
} else if (mediaType.startsWith("image/")) {
payload = {
payload = await addWhatsAppImagePreviewFields({
image: mediaBuffer,
caption: resolvedPayloadText.text || undefined,
mimetype: mediaType,
};
});
} else if (mediaType.startsWith("audio/")) {
payload = { audio: mediaBuffer, ptt: true, mimetype: mediaType };
} else if (mediaType.startsWith("video/")) {

View File

@@ -5,9 +5,9 @@ import { resolveStateDir } from "openclaw/plugin-sdk/state-paths";
import { resolvePreferredOpenClawTmpDir } from "openclaw/plugin-sdk/temp-path";
import { captureEnv } from "openclaw/plugin-sdk/test-env";
import { mockPinnedHostnameResolution } from "openclaw/plugin-sdk/test-env";
import { createNoisyPngBuffer, createSolidPngBuffer } from "openclaw/plugin-sdk/test-fixtures";
import { withMockedWindowsPlatform, withRestoredMocks } from "openclaw/plugin-sdk/test-node-mocks";
import { optimizeImageToPng } from "openclaw/plugin-sdk/web-media";
import sharp from "sharp";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
LocalMediaAccessError,
@@ -36,16 +36,6 @@ async function writeTempFile(buffer: Buffer, ext: string): Promise<string> {
return file;
}
function buildDeterministicBytes(length: number): Buffer {
const buffer = Buffer.allocUnsafe(length);
let seed = 0x12345678;
for (let i = 0; i < length; i++) {
seed = (1103515245 * seed + 12345) & 0x7fffffff;
buffer[i] = seed & 0xff;
}
return buffer;
}
async function createLargeTestJpeg(): Promise<{ buffer: Buffer; file: string }> {
return { buffer: largeJpegBuffer, file: largeJpegFile };
}
@@ -69,41 +59,16 @@ beforeAll(async () => {
fixtureRoot = await fs.mkdtemp(
path.join(resolvePreferredOpenClawTmpDir(), "openclaw-media-test-"),
);
largeJpegBuffer = await sharp({
create: {
width: 400,
height: 400,
channels: 3,
background: "#ff0000",
},
})
.jpeg({ quality: 95 })
.toBuffer();
largeJpegBuffer = await fs.readFile("docs/assets/showcase/roof-camera-sky.jpg");
largeJpegFile = await writeTempFile(largeJpegBuffer, ".jpg");
tinyPngBuffer = await sharp({
create: { width: 10, height: 10, channels: 3, background: "#00ff00" },
})
.png()
.toBuffer();
tinyPngBuffer = createSolidPngBuffer(10, 10, { r: 0, g: 255, b: 0 });
tinyPngFile = await writeTempFile(tinyPngBuffer, ".png");
tinyPngWrongExtFile = await writeTempFile(tinyPngBuffer, ".bin");
alphaPngBuffer = await sharp({
create: {
width: 64,
height: 64,
channels: 4,
background: { r: 255, g: 0, b: 0, alpha: 0.5 },
},
})
.png()
.toBuffer();
alphaPngBuffer = createSolidPngBuffer(64, 64, { r: 255, g: 0, b: 0, a: 128 });
alphaPngFile = await writeTempFile(alphaPngBuffer, ".png");
// Keep this small so the alpha-fallback test stays deterministic but fast.
const size = 24;
const raw = buildDeterministicBytes(size * size * 4);
fallbackPngBuffer = await sharp(raw, { raw: { width: size, height: size, channels: 4 } })
.png()
.toBuffer();
fallbackPngBuffer = createNoisyPngBuffer(size, size);
fallbackPngFile = await writeTempFile(fallbackPngBuffer, ".png");
const smallestPng = await optimizeImageToPng(fallbackPngBuffer, 1);
fallbackPngCap = Math.max(1, smallestPng.optimizedSize - 1);
@@ -317,8 +282,7 @@ describe("web media loading", () => {
expect(result.kind).toBe("image");
expect(result.contentType).toBe("image/png");
const meta = await sharp(result.buffer).metadata();
expect(meta.hasAlpha).toBe(true);
expect(result.buffer[25]).toBe(6);
});
it("falls back to JPEG when PNG alpha cannot fit under cap", async () => {

View File

@@ -18,10 +18,25 @@ import {
} from "./monitor-inbox.test-harness.js";
import type { InboxOnMessage } from "./monitor-inbox.test-harness.js";
const { sleepWithAbortMock } = vi.hoisted(() => ({
const { imageOps, sleepWithAbortMock } = vi.hoisted(() => ({
imageOps: {
getImageMetadata: vi.fn(),
resizeToJpeg: vi.fn(),
},
sleepWithAbortMock: vi.fn(async (_ms: number, _signal?: AbortSignal) => undefined),
}));
vi.mock("openclaw/plugin-sdk/media-runtime", async () => {
const actual = await vi.importActual<typeof import("openclaw/plugin-sdk/media-runtime")>(
"openclaw/plugin-sdk/media-runtime",
);
return {
...actual,
getImageMetadata: imageOps.getImageMetadata,
resizeToJpeg: imageOps.resizeToJpeg,
};
});
vi.mock("./reconnect.js", async () => {
const actual = await vi.importActual<typeof import("./reconnect.js")>("./reconnect.js");
return {
@@ -83,6 +98,10 @@ describe("web monitor inbox", () => {
installWebMonitorInboxUnitTestHooks();
beforeEach(() => {
imageOps.getImageMetadata.mockReset();
imageOps.getImageMetadata.mockResolvedValue(null);
imageOps.resizeToJpeg.mockReset();
imageOps.resizeToJpeg.mockRejectedValue(new Error("unexpected thumbnail generation"));
sleepWithAbortMock.mockReset();
sleepWithAbortMock.mockImplementation(async (_ms: number, _signal?: AbortSignal) => undefined);
});
@@ -486,6 +505,49 @@ describe("web monitor inbox", () => {
await listener.close();
});
it("prepopulates image previews for inbound sendMedia replies", async () => {
const onMessage = vi.fn(async () => undefined);
const { listener, sock } = await startInboxMonitor(onMessage as InboxOnMessage);
sock.ev.emit(
"messages.upsert",
buildNotifyMessageUpsert({
id: nextMessageId("image-preview"),
remoteJid: "999@s.whatsapp.net",
text: "ping",
timestamp: 1_700_000_000,
pushName: "Tester",
}),
);
await waitForMessageCalls(onMessage, 1);
const inbound = inboundMessage(onMessage) as {
sendMedia: (payload: Record<string, unknown>) => Promise<void>;
};
const image = Buffer.from("img");
const thumbnail = Buffer.from("thumb");
imageOps.getImageMetadata.mockResolvedValueOnce({ width: 640, height: 480 });
imageOps.resizeToJpeg.mockResolvedValueOnce(thumbnail);
await inbound.sendMedia({ image, caption: "cap", mimetype: "image/png" });
expect(imageOps.resizeToJpeg).toHaveBeenCalledWith({
buffer: image,
maxSide: 32,
quality: 50,
withoutEnlargement: true,
});
expect(sock.sendMessage).toHaveBeenCalledWith("999@s.whatsapp.net", {
image,
caption: "cap",
mimetype: "image/png",
width: 640,
height: 480,
jpegThumbnail: thumbnail.toString("base64"),
});
await listener.close();
});
it("waits for a replacement socket before sending replies", async () => {
const onMessage = vi.fn(async () => undefined);
const socketRef = createSocketRef();

546
npm-shrinkwrap.json generated
View File

@@ -26,6 +26,7 @@
"@mozilla/readability": "0.6.0",
"@openclaw/fs-safe": "0.2.7",
"@openclaw/proxyline": "0.3.3",
"@silvia-odwyer/photon-node": "0.3.4",
"ajv": "8.20.0",
"chalk": "5.6.2",
"chokidar": "5.0.0",
@@ -68,7 +69,6 @@
"node": ">=22.19.0"
},
"optionalDependencies": {
"sharp": "0.34.5",
"sqlite-vec": "0.1.9"
}
},
@@ -705,16 +705,6 @@
"koffi": "2.16.2"
}
},
"node_modules/@emnapi/runtime": {
"version": "1.10.0",
"resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.10.0.tgz",
"integrity": "sha512-ewvYlk86xUoGI0zQRNq/mC+16R1QeDlKQy21Ki3oSYXNgLb45GV1P6A0M+/s6nyCuNDqe5VpaY84BzXGwVbwFA==",
"license": "MIT",
"optional": true,
"dependencies": {
"tslib": "^2.4.0"
}
},
"node_modules/@google/genai": {
"version": "2.5.0",
"resolved": "https://registry.npmjs.org/@google/genai/-/genai-2.5.0.tgz",
@@ -802,472 +792,6 @@
"hono": "^4"
}
},
"node_modules/@img/colour": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/@img/colour/-/colour-1.1.0.tgz",
"integrity": "sha512-Td76q7j57o/tLVdgS746cYARfSyxk8iEfRxewL9h4OMzYhbW4TAcppl0mT4eyqXddh6L/jwoM75mo7ixa/pCeQ==",
"license": "MIT",
"optional": true,
"engines": {
"node": ">=18"
}
},
"node_modules/@img/sharp-darwin-arm64": {
"version": "0.34.5",
"resolved": "https://registry.npmjs.org/@img/sharp-darwin-arm64/-/sharp-darwin-arm64-0.34.5.tgz",
"integrity": "sha512-imtQ3WMJXbMY4fxb/Ndp6HBTNVtWCUI0WdobyheGf5+ad6xX8VIDO8u2xE4qc/fr08CKG/7dDseFtn6M6g/r3w==",
"cpu": [
"arm64"
],
"license": "Apache-2.0",
"optional": true,
"os": [
"darwin"
],
"engines": {
"node": "^18.17.0 || ^20.3.0 || >=21.0.0"
},
"funding": {
"url": "https://opencollective.com/libvips"
},
"optionalDependencies": {
"@img/sharp-libvips-darwin-arm64": "1.2.4"
}
},
"node_modules/@img/sharp-darwin-x64": {
"version": "0.34.5",
"resolved": "https://registry.npmjs.org/@img/sharp-darwin-x64/-/sharp-darwin-x64-0.34.5.tgz",
"integrity": "sha512-YNEFAF/4KQ/PeW0N+r+aVVsoIY0/qxxikF2SWdp+NRkmMB7y9LBZAVqQ4yhGCm/H3H270OSykqmQMKLBhBJDEw==",
"cpu": [
"x64"
],
"license": "Apache-2.0",
"optional": true,
"os": [
"darwin"
],
"engines": {
"node": "^18.17.0 || ^20.3.0 || >=21.0.0"
},
"funding": {
"url": "https://opencollective.com/libvips"
},
"optionalDependencies": {
"@img/sharp-libvips-darwin-x64": "1.2.4"
}
},
"node_modules/@img/sharp-libvips-darwin-arm64": {
"version": "1.2.4",
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-arm64/-/sharp-libvips-darwin-arm64-1.2.4.tgz",
"integrity": "sha512-zqjjo7RatFfFoP0MkQ51jfuFZBnVE2pRiaydKJ1G/rHZvnsrHAOcQALIi9sA5co5xenQdTugCvtb1cuf78Vf4g==",
"cpu": [
"arm64"
],
"license": "LGPL-3.0-or-later",
"optional": true,
"os": [
"darwin"
],
"funding": {
"url": "https://opencollective.com/libvips"
}
},
"node_modules/@img/sharp-libvips-darwin-x64": {
"version": "1.2.4",
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-x64/-/sharp-libvips-darwin-x64-1.2.4.tgz",
"integrity": "sha512-1IOd5xfVhlGwX+zXv2N93k0yMONvUlANylbJw1eTah8K/Jtpi15KC+WSiaX/nBmbm2HxRM1gZ0nSdjSsrZbGKg==",
"cpu": [
"x64"
],
"license": "LGPL-3.0-or-later",
"optional": true,
"os": [
"darwin"
],
"funding": {
"url": "https://opencollective.com/libvips"
}
},
"node_modules/@img/sharp-libvips-linux-arm": {
"version": "1.2.4",
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm/-/sharp-libvips-linux-arm-1.2.4.tgz",
"integrity": "sha512-bFI7xcKFELdiNCVov8e44Ia4u2byA+l3XtsAj+Q8tfCwO6BQ8iDojYdvoPMqsKDkuoOo+X6HZA0s0q11ANMQ8A==",
"cpu": [
"arm"
],
"license": "LGPL-3.0-or-later",
"optional": true,
"os": [
"linux"
],
"funding": {
"url": "https://opencollective.com/libvips"
}
},
"node_modules/@img/sharp-libvips-linux-arm64": {
"version": "1.2.4",
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm64/-/sharp-libvips-linux-arm64-1.2.4.tgz",
"integrity": "sha512-excjX8DfsIcJ10x1Kzr4RcWe1edC9PquDRRPx3YVCvQv+U5p7Yin2s32ftzikXojb1PIFc/9Mt28/y+iRklkrw==",
"cpu": [
"arm64"
],
"license": "LGPL-3.0-or-later",
"optional": true,
"os": [
"linux"
],
"funding": {
"url": "https://opencollective.com/libvips"
}
},
"node_modules/@img/sharp-libvips-linux-ppc64": {
"version": "1.2.4",
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-ppc64/-/sharp-libvips-linux-ppc64-1.2.4.tgz",
"integrity": "sha512-FMuvGijLDYG6lW+b/UvyilUWu5Ayu+3r2d1S8notiGCIyYU/76eig1UfMmkZ7vwgOrzKzlQbFSuQfgm7GYUPpA==",
"cpu": [
"ppc64"
],
"license": "LGPL-3.0-or-later",
"optional": true,
"os": [
"linux"
],
"funding": {
"url": "https://opencollective.com/libvips"
}
},
"node_modules/@img/sharp-libvips-linux-riscv64": {
"version": "1.2.4",
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-riscv64/-/sharp-libvips-linux-riscv64-1.2.4.tgz",
"integrity": "sha512-oVDbcR4zUC0ce82teubSm+x6ETixtKZBh/qbREIOcI3cULzDyb18Sr/Wcyx7NRQeQzOiHTNbZFF1UwPS2scyGA==",
"cpu": [
"riscv64"
],
"license": "LGPL-3.0-or-later",
"optional": true,
"os": [
"linux"
],
"funding": {
"url": "https://opencollective.com/libvips"
}
},
"node_modules/@img/sharp-libvips-linux-s390x": {
"version": "1.2.4",
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-s390x/-/sharp-libvips-linux-s390x-1.2.4.tgz",
"integrity": "sha512-qmp9VrzgPgMoGZyPvrQHqk02uyjA0/QrTO26Tqk6l4ZV0MPWIW6LTkqOIov+J1yEu7MbFQaDpwdwJKhbJvuRxQ==",
"cpu": [
"s390x"
],
"license": "LGPL-3.0-or-later",
"optional": true,
"os": [
"linux"
],
"funding": {
"url": "https://opencollective.com/libvips"
}
},
"node_modules/@img/sharp-libvips-linux-x64": {
"version": "1.2.4",
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-x64/-/sharp-libvips-linux-x64-1.2.4.tgz",
"integrity": "sha512-tJxiiLsmHc9Ax1bz3oaOYBURTXGIRDODBqhveVHonrHJ9/+k89qbLl0bcJns+e4t4rvaNBxaEZsFtSfAdquPrw==",
"cpu": [
"x64"
],
"license": "LGPL-3.0-or-later",
"optional": true,
"os": [
"linux"
],
"funding": {
"url": "https://opencollective.com/libvips"
}
},
"node_modules/@img/sharp-libvips-linuxmusl-arm64": {
"version": "1.2.4",
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-arm64/-/sharp-libvips-linuxmusl-arm64-1.2.4.tgz",
"integrity": "sha512-FVQHuwx1IIuNow9QAbYUzJ+En8KcVm9Lk5+uGUQJHaZmMECZmOlix9HnH7n1TRkXMS0pGxIJokIVB9SuqZGGXw==",
"cpu": [
"arm64"
],
"license": "LGPL-3.0-or-later",
"optional": true,
"os": [
"linux"
],
"funding": {
"url": "https://opencollective.com/libvips"
}
},
"node_modules/@img/sharp-libvips-linuxmusl-x64": {
"version": "1.2.4",
"resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-x64/-/sharp-libvips-linuxmusl-x64-1.2.4.tgz",
"integrity": "sha512-+LpyBk7L44ZIXwz/VYfglaX/okxezESc6UxDSoyo2Ks6Jxc4Y7sGjpgU9s4PMgqgjj1gZCylTieNamqA1MF7Dg==",
"cpu": [
"x64"
],
"license": "LGPL-3.0-or-later",
"optional": true,
"os": [
"linux"
],
"funding": {
"url": "https://opencollective.com/libvips"
}
},
"node_modules/@img/sharp-linux-arm": {
"version": "0.34.5",
"resolved": "https://registry.npmjs.org/@img/sharp-linux-arm/-/sharp-linux-arm-0.34.5.tgz",
"integrity": "sha512-9dLqsvwtg1uuXBGZKsxem9595+ujv0sJ6Vi8wcTANSFpwV/GONat5eCkzQo/1O6zRIkh0m/8+5BjrRr7jDUSZw==",
"cpu": [
"arm"
],
"license": "Apache-2.0",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": "^18.17.0 || ^20.3.0 || >=21.0.0"
},
"funding": {
"url": "https://opencollective.com/libvips"
},
"optionalDependencies": {
"@img/sharp-libvips-linux-arm": "1.2.4"
}
},
"node_modules/@img/sharp-linux-arm64": {
"version": "0.34.5",
"resolved": "https://registry.npmjs.org/@img/sharp-linux-arm64/-/sharp-linux-arm64-0.34.5.tgz",
"integrity": "sha512-bKQzaJRY/bkPOXyKx5EVup7qkaojECG6NLYswgktOZjaXecSAeCWiZwwiFf3/Y+O1HrauiE3FVsGxFg8c24rZg==",
"cpu": [
"arm64"
],
"license": "Apache-2.0",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": "^18.17.0 || ^20.3.0 || >=21.0.0"
},
"funding": {
"url": "https://opencollective.com/libvips"
},
"optionalDependencies": {
"@img/sharp-libvips-linux-arm64": "1.2.4"
}
},
"node_modules/@img/sharp-linux-ppc64": {
"version": "0.34.5",
"resolved": "https://registry.npmjs.org/@img/sharp-linux-ppc64/-/sharp-linux-ppc64-0.34.5.tgz",
"integrity": "sha512-7zznwNaqW6YtsfrGGDA6BRkISKAAE1Jo0QdpNYXNMHu2+0dTrPflTLNkpc8l7MUP5M16ZJcUvysVWWrMefZquA==",
"cpu": [
"ppc64"
],
"license": "Apache-2.0",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": "^18.17.0 || ^20.3.0 || >=21.0.0"
},
"funding": {
"url": "https://opencollective.com/libvips"
},
"optionalDependencies": {
"@img/sharp-libvips-linux-ppc64": "1.2.4"
}
},
"node_modules/@img/sharp-linux-riscv64": {
"version": "0.34.5",
"resolved": "https://registry.npmjs.org/@img/sharp-linux-riscv64/-/sharp-linux-riscv64-0.34.5.tgz",
"integrity": "sha512-51gJuLPTKa7piYPaVs8GmByo7/U7/7TZOq+cnXJIHZKavIRHAP77e3N2HEl3dgiqdD/w0yUfiJnII77PuDDFdw==",
"cpu": [
"riscv64"
],
"license": "Apache-2.0",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": "^18.17.0 || ^20.3.0 || >=21.0.0"
},
"funding": {
"url": "https://opencollective.com/libvips"
},
"optionalDependencies": {
"@img/sharp-libvips-linux-riscv64": "1.2.4"
}
},
"node_modules/@img/sharp-linux-s390x": {
"version": "0.34.5",
"resolved": "https://registry.npmjs.org/@img/sharp-linux-s390x/-/sharp-linux-s390x-0.34.5.tgz",
"integrity": "sha512-nQtCk0PdKfho3eC5MrbQoigJ2gd1CgddUMkabUj+rBevs8tZ2cULOx46E7oyX+04WGfABgIwmMC0VqieTiR4jg==",
"cpu": [
"s390x"
],
"license": "Apache-2.0",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": "^18.17.0 || ^20.3.0 || >=21.0.0"
},
"funding": {
"url": "https://opencollective.com/libvips"
},
"optionalDependencies": {
"@img/sharp-libvips-linux-s390x": "1.2.4"
}
},
"node_modules/@img/sharp-linux-x64": {
"version": "0.34.5",
"resolved": "https://registry.npmjs.org/@img/sharp-linux-x64/-/sharp-linux-x64-0.34.5.tgz",
"integrity": "sha512-MEzd8HPKxVxVenwAa+JRPwEC7QFjoPWuS5NZnBt6B3pu7EG2Ge0id1oLHZpPJdn3OQK+BQDiw9zStiHBTJQQQQ==",
"cpu": [
"x64"
],
"license": "Apache-2.0",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": "^18.17.0 || ^20.3.0 || >=21.0.0"
},
"funding": {
"url": "https://opencollective.com/libvips"
},
"optionalDependencies": {
"@img/sharp-libvips-linux-x64": "1.2.4"
}
},
"node_modules/@img/sharp-linuxmusl-arm64": {
"version": "0.34.5",
"resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-arm64/-/sharp-linuxmusl-arm64-0.34.5.tgz",
"integrity": "sha512-fprJR6GtRsMt6Kyfq44IsChVZeGN97gTD331weR1ex1c1rypDEABN6Tm2xa1wE6lYb5DdEnk03NZPqA7Id21yg==",
"cpu": [
"arm64"
],
"license": "Apache-2.0",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": "^18.17.0 || ^20.3.0 || >=21.0.0"
},
"funding": {
"url": "https://opencollective.com/libvips"
},
"optionalDependencies": {
"@img/sharp-libvips-linuxmusl-arm64": "1.2.4"
}
},
"node_modules/@img/sharp-linuxmusl-x64": {
"version": "0.34.5",
"resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-x64/-/sharp-linuxmusl-x64-0.34.5.tgz",
"integrity": "sha512-Jg8wNT1MUzIvhBFxViqrEhWDGzqymo3sV7z7ZsaWbZNDLXRJZoRGrjulp60YYtV4wfY8VIKcWidjojlLcWrd8Q==",
"cpu": [
"x64"
],
"license": "Apache-2.0",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": "^18.17.0 || ^20.3.0 || >=21.0.0"
},
"funding": {
"url": "https://opencollective.com/libvips"
},
"optionalDependencies": {
"@img/sharp-libvips-linuxmusl-x64": "1.2.4"
}
},
"node_modules/@img/sharp-wasm32": {
"version": "0.34.5",
"resolved": "https://registry.npmjs.org/@img/sharp-wasm32/-/sharp-wasm32-0.34.5.tgz",
"integrity": "sha512-OdWTEiVkY2PHwqkbBI8frFxQQFekHaSSkUIJkwzclWZe64O1X4UlUjqqqLaPbUpMOQk6FBu/HtlGXNblIs0huw==",
"cpu": [
"wasm32"
],
"license": "Apache-2.0 AND LGPL-3.0-or-later AND MIT",
"optional": true,
"dependencies": {
"@emnapi/runtime": "^1.7.0"
},
"engines": {
"node": "^18.17.0 || ^20.3.0 || >=21.0.0"
},
"funding": {
"url": "https://opencollective.com/libvips"
}
},
"node_modules/@img/sharp-win32-arm64": {
"version": "0.34.5",
"resolved": "https://registry.npmjs.org/@img/sharp-win32-arm64/-/sharp-win32-arm64-0.34.5.tgz",
"integrity": "sha512-WQ3AgWCWYSb2yt+IG8mnC6Jdk9Whs7O0gxphblsLvdhSpSTtmu69ZG1Gkb6NuvxsNACwiPV6cNSZNzt0KPsw7g==",
"cpu": [
"arm64"
],
"license": "Apache-2.0 AND LGPL-3.0-or-later",
"optional": true,
"os": [
"win32"
],
"engines": {
"node": "^18.17.0 || ^20.3.0 || >=21.0.0"
},
"funding": {
"url": "https://opencollective.com/libvips"
}
},
"node_modules/@img/sharp-win32-ia32": {
"version": "0.34.5",
"resolved": "https://registry.npmjs.org/@img/sharp-win32-ia32/-/sharp-win32-ia32-0.34.5.tgz",
"integrity": "sha512-FV9m/7NmeCmSHDD5j4+4pNI8Cp3aW+JvLoXcTUo0IqyjSfAZJ8dIUmijx1qaJsIiU+Hosw6xM5KijAWRJCSgNg==",
"cpu": [
"ia32"
],
"license": "Apache-2.0 AND LGPL-3.0-or-later",
"optional": true,
"os": [
"win32"
],
"engines": {
"node": "^18.17.0 || ^20.3.0 || >=21.0.0"
},
"funding": {
"url": "https://opencollective.com/libvips"
}
},
"node_modules/@img/sharp-win32-x64": {
"version": "0.34.5",
"resolved": "https://registry.npmjs.org/@img/sharp-win32-x64/-/sharp-win32-x64-0.34.5.tgz",
"integrity": "sha512-+29YMsqY2/9eFEiW93eqWnuLcWcufowXewwSNIT6UwZdUUCrM3oFjMWH/Z6/TMmb4hlFenmfAVbpWeup2jryCw==",
"cpu": [
"x64"
],
"license": "Apache-2.0 AND LGPL-3.0-or-later",
"optional": true,
"os": [
"win32"
],
"engines": {
"node": "^18.17.0 || ^20.3.0 || >=21.0.0"
},
"funding": {
"url": "https://opencollective.com/libvips"
}
},
"node_modules/@isaacs/fs-minipass": {
"version": "4.0.1",
"resolved": "https://registry.npmjs.org/@isaacs/fs-minipass/-/fs-minipass-4.0.1.tgz",
@@ -2576,16 +2100,6 @@
"node": ">= 0.8"
}
},
"node_modules/detect-libc": {
"version": "2.1.2",
"resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz",
"integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==",
"license": "Apache-2.0",
"optional": true,
"engines": {
"node": ">=8"
}
},
"node_modules/diff": {
"version": "8.0.4",
"resolved": "https://registry.npmjs.org/diff/-/diff-8.0.4.tgz",
@@ -4465,19 +3979,6 @@
"integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==",
"license": "MIT"
},
"node_modules/semver": {
"version": "7.8.0",
"resolved": "https://registry.npmjs.org/semver/-/semver-7.8.0.tgz",
"integrity": "sha512-AcM7dV/5ul4EekoQ29Agm5vri8JNqRyj39o0qpX6vDF2GZrtutZl5RwgD1XnZjiTAfncsJhMI48QQH3sN87YNA==",
"license": "ISC",
"optional": true,
"bin": {
"semver": "bin/semver.js"
},
"engines": {
"node": ">=10"
}
},
"node_modules/send": {
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/send/-/send-1.2.1.tgz",
@@ -4541,51 +4042,6 @@
"integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==",
"license": "ISC"
},
"node_modules/sharp": {
"version": "0.34.5",
"resolved": "https://registry.npmjs.org/sharp/-/sharp-0.34.5.tgz",
"integrity": "sha512-Ou9I5Ft9WNcCbXrU9cMgPBcCK8LiwLqcbywW3t4oDV37n1pzpuNLsYiAV8eODnjbtQlSDwZ2cUEeQz4E54Hltg==",
"hasInstallScript": true,
"license": "Apache-2.0",
"optional": true,
"dependencies": {
"@img/colour": "^1.0.0",
"detect-libc": "^2.1.2",
"semver": "^7.7.3"
},
"engines": {
"node": "^18.17.0 || ^20.3.0 || >=21.0.0"
},
"funding": {
"url": "https://opencollective.com/libvips"
},
"optionalDependencies": {
"@img/sharp-darwin-arm64": "0.34.5",
"@img/sharp-darwin-x64": "0.34.5",
"@img/sharp-libvips-darwin-arm64": "1.2.4",
"@img/sharp-libvips-darwin-x64": "1.2.4",
"@img/sharp-libvips-linux-arm": "1.2.4",
"@img/sharp-libvips-linux-arm64": "1.2.4",
"@img/sharp-libvips-linux-ppc64": "1.2.4",
"@img/sharp-libvips-linux-riscv64": "1.2.4",
"@img/sharp-libvips-linux-s390x": "1.2.4",
"@img/sharp-libvips-linux-x64": "1.2.4",
"@img/sharp-libvips-linuxmusl-arm64": "1.2.4",
"@img/sharp-libvips-linuxmusl-x64": "1.2.4",
"@img/sharp-linux-arm": "0.34.5",
"@img/sharp-linux-arm64": "0.34.5",
"@img/sharp-linux-ppc64": "0.34.5",
"@img/sharp-linux-riscv64": "0.34.5",
"@img/sharp-linux-s390x": "0.34.5",
"@img/sharp-linux-x64": "0.34.5",
"@img/sharp-linuxmusl-arm64": "0.34.5",
"@img/sharp-linuxmusl-x64": "0.34.5",
"@img/sharp-wasm32": "0.34.5",
"@img/sharp-win32-arm64": "0.34.5",
"@img/sharp-win32-ia32": "0.34.5",
"@img/sharp-win32-x64": "0.34.5"
}
},
"node_modules/shebang-command": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz",

View File

@@ -1816,6 +1816,7 @@
"@mozilla/readability": "0.6.0",
"@openclaw/fs-safe": "0.2.7",
"@openclaw/proxyline": "0.3.3",
"@silvia-odwyer/photon-node": "0.3.4",
"ajv": "8.20.0",
"chalk": "5.6.2",
"chokidar": "5.0.0",
@@ -1877,7 +1878,6 @@
"vitest": "4.1.7"
},
"optionalDependencies": {
"sharp": "0.34.5",
"sqlite-vec": "0.1.9"
},
"overrides": {

814
pnpm-lock.yaml generated

File diff suppressed because it is too large Load Diff

View File

@@ -100,12 +100,15 @@ allowBuilds:
koffi: false
node-llama-cpp: true
protobufjs: true
sharp: true
tree-sitter-bash: false
openclaw: true
"@openclaw/proxyline": true
packageExtensions:
baileys:
peerDependenciesMeta:
sharp:
optional: true
"@earendil-works/pi-coding-agent":
dependencies:
strip-ansi: 7.2.0

View File

@@ -45,6 +45,13 @@ function readWorkspaceOverrides() {
return normalizeOverrides(workspace?.overrides);
}
function readWorkspacePackageExtensions() {
const workspace = parseYaml(readFileSync(path.join(ROOT_DIR, "pnpm-workspace.yaml"), "utf8"));
return workspace?.packageExtensions && typeof workspace.packageExtensions === "object"
? workspace.packageExtensions
: {};
}
function parsePnpmPackageKey(packageKey) {
if (typeof packageKey !== "string") {
return null;
@@ -163,6 +170,88 @@ function runNpm(args, cwd) {
});
}
function packageExtensionAppliesToDependency(selector, dependencyName) {
return selector === dependencyName || selector.startsWith(`${dependencyName}@`);
}
function packageExtensionMarksOptionalPeer(packageExtension) {
const peerDependenciesMeta = packageExtension?.peerDependenciesMeta;
if (
!peerDependenciesMeta ||
typeof peerDependenciesMeta !== "object" ||
Array.isArray(peerDependenciesMeta)
) {
return false;
}
return Object.values(peerDependenciesMeta).some((meta) => meta?.optional === true);
}
function shouldUseLegacyPeerDepsForShrinkwrap(
packageJson,
packageExtensions = readWorkspacePackageExtensions(),
) {
const dependencies = Object.keys(packageJson.dependencies ?? {});
if (dependencies.length === 0) {
return false;
}
for (const dependencyName of dependencies) {
for (const [selector, packageExtension] of Object.entries(packageExtensions)) {
if (
packageExtensionAppliesToDependency(selector, dependencyName) &&
packageExtensionMarksOptionalPeer(packageExtension)
) {
return true;
}
}
}
return false;
}
function applyPackageExtensionPeerMetadata(
lockfile,
packageExtensions = readWorkspacePackageExtensions(),
) {
const packages = lockfile?.packages;
if (!packages || typeof packages !== "object" || Array.isArray(packages)) {
return lockfile;
}
for (const [lockPath, metadata] of Object.entries(packages)) {
if (!metadata || typeof metadata !== "object" || Array.isArray(metadata)) {
continue;
}
const packageName = metadata.name ?? parseLockPackagePath(lockPath).at(-1)?.name;
if (!packageName || !metadata.peerDependencies) {
continue;
}
for (const [selector, packageExtension] of Object.entries(packageExtensions)) {
if (!packageExtensionAppliesToDependency(selector, packageName)) {
continue;
}
const peerDependenciesMeta = packageExtension?.peerDependenciesMeta;
if (
!peerDependenciesMeta ||
typeof peerDependenciesMeta !== "object" ||
Array.isArray(peerDependenciesMeta)
) {
continue;
}
for (const [peerName, peerMeta] of Object.entries(peerDependenciesMeta)) {
if (metadata.peerDependencies[peerName] === undefined) {
continue;
}
metadata.peerDependenciesMeta ??= {};
const existingPeerMeta = metadata.peerDependenciesMeta[peerName];
metadata.peerDependenciesMeta[peerName] = existingPeerMeta
? { ...existingPeerMeta, ...peerMeta }
: { ...peerMeta };
}
}
}
return lockfile;
}
function exactVersionFromOverrideSpec(spec) {
if (!spec || typeof spec !== "string") {
return null;
@@ -276,7 +365,7 @@ function describeOverrideViolations(violations) {
.join("; ");
}
function normalizeShrinkwrapOverrides(tempDir, shrinkwrapOverrides) {
function normalizeShrinkwrapOverrides(tempDir, shrinkwrapOverrides, npmInstallArgs) {
const shrinkwrapPath = path.join(tempDir, "npm-shrinkwrap.json");
const overrideRules = exactOverrideRulesFromOverrides(shrinkwrapOverrides);
if (Object.keys(overrideRules).length === 0) {
@@ -299,10 +388,7 @@ function normalizeShrinkwrapOverrides(tempDir, shrinkwrapOverrides) {
// shrinkwraps as inactive, drop their cached subtree, then ask npm to recalculate this
// package's authoritative lock with registry integrity hashes.
writeFileSync(shrinkwrapPath, `${JSON.stringify(shrinkwrap, null, 2)}\n`);
runNpm(
["install", "--package-lock-only", "--ignore-scripts", "--no-audit", "--no-fund"],
tempDir,
);
runNpm(npmInstallArgs, tempDir);
const normalized = JSON.parse(readFileSync(shrinkwrapPath, "utf8"));
const remaining = collectOverrideViolations(normalized, overrideRules);
@@ -337,18 +423,25 @@ function generateShrinkwrap(packageDir) {
try {
const packageJson = JSON.parse(readFileSync(path.join(packageDir, "package.json"), "utf8"));
const shrinkwrapOverrides = readShrinkwrapOverrides();
const npmInstallArgs = [
"install",
"--package-lock-only",
"--ignore-scripts",
"--no-audit",
"--no-fund",
...(shouldUseLegacyPeerDepsForShrinkwrap(packageJson) ? ["--legacy-peer-deps"] : []),
];
writeFileSync(
path.join(tempDir, "package.json"),
`${JSON.stringify(packageJsonForShrinkwrap(packageJson, shrinkwrapOverrides), null, 2)}\n`,
);
runNpm(
["install", "--package-lock-only", "--ignore-scripts", "--no-audit", "--no-fund"],
tempDir,
);
runNpm(npmInstallArgs, tempDir);
runNpm(["shrinkwrap", "--ignore-scripts", "--no-audit", "--no-fund"], tempDir);
normalizeShrinkwrapOverrides(tempDir, shrinkwrapOverrides);
normalizeShrinkwrapOverrides(tempDir, shrinkwrapOverrides, npmInstallArgs);
const generated = normalizeNpmVersionDrift(
applyPackageExtensionPeerMetadata(
JSON.parse(readFileSync(path.join(tempDir, "npm-shrinkwrap.json"), "utf8")),
),
);
assertShrinkwrapMatchesPnpmLock(generated);
return `${JSON.stringify(generated, null, 2)}\n`;
@@ -608,10 +701,12 @@ export {
disableShrinkwrappedOverrideConflictSources,
exactOverrideRulesFromOverrides,
exactVersionFromOverrideSpec,
applyPackageExtensionPeerMetadata,
normalizeNpmVersionDrift,
packageJsonForShrinkwrap,
parsePnpmPackageKey,
parseLockPackagePath,
readShrinkwrapOverrides,
shouldUseLegacyPeerDepsForShrinkwrap,
shrinkwrapPackageDirsForChangedPaths,
};

View File

@@ -59,7 +59,6 @@ if [[ -n "${OPENCLAW_NODE_VERSION:-}" ]]; then
fi
MIN_NODE_VERSION="22.19.0"
APK_NODE_BIN_DIR="/usr/bin"
SHARP_IGNORE_GLOBAL_LIBVIPS="${SHARP_IGNORE_GLOBAL_LIBVIPS:-1}"
NPM_LOGLEVEL="${OPENCLAW_NPM_LOGLEVEL:-error}"
INSTALL_METHOD="${OPENCLAW_INSTALL_METHOD:-npm}"
GIT_DIR="${OPENCLAW_GIT_DIR:-${OPENCLAW_EFFECTIVE_HOME}/openclaw}"
@@ -85,7 +84,6 @@ Usage: install-cli.sh [options]
--set-npm-prefix Force npm prefix to ~/.npm-global if current prefix is not writable (Linux)
Environment variables:
SHARP_IGNORE_GLOBAL_LIBVIPS=0|1 Default: 1 (avoid sharp building against global libvips)
OPENCLAW_NPM_LOGLEVEL=error|warn|notice Default: error (hide npm deprecation noise)
OPENCLAW_INSTALL_METHOD=git|npm
OPENCLAW_HOME=...
@@ -819,7 +817,7 @@ ensure_pnpm() {
emit_json "{\"event\":\"step\",\"name\":\"pnpm\",\"status\":\"start\",\"method\":\"npm\"}"
log "Installing pnpm via npm..."
SHARP_IGNORE_GLOBAL_LIBVIPS="$SHARP_IGNORE_GLOBAL_LIBVIPS" "$(npm_bin)" install -g --prefix "$PREFIX" pnpm@11
"$(npm_bin)" install -g --prefix "$PREFIX" pnpm@11
detect_pnpm_cmd || true
emit_json "{\"event\":\"step\",\"name\":\"pnpm\",\"status\":\"ok\"}"
return 0
@@ -967,14 +965,14 @@ install_openclaw() {
fi
if [[ "${requested}" == "latest" ]]; then
if ! env -u NPM_CONFIG_BEFORE -u npm_config_before -u NPM_CONFIG_MIN_RELEASE_AGE -u npm_config_min_release_age -u npm_config_min-release-age "SHARP_IGNORE_GLOBAL_LIBVIPS=$SHARP_IGNORE_GLOBAL_LIBVIPS" "$(npm_bin)" install -g --prefix "$(node_dir)" "${npm_args[@]}" "openclaw@latest"; then
if ! env -u NPM_CONFIG_BEFORE -u npm_config_before -u NPM_CONFIG_MIN_RELEASE_AGE -u npm_config_min_release_age -u npm_config_min-release-age "$(npm_bin)" install -g --prefix "$(node_dir)" "${npm_args[@]}" "openclaw@latest"; then
log "npm install openclaw@latest failed; retrying openclaw@next"
emit_json "{\"event\":\"step\",\"name\":\"openclaw\",\"status\":\"retry\",\"version\":\"next\"}"
env -u NPM_CONFIG_BEFORE -u npm_config_before -u NPM_CONFIG_MIN_RELEASE_AGE -u npm_config_min_release_age -u npm_config_min-release-age "SHARP_IGNORE_GLOBAL_LIBVIPS=$SHARP_IGNORE_GLOBAL_LIBVIPS" "$(npm_bin)" install -g --prefix "$(node_dir)" "${npm_args[@]}" "openclaw@next"
env -u NPM_CONFIG_BEFORE -u npm_config_before -u NPM_CONFIG_MIN_RELEASE_AGE -u npm_config_min_release_age -u npm_config_min-release-age "$(npm_bin)" install -g --prefix "$(node_dir)" "${npm_args[@]}" "openclaw@next"
requested="next"
fi
else
env -u NPM_CONFIG_BEFORE -u npm_config_before -u NPM_CONFIG_MIN_RELEASE_AGE -u npm_config_min_release_age -u npm_config_min-release-age "SHARP_IGNORE_GLOBAL_LIBVIPS=$SHARP_IGNORE_GLOBAL_LIBVIPS" "$(npm_bin)" install -g --prefix "$(node_dir)" "${npm_args[@]}" "openclaw@${requested}"
env -u NPM_CONFIG_BEFORE -u npm_config_before -u NPM_CONFIG_MIN_RELEASE_AGE -u npm_config_min_release_age -u npm_config_min-release-age "$(npm_bin)" install -g --prefix "$(node_dir)" "${npm_args[@]}" "openclaw@${requested}"
fi
mkdir -p "${PREFIX}/bin"
@@ -1070,7 +1068,7 @@ install_openclaw_from_git() {
local install_lockfile_flag
install_lockfile_flag="$(git_install_lockfile_flag "$repo_dir" "$git_ref")"
CI="${CI:-true}" SHARP_IGNORE_GLOBAL_LIBVIPS="$SHARP_IGNORE_GLOBAL_LIBVIPS" run_pnpm -C "$repo_dir" install "$install_lockfile_flag"
CI="${CI:-true}" run_pnpm -C "$repo_dir" install "$install_lockfile_flag"
if ! run_pnpm -C "$repo_dir" ui:build; then
log "UI build failed; continuing (CLI may still work)"

View File

@@ -842,7 +842,7 @@ run_npm_global_install() {
fi
local -a cmd
cmd=(env -u NPM_CONFIG_BEFORE -u npm_config_before -u NPM_CONFIG_MIN_RELEASE_AGE -u npm_config_min_release_age -u npm_config_min-release-age "SHARP_IGNORE_GLOBAL_LIBVIPS=$SHARP_IGNORE_GLOBAL_LIBVIPS" npm --loglevel "$NPM_LOGLEVEL")
cmd=(env -u NPM_CONFIG_BEFORE -u npm_config_before -u NPM_CONFIG_MIN_RELEASE_AGE -u npm_config_min_release_age -u npm_config_min-release-age npm --loglevel "$NPM_LOGLEVEL")
if [[ -n "$NPM_SILENT_FLAG" ]]; then
cmd+=("$NPM_SILENT_FLAG")
fi
@@ -1134,7 +1134,6 @@ USE_BETA=${OPENCLAW_BETA:-0}
GIT_DIR_DEFAULT="$(resolve_openclaw_effective_home)/openclaw"
GIT_DIR=${OPENCLAW_GIT_DIR:-$GIT_DIR_DEFAULT}
GIT_UPDATE=${OPENCLAW_GIT_UPDATE:-1}
SHARP_IGNORE_GLOBAL_LIBVIPS="${SHARP_IGNORE_GLOBAL_LIBVIPS:-1}"
NPM_LOGLEVEL="${OPENCLAW_NPM_LOGLEVEL:-error}"
NPM_SILENT_FLAG="--silent"
VERBOSE="${OPENCLAW_VERBOSE:-0}"
@@ -1177,8 +1176,6 @@ Environment variables:
OPENCLAW_NO_ONBOARD=1
OPENCLAW_VERBOSE=1
OPENCLAW_NPM_LOGLEVEL=error|warn|notice Default: error (hide npm deprecation noise)
SHARP_IGNORE_GLOBAL_LIBVIPS=0|1 Default: 1 (avoid sharp building against global libvips)
Examples:
curl -fsSL --proto '=https' --tlsv1.2 https://openclaw.ai/install.sh | bash
curl -fsSL --proto '=https' --tlsv1.2 https://openclaw.ai/install.sh | bash -s -- --no-onboard
@@ -2535,7 +2532,7 @@ install_openclaw_from_git() {
local install_lockfile_flag
install_lockfile_flag="$(git_install_lockfile_flag "$repo_dir" "$git_ref")"
CI="${CI:-true}" SHARP_IGNORE_GLOBAL_LIBVIPS="$SHARP_IGNORE_GLOBAL_LIBVIPS" run_quiet_step "Installing dependencies" run_pnpm -C "$repo_dir" install "$install_lockfile_flag"
CI="${CI:-true}" run_quiet_step "Installing dependencies" run_pnpm -C "$repo_dir" install "$install_lockfile_flag"
if ! run_quiet_step "Building UI" run_pnpm -C "$repo_dir" ui:build; then
ui_warn "UI build failed; continuing (CLI may still work)"

View File

@@ -168,11 +168,11 @@
"class": "default-runtime-initially",
"risk": ["terminal-rendering", "png-encoding"]
},
"sharp": {
"@silvia-odwyer/photon-node": {
"owner": "plugin:media-understanding-core",
"class": "plugin-runtime",
"activation": ["media-understanding-core.image-ops"],
"risk": ["native", "parser", "untrusted-files"]
"risk": ["wasm", "parser", "untrusted-files"]
},
"sqlite-vec": {
"owner": "capability:memory-sqlite-vec",

View File

@@ -28,6 +28,10 @@ const ROOT_OWNED_EXTENSION_RUNTIME_DEPENDENCIES = new Map([
"playwright-core",
"keep at root; the internal browser runtime is shipped with core even though downloadable browser-adjacent plugins also declare it",
],
[
"@silvia-odwyer/photon-node",
"keep at root; the internal media understanding runtime is shipped with packaged image-processing surfaces even though the bundled plugin also declares it",
],
]);
function readJson(filePath) {

View File

@@ -2,6 +2,7 @@ import fs from "node:fs/promises";
import path from "node:path";
import type { ImageContent } from "@earendil-works/pi-ai";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { createSolidPngBuffer } from "../../test/helpers/image-fixtures.js";
import { resolvePreferredOpenClawTmpDir } from "../infra/tmp-openclaw-dir.js";
import { MAX_IMAGE_BYTES } from "../media/constants.js";
import { escapeRegExp } from "../shared/regexp.js";
@@ -269,13 +270,7 @@ describe("writeCliImages", () => {
path.join(resolvePreferredOpenClawTmpDir(), "openclaw-cli-prompt-image-"),
);
const sourceImage = path.join(tempDir, "bb-image.png");
await fs.writeFile(
sourceImage,
Buffer.from(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=",
"base64",
),
);
await fs.writeFile(sourceImage, createSolidPngBuffer(1, 1, { r: 255, g: 255, b: 255 }));
try {
const prepared = await prepareCliPromptImagePayload({
@@ -321,13 +316,7 @@ describe("writeCliImages", () => {
path.join(resolvePreferredOpenClawTmpDir(), "openclaw-cli-prompt-image-generic-"),
);
const sourceImage = path.join(tempDir, "claude-image.png");
await fs.writeFile(
sourceImage,
Buffer.from(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=",
"base64",
),
);
await fs.writeFile(sourceImage, createSolidPngBuffer(1, 1, { r: 255, g: 255, b: 255 }));
try {
const prompt = `[media attached: ${sourceImage} (image/png)]\n\n<media:image>`;
@@ -407,13 +396,7 @@ describe("writeCliImages", () => {
path.join(resolvePreferredOpenClawTmpDir(), "openclaw-cli-explicit-images-"),
);
const sourceImage = path.join(tempDir, "ignored-prompt-image.png");
await fs.writeFile(
sourceImage,
Buffer.from(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=",
"base64",
),
);
await fs.writeFile(sourceImage, createSolidPngBuffer(1, 1, { r: 255, g: 255, b: 255 }));
const explicitImage: ImageContent = {
type: "image",
data: "c29tZS1leHBsaWNpdC1pbWFnZQ==",

View File

@@ -14,6 +14,11 @@ import {
splitPromptAndAttachmentRefs,
} from "./images.js";
const TINY_PNG_BASE64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAACXBIWXMAAAsTAAALEwEAmpwYAAAADUlEQVR4nGP4////KwAJ5gPoxLp9owAAAABJRU5ErkJggg==";
const OPTIMIZED_TINY_PNG_BASE64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR4nGP4////KwAJ5gPoxLp9owAAAABJRU5ErkJggg==";
function expectNoPromptImages(result: { detectedRefs: unknown[]; images: unknown[] }) {
expect(result.detectedRefs).toHaveLength(0);
expect(result.images).toHaveLength(0);
@@ -358,8 +363,7 @@ describe("loadImageFromRef", () => {
const sandboxRoot = path.join(sandboxParent, "sandbox");
await fs.mkdir(sandboxRoot, { recursive: true });
const imagePath = path.join(sandboxRoot, "photo.png");
const pngB64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=";
const pngB64 = TINY_PNG_BASE64;
await fs.writeFile(imagePath, Buffer.from(pngB64, "base64"));
const image = await loadImageFromRef(
@@ -379,9 +383,7 @@ describe("loadImageFromRef", () => {
expect(image?.type).toBe("image");
expect(image?.mimeType).toBe("image/png");
expect(image?.data).toBe(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAACXBIWXMAAAsTAAALEwEAmpwYAAAADUlEQVR4nGP4////KwAJ5gPoxLp9owAAAABJRU5ErkJggg==",
);
expect(image?.data).toBe(OPTIMIZED_TINY_PNG_BASE64);
} finally {
await fs.rm(sandboxParent, { recursive: true, force: true });
}
@@ -425,8 +427,7 @@ describe("detectAndLoadPromptImages", () => {
it("skips generated media-note refs already supplied inline", async () => {
const stateDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-native-image-dedupe-"));
const imagePath = path.join(stateDir, "photo.png");
const pngB64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+/p9sAAAAASUVORK5CYII=";
const pngB64 = TINY_PNG_BASE64;
await fs.writeFile(imagePath, Buffer.from(pngB64, "base64"));
try {
@@ -449,8 +450,7 @@ describe("detectAndLoadPromptImages", () => {
});
it("keeps distinct inline attachments with identical bytes", async () => {
const pngB64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+/p9sAAAAASUVORK5CYII=";
const pngB64 = TINY_PNG_BASE64;
const image = { type: "image" as const, data: pngB64, mimeType: "image/png" };
const result = await detectAndLoadPromptImages({
@@ -512,8 +512,7 @@ describe("detectAndLoadPromptImages", () => {
const agentRoot = path.join(stateDir, "agent");
await fs.mkdir(sandboxRoot, { recursive: true });
await fs.mkdir(agentRoot, { recursive: true });
const pngB64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=";
const pngB64 = TINY_PNG_BASE64;
await fs.writeFile(path.join(agentRoot, "secret.png"), Buffer.from(pngB64, "base64"));
const sandbox = createUnsafeMountedSandbox({ sandboxRoot, agentRoot });
const bridge = sandbox.fsBridge;
@@ -546,8 +545,7 @@ describe("detectAndLoadPromptImages", () => {
await fs.mkdir(workspaceDir, { recursive: true });
await fs.mkdir(inboundDir, { recursive: true });
const imagePath = path.join(inboundDir, "signal-replay.png");
const pngB64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=";
const pngB64 = TINY_PNG_BASE64;
await fs.writeFile(imagePath, Buffer.from(pngB64, "base64"));
vi.stubEnv("OPENCLAW_STATE_DIR", stateDir);

View File

@@ -1,5 +1,5 @@
import sharp from "sharp";
import { beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { createSolidPngBuffer } from "../../test/helpers/image-fixtures.js";
const { infoMock, warnMock } = vi.hoisted(() => ({
infoMock: vi.fn(),
@@ -25,14 +25,7 @@ vi.mock("../logging/subsystem.js", () => {
import { sanitizeContentBlocksImages } from "./tool-images.js";
async function createLargePng(): Promise<Buffer> {
const width = 2001;
const height = 8;
const raw = Buffer.alloc(width * height * 3, 0x7f);
return await sharp(raw, {
raw: { width, height, channels: 3 },
})
.png({ compressionLevel: 0 })
.toBuffer();
return createSolidPngBuffer(2001, 8, { r: 0x7f, g: 0x7f, b: 0x7f });
}
describe("tool-images log context", () => {

View File

@@ -1,5 +1,10 @@
import sharp from "sharp";
import { describe, expect, it } from "vitest";
import {
createNoisyPngBuffer,
createSolidPngBuffer,
createTinyJpegBuffer,
} from "../../test/helpers/image-fixtures.js";
import { getImageMetadata } from "../media/image-ops.js";
import { sanitizeContentBlocksImages, sanitizeImageBlocks } from "./tool-images.js";
describe("tool image sanitizing", () => {
@@ -30,26 +35,14 @@ describe("tool image sanitizing", () => {
};
const createWidePng = async () => {
const width = 420;
const height = 120;
const raw = Buffer.alloc(width * height * 3, 0x7f);
return sharp(raw, {
raw: { width, height, channels: 3 },
})
.png({ compressionLevel: 9 })
.toBuffer();
return createSolidPngBuffer(420, 120, { r: 0x7f, g: 0x7f, b: 0x7f });
};
it("shrinks oversized images to the configured byte limit", async () => {
const maxBytes = 16 * 1024;
const maxBytes = 64 * 1024;
const width = 300;
const height = 300;
const raw = Buffer.alloc(width * height * 3, 0xff);
const bigPng = await sharp(raw, {
raw: { width, height, channels: 3 },
})
.png({ compressionLevel: 0 })
.toBuffer();
const bigPng = createNoisyPngBuffer(width, height);
expect(bigPng.byteLength).toBeGreaterThan(maxBytes);
const blocks = [
@@ -78,9 +71,9 @@ describe("tool image sanitizing", () => {
});
expect(dropped).toBe(0);
expect(out.length).toBe(1);
const meta = await sharp(Buffer.from(out[0].data, "base64")).metadata();
expect(meta.width).toBeLessThanOrEqual(120);
expect(meta.height).toBeLessThanOrEqual(120);
const meta = await getImageMetadata(Buffer.from(out[0].data, "base64"));
expect(meta?.width).toBeLessThanOrEqual(120);
expect(meta?.height).toBeLessThanOrEqual(120);
}, 20_000);
it("shrinks images that exceed max dimension even if size is small", async () => {
@@ -96,9 +89,9 @@ describe("tool image sanitizing", () => {
const out = await sanitizeContentBlocksImages(blocks, "test", { maxDimensionPx: 120 });
const image = getImageBlock(out);
const meta = await sharp(Buffer.from(image.data, "base64")).metadata();
expect(meta.width).toBeLessThanOrEqual(120);
expect(meta.height).toBeLessThanOrEqual(120);
const meta = await getImageMetadata(Buffer.from(image.data, "base64"));
expect(meta?.width).toBeLessThanOrEqual(120);
expect(meta?.height).toBeLessThanOrEqual(120);
expect(image.mimeType).toBe("image/jpeg");
}, 20_000);
@@ -126,16 +119,7 @@ describe("tool image sanitizing", () => {
}, 20_000);
it("corrects mismatched jpeg mimeType", async () => {
const jpeg = await sharp({
create: {
width: 10,
height: 10,
channels: 3,
background: { r: 255, g: 0, b: 0 },
},
})
.jpeg()
.toBuffer();
const jpeg = createTinyJpegBuffer();
const blocks = [
{

View File

@@ -1963,8 +1963,6 @@ describe("image tool data URL support", () => {
});
describe("image tool MiniMax VLM routing", () => {
const pngB64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=";
const priorFetch = global.fetch;
registerImageToolEnvReset(priorFetch, [
"MINIMAX_API_KEY",
@@ -1997,7 +1995,7 @@ describe("image tool MiniMax VLM routing", () => {
const res = await tool.execute("t1", {
prompt: "Describe the image.",
image: `data:image/png;base64,${pngB64}`,
image: `data:image/png;base64,${ONE_PIXEL_PNG_B64}`,
});
expect(fetch).toHaveBeenCalledTimes(1);
@@ -2021,7 +2019,10 @@ describe("image tool MiniMax VLM routing", () => {
const res = await tool.execute("t1", {
prompt: "Compare these images.",
images: [`data:image/png;base64,${pngB64}`, `data:image/png;base64,${secondPngB64}`],
images: [
`data:image/png;base64,${ONE_PIXEL_PNG_B64}`,
`data:image/png;base64,${secondPngB64}`,
],
});
expect(fetch).toHaveBeenCalledTimes(2);
@@ -2039,9 +2040,9 @@ describe("image tool MiniMax VLM routing", () => {
const deduped = await tool.execute("t1", {
prompt: "Compare these images.",
image: `data:image/png;base64,${pngB64}`,
image: `data:image/png;base64,${ONE_PIXEL_PNG_B64}`,
images: [
`data:image/png;base64,${pngB64}`,
`data:image/png;base64,${ONE_PIXEL_PNG_B64}`,
`data:image/png;base64,${secondPngB64}`,
`data:image/png;base64,${secondPngB64}`,
],
@@ -2057,7 +2058,7 @@ describe("image tool MiniMax VLM routing", () => {
const tooMany = await tool.execute("t2", {
prompt: "Compare these images.",
image: `data:image/png;base64,${pngB64}`,
image: `data:image/png;base64,${ONE_PIXEL_PNG_B64}`,
images: [`data:image/gif;base64,${ONE_PIXEL_GIF_B64}`],
maxImages: 1,
});
@@ -2081,7 +2082,7 @@ describe("image tool MiniMax VLM routing", () => {
await expect(
tool.execute("t1", {
prompt: "Describe the image.",
image: `data:image/png;base64,${pngB64}`,
image: `data:image/png;base64,${ONE_PIXEL_PNG_B64}`,
}),
).rejects.toThrow(/MiniMax VLM API error/i);
});

View File

@@ -4,6 +4,10 @@ import type { AddressInfo } from "node:net";
import os from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import {
createNoisyPngBuffer as createNoisyPngFixtureBuffer,
createSolidPngBuffer,
} from "../../test/helpers/image-fixtures.js";
import { createPinnedLookup } from "../infra/net/ssrf.js";
import { setMediaStoreNetworkDepsForTest } from "../media/store.js";
@@ -43,33 +47,12 @@ const TINY_PNG_BASE64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAusB9WnXcZ0AAAAASUVORK5CYII=";
async function createPngDataUrl(width: number, height: number): Promise<string> {
const sharp = (await import("sharp")).default;
const buffer = await sharp({
create: {
width,
height,
channels: 4,
background: { r: 24, g: 64, b: 128, alpha: 1 },
},
})
.png()
.toBuffer();
const buffer = createSolidPngBuffer(width, height, { r: 24, g: 64, b: 128 });
return `data:image/png;base64,${buffer.toString("base64")}`;
}
async function createNoisyPngBuffer(width: number, height: number): Promise<Buffer> {
const sharp = (await import("sharp")).default;
const pixels = Buffer.alloc(width * height * 4);
for (let i = 0; i < pixels.length; i += 4) {
const seed = i / 4;
pixels[i] = seed % 251;
pixels[i + 1] = (seed * 17) % 253;
pixels[i + 2] = (seed * 29) % 255;
pixels[i + 3] = 255;
}
return sharp(pixels, { raw: { width, height, channels: 4 } })
.png({ compressionLevel: 0 })
.toBuffer();
return createNoisyPngFixtureBuffer(width, height);
}
function requireAttachmentIdFromUrl(url: unknown): string {

View File

@@ -3,6 +3,7 @@ import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
import { createGrayscaleAlphaPngBuffer } from "../../test/helpers/image-fixtures.js";
import { resolveSystemBin } from "../infra/resolve-system-bin.js";
import {
convertHeicToJpeg,
@@ -12,6 +13,7 @@ import {
isImageProcessorUnavailableError,
MAX_IMAGE_INPUT_PIXELS,
resizeToJpeg,
resizeToPng,
} from "./image-ops.js";
import { createPngBufferWithDimensions } from "./test-helpers.js";
@@ -41,6 +43,32 @@ function createHeifLikeBuffer(...sizes: Array<{ width: number; height: number }>
return Buffer.concat([isoBox("ftyp", ftypPayload), meta]);
}
function createBmpHeaderBuffer(width: number, height: number): Buffer {
const buffer = Buffer.alloc(26);
buffer.write("BM", 0, "ascii");
buffer.writeUInt32LE(40, 14);
buffer.writeInt32LE(width, 18);
buffer.writeInt32LE(height, 22);
return buffer;
}
function createTiffHeaderBuffer(width: number, height: number): Buffer {
const buffer = Buffer.alloc(38);
buffer.write("II", 0, "ascii");
buffer.writeUInt16LE(42, 2);
buffer.writeUInt32LE(8, 4);
buffer.writeUInt16LE(2, 8);
buffer.writeUInt16LE(256, 10);
buffer.writeUInt16LE(4, 12);
buffer.writeUInt32LE(1, 14);
buffer.writeUInt32LE(width, 18);
buffer.writeUInt16LE(257, 22);
buffer.writeUInt16LE(4, 24);
buffer.writeUInt32LE(1, 26);
buffer.writeUInt32LE(height, 30);
return buffer;
}
describe("image input pixel guard", () => {
const oversizedPng = createPngBufferWithDimensions({ width: 8_000, height: 4_000 });
const overflowedPng = createPngBufferWithDimensions({
@@ -82,6 +110,17 @@ describe("image input pixel guard", () => {
});
});
it("reads BMP and TIFF dimensions before selecting an image backend", async () => {
await expect(getImageMetadata(createBmpHeaderBuffer(640, 480))).resolves.toEqual({
width: 640,
height: 480,
});
await expect(getImageMetadata(createTiffHeaderBuffer(320, 240))).resolves.toEqual({
width: 320,
height: 240,
});
});
it("rejects oversized HEIF-style ISO BMFF images before fallback tools run", async () => {
const oversizedHeif = createHeifLikeBuffer(
{ width: 64, height: 64 },
@@ -123,7 +162,7 @@ describe("image input pixel guard", () => {
).toBe(true);
expect(
isImageProcessorUnavailableError(
new Error("Optional dependency sharp is required for image attachment processing"),
new Error("Photon did not expose the required image processor API"),
),
).toBe(true);
});
@@ -137,6 +176,39 @@ describe("image input pixel guard", () => {
await expect(hasAlphaChannel(opaquePng)).resolves.toBe(false);
});
it("resizes grayscale alpha PNGs through the Photon backend", async () => {
const source = createGrayscaleAlphaPngBuffer(64, 32);
await expect(hasAlphaChannel(source)).resolves.toBe(true);
const jpeg = await resizeToJpeg({
buffer: source,
maxSide: 16,
quality: 80,
withoutEnlargement: true,
});
await expect(getImageMetadata(jpeg)).resolves.toEqual({ width: 16, height: 8 });
});
it("honors PNG compression levels in the Photon backend", async () => {
const source = createGrayscaleAlphaPngBuffer(128, 128);
const uncompressed = await resizeToPng({
buffer: source,
maxSide: 128,
compressionLevel: 0,
withoutEnlargement: true,
});
const compressed = await resizeToPng({
buffer: source,
maxSide: 128,
compressionLevel: 9,
withoutEnlargement: true,
});
expect(compressed.length).toBeLessThan(uncompressed.length);
await expect(getImageMetadata(compressed)).resolves.toEqual({ width: 128, height: 128 });
});
const itIfFfmpeg = resolveSystemBin("ffmpeg", { trust: "standard" }) ? it : it.skip;
itIfFfmpeg("honors enlargement when the ffmpeg fallback is selected", async () => {

View File

@@ -37,7 +37,7 @@ type ResizeToPngParams = {
};
type ImageBackend =
| "sharp"
| "photon"
| "sips"
| "windows-native"
| "imagemagick"
@@ -93,11 +93,12 @@ export function isImageProcessorUnavailableError(err: unknown): boolean {
const detail = messages.join("\n").toLowerCase();
return (
detail.includes("image processor unavailable") ||
detail.includes("optional dependency sharp is required") ||
detail.includes("cannot find package 'sharp'") ||
detail.includes('cannot find package "sharp"') ||
detail.includes("cannot find module 'sharp'") ||
detail.includes('cannot find module "sharp"')
detail.includes("photon did not expose") ||
detail.includes("photon backend skipped") ||
detail.includes("cannot find package '@silvia-odwyer/photon-node'") ||
detail.includes('cannot find package "@silvia-odwyer/photon-node"') ||
detail.includes("cannot find module '@silvia-odwyer/photon-node'") ||
detail.includes('cannot find module "@silvia-odwyer/photon-node"')
);
}
@@ -111,7 +112,7 @@ export function buildImageResizeSideGrid(maxSide: number, sideStart: number): nu
function getImageBackendPreference(): ImageBackendPreference {
const raw = process.env.OPENCLAW_IMAGE_BACKEND?.trim().toLowerCase();
switch (raw) {
case "sharp":
case "photon":
case "sips":
case "windows-native":
case "imagemagick":
@@ -134,7 +135,7 @@ function getImageBackendPreference(): ImageBackendPreference {
}
function shouldFailClosedOnUnknownMetadata(): boolean {
return getImageBackendPreference() !== "auto";
return true;
}
function imageBackendsForOperation(operation: ImageOperation): ImageBackend[] {
@@ -145,32 +146,35 @@ function imageBackendsForOperation(operation: ImageOperation): ImageBackend[] {
if (operation === "resizeToPng") {
if (process.platform === "win32") {
return ["sharp", "windows-native", "imagemagick", "graphicsmagick"];
return ["photon", "windows-native", "imagemagick", "graphicsmagick"];
}
return ["sharp", "imagemagick", "graphicsmagick"];
return ["photon", "imagemagick", "graphicsmagick"];
}
if (operation === "normalizeExifOrientation") {
if (process.platform === "win32") {
return ["sharp", "imagemagick", "graphicsmagick"];
return ["photon", "imagemagick", "graphicsmagick"];
}
return process.platform === "darwin"
? ["sharp", "sips", "imagemagick", "graphicsmagick"]
: ["sharp", "imagemagick", "graphicsmagick"];
? ["photon", "sips", "imagemagick", "graphicsmagick"]
: ["photon", "imagemagick", "graphicsmagick"];
}
if (process.platform === "win32") {
if (operation === "convertHeicToJpeg") {
return ["sharp", "imagemagick", "graphicsmagick", "ffmpeg"];
return ["imagemagick", "graphicsmagick", "ffmpeg"];
}
return ["sharp", "windows-native", "imagemagick", "graphicsmagick", "ffmpeg"];
return ["photon", "windows-native", "imagemagick", "graphicsmagick", "ffmpeg"];
}
const fallbacks =
process.platform === "darwin"
? (["sips", "imagemagick", "graphicsmagick", "ffmpeg"] as const)
: (["imagemagick", "graphicsmagick", "ffmpeg"] as const);
return ["sharp", ...fallbacks];
if (operation === "convertHeicToJpeg") {
return [...fallbacks];
}
return ["photon", ...fallbacks];
}
function createImageProcessorUnavailableError(
@@ -180,10 +184,10 @@ function createImageProcessorUnavailableError(
const backends = imageBackendsForOperation(operation).join(", ");
const hint =
process.platform === "win32"
? "Install Sharp, ImageMagick, GraphicsMagick, or ffmpeg; Windows native image resizing is tried automatically when available."
? "Install ImageMagick, GraphicsMagick, or ffmpeg; Windows native image resizing is tried automatically when available."
: process.platform === "darwin"
? "Install Sharp or a system image tool such as sips, ImageMagick, GraphicsMagick, or ffmpeg."
: "Install Sharp, ImageMagick, GraphicsMagick, or ffmpeg.";
? "Install a system image tool such as sips, ImageMagick, GraphicsMagick, or ffmpeg."
: "Install ImageMagick, GraphicsMagick, or ffmpeg.";
return new ImageProcessorUnavailableError(
operation,
`Image processor unavailable for ${operation}; tried: ${backends}. ${hint}`,
@@ -200,11 +204,12 @@ function isImageBackendUnavailableCause(error: unknown): boolean {
}
const detail = messages.join("\n").toLowerCase();
return (
detail.includes("optional dependency sharp is required") ||
detail.includes("cannot find package 'sharp'") ||
detail.includes('cannot find package "sharp"') ||
detail.includes("cannot find module 'sharp'") ||
detail.includes('cannot find module "sharp"') ||
detail.includes("photon did not expose") ||
detail.includes("photon backend skipped") ||
detail.includes("cannot find package '@silvia-odwyer/photon-node'") ||
detail.includes('cannot find package "@silvia-odwyer/photon-node"') ||
detail.includes("cannot find module '@silvia-odwyer/photon-node'") ||
detail.includes('cannot find module "@silvia-odwyer/photon-node"') ||
detail.includes("support for this compression format has not been built in") ||
detail.includes("is not available") ||
detail.includes("command not found") ||
@@ -374,6 +379,80 @@ function readWebpMetadata(buffer: Buffer): ImageMetadata | null {
return null;
}
function readBmpMetadata(buffer: Buffer): ImageMetadata | null {
if (buffer.length < 26 || buffer.toString("ascii", 0, 2) !== "BM") {
return null;
}
const dibHeaderSize = buffer.readUInt32LE(14);
if (dibHeaderSize === 12) {
return buildImageMetadata(buffer.readUInt16LE(18), buffer.readUInt16LE(20));
}
if (dibHeaderSize < 40 || buffer.length < 26) {
return null;
}
return buildImageMetadata(buffer.readInt32LE(18), Math.abs(buffer.readInt32LE(22)));
}
function readTiffUnsignedInteger(buffer: Buffer, offset: number, littleEndian: boolean): number {
return littleEndian ? buffer.readUInt16LE(offset) : buffer.readUInt16BE(offset);
}
function readTiffUnsignedLong(buffer: Buffer, offset: number, littleEndian: boolean): number {
return littleEndian ? buffer.readUInt32LE(offset) : buffer.readUInt32BE(offset);
}
function readTiffMetadata(buffer: Buffer): ImageMetadata | null {
if (buffer.length < 8) {
return null;
}
const byteOrder = buffer.toString("ascii", 0, 2);
const littleEndian = byteOrder === "II";
if (!littleEndian && byteOrder !== "MM") {
return null;
}
if (readTiffUnsignedInteger(buffer, 2, littleEndian) !== 42) {
return null;
}
const ifdOffset = readTiffUnsignedLong(buffer, 4, littleEndian);
if (ifdOffset + 2 > buffer.length) {
return null;
}
const entryCount = readTiffUnsignedInteger(buffer, ifdOffset, littleEndian);
let width: number | null = null;
let height: number | null = null;
for (let index = 0; index < entryCount; index += 1) {
const entryOffset = ifdOffset + 2 + index * 12;
if (entryOffset + 12 > buffer.length) {
return null;
}
const tag = readTiffUnsignedInteger(buffer, entryOffset, littleEndian);
if (tag !== 256 && tag !== 257) {
continue;
}
const type = readTiffUnsignedInteger(buffer, entryOffset + 2, littleEndian);
const count = readTiffUnsignedLong(buffer, entryOffset + 4, littleEndian);
if (count !== 1 || (type !== 3 && type !== 4)) {
continue;
}
const value =
type === 3
? readTiffUnsignedInteger(buffer, entryOffset + 8, littleEndian)
: readTiffUnsignedLong(buffer, entryOffset + 8, littleEndian);
if (tag === 256) {
width = value;
} else {
height = value;
}
}
return width === null || height === null ? null : buildImageMetadata(width, height);
}
const ISO_BMFF_IMAGE_BRANDS = new Set([
"avif",
"avis",
@@ -541,11 +620,28 @@ export function readImageMetadataFromHeader(buffer: Buffer): ImageMetadata | nul
readPngMetadata(buffer) ??
readGifMetadata(buffer) ??
readWebpMetadata(buffer) ??
readBmpMetadata(buffer) ??
readTiffMetadata(buffer) ??
readIsoBmffImageMetadata(buffer) ??
readJpegMetadata(buffer)
);
}
function hasPhotonDecodableHeader(buffer: Buffer): boolean {
return (
readPngMetadata(buffer) !== null ||
readGifMetadata(buffer) !== null ||
readWebpMetadata(buffer) !== null ||
readJpegMetadata(buffer) !== null
);
}
function assertPhotonDecodableHeader(buffer: Buffer): void {
if (!hasPhotonDecodableHeader(buffer)) {
throw new Error("Photon backend skipped for image format handled by external tools");
}
}
function countImagePixels(meta: ImageMetadata): number | null {
const pixels = meta.width * meta.height;
return Number.isSafeInteger(pixels) ? pixels : null;
@@ -704,7 +800,7 @@ function clampInteger(value: number, min: number, max: number): number {
return Math.max(min, Math.min(max, Math.round(value)));
}
function resolveImageTool(backend: Exclude<ImageBackend, "sharp">): ExternalImageTool | null {
function resolveImageTool(backend: Exclude<ImageBackend, "photon">): ExternalImageTool | null {
if (backend === "sips") {
return process.platform === "darwin"
? { backend, flavor: "sips", command: "/usr/bin/sips" }
@@ -888,7 +984,7 @@ function buildFfmpegResizeFilter(maxSide: number, withoutEnlargement?: boolean):
}
async function externalResizeToJpeg(
backend: Exclude<ImageBackend, "sharp">,
backend: Exclude<ImageBackend, "photon">,
params: ResizeToJpegParams,
): Promise<Buffer> {
const tool = resolveImageTool(backend);
@@ -959,7 +1055,7 @@ async function externalResizeToJpeg(
}
async function externalConvertToJpeg(
backend: Exclude<ImageBackend, "sharp">,
backend: Exclude<ImageBackend, "photon">,
buffer: Buffer,
): Promise<Buffer> {
const tool = resolveImageTool(backend);
@@ -988,7 +1084,7 @@ async function externalConvertToJpeg(
}
async function externalNormalizeExifOrientation(
backend: Exclude<ImageBackend, "sharp" | "ffmpeg">,
backend: Exclude<ImageBackend, "photon" | "ffmpeg">,
buffer: Buffer,
): Promise<Buffer> {
if (backend === "sips") {
@@ -1010,7 +1106,7 @@ async function externalNormalizeExifOrientation(
}
async function externalResizeToPng(
backend: Exclude<ImageBackend, "sharp" | "sips" | "ffmpeg">,
backend: Exclude<ImageBackend, "photon" | "sips" | "ffmpeg">,
params: ResizeToPngParams,
): Promise<Buffer> {
const tool = resolveImageTool(backend);
@@ -1091,7 +1187,7 @@ export async function getImageMetadata(buffer: Buffer): Promise<ImageMetadata |
}
const preference = getImageBackendPreference();
if (preference !== "auto" && preference !== "sharp") {
if (preference !== "auto" && preference !== "photon") {
return null;
}
@@ -1156,7 +1252,8 @@ export async function normalizeExifOrientation(buffer: Buffer): Promise<Buffer>
for (const backend of imageBackendsForOperation("normalizeExifOrientation")) {
try {
if (backend === "sharp") {
if (backend === "photon") {
assertPhotonDecodableHeader(buffer);
const ops = await loadMediaAttachmentImageOps();
return await ops.normalizeExifOrientation(buffer);
}
@@ -1175,7 +1272,8 @@ export async function normalizeExifOrientation(buffer: Buffer): Promise<Buffer>
export async function resizeToJpeg(params: ResizeToJpegParams): Promise<Buffer> {
await assertImagePixelLimit(params.buffer);
return await runWithImageBackends("resizeToJpeg", async (backend) => {
if (backend === "sharp") {
if (backend === "photon") {
assertPhotonDecodableHeader(params.buffer);
return await (await loadMediaAttachmentImageOps()).resizeToJpeg(params);
}
assertKnownImagePixelLimitBeforeExternalFallback(params.buffer);
@@ -1186,8 +1284,8 @@ export async function resizeToJpeg(params: ResizeToJpegParams): Promise<Buffer>
export async function convertHeicToJpeg(buffer: Buffer): Promise<Buffer> {
await assertImagePixelLimit(buffer);
return await runWithImageBackends("convertHeicToJpeg", async (backend) => {
if (backend === "sharp") {
return await (await loadMediaAttachmentImageOps()).convertHeicToJpeg(buffer);
if (backend === "photon") {
throw new Error("Photon does not support HEIC/AVIF conversion");
}
assertKnownImagePixelLimitBeforeExternalFallback(buffer);
return await externalConvertToJpeg(backend, buffer);
@@ -1221,7 +1319,8 @@ export async function hasAlphaChannel(buffer: Buffer): Promise<boolean> {
export async function resizeToPng(params: ResizeToPngParams): Promise<Buffer> {
await assertImagePixelLimit(params.buffer);
return await runWithImageBackends("resizeToPng", async (backend) => {
if (backend === "sharp") {
if (backend === "photon") {
assertPhotonDecodableHeader(params.buffer);
return await (await loadMediaAttachmentImageOps()).resizeToPng(params);
}
if (backend === "windows-native" || backend === "imagemagick" || backend === "graphicsmagick") {
@@ -1299,7 +1398,7 @@ export async function optimizeImageToPng(
}
/**
* Internal sips-only EXIF normalization (no sharp fallback).
* Internal sips-only EXIF normalization (no Photon fallback).
* Used by resizeToJpeg to normalize before sips resize.
*/
async function normalizeExifOrientationSips(buffer: Buffer): Promise<Buffer> {

View File

@@ -60,9 +60,8 @@ export function fillPixel(
buf[idx + 3] = a;
}
/** Encode an RGBA buffer as a PNG image. */
export function encodePngRgba(buffer: Buffer, width: number, height: number): Buffer {
const stride = width * 4;
function encodePng(buffer: Buffer, width: number, height: number, channels: 3 | 4): Buffer {
const stride = width * channels;
const raw = Buffer.alloc((stride + 1) * height);
for (let row = 0; row < height; row += 1) {
const rawOffset = row * (stride + 1);
@@ -76,7 +75,7 @@ export function encodePngRgba(buffer: Buffer, width: number, height: number): Bu
ihdr.writeUInt32BE(width, 0);
ihdr.writeUInt32BE(height, 4);
ihdr[8] = 8; // bit depth
ihdr[9] = 6; // color type RGBA
ihdr[9] = channels === 4 ? 6 : 2; // color type RGB/RGBA
ihdr[10] = 0; // compression
ihdr[11] = 0; // filter
ihdr[12] = 0; // interlace
@@ -88,3 +87,13 @@ export function encodePngRgba(buffer: Buffer, width: number, height: number): Bu
pngChunk("IEND", Buffer.alloc(0)),
]);
}
/** Encode an RGB buffer as a PNG image. */
export function encodePngRgb(buffer: Buffer, width: number, height: number): Buffer {
return encodePng(buffer, width, height, 3);
}
/** Encode an RGBA buffer as a PNG image. */
export function encodePngRgba(buffer: Buffer, width: number, height: number): Buffer {
return encodePng(buffer, width, height, 4);
}

View File

@@ -3,8 +3,8 @@ import path from "node:path";
import { Readable } from "node:stream";
import JSZip from "jszip";
import { importFreshModule } from "openclaw/plugin-sdk/test-fixtures";
import sharp from "sharp";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { createSolidPngBuffer, createTinyJpegBuffer } from "../../test/helpers/image-fixtures.js";
import { isPathWithinBase } from "../../test/helpers/paths.js";
import { createTempHomeEnv, type TempHomeEnv } from "../test-utils/temp-home.js";
@@ -644,11 +644,7 @@ describe("media store", () => {
{
name: "saves jpeg buffers with the detected extension",
bufferFactory: async () => {
return await sharp({
create: { width: 2, height: 2, channels: 3, background: "#123456" },
})
.jpeg({ quality: 80 })
.toBuffer();
return createTinyJpegBuffer();
},
contentType: "image/jpeg",
expectedContentType: "image/jpeg",
@@ -845,11 +841,7 @@ describe("media store", () => {
name: "renames media based on detected mime even when extension is wrong",
relativeSourcePath: "image-wrong.bin",
contentsFactory: async () => {
return await sharp({
create: { width: 2, height: 2, channels: 3, background: "#00ff00" },
})
.png()
.toBuffer();
return createSolidPngBuffer(2, 2, { r: 0, g: 255, b: 0 });
},
expectedContentType: "image/png",
expectedExtension: ".png",

View File

@@ -3,6 +3,7 @@ import path from "node:path";
import { pathToFileURL } from "node:url";
import JSZip from "jszip";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { createSolidPngBuffer } from "../../test/helpers/image-fixtures.js";
import { resolveStateDir } from "../config/paths.js";
import { resolvePreferredOpenClawTmpDir } from "../infra/tmp-openclaw-dir.js";
import { createEmptyPluginRegistry } from "../plugins/registry-empty.js";
@@ -17,8 +18,8 @@ let loadWebMediaRaw: typeof import("./web-media.js").loadWebMediaRaw;
let optimizeImageToJpeg: typeof import("./web-media.js").optimizeImageToJpeg;
let resolveImageCompressionGrid: typeof import("./web-media.js").resolveImageCompressionGrid;
const TINY_PNG_BASE64 =
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/woAAn8B9FD5fHAAAAAASUVORK5CYII=";
const TINY_PNG_BUFFER = createSolidPngBuffer(1, 1, { r: 255, g: 255, b: 255 });
const TINY_PNG_BASE64 = TINY_PNG_BUFFER.toString("base64");
const CANVAS_HOST_PATH = "/__openclaw__/canvas";
let fixtureRoot = "";
@@ -448,19 +449,19 @@ describe("loadWebMedia", () => {
convertHeicToJpeg: vi.fn(async (buffer: Buffer) => buffer),
hasAlphaChannel: vi.fn(async () => {
throw new Error(
"Optional dependency sharp is required for image attachment processing | Cannot find package 'sharp' imported from image-ops.js",
"Photon did not expose the required image processor API | Cannot find package '@silvia-odwyer/photon-node' imported from image-ops.js",
);
}),
isImageProcessorUnavailableError: (err: unknown) =>
err instanceof Error && err.message.includes("Optional dependency sharp is required"),
err instanceof Error && err.message.includes("Photon did not expose"),
optimizeImageToPng: vi.fn(async () => {
throw new Error(
"Optional dependency sharp is required for image attachment processing | Cannot find package 'sharp' imported from image-ops.js",
"Photon did not expose the required image processor API | Cannot find package '@silvia-odwyer/photon-node' imported from image-ops.js",
);
}),
resizeToJpeg: vi.fn(async () => {
throw new Error(
"Optional dependency sharp is required for image attachment processing | Cannot find package 'sharp' imported from image-ops.js",
"Photon did not expose the required image processor API | Cannot find package '@silvia-odwyer/photon-node' imported from image-ops.js",
);
}),
}));
@@ -472,7 +473,7 @@ describe("loadWebMedia", () => {
}
}
it("sends an in-limit original image when optional sharp optimization is unavailable", async () => {
it("sends an in-limit original image when image optimization is unavailable", async () => {
await withUnavailableImageOptimizer(async () => {
const { loadWebMedia: loadWebMediaWithMissingOptimizer } = await import("./web-media.js");
const result = await loadWebMediaWithMissingOptimizer(
@@ -486,16 +487,16 @@ describe("loadWebMedia", () => {
});
});
it("does not bypass the size cap when optional sharp optimization is unavailable", async () => {
it("does not bypass the size cap when image optimization is unavailable", async () => {
await withUnavailableImageOptimizer(async () => {
const { loadWebMedia: loadWebMediaWithMissingOptimizer } = await import("./web-media.js");
await expect(
loadWebMediaWithMissingOptimizer(tinyPngFile, { maxBytes: 8, localRoots: [fixtureRoot] }),
).rejects.toThrow(/Optional dependency sharp is required/);
).rejects.toThrow(/Photon did not expose/);
});
});
it("sends an in-limit data URL image when optional sharp optimization is unavailable", async () => {
it("sends an in-limit data URL image when image optimization is unavailable", async () => {
await withUnavailableImageOptimizer(async () => {
const { optimizeImageBufferForWebMedia } = await import("./web-media.js");
const buffer = Buffer.from(TINY_PNG_BASE64, "base64");
@@ -511,7 +512,7 @@ describe("loadWebMedia", () => {
});
});
it("does not bypass the data URL image cap when optional sharp optimization is unavailable", async () => {
it("does not bypass the data URL image cap when image optimization is unavailable", async () => {
await withUnavailableImageOptimizer(async () => {
const { optimizeImageBufferForWebMedia } = await import("./web-media.js");
await expect(
@@ -521,11 +522,11 @@ describe("loadWebMedia", () => {
maxBytes: 8,
imageCompression: { models: [{ maxSidePx: 1024 }] },
}),
).rejects.toThrow(/Optional dependency sharp is required/);
).rejects.toThrow(/Photon did not expose/);
});
});
it("does not bypass model dimensions when optional sharp optimization is unavailable", async () => {
it("does not bypass model dimensions when image optimization is unavailable", async () => {
await withUnavailableImageOptimizer(async () => {
const { optimizeImageBufferForWebMedia } = await import("./web-media.js");
await expect(
@@ -535,7 +536,7 @@ describe("loadWebMedia", () => {
maxBytes: 16 * 1024 * 1024,
imageCompression: { models: [{ maxSidePx: 512 }] },
}),
).rejects.toThrow(/Optional dependency sharp is required/);
).rejects.toThrow(/Photon did not expose/);
});
});
@@ -644,14 +645,14 @@ describe("loadWebMedia", () => {
).toBe(2048);
});
it("does not send original HEIC media when optional sharp conversion is unavailable", async () => {
it("does not send original HEIC media when image conversion is unavailable", async () => {
await withUnavailableImageOptimizer(async () => {
const heicFile = path.join(fixtureRoot, "photo.heic");
await fs.writeFile(heicFile, Buffer.from("heic-source"));
const { loadWebMedia: loadWebMediaWithMissingOptimizer } = await import("./web-media.js");
await expect(
loadWebMediaWithMissingOptimizer(heicFile, createLocalWebMediaOptions()),
).rejects.toThrow(/Optional dependency sharp is required/);
).rejects.toThrow(/Photon did not expose/);
});
});
@@ -686,7 +687,7 @@ describe("loadWebMedia", () => {
});
expect(result.kind).toBe("image");
expect(result.contentType).toBe("image/png");
expect(result.contentType).toBe("image/jpeg");
expect(result.fileName).toBe("tiny.png");
});

View File

@@ -42,3 +42,9 @@ export {
repoInstallSpec,
} from "./test-helpers/bundled-plugin-paths.js";
export { importFreshModule } from "./test-helpers/import-fresh.js";
export {
createGrayscaleAlphaPngBuffer,
createNoisyPngBuffer,
createNoisyRgbaBuffer,
createSolidPngBuffer,
} from "./test-helpers/image-fixtures.js";

View File

@@ -0,0 +1,129 @@
import { deflateSync } from "node:zlib";
import { encodePngRgb, encodePngRgba } from "../../media/png-encode.js";
type Rgba = {
r: number;
g: number;
b: number;
a?: number;
};
const PNG_SIGNATURE = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
const TINY_JPEG = Buffer.from(
"/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAP//////////////////////////////////////////////////////////////////////////////////////2wBDAf//////////////////////////////////////////////////////////////////////////////////////wAARCAABAAEDASIAAhEBAxEB/8QAFQABAQAAAAAAAAAAAAAAAAAAAAX/xAAUEAEAAAAAAAAAAAAAAAAAAAAA/9oADAMBAAIQAxAAAAH/xAAUEAEAAAAAAAAAAAAAAAAAAAAA/9oACAEBAAEFAqf/xAAUEQEAAAAAAAAAAAAAAAAAAAAA/9oACAEDAQE/ASP/xAAUEQEAAAAAAAAAAAAAAAAAAAAA/9oACAECAQE/ASP/xAAUEAEAAAAAAAAAAAAAAAAAAAAA/9oACAEBAAY/Aqf/xAAUEAEAAAAAAAAAAAAAAAAAAAAA/9oACAEBAAE/IV//2gAMAwEAAgADAAAAEP/EFBQRAQAAAAAAAAAAAAAAAAAAABD/2gAIAQMBAT8QH//EFBQRAQAAAAAAAAAAAAAAAAAAABD/2gAIAQIBAT8QH//EFBABAQAAAAAAAAAAAAAAAAAAABD/2gAIAQEAAT8QH//Z",
"base64",
);
const CRC_TABLE = (() => {
const table = new Uint32Array(256);
for (let index = 0; index < table.length; index += 1) {
let value = index;
for (let bit = 0; bit < 8; bit += 1) {
value = value & 1 ? 0xedb88320 ^ (value >>> 1) : value >>> 1;
}
table[index] = value >>> 0;
}
return table;
})();
function crc32(buffer: Buffer): number {
let crc = 0xffffffff;
for (const byte of buffer) {
crc = CRC_TABLE[(crc ^ byte) & 0xff] ^ (crc >>> 8);
}
return (crc ^ 0xffffffff) >>> 0;
}
function pngChunk(type: string, data: Buffer): Buffer {
const typeBuffer = Buffer.from(type, "ascii");
const length = Buffer.alloc(4);
length.writeUInt32BE(data.length, 0);
const crc = Buffer.alloc(4);
crc.writeUInt32BE(crc32(Buffer.concat([typeBuffer, data])), 0);
return Buffer.concat([length, typeBuffer, data, crc]);
}
function fillSolidRgba(width: number, height: number, color: Rgba): Buffer {
const pixels = Buffer.alloc(width * height * 4);
for (let offset = 0; offset < pixels.length; offset += 4) {
pixels[offset] = color.r;
pixels[offset + 1] = color.g;
pixels[offset + 2] = color.b;
pixels[offset + 3] = color.a ?? 255;
}
return pixels;
}
function fillSolidRgb(width: number, height: number, color: Rgba): Buffer {
const pixels = Buffer.alloc(width * height * 3);
for (let offset = 0; offset < pixels.length; offset += 3) {
pixels[offset] = color.r;
pixels[offset + 1] = color.g;
pixels[offset + 2] = color.b;
}
return pixels;
}
export function createSolidPngBuffer(width: number, height: number, color: Rgba): Buffer {
if (color.a === undefined || color.a === 255) {
return encodePngRgb(fillSolidRgb(width, height, color), width, height);
}
return encodePngRgba(fillSolidRgba(width, height, color), width, height);
}
export function createNoisyPngBuffer(width: number, height: number): Buffer {
const rgba = createNoisyRgbaBuffer(width, height);
const rgb = Buffer.alloc(width * height * 3);
for (let source = 0, target = 0; source < rgba.length; source += 4, target += 3) {
rgb[target] = rgba[source] ?? 0;
rgb[target + 1] = rgba[source + 1] ?? 0;
rgb[target + 2] = rgba[source + 2] ?? 0;
}
return encodePngRgb(rgb, width, height);
}
export function createGrayscaleAlphaPngBuffer(width: number, height: number): Buffer {
const stride = width * 2;
const raw = Buffer.alloc((stride + 1) * height);
for (let row = 0; row < height; row += 1) {
const rawOffset = row * (stride + 1);
raw[rawOffset] = 0;
for (let column = 0; column < width; column += 1) {
const pixel = rawOffset + 1 + column * 2;
const seed = row * width + column;
raw[pixel] = seed % 256;
raw[pixel + 1] = seed % 5 === 0 ? 96 : 255;
}
}
const ihdr = Buffer.alloc(13);
ihdr.writeUInt32BE(width, 0);
ihdr.writeUInt32BE(height, 4);
ihdr[8] = 8;
ihdr[9] = 4;
ihdr[10] = 0;
ihdr[11] = 0;
ihdr[12] = 0;
return Buffer.concat([
PNG_SIGNATURE,
pngChunk("IHDR", ihdr),
pngChunk("IDAT", deflateSync(raw)),
pngChunk("IEND", Buffer.alloc(0)),
]);
}
export function createTinyJpegBuffer(): Buffer {
return Buffer.from(TINY_JPEG);
}
export function createNoisyRgbaBuffer(width: number, height: number): Buffer {
const pixels = Buffer.alloc(width * height * 4);
for (let offset = 0; offset < pixels.length; offset += 4) {
const seed = offset / 4;
pixels[offset] = seed % 251;
pixels[offset + 1] = (seed * 17) % 253;
pixels[offset + 2] = (seed * 29) % 255;
pixels[offset + 3] = 255;
}
return pixels;
}

View File

@@ -76,7 +76,7 @@ const packageManifestContractTests: PackageManifestContractParams[] = [
{ pluginId: "voice-call", minHostVersionBaseline: "2026.3.22" },
{
pluginId: "whatsapp",
pluginLocalRuntimeDeps: ["audio-decode", "baileys", "jimp"],
pluginLocalRuntimeDeps: ["audio-decode", "baileys"],
minHostVersionBaseline: "2026.3.22",
},
{ pluginId: "zalo", minHostVersionBaseline: "2026.3.22" },

View File

@@ -0,0 +1,7 @@
export {
createGrayscaleAlphaPngBuffer,
createNoisyPngBuffer,
createNoisyRgbaBuffer,
createSolidPngBuffer,
createTinyJpegBuffer,
} from "../../src/plugin-sdk/test-helpers/image-fixtures.js";

View File

@@ -570,12 +570,12 @@ describe("collectPackedTestCargoErrors", () => {
collectPackedTestCargoErrors([
"dist/extensions/webhooks/node_modules/zod/src/v3/tests/all-errors.test.ts",
"dist/extensions/whatsapp/node_modules/pino/test/basic.test.js",
"dist/extensions/whatsapp/node_modules/@jimp/plugin-crop/src/__snapshots__/crop.test.ts.snap",
"dist/extensions/whatsapp/node_modules/example-codec/src/__snapshots__/codec.test.ts.snap",
"dist/index.js",
]),
).toEqual([
'npm package must not include test cargo "dist/extensions/webhooks/node_modules/zod/src/v3/tests/all-errors.test.ts".',
'npm package must not include test cargo "dist/extensions/whatsapp/node_modules/@jimp/plugin-crop/src/__snapshots__/crop.test.ts.snap".',
'npm package must not include test cargo "dist/extensions/whatsapp/node_modules/example-codec/src/__snapshots__/codec.test.ts.snap".',
'npm package must not include test cargo "dist/extensions/whatsapp/node_modules/pino/test/basic.test.js".',
]);
});

View File

@@ -1,6 +1,7 @@
import path from "node:path";
import { describe, expect, it } from "vitest";
import {
applyPackageExtensionPeerMetadata,
collectOverrideViolations,
collectPnpmLockViolations,
createNpmShrinkwrapCommand,
@@ -10,6 +11,7 @@ import {
normalizeNpmVersionDrift,
parsePnpmPackageKey,
parseLockPackagePath,
shouldUseLegacyPeerDepsForShrinkwrap,
shrinkwrapPackageDirsForChangedPaths,
} from "../../scripts/generate-npm-shrinkwrap.mjs";
@@ -172,6 +174,57 @@ describe("generate-npm-shrinkwrap", () => {
});
});
it("uses legacy peer resolution when package extensions mark dependency peers optional", () => {
expect(
shouldUseLegacyPeerDepsForShrinkwrap(
{ dependencies: { baileys: "7.0.0-rc13" } },
{ baileys: { peerDependenciesMeta: { sharp: { optional: true } } } },
),
).toBe(true);
expect(
shouldUseLegacyPeerDepsForShrinkwrap(
{ dependencies: { "not-baileys": "1.0.0" } },
{ baileys: { peerDependenciesMeta: { sharp: { optional: true } } } },
),
).toBe(false);
});
it("applies package extension peer metadata to generated shrinkwrap packages", () => {
expect(
applyPackageExtensionPeerMetadata(
{
packages: {
"node_modules/baileys": {
version: "7.0.0-rc13",
peerDependencies: {
"audio-decode": "^2.1.3",
sharp: "*",
},
peerDependenciesMeta: {
"audio-decode": { optional: true },
},
},
},
},
{ baileys: { peerDependenciesMeta: { sharp: { optional: true } } } },
),
).toEqual({
packages: {
"node_modules/baileys": {
version: "7.0.0-rc13",
peerDependencies: {
"audio-decode": "^2.1.3",
sharp: "*",
},
peerDependenciesMeta: {
"audio-decode": { optional: true },
sharp: { optional: true },
},
},
},
});
});
it("targets changed publishable plugin shrinkwraps", () => {
expect(
shrinkwrapPackageDirsForChangedPaths([

View File

@@ -199,7 +199,7 @@ describe("install-cli.sh", () => {
expect(result.stdout).toContain("branch=--no-frozen-lockfile");
expect(result.stdout).toContain("tag=--frozen-lockfile");
expect(script).toContain(
'CI="${CI:-true}" SHARP_IGNORE_GLOBAL_LIBVIPS="$SHARP_IGNORE_GLOBAL_LIBVIPS" run_pnpm -C "$repo_dir" install "$install_lockfile_flag"',
'CI="${CI:-true}" run_pnpm -C "$repo_dir" install "$install_lockfile_flag"',
);
});

View File

@@ -473,7 +473,6 @@ describe("install.sh", () => {
`PATH=${JSON.stringify(`${bin}:/usr/bin:/bin`)}`,
"NPM_LOGLEVEL=error",
"NPM_SILENT_FLAG=",
"SHARP_IGNORE_GLOBAL_LIBVIPS=1",
`run_npm_global_install openclaw@latest ${JSON.stringify(join(tmp, "install.log"))}`,
].join("\n"),
);
@@ -512,7 +511,6 @@ describe("install.sh", () => {
`PATH=${JSON.stringify(`${bin}:/usr/bin:/bin`)}`,
"NPM_LOGLEVEL=error",
"NPM_SILENT_FLAG=",
"SHARP_IGNORE_GLOBAL_LIBVIPS=1",
`run_npm_global_install openclaw@latest ${JSON.stringify(join(tmp, "install.log"))}`,
].join("\n"),
);
@@ -926,7 +924,7 @@ describe("install.sh", () => {
expect(result.stdout).toContain("branch=--no-frozen-lockfile");
expect(result.stdout).toContain("tag=--frozen-lockfile");
expect(script).toContain(
'CI="${CI:-true}" SHARP_IGNORE_GLOBAL_LIBVIPS="$SHARP_IGNORE_GLOBAL_LIBVIPS" run_quiet_step "Installing dependencies" run_pnpm -C "$repo_dir" install "$install_lockfile_flag"',
'CI="${CI:-true}" run_quiet_step "Installing dependencies" run_pnpm -C "$repo_dir" install "$install_lockfile_flag"',
);
});