mirror of
https://github.com/openclaw/openclaw.git
synced 2026-06-06 05:51:15 +08:00
test(qa-lab): trace scenario issue evidence
This commit is contained in:
@@ -5,8 +5,8 @@ Single source of truth for repo-backed QA suite bootstrap data.
|
||||
|
||||
- `index.md` defines pack-level bootstrap data
|
||||
- each nested `*.md` scenario defines one runnable test via `qa-scenario` + `qa-flow`
|
||||
- scenario markdown may also define coverage IDs, category metadata, required plugins,
|
||||
lane filters, runtime parity tiers, and gateway config patching
|
||||
- scenario markdown may also define coverage IDs, evidence links, category metadata,
|
||||
required plugins, lane filters, runtime parity tiers, and gateway config patching
|
||||
|
||||
- kickoff mission
|
||||
- QA operator identity
|
||||
@@ -20,6 +20,9 @@ Coverage tracking:
|
||||
- prefer reusing an existing feature ID over minting a scenario-shaped ID
|
||||
- avoid copying the scenario title into coverage IDs
|
||||
- use `pnpm openclaw qa coverage` to render the current inventory
|
||||
- use `evidence.github` for full `https://github.com/openclaw/openclaw/issues/<n>` or
|
||||
`https://github.com/openclaw/openclaw/pull/<n>` links when a scenario directly protects
|
||||
a reported regression, RFC, or accepted PR behavior
|
||||
- use `runtimeParityTier` for runtime-pair gate membership: `standard`,
|
||||
`optional`, `live-only`, or `soak`
|
||||
- treat the old `coverage: ["id"]` / `coverage: - id` list shape as invalid
|
||||
|
||||
@@ -11,6 +11,10 @@ coverage:
|
||||
secondary:
|
||||
- runtime.prompt-compatibility
|
||||
- tools.fs.read
|
||||
evidence:
|
||||
github:
|
||||
- https://github.com/openclaw/openclaw/pull/80323
|
||||
- https://github.com/openclaw/openclaw/issues/81734
|
||||
objective: Verify Codex-mode agents can satisfy legacy Pi-shaped "Read tool" wording through the native Codex workspace-read capability instead of stopping because duplicate OpenClaw dynamic read is intentionally filtered.
|
||||
successCriteria:
|
||||
- Agent reads the seeded workspace file and replies with the exact marker line.
|
||||
|
||||
@@ -10,6 +10,11 @@ coverage:
|
||||
- runtime.first-hour-20
|
||||
secondary:
|
||||
- runtime.long-context
|
||||
evidence:
|
||||
github:
|
||||
- https://github.com/openclaw/openclaw/issues/80171
|
||||
- https://github.com/openclaw/openclaw/issues/80337
|
||||
- https://github.com/openclaw/openclaw/issues/80364
|
||||
objective: Verify both runtimes preserve a same-session conversation across the required 20-turn maintainer gate.
|
||||
successCriteria:
|
||||
- The same QA session accepts 20 sequential user turns.
|
||||
|
||||
@@ -10,6 +10,11 @@ coverage:
|
||||
- runtime.soak-100
|
||||
secondary:
|
||||
- runtime.long-context
|
||||
evidence:
|
||||
github:
|
||||
- https://github.com/openclaw/openclaw/issues/80171
|
||||
- https://github.com/openclaw/openclaw/issues/80338
|
||||
- https://github.com/openclaw/openclaw/issues/80395
|
||||
objective: Provide an optional long-run soak that can be scheduled or run in Testbox without entering the maintainer default gate.
|
||||
successCriteria:
|
||||
- The same QA session accepts 100 sequential user turns.
|
||||
|
||||
@@ -8,6 +8,9 @@ runtimeParityTier: standard
|
||||
coverage:
|
||||
primary:
|
||||
- tools.apply-patch
|
||||
evidence:
|
||||
github:
|
||||
- https://github.com/openclaw/openclaw/issues/80320
|
||||
objective: Verify apply_patch behavior is tracked across Pi and Codex while Codex owns patching natively.
|
||||
successCriteria:
|
||||
- Pi may expose OpenClaw apply_patch while Codex app-server mode may omit duplicate OpenClaw dynamic apply_patch.
|
||||
|
||||
@@ -8,6 +8,9 @@ runtimeParityTier: standard
|
||||
coverage:
|
||||
primary:
|
||||
- tools.fs.read
|
||||
evidence:
|
||||
github:
|
||||
- https://github.com/openclaw/openclaw/issues/80312
|
||||
objective: Verify file read behavior is tracked across Pi and Codex while Codex owns read natively.
|
||||
successCriteria:
|
||||
- Pi may expose OpenClaw read while Codex app-server mode may omit duplicate OpenClaw dynamic read.
|
||||
|
||||
Reference in New Issue
Block a user