Compare commits

..

860 Commits

Author SHA1 Message Date
Peter Steinberger
6b3f9a5934 docs: note android sms capability 2026-01-04 13:57:41 +01:00
Peter Steinberger
0372457ba8 fix(android): refresh hello on sms permission grant 2026-01-04 13:56:20 +01:00
Peter Steinberger
f28a03c407 fix(android): add sms permission flow and tests 2026-01-04 13:27:30 +01:00
Vasanth Rao Naik Sabavat
f5abc8e9c9 feat(android): add SMS sending capability to Android node
Add sms.send command to allow sending text messages via the paired Android device.

Changes:
- Add SmsManager class to handle SMS sending via Android SmsManager API
- Add ClawdisSmsCommand enum and Sms capability to protocol constants
- Wire sms.send command into NodeRuntime invoke handler
- Add SEND_SMS permission to AndroidManifest.xml
- Advertise sms capability when SEND_SMS permission is granted

The SMS capability is only advertised when the user has granted SEND_SMS
permission. Messages longer than 160 chars are automatically split into
multipart messages.
2026-01-03 22:16:46 -08:00
Peter Steinberger
4b3ca29404 build: add homebrew to sandbox image 2026-01-04 06:12:06 +00:00
Peter Steinberger
259b14d66a chore: refresh protocol models 2026-01-04 07:07:21 +01:00
Peter Steinberger
c9504a6f20 refactor: split config module 2026-01-04 07:05:17 +01:00
Peter Steinberger
5e36e2c3f3 fix: allow elevated via discord username 2026-01-04 05:47:28 +00:00
Peter Steinberger
d2da305190 feat: fallback elevated allowlist to discord dms 2026-01-04 05:31:00 +00:00
Peter Steinberger
be9fa124df build: add pkg-config + libasound2-dev to sandbox image 2026-01-04 05:28:08 +00:00
Peter Steinberger
ff88f3c075 style: fix lint ordering 2026-01-04 06:27:54 +01:00
Peter Steinberger
1315fc4caf docs: split elevated directives 2026-01-04 05:21:12 +00:00
Peter Steinberger
a03895dfa9 fix: default elevated mode to on 2026-01-04 05:19:28 +00:00
Peter Steinberger
40c3898ca1 docs: update changelog for #166 2026-01-04 06:17:07 +01:00
Peter Steinberger
6ea0eb438c style: fix lint formatting 2026-01-04 06:17:07 +01:00
Peter Steinberger
04cd1bd11a fix(macos): bridge wizard option values 2026-01-04 06:17:07 +01:00
Peter Steinberger
fe0b3500cc feat: add elevated bash mode 2026-01-04 05:15:59 +00:00
Tu Nombre Real
b978cc4e91 feat(macos): add Swift 6 strict concurrency compatibility
Prepares the macOS app for Swift 6 strict concurrency mode by:

1. Adding Sendable conformance to WizardNextResult, WizardStartResult,
   and WizardStatusResult in GatewayModels.swift

2. Adding AnyCodable bridging helpers in OnboardingWizard.swift to
   handle type conflicts between ClawdisProtocol and local module

3. Making CLLocationManagerDelegate methods nonisolated in:
   - MacNodeLocationService.swift
   - PermissionManager.swift (LocationPermissionRequester)

   Using Task { @MainActor in } pattern to safely access MainActor
   state from nonisolated protocol requirements.

These changes are forward-compatible and don't affect behavior on
current Swift versions.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-04 06:09:52 +01:00
Peter Steinberger
72a9e58777 refactor(auto-reply): split reply flow 2026-01-04 05:47:37 +01:00
Peter Steinberger
fd91da2b7f fix: log dynamic config reloads 2026-01-04 04:24:50 +00:00
Peter Steinberger
5673f4299a build: add sandbox common image builder 2026-01-04 04:17:13 +00:00
Peter Steinberger
770daadaf7 chore: bump Peekaboo submodule 2026-01-04 05:15:57 +01:00
Peter Steinberger
13c2f22240 refactor: split agent tools 2026-01-04 05:07:44 +01:00
Peter Steinberger
f2ce455c8c fix: set writable home for sandbox browser 2026-01-04 03:49:39 +00:00
Peter Steinberger
640ec465d7 chore: bump Peekaboo submodule 2026-01-04 04:46:07 +01:00
Peter Steinberger
70f79bd926 fix: stabilize sandbox browser startup 2026-01-04 03:45:14 +00:00
Peter Steinberger
7d95f43a75 style: fix lint 2026-01-04 03:37:08 +00:00
Peter Steinberger
c2f3b653c2 docs: thank scald for Notion skill 2026-01-04 04:36:28 +01:00
Peter Steinberger
12ba32c724 feat(browser): add remote-capable profiles
Co-authored-by: James Groat <james@groat.com>
2026-01-04 03:33:07 +00:00
Peter Steinberger
0e75aa2716 test: add sessions_send loopback test 2026-01-04 04:30:57 +01:00
Steve Caldwell
44990d837f feat: add Notion API skill
Create and manage Notion pages, databases, and blocks via API.
2026-01-04 04:29:44 +01:00
Shadow
3a28e3562c Discord: tools for uploading emojis and stickers! 2026-01-03 21:20:01 -06:00
Peter Steinberger
24aa3e3311 test: stabilize gateway tests 2026-01-04 04:16:38 +01:00
Peter Steinberger
3c4c2aa98c refactor: split gateway server methods 2026-01-04 04:05:18 +01:00
Peter Steinberger
3ebee63cb3 feat: add clawdhub skill 2026-01-04 04:05:10 +01:00
Peter Steinberger
6d6038b855 docs: tighten wacli skill guidance 2026-01-04 03:45:49 +01:00
Peter Steinberger
55876f7be0 test(agents): cover ping-pong announce flow 2026-01-04 03:41:58 +01:00
Peter Steinberger
cd3c42d0c0 feat(sessions): add agent-to-agent ping-pong 2026-01-04 03:37:44 +01:00
Peter Steinberger
add1301a51 feat(sessions): add agent-to-agent post step 2026-01-04 03:04:55 +01:00
Peter Steinberger
052cec70ae fix: render thinking text in italics 2026-01-04 02:44:11 +01:00
Peter Steinberger
534de59f7c docs: clarify menu bar sessionKey usage 2026-01-04 02:10:22 +01:00
Peter Steinberger
1d06164e18 refactor: use per-send run ids for gateway agent 2026-01-04 02:08:52 +01:00
Peter Steinberger
fe67073b74 fix: avoid sessions_send timeouts 2026-01-04 01:52:01 +01:00
Peter Steinberger
cbf41859aa test: relax cron default scheduler timeout 2026-01-04 01:45:50 +01:00
Cash Williams
12186e14a9 fix(android): handle unreachable gateway gracefully
Previously, if the gateway was unreachable (wrong IP, offline, etc.),
the Android app would crash with an unhandled socket exception.

Changes:
- Wrap socket.connect() in try/catch to handle connection failures
- Return PairResult with error message instead of crashing
- Display actual error message in status text instead of generic 'pairing required'

This gives users useful feedback like 'Connection refused' or
'Network is unreachable' instead of a crash.
2026-01-04 01:44:43 +01:00
Peter Steinberger
fbaa109a3a fix: stabilize lint and test timeouts 2026-01-04 01:42:08 +01:00
Peter Steinberger
70d68d29d0 fix: warm agent.wait cache 2026-01-04 01:35:02 +01:00
Peter Steinberger
e7615c464a docs: update apple-reminders skill for remindctl 2026-01-04 01:33:47 +01:00
Peter Steinberger
a1780efb9f fix: adjust typing TTL 2026-01-04 00:26:31 +00:00
Peter Steinberger
53d954695e style: format agent.wait imports 2026-01-04 01:22:22 +01:00
Peter Steinberger
44bdd4ca0c chore: regen Swift protocol models 2026-01-04 01:20:20 +01:00
Peter Steinberger
8724c2aea8 fix: satisfy gate checks 2026-01-04 01:16:53 +01:00
Peter Steinberger
e3c543ec06 fix: wait on agent.wait for sessions_send 2026-01-04 01:15:23 +01:00
Peter Steinberger
412e8b3aee test: cover gif playback send params 2026-01-03 23:57:43 +00:00
Peter Steinberger
5862f95bd2 fix: lock main session deletion 2026-01-03 23:57:17 +00:00
Peter Steinberger
e17c038d18 fix: add gif playback for WhatsApp sends 2026-01-03 23:56:40 +00:00
Peter Steinberger
e1dd764504 feat: add node location support 2026-01-04 00:54:44 +01:00
Peter Steinberger
52f59e6dc1 fix: drop stale ClawdisCLI build flag 2026-01-04 00:42:22 +01:00
Peter Steinberger
3bc24bf179 fix: wait for final agent response in sessions_send 2026-01-04 00:40:40 +01:00
Peter Steinberger
e07fdd117d docs: migrate Mintlify config 2026-01-04 00:36:55 +01:00
Peter Steinberger
7c062e0ef2 fix: clarify provider requirements in onboarding 2026-01-03 23:29:38 +00:00
Peter Steinberger
0f1781fc2c docs: add Mintlify config 2026-01-04 00:25:42 +01:00
Peter Steinberger
0f6e566a20 fix: make sessions_send wait via agent events 2026-01-04 00:12:14 +01:00
Peter Steinberger
03ee77b0e1 docs: add mac config sync note 2026-01-04 00:09:18 +01:00
Peter Steinberger
86038ec165 chore: apply lint fixes 2026-01-04 00:06:02 +01:00
Peter Steinberger
e7c9b9a749 feat: add sessions tools and send policy 2026-01-03 23:44:42 +01:00
Peter Steinberger
919d5d1dbb fix: restore sandbox PATH default 2026-01-03 22:36:16 +00:00
Peter Steinberger
3f7c69fa7f docs: note mac app config sync 2026-01-03 23:34:25 +01:00
Shadow
cc07ea82a4 CI: split macOS/android checks 2026-01-03 23:25:51 +01:00
Peter Steinberger
30e22769bb docs: update changelog for #144 2026-01-03 22:25:30 +00:00
Peter Steinberger
6c406b488d ci: consolidate check jobs 2026-01-03 22:25:29 +00:00
Peter Steinberger
f13f89e8b9 docs: update changelog for PR 156 2026-01-03 22:59:11 +01:00
Peter Steinberger
8b069e62fc fix: appease lint after merge 2026-01-03 22:59:11 +01:00
Shadow
e2709a3ebd CI: split macOS/android checks 2026-01-03 21:55:39 +00:00
Azade
18a89a31af fix(browser): avoid esbuild __name helper in evaluateViaPlaywright
When tsx/esbuild compiles arrow functions, it adds a __name helper
for debugging. This helper doesn't exist in the browser context,
causing 'ReferenceError: __name is not defined' when using
page.evaluate() with inline functions.

The fix uses new Function() constructed at runtime, which esbuild
doesn't transform, avoiding the __name injection.
2026-01-03 22:37:21 +01:00
Peter Steinberger
934f891932 fix: include embedded agent error cause in reply 2026-01-03 21:30:43 +00:00
Peter Steinberger
5493772910 fix: tolerate missing sandbox config in embedded runner 2026-01-03 21:30:40 +00:00
Peter Steinberger
c533593d8e fix: add ~/.local/bin to PATH bootstrap for uv-installed tools (fixes #78) (#150) 2026-01-03 22:25:52 +01:00
Mariano Belinky
fe1b894676 docs: clarify personal vs private in README (#125) 2026-01-03 22:21:55 +01:00
Mariano Belinky
d88581eb7c fix: add ~/.local/bin to PATH for uv tool binaries (#78) 2026-01-03 22:21:16 +01:00
Peter Steinberger
3d39e2ad75 feat(macos): sync gateway config 2026-01-03 22:17:04 +01:00
Peter Steinberger
2dc10ce337 docs: expand peekaboo skill docs 2026-01-03 22:14:21 +01:00
Peter Steinberger
d8a417f7ff feat: add sandbox browser support 2026-01-03 22:14:18 +01:00
Peter Steinberger
107dc1aa42 style(logging): organize embedded log imports 2026-01-03 21:09:44 +00:00
Peter Steinberger
9d2d0c64c2 test(gateway): cover config reload 2026-01-03 21:01:26 +00:00
Peter Steinberger
3872f32419 fix(logging): quiet embedded run console logs 2026-01-03 20:57:39 +00:00
Peter Steinberger
3b075dff8a feat: add per-session agent sandbox 2026-01-03 21:41:58 +01:00
Peter Steinberger
7bad9f3fbd fix: drop embedded sandbox wiring 2026-01-03 20:16:53 +00:00
Peter Steinberger
16e3535ac0 refactor: remove bash pty mode 2026-01-03 20:15:10 +00:00
Peter Steinberger
a15cffb7de fix: stream tool summaries early and tool output 2026-01-03 21:04:40 +01:00
Peter Steinberger
03c1599544 docs(templates): add platform formatting tips (Discord embeds, tables) 2026-01-03 20:01:17 +00:00
Shadow
6464d93bbb Discord: add forwarded message handling 2026-01-03 13:56:09 -06:00
Peter Steinberger
424d31af1f docs(templates): add voice storytelling tip for sag users 2026-01-03 19:55:32 +00:00
Peter Steinberger
e9d7ac8e84 feat(gateway): add config hot reload 2026-01-03 19:52:24 +00:00
Peter Steinberger
fac694fc03 docs(skills): add parallel Codex orchestration learnings
- coding-agent: document --yolo flag, git worktree + tmux pattern
- tmux: add section on orchestrating coding agents in parallel

Learnings from running 5 parallel Codex sessions to analyze GitHub issues
2026-01-03 19:45:18 +00:00
Shadow
3e84b9632d Discord: handle system message types 2026-01-03 13:15:19 -06:00
Peter Steinberger
ce3fd09e14 docs(faq): add alternative providers section (OpenRouter, Z.AI)
- Added OpenRouter and Z.AI setup examples
- Emphasized using latest Claude 4.5 models, not deprecated 3.x

🦞
2026-01-03 19:14:05 +00:00
Peter Steinberger
641080a0b6 fix: document macOS permission requirements 2026-01-03 20:05:22 +01:00
Jake
99c3fc1128 Scripts: Make ad-hoc fallback opt-in with stronger TCC warnings 2026-01-03 20:05:22 +01:00
Jake
8c7b2aa2d3 Scripts: Fallback to ad-hoc signing in codesign-mac-app.sh 2026-01-03 20:05:22 +01:00
Peter Steinberger
55a07a0ef0 style: fix lint formatting 2026-01-03 18:51:25 +00:00
Peter Steinberger
9899ba53a3 Docs: add PR number for Discord reactions 2026-01-03 18:48:36 +00:00
Peter Steinberger
52458a5628 Discord: default reaction notifications to own 2026-01-03 18:48:36 +00:00
Shadow
7abd6713c8 Docs: clarify discord reaction notifications 2026-01-03 18:48:36 +00:00
Shadow
451174ca10 Discord: add reaction notification allowlist 2026-01-03 18:48:36 +00:00
Peter Steinberger
cdfbd6e7eb test(gateway): align config constants in auth test 2026-01-03 19:37:09 +01:00
Peter Steinberger
350e007a5c test(agents): extend text_end coverage 2026-01-03 19:37:09 +01:00
Peter Steinberger
5e156135a1 test(gateway): avoid hoisted export errors 2026-01-03 19:37:09 +01:00
Peter Steinberger
b7ec9ae475 fix(gateway): format status/code errors 2026-01-03 19:37:09 +01:00
Peter Steinberger
8a18af409d test(gateway): cover helper registries 2026-01-03 19:37:09 +01:00
Peter Steinberger
6a125b554b refactor(gateway): split server helpers 2026-01-03 19:37:09 +01:00
Shadow
ce92fac983 chore: formatting 2026-01-03 12:35:16 -06:00
Peter Steinberger
341a224301 docs: credit Hyaxia in changelog and credits
Co-authored-by: Maxim Vovshin <36747317+Hyaxia@users.noreply.github.com>
2026-01-03 18:05:46 +00:00
Peter Steinberger
95cd153f33 feat: add blogwatcher skill 2026-01-03 18:00:08 +00:00
Peter Steinberger
0af89022ff fix: avoid Swift compiler crash in onboarding wizard 2026-01-03 17:59:37 +00:00
Peter Steinberger
27a8f3d061 chore: add inline guidance for block streaming 2026-01-03 18:46:59 +01:00
Peter Steinberger
72b34f7d03 fix: harden block stream dedupe 2026-01-03 18:44:07 +01:00
Peter Steinberger
73fa2e10bc refactor: split gateway server methods 2026-01-03 18:14:07 +01:00
Peter Steinberger
4a6b33d799 refactor: add gateway server helper modules 2026-01-03 18:00:45 +01:00
Peter Steinberger
145964c85e feat: add github skill 2026-01-03 17:57:13 +01:00
Peter Steinberger
217b84f2ac fix: drop final payloads after block streaming 2026-01-03 17:55:31 +01:00
Peter Steinberger
1d6de24ab3 feat: configurable control ui base path 2026-01-03 17:55:31 +01:00
Peter Steinberger
822def84d2 docs(faq): add Tailscale bind conflict + model/thinking compatibility
- Added Tailscale serve requires bind: loopback (not lan)
- Added model + thinking mode issues section (Gemini Flash, Opus, local LLMs)

From Discord #help session learnings 🦞
2026-01-03 16:53:56 +00:00
Peter Steinberger
f313af75e9 fix: avoid duplicate block-stream payloads 2026-01-03 16:53:56 +00:00
Peter Steinberger
591773715e fix: honor whatsapp per-group mention overrides 2026-01-03 17:51:10 +01:00
Peter Steinberger
dd6b9b510b docs: update changelog for gateway refactor 2026-01-03 17:35:29 +01:00
Peter Steinberger
6ae51ae3de refactor: split gateway server helpers and tests 2026-01-03 17:34:52 +01:00
Peter Steinberger
00c3e98431 docs: add tmux skill guidance 2026-01-03 17:31:26 +01:00
Peter Steinberger
dd561f58d1 docs: expand coding-agent Pi usage 2026-01-03 17:21:17 +01:00
Peter Steinberger
200dd634fb fix: preserve block streaming order 2026-01-03 17:14:01 +01:00
Peter Steinberger
3bbdcaf87f fix: avoid duplicate block streaming 2026-01-03 17:10:47 +01:00
Peter Steinberger
abff5e3b1f docs: thank @ratulsarna for control UI UUID fallback 2026-01-03 15:56:36 +00:00
Peter Steinberger
40ee0f0672 build: lock x86_64 relay to AVX2 2026-01-03 16:52:06 +01:00
Peter Steinberger
9f8eeceae7 feat: soften block streaming chunking 2026-01-03 16:48:26 +01:00
Peter Steinberger
53baba71fa feat: unify onboarding + config schema 2026-01-03 16:48:08 +01:00
Peter Steinberger
0f85080d81 Merge pull request #133 from ratulsarna/fix/ui-http-uuid
fix(ui): robust UUID generation for HTTP Control UI
2026-01-03 16:16:43 +01:00
Peter Steinberger
72f8148080 fix: clean up embedded lint 2026-01-03 15:09:07 +00:00
Peter Steinberger
be3da5b856 fix: update protocol models and android parsing 2026-01-03 15:04:24 +00:00
Peter Steinberger
9a9b429f74 fix: elevate embedded run logs to info 2026-01-03 15:03:03 +00:00
Peter Steinberger
733e86516e fix: address runtime warnings in build 2026-01-03 15:01:38 +00:00
Peter Steinberger
1a00175eb7 chore: fix lint formatting 2026-01-03 14:57:49 +00:00
Peter Steinberger
77c76ca52f test: fix transcription and tool schema assertions 2026-01-03 14:55:05 +00:00
Peter Steinberger
5de3395204 fix: resolve gcloud python path 2026-01-03 14:36:48 +00:00
Peter Steinberger
4e4655f607 docs(faq): use correct codex login --device-auth command 2026-01-03 14:13:18 +00:00
Peter Steinberger
48731f494b fix: add embedded run logs and typing ttl 2026-01-03 14:09:19 +00:00
Peter Steinberger
4fcd89c3d9 docs(faq): add stop/cancel task + Codex subscription auth sections
- Added FAQ for /stop and other abort commands
- Added FAQ explaining Codex CLI browser auth vs API key
- Browser OAuth uses ChatGPT Pro subscription, API key is pay-per-token

Co-authored-by: Clawd <clawdbot@gmail.com>
2026-01-03 14:08:24 +00:00
Peter Steinberger
a4f433a1b1 docs: update onboarding steps 2026-01-03 14:08:24 +00:00
Ratul Sarna
84a7ee491b fix(ui): robust UUID generation on HTTP
Fixes #131
2026-01-03 13:43:20 +00:00
Peter Steinberger
3043dd3a0c fix: restructure macOS connections settings 2026-01-03 14:25:03 +01:00
Jake
81f4a7cdb7 Agents: Fix Gemini schema compatibility and robust model discovery 2026-01-03 13:57:29 +01:00
Peter Steinberger
c2a74d6d2a docs(template): add 'Write It Down' rule to AGENTS.md template
Mental notes don't survive sessions. Files do. Text > Brain 📝
2026-01-03 12:52:11 +00:00
Peter Steinberger
861e1b33f5 docs(skill): add PR review safety rules for coding-agent
- Never checkout branches in live Clawdis repo
- Clone to temp folder or use git worktree for reviews
- Added explicit examples for safe PR review workflow
2026-01-03 12:49:03 +00:00
Peter Steinberger
0647d56555 fix(build): repair tool-meta regex literal 2026-01-03 12:46:33 +00:00
Peter Steinberger
ea6aea8532 docs: warn about gmail watcher port conflict 2026-01-03 12:41:44 +00:00
Peter Steinberger
6eca2edd79 chore(swift): update Swabble package lock 2026-01-03 13:38:18 +01:00
Peter Steinberger
d31dfbc565 chore(canvas): refresh a2ui bundle hash 2026-01-03 13:38:12 +01:00
Peter Steinberger
1e0f776824 test(gateway): add multi-instance e2e suite 2026-01-03 13:37:46 +01:00
Peter Steinberger
db36f0105d fix(gateway): validate event/response frames 2026-01-03 13:37:40 +01:00
Peter Steinberger
5377e2400a fix: avoid red gmail-watcher prefix 2026-01-03 12:36:15 +00:00
Peter Steinberger
72c0aa63fb style: tidy imports and formatting 2026-01-03 12:35:23 +00:00
Peter Steinberger
933bee220f fix(cron): pass resolved channel to agent tools 2026-01-03 12:35:23 +00:00
Peter Steinberger
bd2dabfa8f fix(agents): load tool display config from disk 2026-01-03 12:35:23 +00:00
Peter Steinberger
f11b352089 fix(macos): expand onboarding window height 2026-01-03 13:34:30 +01:00
Peter Steinberger
bb54e60179 fix(logging): decouple file logs from console verbose 2026-01-03 12:32:14 +00:00
Peter Steinberger
e52bdaa2a2 fix: repair tool meta regex 2026-01-03 12:30:46 +00:00
Peter Steinberger
b6301c719b fix: default low thinking for reasoning models 2026-01-03 12:19:06 +00:00
Peter Steinberger
6e16c0699a feat: centralize tool display metadata 2026-01-03 13:18:27 +01:00
Peter Steinberger
bf4ad295af docs(faq): add media/vision troubleshooting section
- Added FAQ entry for images/media not being understood
- Covers vision-capable models checklist
- Debugging steps for media pipeline
- Link to summarize.sh for exotic files

Co-authored-by: Clawd <clawdbot@gmail.com>
2026-01-03 11:43:40 +00:00
Peter Steinberger
7a80e8fe77 refactor: centralize home path shortening 2026-01-03 12:42:27 +01:00
Peter Steinberger
1ec3512925 refactor!: drop clawdis_ tool prefix 2026-01-03 12:39:52 +01:00
Peter Steinberger
772ada4308 fix: refine tool summaries and scope discord tool 2026-01-03 12:33:42 +01:00
Peter Steinberger
7165c8a7e5 refactor: rename bundle identifiers to com.clawdis 2026-01-03 12:26:22 +01:00
Peter Steinberger
daa1460502 docs(discord): document sendMessage mediaUrl and to format
- Add example for sendMessage with media attachment (file:// and https://)
- Clarify that sendMessage uses 'to: channel:<id>' not 'channelId'
- Document replyTo parameter for replying to specific messages
- Add mediaUrl to inputs section
2026-01-03 11:05:09 +00:00
Peter Steinberger
f47c7ac369 feat: support configurable gateway port 2026-01-03 12:00:17 +01:00
Peter Steinberger
7199813969 docs: document gateway port configuration 2026-01-03 11:46:58 +01:00
Peter Steinberger
87d5fa516d docs(skills): correct bear-notes instructions
Co-authored-by: Tyler Wince <tylerwince@users.noreply.github.com>
2026-01-03 11:34:31 +01:00
Claude
10340d2a3f feat(skills): add bear-notes skill using grizzly CLI 2026-01-03 11:29:14 +01:00
Peter Steinberger
508c4d362f docs: update changelog for gog skill 2026-01-03 11:20:17 +01:00
Mariano Belinky
f73b008251 docs: add Sheets/Docs examples to gog skill 2026-01-03 11:20:17 +01:00
Peter Steinberger
c583e64bb7 chore: update changelog 2026-01-03 11:17:00 +01:00
Peter Steinberger
9df63b008d docs: move telegram chunking fix to beta6 2026-01-03 11:15:57 +01:00
Peter Steinberger
3daecc092c docs: add changelog entry for telegram block replies 2026-01-03 11:12:15 +01:00
Muhammed Mukhthar CM
4d42811ecf fix(telegram): add textLimit to block reply chunking
Block streaming replies were missing the textLimit parameter in
deliverReplies(), causing long messages to fail with 'message is too
long' error instead of being chunked properly.

The final reply path already included textLimit, but the onBlockReply
callback path did not.
2026-01-03 11:12:15 +01:00
Peter Steinberger
1bebcf8033 chore: update appcast and TUI streaming handling 2026-01-03 11:06:49 +01:00
Peter Steinberger
45c555a4bd fix: use x86_64 bun for relay builds 2026-01-03 11:06:49 +01:00
Peter Steinberger
5986a83e80 fix: skip duplicate arch merge for Sparkle 2026-01-03 11:06:49 +01:00
Peter Steinberger
732de4acf0 fix: make Sparkle builds numeric + universal 2026-01-03 11:06:48 +01:00
Shadow
7400c0946e Discord: update UIs to use the new config 2026-01-03 01:02:22 -06:00
Peter Steinberger
14ee2b2c11 FAQ: Add common questions from Discord (Jan 3)
- Linux/VPS installation without Homebrew
- Minimum system requirements (runs on 1GB RAM!)
- Enterprise OAuth status (not supported yet)
- Discord DM allowlist config
- Model switching with /model
- Message queue modes with /queue
2026-01-03 06:09:51 +00:00
Peter Steinberger
c3e1b8cfd9 chore: update protocol swift models 2026-01-03 06:44:21 +01:00
Peter Steinberger
67a67df35a fix: avoid unsafe string coercion in tui 2026-01-03 06:44:17 +01:00
Peter Steinberger
0f0578b268 docs: check off tui gate 2026-01-03 06:37:44 +01:00
Peter Steinberger
662208949f fix: align sessions.patch and tui typing 2026-01-03 06:37:40 +01:00
Peter Steinberger
e41821342b docs: refresh tui guide 2026-01-03 06:28:36 +01:00
Peter Steinberger
d3458a4fc3 feat: overhaul tui controller 2026-01-03 06:27:38 +01:00
Peter Steinberger
32c91bbb25 feat: add tui ui kit 2026-01-03 06:22:20 +01:00
Peter Steinberger
aee13507f9 feat: expand tui gateway client 2026-01-03 06:17:33 +01:00
Peter Steinberger
61b67f6301 feat: extend gateway session patch 2026-01-03 06:16:49 +01:00
Peter Steinberger
b86619bcd0 docs: fix appcast to only ship beta5 2026-01-03 06:12:01 +01:00
Peter Steinberger
31b5b45581 docs: refresh appcast for notarized beta5 2026-01-03 06:04:20 +01:00
Peter Steinberger
33cdb16b9e docs: update appcast for 2.0.0-beta5 2026-01-03 05:55:31 +01:00
Peter Steinberger
53fd7a4473 chore: fix lint ordering 2026-01-03 05:38:29 +01:00
Peter Steinberger
10d56d31e9 docs: date 2.0.0-beta5 changelog 2026-01-03 05:37:04 +01:00
Peter Steinberger
3633c829ae fix: repair discord action typing 2026-01-03 05:33:57 +01:00
Peter Steinberger
6cda84432e fix: stabilize pi-ai patch + tests 2026-01-03 05:22:09 +01:00
Peter Steinberger
b914eaa6fa chore: apply biome lint fixes 2026-01-03 05:10:09 +01:00
Peter Steinberger
988b67aa30 test: refresh auto-reply expectations 2026-01-03 05:09:59 +01:00
Peter Steinberger
0ed5b82389 fix: prefer explicit hook mappings 2026-01-03 05:09:54 +01:00
Peter Steinberger
b417fe5727 fix: show rich session names in chat UIs 2026-01-03 05:07:13 +01:00
Peter Steinberger
fabad7aa7a docs: update changelog for antigravity oauth 2026-01-03 05:01:42 +01:00
Peter Steinberger
3c54da952a chore: refresh pi-ai patch hash 2026-01-03 05:01:42 +01:00
Peter Steinberger
2ef2646b31 chore: note lossy google schema scrub 2026-01-03 05:01:42 +01:00
mukhtharcm
82ad7e29a6 fix: reject antigravity auth in non-interactive onboarding mode 2026-01-03 05:01:42 +01:00
mukhtharcm
2290a3c8af feat: add VPS-aware Antigravity OAuth with manual URL paste fallback
Detects SSH/VPS/headless environments and prompts user to paste
the OAuth callback URL instead of relying on localhost server.

- Add antigravity-oauth.ts with VPS detection and manual OAuth flow
- Update onboard-interactive.ts to use VPS-aware flow
- Update configure.ts to use VPS-aware flow
2026-01-03 05:01:42 +01:00
mukhtharcm
d216cebff5 fix: use claude-opus-4-5-thinking as default antigravity model 2026-01-03 05:01:42 +01:00
mukhtharcm
05bd345828 feat: add Google Antigravity authentication support
- Add 'antigravity' as new auth choice in onboard and configure wizards
- Implement Google Antigravity OAuth flow using loginAntigravity from pi-ai
- Update writeOAuthCredentials to accept any OAuthProvider (not just 'anthropic')
- Add schema sanitization for Google Cloud Code Assist API to fix tool call errors
- Default model set to google-antigravity/claude-opus-4-5 after successful auth

The schema sanitization removes unsupported JSON Schema keywords (patternProperties,
const, anyOf, etc.) that Google's Cloud Code Assist API doesn't understand.
2026-01-03 05:01:42 +01:00
Peter Steinberger
5eff541da8 docs: prefer spogo or spotify_player 2026-01-03 04:47:34 +01:00
Peter Steinberger
598a27cc96 docs: update changelog for tui 2026-01-03 04:47:34 +01:00
Peter Steinberger
08ce608ae7 feat: add gateway TUI 2026-01-03 04:47:34 +01:00
Peter Steinberger
928631309e docs: note queue tests 2026-01-03 04:47:34 +01:00
Peter Steinberger
971b98c96d test: cover new queue modes 2026-01-03 04:47:34 +01:00
Peter Steinberger
a72da30c9a sag skill: add chat voice response guidance
When Peter asks for voice replies, generate audio with sag and send via MEDIA:
2026-01-03 03:34:31 +00:00
Peter Steinberger
f7eabcb2d9 docs: note new queue modes 2026-01-03 04:27:22 +01:00
Peter Steinberger
ac36eba822 feat: expand queue modes and followup backlog 2026-01-03 04:26:49 +01:00
Peter Steinberger
6160521f2f fix: guard bash pty cwd 2026-01-03 03:05:51 +00:00
Jared Verdi
ca9b0dbc88 Gmail watcher: start when gateway (re)starts 2026-01-03 03:49:53 +01:00
Peter Steinberger
11c7e05f43 fix: harden pty spawn path 2026-01-03 02:36:01 +00:00
Peter Steinberger
1781105438 group chat: hint that reactions are welcome while lurking
Even when staying silent, emoji reactions show engagement without cluttering chat.
2026-01-03 02:29:32 +00:00
Peter Steinberger
632ca01fbf style: format linted files 2026-01-03 03:10:17 +01:00
Peter Steinberger
b8fd22bfd8 docs: update changelog for discord actions 2026-01-03 03:07:13 +01:00
Shadow
98a1deb129 UI: add discord action toggles 2026-01-03 03:07:13 +01:00
Shadow
0c38f2df2a Discord: drop enableReactions config 2026-01-03 03:07:13 +01:00
Shadow
6bab813bb3 Discord: add reactions, stickers, and polls skill 2026-01-03 03:07:13 +01:00
Peter Steinberger
d8201f8436 fix: handle null action in hooks-mapping mergeAction call 2026-01-03 02:05:01 +00:00
Peter Steinberger
b28e4e95c2 docs: note gmail watcher auto-start 2026-01-03 03:04:15 +01:00
Peter Steinberger
a3865f1417 group chat: add lurking guidance to system prompt
Be a good group participant: lurk and follow the conversation,
but only chime in when genuinely helpful. Quality over quantity.
2026-01-03 02:02:55 +00:00
Peter Steinberger
fb10bf5f75 feat: add bash pty diagnostics 2026-01-03 01:56:54 +00:00
Peter Steinberger
a9eb31e8fe fix: satisfy discord and gateway typing 2026-01-03 02:55:43 +01:00
Peter Steinberger
3ec5ce8349 docs: note onboarding scroll gutter 2026-01-03 02:55:43 +01:00
Peter Steinberger
c5d70019bb fix: respect onboarding scroll indicator preference 2026-01-03 02:55:43 +01:00
Shadow
a35fb3a9b4 macOS: add onboarding scroll gutter 2026-01-03 02:55:43 +01:00
Peter Steinberger
79403f9083 docs: update apple notes/reminders skill setup 2026-01-03 02:41:12 +01:00
Claude
7a44c19362 feat(skills): add Apple Notes and Reminders skills via memo CLI 2026-01-03 02:41:12 +01:00
Peter Steinberger
11fc10ea47 docs: thank contributor for telegram group gating 2026-01-03 02:34:48 +01:00
Peter Steinberger
7e4e9ecdea templates: add qmd semantic memory recall to AGENTS.md 2026-01-03 01:33:10 +00:00
Peter Steinberger
0c013a237f fix: default block streaming break to message_end 2026-01-03 01:33:10 +00:00
Peter Steinberger
f85951bc65 docs: add changelog entry for gog calendar fix 2026-01-03 02:32:50 +01:00
Jared Verdi
12e27f9e5e Gateway: ack skipped hook transforms with 204 2026-01-03 02:32:50 +01:00
Jay Hickey
7e9be3c28c Update gog Calendar command date format to RFC3339
I am seeing instances where Clawdis is not including timezone in the gog calendar range requests. This results in a 400 bad request from the Google API, e.g.

```
gog calendar events primary --from 2026-01-02T00:00:00 --to 2026-01-03T23:59:59 --account <email>

Google API error (400 badRequest): Bad Request
```

While this is a valid ISO 8601 format, Google Calendar API requires a stricter RFC 3339 format like the following:

```
gog calendar events primary --from 2026-01-02T00:00:00Z --to 2026-01-03T23:59:59Z --account <email>

<calendar events listed successfully>
```
2026-01-03 02:30:32 +01:00
Peter Steinberger
3368fcf31e fix: avoid duplicate replies with block streaming 2026-01-03 02:16:01 +01:00
Peter Steinberger
32877afe55 docs: note self-chat config docs update 2026-01-03 02:05:37 +01:00
rafaelreis-r
efe7eca726 docs: clarify routing.allowFrom and self-chat mode for group mentions
- Add new section explaining self-chat mode for group control
- Document routing.allowFrom as the key setting for controlling metadata mentions
- Clarify difference between whatsapp.allowFrom (DM allowlist) and routing.allowFrom (self-chat mode)
- Explain metadata mentions vs text patterns in routing.groupChat
- Add example config for responding only to specific text triggers

When routing.allowFrom contains the bot's own number, WhatsApp native
@-mentions are ignored in groups, and only mentionPatterns trigger responses.
This prevents unwanted responses when users tap-to-mention the bot owner.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-03 02:05:37 +01:00
Peter Steinberger
72d1fa4da5 fix: dedupe repeated block replies 2026-01-03 01:49:27 +01:00
Peter Steinberger
2042013360 test: cover provider textChunkLimit config 2026-01-03 01:49:27 +01:00
Peter Steinberger
f5189cc897 refactor: move text chunk limits to providers 2026-01-03 01:49:27 +01:00
Peter Steinberger
75a9cd83a0 fix(mac): resolve camera type deprecation 2026-01-03 01:49:27 +01:00
Peter Steinberger
5684e2d658 feat: configurable outbound text chunk limits 2026-01-03 01:49:27 +01:00
Peter Steinberger
2d28fa34f5 feat: make block streaming break configurable 2026-01-03 01:49:27 +01:00
Peter Steinberger
ea7d967625 Update Discord invite to vanity URL discord.gg/clawd 🦞
Thanks camerondare for the boosts! Level 3 unlocked.
2026-01-03 00:47:22 +00:00
Peter Steinberger
5dfb2b1128 coding-agent: add temp space pattern, never start in ~/clawd
Learnings from tonight:
- Codex reads AGENTS.md/SOUL.md and gets ideas about org hierarchy
- Use mktemp -d for scratch/chat sessions
- Never start in ~/clawd or agent home dirs
- Keep agents in their 'little box' 📦🦞
2026-01-03 00:35:51 +00:00
Peter Steinberger
cbc599a5b8 coding-agent: add batch PR review pattern
Tonight's learnings:
- Parallel Codex army for batch PR reviews
- Fetch PR refs: git fetch origin '+refs/pull/*/head:refs/remotes/origin/pr/*'
- Use git diff origin/main...origin/pr/XX (don't checkout)
- Post results with gh pr comment
- Successfully reviewed 13 PRs in parallel! 🦞
2026-01-03 00:24:34 +00:00
Peter Steinberger
1354d0836f coding-agent: comprehensive update from Jan 2 learnings
- workdir 'little box' pattern (don't read unrelated files)
- background mode replaces tmux
- --full-auto for building, vanilla for reviewing
- parallel Codex processes supported
- PR review tips (fetch refs, use git diff, don't checkout)
- patience rules (don't kill slow sessions!)
2026-01-03 00:11:21 +00:00
Peter Steinberger
b313250638 coding-agent: switch to native background mode, drop tmux requirement
- Use bash background:true instead of tmux
- Full programmatic control: log/poll/write/kill
- Simpler, no shell escaping issues
- workdir still critical for 'little box' pattern
2026-01-03 00:00:37 +00:00
Peter Steinberger
e37c147ea9 coding-agent: unified workdir+tmux pattern for all tools 2026-01-02 23:58:33 +00:00
Peter Steinberger
feb4f9028d coding-agent: choose reasoning effort based on task complexity 2026-01-02 23:57:46 +00:00
Peter Steinberger
4804ce5678 coding-agent: simplify to gpt-5.2-codex only, remove old models 2026-01-02 23:57:24 +00:00
Peter Steinberger
001a342f20 coding-agent: workdir pattern + patience rules
- Use bash workdir param so Codex wakes up in a 'little box'
- Prevents reading unrelated files (like my soul.md lol)
- Added rule: NEVER offer to build it yourself when user asks for Codex
- gpt-5.2-codex requires medium reasoning effort
2026-01-02 23:56:10 +00:00
Peter Steinberger
fe040b84d9 chore: sync lockfile and bundle hash 2026-01-03 00:40:39 +01:00
Sreekaran Srinath
0ac30afb29 feat: add coding-agent skill and anyBins gating
Co-authored-by: Sreekaran Srinath <ss@sreekaran.com>
2026-01-03 00:40:03 +01:00
Peter Steinberger
59601eb99c fix: preserve newlines in reply tags 2026-01-02 23:36:43 +00:00
Peter Steinberger
9616f4b2b1 feat: stream reply blocks immediately 2026-01-03 00:28:33 +01:00
Peter Steinberger
9dd613edf7 fix(mac): harden remote tunnel recovery 2026-01-03 00:02:27 +01:00
Peter Steinberger
88ed58b3d0 chore: update deps and extend read tool tests 2026-01-02 23:47:28 +01:00
Peter Steinberger
fc54e905c0 chore: upgrade pi-mono deps to 0.31.1 2026-01-02 23:37:08 +01:00
Peter Steinberger
d1b76cb1b2 test: cover replyToMode behavior 2026-01-02 23:20:52 +01:00
Peter Steinberger
2c92ccd66e feat: add reply tags and replyToMode 2026-01-02 23:18:41 +01:00
Peter Steinberger
a9ff03acaf feat: unify group mention defaults 2026-01-02 22:50:58 +01:00
Shadow
281dc10b2f Changelog: mention Discord reply context 2026-01-02 15:41:45 -06:00
Peter Steinberger
fd32fc8d8d feat: add discord guild wildcard defaults 2026-01-02 22:33:26 +01:00
Peter Steinberger
47f4f59692 chore: remove stray ds_store files 2026-01-02 22:24:26 +01:00
Peter Steinberger
5cf1a9535e feat: move group mention gating to provider groups 2026-01-02 22:24:26 +01:00
Peter Steinberger
e93102b276 chore: bump peekaboo submodule 2026-01-02 22:24:26 +01:00
Shadow
da57c314ef Discord: clarify docs and drop legacy guild schema 2026-01-02 15:21:13 -06:00
Shadow
2676636316 Discord: fix reply context formatting 2026-01-02 14:55:07 -06:00
Shadow
f3a973dc9e Discord: include reply context 2026-01-02 14:49:16 -06:00
Peter Steinberger
f4a1190bdd docs: add CONTRIBUTING.md with maintainers and guidelines
- List maintainers with GitHub/X links
- Link to Discord and GitHub Discussions
- AI/vibe-coded PRs welcome with transparency guidelines
- Link from README

Co-authored-by: Clawd <clawdbot@gmail.com>
2026-01-02 20:31:41 +00:00
Peter Steinberger
118a6d7421 fix: align discord config ui 2026-01-02 21:15:59 +01:00
Peter Steinberger
4541bb2716 Merge pull request #108 from thewilloftheshadow/shadow/ui-connection-update
UI: Update connections UIs
2026-01-02 21:04:45 +01:00
Peter Steinberger
505c4262c6 docs: note optional docker setup 2026-01-02 20:59:58 +01:00
Peter Steinberger
3104b088e4 chore(canvas): update a2ui bundle hash 2026-01-02 19:58:46 +00:00
Peter Steinberger
f12f814816 docs(whatsapp): add number guidance 2026-01-02 19:58:44 +00:00
Peter Steinberger
3b0ad719c9 chore(discord): add verbose diagnostics 2026-01-02 19:58:42 +00:00
Peter Steinberger
e368e56246 chore(gateway): quiet provider startup logs 2026-01-02 19:58:40 +00:00
Peter Steinberger
675420013d fix(macos): resolve gateway launch args 2026-01-02 19:58:38 +00:00
Peter Steinberger
eaa69fb6b2 test: silence docker onboarding noise 2026-01-02 20:46:26 +01:00
Peter Steinberger
e0795cf18c test: annotate onboarding docker e2e 2026-01-02 20:41:47 +01:00
Peter Steinberger
8ed878e73c test: stabilize docker onboarding e2e 2026-01-02 20:40:33 +01:00
Peter Steinberger
08b95411df chore: add goplaces skill 2026-01-02 20:33:06 +01:00
Peter Steinberger
460fafff7f docs: thank @dan-dr for docker setup 2026-01-02 20:24:44 +01:00
Peter Steinberger
7b4fa9e1a1 test: cover camera list invoke 2026-01-02 20:24:41 +01:00
Peter Steinberger
7e4ebb22a0 Merge pull request #107 from dan-dr/main
Add Docker setup script
2026-01-02 20:24:21 +01:00
Peter Steinberger
8b47315845 fix(macos): improve session preview loading 2026-01-02 19:55:19 +01:00
Shadow
96a5e01878 macOS: swiftformat connections settings 2026-01-02 12:30:59 -06:00
Shadow
5e36390a27 macOS: fix swiftlint param count 2026-01-02 12:25:47 -06:00
Shadow
729a545173 Update connections UIs 2026-01-02 12:06:05 -06:00
Dan
488f5e2dac Merge branch 'steipete:main' into main 2026-01-02 19:53:16 +02:00
Peter Steinberger
49e89cf3f1 fix: satisfy swiftformat for ios build 2026-01-02 18:48:05 +01:00
Peter Steinberger
43f6b9ef32 fix: resolve camera tool handling 2026-01-02 17:44:25 +00:00
Peter Steinberger
8e48cffe3b fix(macos): decode session preview payload 2026-01-02 18:32:03 +01:00
Peter Steinberger
3ed01adabc feat(macos): add session previews in menu 2026-01-02 18:29:47 +01:00
Dan
4239de8060 Merge branch 'steipete:main' into main 2026-01-02 19:26:14 +02:00
Peter Steinberger
cba37f99b6 test: cover camera device selection 2026-01-02 18:25:22 +01:00
Peter Steinberger
74db53d939 feat: add camera list and device selection 2026-01-02 18:23:26 +01:00
Peter Steinberger
2b34bf08da fix: resolve mac camera continuation isolation 2026-01-02 18:02:24 +01:00
Dan
b92f70c52b Merge branch 'steipete:main' into main 2026-01-02 19:00:21 +02:00
Peter Steinberger
34d2e1e2e8 fix: wait for camera exposure to settle 2026-01-02 17:57:34 +01:00
Peter Steinberger
5f82739e2b test: cover camera snap mime mapping 2026-01-02 17:49:20 +01:00
Peter Steinberger
d79dc4d742 fix: correct camera snap mime mapping 2026-01-02 17:43:34 +01:00
Peter Steinberger
1d12a844c2 docs: add WhatsApp disconnect workaround to FAQ
When using macOS app with WhatsApp issues:
1. Run pnpm gateway:watch (Node instead of bun)
2. Enable 'External gateway' in app debug settings

Verified gateway:watch command exists in package.json
2026-01-02 16:34:27 +00:00
Peter Steinberger
2d16450869 feat: add weather skill (wttr.in + Open-Meteo fallback)
No API key required. Two services:
- wttr.in: human-readable, emoji, ASCII art, PNG
- Open-Meteo: JSON API fallback for programmatic use

🌤️🦞
2026-01-02 16:33:31 +00:00
Peter Steinberger
2a6248dad6 fix: add camera entitlement to macOS signing 2026-01-02 17:31:59 +01:00
Peter Steinberger
8b27c03472 docs(skills/local-places): add emoji and tagline
📍 Find places, Go fast

🦞
2026-01-02 16:22:26 +00:00
Peter Steinberger
baf3bea574 docs(changelog): note macOS config actor fix 2026-01-02 17:16:49 +01:00
Peter Steinberger
868b438e67 test(gateway): fix nix mode mock toggle 2026-01-02 17:15:26 +01:00
Peter Steinberger
8989bd9fd7 fix(auto-reply): default whatsapp self-only on empty config 2026-01-02 17:15:26 +01:00
Peter Steinberger
a4f12babb7 test(macos): cover gateway password whitespace 2026-01-02 17:15:26 +01:00
Peter Steinberger
97e06a8eb4 chore(canvas): regenerate a2ui bundle hash 2026-01-02 17:15:26 +01:00
Peter Steinberger
0de6e38ce9 fix(macos): keep config writes on main actor 2026-01-02 17:15:26 +01:00
Peter Steinberger
314164fb8a chore: fix lint and add gateway auth tests 2026-01-02 17:15:26 +01:00
Peter Steinberger
8d925226cb docs: expand FAQ with Docker, OAuth, bun vs Node, debugging
- Docker/container setup (volumes, pnpm persistence, startup script)
- OAuth vs API key billing differences
- OAuth callback workarounds for headless/containers
- Terminal onboarding vs macOS app (terminal more stable)
- bun binary vs Node runtime (Node more stable for WhatsApp)
- gateway:watch for debugging
- Tailscale link and setup clarification

Based on community questions from Discord #help
2026-01-02 16:04:02 +00:00
Peter Steinberger
f2eb2004aa docs: thank @jeffersonwarrior for gateway auth 2026-01-02 16:51:48 +01:00
Peter Steinberger
bf37015c23 Merge pull request #85 from jeffersonwarrior/main
feat: add gateway password auth support and fix Swift 6.2 concurrency
2026-01-02 16:50:57 +01:00
Peter Steinberger
f489b6e7a5 chore: merge origin/main 2026-01-02 16:50:29 +01:00
Peter Steinberger
921e5be8e6 fix(skills/local-places): copy files instead of submodule
Submodules are pain. Just copy the Python code directly.

🦞
2026-01-02 15:48:24 +00:00
Peter Steinberger
a8bc974a2e fix: harden gateway password auth 2026-01-02 16:47:52 +01:00
Peter Steinberger
100a022ab7 feat(skills/local-places): add server as submodule
- Links to Hyaxia/local_places for easy upstream updates
- Updated SKILL.md with {baseDir}/server path

🦞
2026-01-02 15:47:42 +00:00
Peter Steinberger
6b7484a885 feat(skills): add local-places skill for Google Places search
- Wraps Hyaxia/local_places FastAPI server
- Two-step flow: resolve location → search places
- Supports filters: type, rating, price, open_now

🦞
2026-01-02 15:46:08 +00:00
Peter Steinberger
8de40e0c10 feat(macos): add Camera permission to onboarding flow
- Add 'camera' case to Capability enum
- Add UI strings (title, subtitle, icon) in PermissionsSettings
- Add ensureCamera() and camera status check in PermissionManager
- Add CameraPermissionHelper for opening System Settings

🦞 Clawd's first code contribution!
2026-01-02 15:27:54 +00:00
Peter Steinberger
9b3aef3567 fix: show skill descriptions in onboarding list 2026-01-02 16:25:28 +01:00
Peter Steinberger
25e52e19dc fix(macos): return node name 2026-01-02 15:28:34 +01:00
Peter Steinberger
68806902ff fix(macos): show gateway in devices list 2026-01-02 15:27:21 +01:00
Peter Steinberger
ebf8649940 feat: add songsee skill 2026-01-02 15:22:23 +01:00
Peter Steinberger
c93d02891a test: cover control ui token url 2026-01-02 15:13:05 +01:00
Peter Steinberger
87be5c737c fix(macos): suppress cancelled node refresh 2026-01-02 15:12:57 +01:00
Peter Steinberger
ad9d6f616d fix: improve onboarding auth UX 2026-01-02 15:03:38 +01:00
Peter Steinberger
f57f892409 fix(macos): clarify gateway error state 2026-01-02 13:48:19 +01:00
Peter Steinberger
5ecb65cbbe fix: persist gateway token for local CLI auth 2026-01-02 13:46:48 +01:00
Peter Steinberger
1e04481aaf style: format discord slash handler 2026-01-02 13:38:36 +01:00
Peter Steinberger
5f103e32bd fix: gate discord slash commands 2026-01-02 13:38:35 +01:00
Shadow
fff9efe8a8 Discord: auto-register slash command 2026-01-02 13:38:35 +01:00
Shadow
b135b3efb9 Discord: add slash command handling 2026-01-02 13:38:35 +01:00
Peter Steinberger
17e17f85ae docs: note gateway auto-migrate 2026-01-02 13:10:09 +01:00
Peter Steinberger
ecef49605b test: cover gateway legacy auto-migrate 2026-01-02 13:09:41 +01:00
Peter Steinberger
7b72b35cca chore: update doctor migration hash 2026-01-02 13:07:26 +01:00
Peter Steinberger
16420e5b31 refactor: auto-migrate legacy config in gateway 2026-01-02 13:07:14 +01:00
Peter Steinberger
55665246bb chore: refresh doctor migration commit 2026-01-02 13:00:44 +01:00
Peter Steinberger
b9b862a380 chore: note doctor migration commit 2026-01-02 13:00:29 +01:00
Peter Steinberger
0766c5e3cb refactor: move whatsapp allowFrom config 2026-01-02 13:00:29 +01:00
ddyo
8d4c6d41ab Docker: add root-level setup 2026-01-02 13:53:06 +02:00
Peter Steinberger
58d32d4542 docs: expand FAQ with skills, Tailscale, troubleshooting
- How to add/reload skills (/reset)
- Tailscale for multi-machine setups
- Using Codex to debug
- Handling supervised processes on Linux
- Clean uninstall steps
2026-01-02 11:50:09 +00:00
Peter Steinberger
6bad75827a docs: clarify Signal setup and env-token gating 2026-01-02 11:41:08 +00:00
Peter Steinberger
2b3ddabe90 fix(gateway): gate providers by config presence 2026-01-02 11:41:01 +00:00
Peter Steinberger
e92b480629 fix(signal): surface signal-cli failures as errors 2026-01-02 11:40:55 +00:00
Peter Steinberger
a53cdbf1b4 docs: clarify Windows is untested in FAQ 2026-01-02 11:30:27 +00:00
Peter Steinberger
21a64a9957 docs: link FAQ and add platforms note 2026-01-02 11:24:41 +00:00
Peter Steinberger
d656db4d04 fix: widen discord channel type check 2026-01-02 12:23:35 +01:00
Peter Steinberger
506b66a852 docs: add FAQ with common questions from Discord
Covers:
- Installation & setup (data locations, unauthorized errors, fresh start, doctor)
- Migration & deployment (new machine, VPS, Docker)
- Multi-instance & contexts (one Clawd philosophy, groups for separation)
- Context & memory (200k tokens, autocompaction, workspace location)
- Platforms (supported platforms, multi-platform, WhatsApp numbers)
- Troubleshooting (build errors, WhatsApp logout, gateway issues)
- Chat commands reference

Based on community questions from #help channel.

🦞
2026-01-02 11:22:06 +00:00
Peter Steinberger
95f03d63ad style(ui): refresh dashboard theme 2026-01-02 11:22:06 +00:00
Peter Steinberger
7f8af736dd chore(canvas): regenerate a2ui bundle hash 2026-01-02 11:22:06 +00:00
Peter Steinberger
eaacebeecc fix: improve onboarding/imessage errors 2026-01-02 12:20:48 +01:00
Peter Steinberger
fd4cff06ca test: fix discord/config test lint 2026-01-02 12:20:43 +01:00
Peter Steinberger
b50df6eb1d style: format linted files 2026-01-02 12:20:38 +01:00
Peter Steinberger
fa16304e4f docs: note discord ignore-list removal 2026-01-02 11:54:30 +01:00
Peter Steinberger
eda74d3a55 test: cover every schedule anchor boundary 2026-01-02 11:33:49 +01:00
Peter Steinberger
25762c0ac6 docs(discord): note from label includes tag/id 2026-01-02 11:32:59 +01:00
Peter Steinberger
2d7289bcad docs: update changelog for cron fix 2026-01-02 11:29:35 +01:00
Peter Steinberger
2d1d5d603d Merge pull request #80 from jamesgroat/fix/cron-every-schedule-infinite-loop
fix(cron): prevent every schedule from firing in infinite loop
2026-01-02 11:29:08 +01:00
Peter Steinberger
94206cf10f Merge pull request #92 from thewilloftheshadow/shadow/discord-id
Discord: pass user id to clawd so it can ping users
2026-01-02 11:27:37 +01:00
Peter Steinberger
dc2521a1cf merge main into shadow/discord-id 2026-01-02 11:27:24 +01:00
Peter Steinberger
30b5955f22 fix(discord): add tag/id to from label 2026-01-02 11:26:09 +01:00
Peter Steinberger
4267a1b87d test: cover discord config + slug routing 2026-01-02 11:19:10 +01:00
Peter Steinberger
eb44ae76f1 feat: add discord guild map + group dm controls 2026-01-02 11:15:52 +01:00
Peter Steinberger
bd3d18f660 fix: unbreak TypeScript build 2026-01-02 11:02:06 +01:00
Peter Steinberger
8bd5f1b9f2 fix: improve onboarding allowlist + Control UI link 2026-01-02 10:57:04 +01:00
Peter Steinberger
71b0dcc922 Merge pull request #100 from steipete/feat/trello-skill
feat(skills): add Trello skill for board/list/card management
2026-01-02 10:47:45 +01:00
Peter Steinberger
1bf7d2f3bd docs: update trello skill requirements 2026-01-02 10:47:31 +01:00
Peter Steinberger
87127fd133 fix: refine web chat session selector 2026-01-02 10:40:24 +01:00
Peter Steinberger
e85c15d178 docs: note mac app rebuilds need local 2026-01-02 10:38:18 +01:00
Peter Steinberger
0f56dce748 feat: add discord dm/guild allowlists 2026-01-02 10:32:21 +01:00
Peter Steinberger
d2e2077ada fix: add top padding before first chat message 2026-01-02 10:23:40 +01:00
Peter Steinberger
9adbf47773 refactor: normalize group session keys 2026-01-02 10:14:58 +01:00
Peter Steinberger
e5ee041d4e feat(skills): add Trello skill for board/list/card management 2026-01-02 08:37:15 +00:00
Shadow
63a46a85f6 feat: pass discord id to clawd so it can ping users 2026-01-01 23:30:03 -06:00
Jefferson Nunn
fe87d6d8be feat(macOS): add gateway password auth support and fix Swift 6.2 concurrency
- Add CLAWDIS_GATEWAY_PASSWORD to launchd plist environment
- Read password from gateway.remote.password config in client
- Fix Swift 6.2 sending parameter violations in config save functions
- Add password parameter to GatewayConnection.Config type
- GatewayChannel now sends password in connect auth params
- GatewayEndpointStore and GatewayLaunchAgentManager read password from config
- CLI gateway client reads password from remote config and env
2026-01-01 21:34:46 -06:00
jeffersonwarrior
9387ecf043 fix(macos): support password auth mode for gateway connections
GatewayChannel now sends both 'token' and 'password' fields in the auth
payload to support both authentication modes. Gateway checks the field
matching its auth.mode configuration ('token' or 'password').

Also adds config file password fallback for remote mode, allowing
gateway password to be configured in ~/.clawdis/clawdis.json without
requiring environment variables.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-01-01 21:26:37 -06:00
Peter Steinberger
35582cfe8a docs: fix broken clawd.md link in index 2026-01-02 02:45:01 +00:00
Peter Steinberger
76e24653e9 fix(media): preserve GIF animation, skip JPEG optimization
- Skip JPEG optimization for image/gif content type (both local and URL)
- Preserves animation in uploaded GIFs to Discord/other providers
- Added tests for GIF preservation from local files and URLs
- Updated changelog
2026-01-02 00:56:04 +00:00
Peter Steinberger
4c2812b429 fix: refine HEARTBEAT_OK handling 2026-01-02 01:42:27 +01:00
James Groat
7154bc6857 fix(cron): prevent every schedule from firing in infinite loop
When anchorMs is not provided (always in production), the schedule
computed nextRunAtMs as nowMs, causing jobs to fire immediately and
repeatedly instead of at the configured interval.

- Change nowMs <= anchor to nowMs < anchor to prevent early return
- Add Math.max(1, ...) to ensure steps is always at least 1
- Add test for anchorMs not provided case
2026-01-01 17:30:05 -07:00
Peter Steinberger
c31070db24 style: apply biome formatting 2026-01-02 01:29:05 +01:00
Peter Steinberger
336048441c docs: add imessage rpc and groups docs 2026-01-02 01:19:40 +01:00
Peter Steinberger
cbac34347b feat: add imessage rpc adapter 2026-01-02 01:19:40 +01:00
Peter Steinberger
3ee27a00c7 docs(changelog): note log prefix cleanup 2026-01-02 00:15:03 +00:00
Peter Steinberger
4ec020a86d fix(logging): trim provider log prefixes 2026-01-02 00:15:01 +00:00
Peter Steinberger
464dabdc16 docs: default discord reactions to on 2026-01-02 01:11:04 +01:00
Peter Steinberger
c0976ec099 fix(gateway): stream chat events for agent runs 2026-01-02 01:04:59 +01:00
Peter Steinberger
7f3113b8d4 feat: add discord reaction tool 2026-01-02 00:29:32 +01:00
Peter Steinberger
9180cbe821 fix: keep chat scrolled to bottom on session switch 2026-01-02 00:21:48 +01:00
Peter Steinberger
c5daa754ff chore: refresh a2ui bundle hash 2026-01-02 00:17:59 +01:00
Peter Steinberger
23a29216d3 fix: allow remote gateway password config 2026-01-02 00:17:54 +01:00
Peter Steinberger
8a2168ecf6 style: fix swiftlint warnings 2026-01-02 00:17:49 +01:00
Peter Steinberger
38d8a669b4 fix: add discord mention context history 2026-01-01 23:58:35 +01:00
Peter Steinberger
06e379a239 fix: suppress stray HEARTBEAT_OK replies 2026-01-01 23:53:29 +01:00
Peter Steinberger
7c0379ce05 feat: add recent session switchers 2026-01-01 23:50:26 +01:00
Peter Steinberger
c7c13f2d5e fix(gateway): read CLAWDIS_GATEWAY_PASSWORD from env
The CLI client (callGateway) now reads password from:
1. opts.password (explicit parameter)
2. CLAWDIS_GATEWAY_PASSWORD env var (NEW)
3. remote.password from config

This allows CLI commands like doctor/health to authenticate
without needing a --password flag when the env var is set.

Fixes auth issues for users with password-protected gateways.
2026-01-01 22:40:36 +00:00
Peter Steinberger
6df9b3f38c docs: update changelog 2026-01-01 23:30:02 +01:00
Peter Steinberger
ca81d94b90 feat(cli): hint gateway reachability for local/remote 2026-01-01 23:30:02 +01:00
Peter Steinberger
a39ef7181d feat(cli): add provider setup primers 2026-01-01 23:22:52 +01:00
Peter Steinberger
93b7e3431b docs: update changelog 2026-01-01 23:22:52 +01:00
Peter Steinberger
dd02cc0747 docs: update changelog 2026-01-01 23:19:30 +01:00
Peter Steinberger
867883453e fix(cli): allow skipping skill dependency installs 2026-01-01 23:19:26 +01:00
Peter Steinberger
a68784c319 docs: update changelog 2026-01-01 23:16:42 +01:00
Peter Steinberger
46c763410f fix(cli): colorize provider status note 2026-01-01 23:16:36 +01:00
Peter Steinberger
815d4572f6 feat(cli): explain Tailscale exposure options 2026-01-01 23:16:28 +01:00
Peter Steinberger
279a191b86 fix(macos): colorize provider status subtitles 2026-01-01 23:16:22 +01:00
Peter Steinberger
f0da42917b feat(macos): verify Claude OAuth in onboarding 2026-01-01 23:16:15 +01:00
Peter Steinberger
6e87fd2d4c docs: update changelog 2026-01-01 22:55:25 +01:00
Peter Steinberger
fbf5efb570 feat(process): support env overrides in exec 2026-01-01 22:55:21 +01:00
Peter Steinberger
1a3323a261 fix(cli): improve skill install failure output 2026-01-01 22:55:15 +01:00
Peter Steinberger
b858fdd755 feat(macos): show skills in onboarding 2026-01-01 22:55:10 +01:00
Peter Steinberger
0aff827414 fix: preserve webchat run ordering 2026-01-01 22:46:43 +01:00
Peter Steinberger
bd8a0a9f8f feat: add remote CDP browser support 2026-01-01 22:44:52 +01:00
Peter Steinberger
73d0e2cb81 fix: gate skills by OS 2026-01-01 22:25:37 +01:00
Peter Steinberger
47f816696c fix: refine A2UI status HUD styling 2026-01-01 20:47:51 +00:00
Peter Steinberger
1cf455e91c fix: use brew installer for imsg skill 2026-01-01 21:41:39 +01:00
Peter Steinberger
952c8c2d64 fix: improve canvas debug status in remote mode 2026-01-01 20:41:13 +00:00
Peter Steinberger
dce3bf01fd build: refresh a2ui bundle hash 2026-01-01 20:41:09 +00:00
Peter Steinberger
7b1687d7e5 fix: resolve macOS config store concurrency 2026-01-01 21:31:37 +01:00
Peter Steinberger
9ad6863567 docs: trim changelog 2026-01-01 21:31:13 +01:00
Peter Steinberger
4c1424bb83 chore: fix lint warnings 2026-01-01 21:25:29 +01:00
Peter Steinberger
c7364de2f0 fix: align telegram token resolution 2026-01-01 21:22:59 +01:00
Peter Steinberger
e0043906be docs: add Discord badge 2026-01-01 21:22:00 +01:00
Peter Steinberger
eda9fb5522 feat(skills): add things-mac 2026-01-01 21:12:37 +01:00
Peter Steinberger
8a775144bf docs: update changelog 2026-01-01 21:09:36 +01:00
Peter Steinberger
9b65534561 test: harden wizard e2e flow 2026-01-01 21:09:32 +01:00
Peter Steinberger
f6c0618596 fix: improve web chat scroll and text 2026-01-01 21:09:28 +01:00
Peter Steinberger
15fd030fa4 docs: refresh onboarding wizard docs 2026-01-01 21:09:24 +01:00
Peter Steinberger
693be03dcc test: cover remote config routing 2026-01-01 20:29:53 +01:00
Peter Steinberger
6e3cb34024 chore: pin ElevenLabsKit + wizard note 2026-01-01 20:19:00 +01:00
Peter Steinberger
bd7cd33b02 feat: add remote gateway client config 2026-01-01 20:10:50 +01:00
Peter Steinberger
a72fdf7c26 feat: expand wizard setup flow 2026-01-01 19:14:14 +01:00
Peter Steinberger
850cbfe369 fix: route macOS remote config via gateway 2026-01-01 18:58:41 +01:00
Peter Steinberger
351db0632d fix(signal): map stderr INFO to log 2026-01-01 17:30:51 +00:00
Peter Steinberger
d642e90cdd style: format onboarding commands 2026-01-01 17:30:51 +00:00
Peter Steinberger
c454f7ac0d fix: detect bun relay assets 2026-01-01 18:30:16 +01:00
Peter Steinberger
b5b47d7273 docs: update changelog 2026-01-01 17:24:42 +00:00
Peter Steinberger
7c2c541729 feat: expand onboarding wizard 2026-01-01 18:23:59 +01:00
Peter Steinberger
f10abc8ee0 fix: narrow onboarding prompt types 2026-01-01 17:14:02 +00:00
Peter Steinberger
8ea27968d7 style: apply biome formatting 2026-01-01 17:06:47 +00:00
Peter Steinberger
956db9c182 fix: keep pi-ai tool types for published sdk 2026-01-01 17:02:02 +00:00
Peter Steinberger
3eb3f38adf test: add onboarding e2e harness 2026-01-01 18:01:42 +01:00
Peter Steinberger
35b66e5ad1 feat: add onboarding wizard 2026-01-01 17:58:07 +01:00
Peter Steinberger
d83ea305b5 fix: satisfy bun biome formatting 2026-01-01 16:54:46 +00:00
Peter Steinberger
c1d8508748 fix: clean up pi-agent-core lint 2026-01-01 16:51:08 +00:00
Peter Steinberger
dc8f2bda2a fix: restart via systemd on linux 2026-01-01 17:48:28 +01:00
Peter Steinberger
f0f5acfa42 fix: update pi-agent-core integration 2026-01-01 16:46:40 +00:00
Peter Steinberger
4e00edf8a7 docs: update changelog for macOS rpath fix 2026-01-01 17:44:53 +01:00
Petter Blomberg
02d5c00873 macOS: move rpath configuration to build step for reliability 2026-01-01 17:44:39 +01:00
Petter Blomberg
17009d28cf build: fix hardcoded dependency path for ElevenLabsKit 2026-01-01 17:43:27 +01:00
Peter Steinberger
325a6a4e02 docs: update changelog for chat duplicate fix 2026-01-01 17:42:30 +01:00
Marc Beaupre
b51b24955c fix(chat): clear input immediately after send to prevent duplicate messages
Two issues were causing the input field to retain text after sending:

1. ChatComposer's NSViewRepresentable was skipping all updates while the
   text view was first responder. Now it allows clearing (empty binding)
   even during editing, only skipping other updates to avoid cursor jumps.

2. ChatViewModel cleared input after awaiting the network response, leaving
   text visible during the round trip. Now clears immediately after capturing
   the message content, before the async send.

Together these prevent users from accidentally re-sending messages when
the input appeared unchanged after pressing Enter.
2026-01-01 17:42:05 +01:00
Peter Steinberger
a954aaa507 docs: thank contributor for macOS device resource fix 2026-01-01 17:39:54 +01:00
Petter Blomberg
ad475239a5 fix(macos): prioritize main bundle for device resources to prevent crash 2026-01-01 17:39:33 +01:00
Peter Steinberger
5e280674f9 docs: require Xcode 26.2+ 2026-01-01 17:38:16 +01:00
Petter Blomberg
6cdfd143b0 docs: add macOS developer setup and troubleshooting guides 2026-01-01 17:37:19 +01:00
Petter Blomberg
da454fa376 build: update A2UI bundle hash 2026-01-01 17:37:19 +01:00
Peter Steinberger
358dd4f791 merge: fix/codesign-adhoc 2026-01-01 17:34:46 +01:00
Peter Steinberger
2401abe17e docs: update changelog for codesign fix 2026-01-01 17:30:22 +01:00
Peter Steinberger
56ea6b6e43 fix: align tool schemas and health snapshot 2026-01-01 17:30:19 +01:00
Peter Steinberger
04691ed598 chore: apply biome formatting 2026-01-01 17:30:15 +01:00
William Stock
7366b55b14 docs: Add manual OAuth setup for remote/headless deployments
Expand "Remote mode note" section with:
- Exact oauth.json format required (access, refresh, expires)
- Note that auto-import doesn't work with Claude Code credentials
- jq script to convert Claude Code credentials to Clawdis format

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-01 17:21:27 +01:00
Peter Steinberger
a248bea50f chore(browser): format CDP helpers 2026-01-01 16:19:37 +00:00
Peter Steinberger
c8c84bc419 test(browser): fix chrome reachability mock 2026-01-01 16:16:55 +00:00
Peter Steinberger
5f990fb3a2 docs: note browser resiliency and reset 2026-01-01 16:15:17 +00:00
Peter Steinberger
538c1eb660 fix(browser): harden CDP readiness 2026-01-01 16:15:12 +00:00
Peter Steinberger
9f704d7aa7 docs: note macos app logging menu icon 2026-01-01 17:12:49 +01:00
Peter Steinberger
a5777300d8 fix(macos): add icon to app logging menu 2026-01-01 16:48:17 +01:00
Peter Steinberger
57e1362344 docs(signal): explain bot-number setup 2026-01-01 15:37:45 +00:00
Peter Steinberger
c1ccbd58f5 fix(signal): stabilize daemon + add signal delivery 2026-01-01 15:31:41 +00:00
Peter Steinberger
09a2ab420b style: biome formatting 2026-01-01 15:31:36 +00:00
Peter Steinberger
596770942a feat: add Signal provider support 2026-01-01 15:43:15 +01:00
Petter Blomberg
fe5e58af91 scripts: fix ad-hoc signing crashes and bash unbound variable error 2026-01-01 15:29:01 +01:00
Peter Steinberger
0a4c2f91f5 fix: add bottom padding to macos web chat 2026-01-01 13:20:27 +01:00
Peter Steinberger
5b33a7dcbe fix: polish macos web chat composer 2026-01-01 12:49:05 +01:00
Peter Steinberger
c7e2b1230c fix: make composer pill full-width 2026-01-01 12:18:18 +01:00
Peter Steinberger
bdf6a23de9 fix: polish web chat empty/error state 2026-01-01 11:40:11 +01:00
Peter Steinberger
1a539b9830 fix(macos): restore swift test build 2026-01-01 11:05:14 +01:00
Peter Steinberger
3addd3420b fix: tidy web chat composer layout 2026-01-01 11:05:14 +01:00
Peter Steinberger
6ea10dd153 fix: allow direct file input uploads 2026-01-01 09:44:29 +00:00
Peter Steinberger
bf0bee58b3 fix: improve browser upload triggering 2026-01-01 09:35:20 +00:00
Peter Steinberger
fbcbc60e85 feat: unify skills config 2026-01-01 10:07:31 +01:00
Peter Steinberger
0a9f06d60f docs: annotate nix path resolution 2026-01-01 09:30:12 +01:00
Peter Steinberger
f6956320f9 feat: centralize config paths and expose in snapshot 2026-01-01 09:22:37 +01:00
Peter Steinberger
20bc323963 docs: note nix support 2026-01-01 09:17:24 +01:00
Peter Steinberger
bcead5f0f4 fix: honor nix config overrides in mac app 2026-01-01 09:17:21 +01:00
Peter Steinberger
cf3049ae34 Merge pull request #40 from joshp123/upstream-preview-nix-2025-12-20
Nix mode support + macOS Info.plist template
2026-01-01 09:15:41 +01:00
Peter Steinberger
ad9a9d8d35 Merge remote-tracking branch 'origin/main' into upstream-preview-nix-2025-12-20 2026-01-01 09:15:28 +01:00
Peter Steinberger
14e9077584 chore: add bench-model script 2026-01-01 08:59:31 +01:00
Peter Steinberger
43cf526b5f docs: thank contributor for PR #64 2026-01-01 08:59:24 +01:00
Peter Steinberger
2d5c401d11 fix: prefer module bundle for device models 2026-01-01 08:58:54 +01:00
Peter Steinberger
78cf68549f Merge pull request #64 from mbelinky/fix-instances-crash
Fix Instances crash by bundling device model resources
2026-01-01 08:58:35 +01:00
Peter Steinberger
dececccd8e docs: thank contributor for PR #65 2026-01-01 08:55:51 +01:00
Mariano Belinky
941ad27551 Bundle Control UI in Mac app 2026-01-01 08:55:09 +01:00
Peter Steinberger
24e95ab38e docs: update changelog for PR #66 2026-01-01 08:37:49 +01:00
Mariano Belinky
c4de0b8255 Use user home for pnpm path 2026-01-01 08:35:54 +01:00
Peter Steinberger
7baaca4a76 docs: add model latency bench notes 2025-12-31 22:39:42 +01:00
Mariano Belinky
ea248f6743 Fix device model resources for Instances 2025-12-31 16:45:35 +01:00
Peter Steinberger
f03605d8ae test: add minimax live test 2025-12-31 16:31:23 +01:00
Peter Steinberger
0babf08926 chore: add mac app logging coverage 2025-12-31 16:28:51 +01:00
Peter Steinberger
6517b05abe feat: add swift-log app logging controls 2025-12-31 16:03:18 +01:00
Peter Steinberger
fa91b5fd03 docs: update changelog for Android chat bubble 2025-12-31 12:50:34 +01:00
Manuel Jiménez Torres
f831ccfc63 fix(android): wrong text color in user chat bubbles 2025-12-31 12:48:59 +01:00
Peter Steinberger
12084fc4f9 test: extend Z.AI live test timeout 2025-12-31 12:43:34 +01:00
Peter Steinberger
21237dae98 feat: add Z.AI env support and live test 2025-12-31 11:36:57 +01:00
Peter Steinberger
4bdc25d072 docs: link Anthropic OAuth setup 2025-12-31 11:35:42 +01:00
Peter Steinberger
2f55abace2 fix: add brew installer for ordercli skill 2025-12-31 04:52:40 +01:00
Peter Steinberger
3213e5df2d feat: add gifgrep skill 2025-12-31 04:52:37 +01:00
Peter Steinberger
7e40147aa3 fix: gate web chat/talk on mobile nodes 2025-12-30 22:05:17 +01:00
Peter Steinberger
a2a26b26fb fix: satisfy swiftformat in chat view 2025-12-30 20:41:12 +01:00
Peter Steinberger
b3cf07d6cb feat: add ui theme toggle 2025-12-30 20:25:58 +01:00
Peter Steinberger
ed76cd7574 fix: restore talk orb hit testing 2025-12-30 20:25:52 +01:00
Peter Steinberger
01b8a71ee6 docs: clarify browser wait guidance 2025-12-30 19:22:38 +00:00
Peter Steinberger
cc86bbf27d feat: add food-order skill 2025-12-30 15:43:13 +01:00
Peter Steinberger
42cbb11de8 build: update a2ui bundle 2025-12-30 14:43:34 +01:00
Peter Steinberger
52303e8eda docs: update changelog for status pill 2025-12-30 14:39:33 +01:00
Peter Steinberger
cf903be4a7 fix: avoid duplicate gateway reconnecting pill 2025-12-30 14:37:59 +01:00
Peter Steinberger
6306786645 fix: allow mp3 fallback result 2025-12-30 14:35:53 +01:00
Peter Steinberger
d7b267843e fix: fallback mp3 when pcm blocked 2025-12-30 14:32:47 +01:00
Peter Steinberger
3aefe375c1 chore: update deps and add control ui routing tests 2025-12-30 14:30:46 +01:00
Peter Steinberger
3d6cc435ef fix: hop audio to main actor 2025-12-30 14:22:03 +01:00
Peter Steinberger
973bd3a427 fix: improve talk overlay input + drag 2025-12-30 14:18:51 +01:00
Peter Steinberger
7d1ec51df5 fix: modernize chat scroll position 2025-12-30 13:52:12 +01:00
Peter Steinberger
9fb74399c8 refactor: inject audio players 2025-12-30 13:46:14 +01:00
Peter Steinberger
bc0a6fffd1 fix: tighten macOS menu device rows 2025-12-30 13:31:11 +01:00
Peter Steinberger
fa85dd6527 docs: note macOS menu layout 2025-12-30 12:57:10 +01:00
Peter Steinberger
73d595eecc chore: sync local changes 2025-12-30 12:53:17 +01:00
Peter Steinberger
3bf8b9ccf4 fix: default android talk pcm_24000 2025-12-30 12:52:56 +01:00
Peter Steinberger
83262a67b1 refactor: extract elevenlabs kit 2025-12-30 12:48:09 +01:00
Peter Steinberger
66952a682d test: add pcm streaming smoke 2025-12-30 12:27:06 +01:00
Peter Steinberger
9df22c0093 fix: address talk streaming build 2025-12-30 12:20:32 +01:00
Peter Steinberger
27adfb76fa fix: stream elevenlabs tts playback 2025-12-30 12:17:40 +01:00
Peter Steinberger
9c532eac07 feat(talk): pause + drag overlay orb 2025-12-30 11:37:52 +01:00
Peter Steinberger
2814815312 feat: add talk voice alias map 2025-12-30 11:35:29 +01:00
Peter Steinberger
ab27586674 test: cover external chat completion 2025-12-30 11:23:45 +01:00
Peter Steinberger
2749c5cac3 fix: clear external streaming bubbles 2025-12-30 11:21:57 +01:00
Peter Steinberger
715cf311df fix(ui): move mac talk orb to corner 2025-12-30 11:20:14 +01:00
Peter Steinberger
312443235d fix(ios): unblock device builds 2025-12-30 11:16:15 +01:00
Peter Steinberger
0d95d63258 fix(macos): await-safe session key selection 2025-12-30 11:07:34 +01:00
Peter Steinberger
f86772f26c fix(talk): harden TTS + add system fallback 2025-12-30 07:40:02 +01:00
Peter Steinberger
a7617e4d79 fix(ui): refine talk overlays 2025-12-30 06:47:35 +01:00
Peter Steinberger
7612a83fa2 fix(talk): align sessions and chat UI 2025-12-30 06:47:19 +01:00
Peter Steinberger
afbd18e8df fix(talk): harden playback, interrupts, and timeouts 2025-12-30 06:05:43 +01:00
Peter Steinberger
be2bc61d38 fix(talk): hard-timeout ElevenLabs synthesis 2025-12-30 05:46:47 +01:00
Peter Steinberger
dcee8beb99 style: biome format gateway server tests 2025-12-30 05:34:53 +01:00
Peter Steinberger
fb8f72d5a9 feat(ui): add centered talk orb 2025-12-30 05:27:29 +01:00
Peter Steinberger
b3f2416a09 test: reduce flaky timeouts 2025-12-30 05:27:18 +01:00
Peter Steinberger
b5ae2ccc3c fix(voice): sync talk mode chat events 2025-12-30 05:27:11 +01:00
Peter Steinberger
05efc3eace fix: avoid iOS talk mode audio tap crash 2025-12-30 04:52:57 +01:00
Peter Steinberger
24f8ff7548 chore(protocol): regenerate Swift gateway models 2025-12-30 04:42:08 +01:00
Peter Steinberger
c0c6782a17 fix(android): stabilize BridgeSession shutdown 2025-12-30 04:42:02 +01:00
Peter Steinberger
d2ac672f47 feat: add ui.seamColor accent 2025-12-30 04:14:36 +01:00
Peter Steinberger
e3d8d5f300 fix(macos): prevent Talk Mode audio hang 2025-12-30 04:14:16 +01:00
Peter Steinberger
c5d5c9fcb5 fix: make android canvas background visible 2025-12-30 04:02:52 +01:00
Peter Steinberger
2e040ee07a fix: brighten android canvas 2025-12-30 03:58:18 +01:00
Peter Steinberger
9846c46434 fix: tag A2UI platform and boost Android canvas 2025-12-30 03:49:24 +01:00
Peter Steinberger
5c7c1af44e fix: android talk timestamp parsing 2025-12-30 02:05:14 +01:00
Peter Steinberger
e119a82334 feat: talk mode key distribution and tts polling 2025-12-30 01:57:58 +01:00
Peter Steinberger
02db68aa67 fix(macos): hide Restart Gateway when remote 2025-12-30 01:57:58 +01:00
Peter Steinberger
10e1e7fd44 chore: apply biome formatting 2025-12-30 00:16:07 +00:00
Peter Steinberger
7aabe73521 chore: sync pending changes 2025-12-30 00:59:30 +01:00
Peter Steinberger
37f85bb2d1 fix: expand talk overlay bounds 2025-12-30 00:58:58 +01:00
Peter Steinberger
39fccc3699 fix: talk overlay + elevenlabs defaults 2025-12-30 00:51:17 +01:00
Peter Steinberger
53eccc1c1e fix: wire talk menu + mac build 2025-12-30 00:17:10 +01:00
Peter Steinberger
c56292a6ec feat: move talk mode to overlay button 2025-12-30 00:01:21 +01:00
Peter Steinberger
857cd6a28a fix: align ios lint and android build 2025-12-29 23:45:58 +01:00
Peter Steinberger
303954ae8c feat: extend status activity indicators 2025-12-29 23:42:22 +01:00
Peter Steinberger
3c338d1858 fix: adjust android talk parser for kotlin json 2025-12-29 23:26:38 +01:00
Peter Steinberger
20d7882033 feat: add talk mode across nodes 2025-12-29 23:21:05 +01:00
Peter Steinberger
6927b0fb8d fix: align camera payload caps 2025-12-29 23:20:55 +01:00
Peter Steinberger
6e83f95c83 fix: clamp tool images to 5MB 2025-12-29 22:13:39 +00:00
Peter Steinberger
8f0c8a6561 fix: cap camera snap payload size 2025-12-29 23:12:20 +01:00
Peter Steinberger
a61b7056d5 feat: surface camera activity in status pill 2025-12-29 23:12:03 +01:00
Peter Steinberger
f41ade9417 feat(skills): add obsidian skill 2025-12-29 22:51:42 +01:00
Peter Steinberger
b0396e196f fix: refresh bridge tokens and enrich node settings 2025-12-29 22:11:12 +01:00
Peter Steinberger
cf42fabfd8 test: add ios swift testing + android kotest 2025-12-29 21:10:44 +01:00
Peter Steinberger
52263bd5a3 fix: avoid cli gateway close race 2025-12-29 20:45:50 +01:00
Peter Steinberger
24151a2028 fix: mark screen recorder sendable 2025-12-29 20:28:06 +01:00
Peter Steinberger
c11e2d9e5e fix: avoid self capture in ReplayKit start 2025-12-29 20:26:49 +01:00
Peter Steinberger
a8c9b2810b fix: align ReplayKit stopCapture call 2025-12-29 20:25:44 +01:00
Peter Steinberger
7a849ab7d1 fix: isolate ReplayKit capture state 2025-12-29 20:24:34 +01:00
Peter Steinberger
c14d738d37 fix: avoid screen recorder data races 2025-12-29 20:22:26 +01:00
Peter Steinberger
65478a6ff3 fix: avoid main-actor stopCapture error 2025-12-29 20:20:14 +01:00
Peter Steinberger
41be9232fe fix: prevent iOS screen capture crash 2025-12-29 20:10:36 +01:00
Peter Steinberger
653932e50d fix: show connected nodes only 2025-12-29 18:35:52 +01:00
Peter Steinberger
09ef991e1a chore: harden restart script 2025-12-29 18:09:27 +01:00
Josh Palmer
0f7029583c macOS: load device models from bundle resources 2025-12-29 17:49:13 +01:00
Josh Palmer
10eced9971 fix: use telegram token file for sends and guard console EPIPE 2025-12-29 17:49:13 +01:00
Josh Palmer
1d8b47785c feat(macos): add current TeamID to Peekaboo allowlist
Problem: The bridge only accepts the upstream TeamID, so packaged builds signed locally (Nix/CI) can’t use the bridge even though they are the same app.

Fix: Include the running app’s TeamID (from its code signature) in the allowlist.

Safety: TeamID gating remains; this just adds the app’s own TeamID to preserve permissions/automation in reproducible installs.
2025-12-29 17:49:13 +01:00
Josh Palmer
ced271bec1 chore(macos): harden mktemp templates in codesign 2025-12-29 17:49:13 +01:00
Josh Palmer
5d19afd422 feat: improve health checks (telegram tokenFile + hints) 2025-12-29 17:49:13 +01:00
Josh Palmer
b7363f7c18 feat: Nix mode config, UX, onboarding, SwiftPM plist, docs 2025-12-29 17:49:13 +01:00
Peter Steinberger
aa2700ffa7 chore: set ios signing team for device builds 2025-12-29 17:38:21 +01:00
Peter Steinberger
510e2a1d17 fix: menu devices list 2025-12-29 17:31:23 +01:00
Peter Steinberger
ebfe55f909 fix: enable canvas webview scrolling on mobile nodes 2025-12-29 17:13:31 +01:00
Peter Steinberger
26fa9dea97 chore: bump version to 2.0.0-beta5 2025-12-28 14:38:48 +00:00
Peter Steinberger
3bb4c0c237 fix: report macos product version in presence 2025-12-28 14:34:07 +00:00
Peter Steinberger
255a875a2a chore: refresh a2ui bundle hash 2025-12-28 12:06:48 +00:00
Peter Steinberger
2b5f3f1361 docs: clarify watchdog reconnect note 2025-12-28 12:05:06 +00:00
Peter Steinberger
eb158545fc fix: force web reconnect on stalled close 2025-12-28 12:04:20 +00:00
Peter Steinberger
cade7b1132 docs: clarify gateway readiness in changelog 2025-12-28 10:30:40 +00:00
Peter Steinberger
d529736597 fix(macos): fully stop Voice Wake runtime when disabled 2025-12-28 10:17:30 +00:00
Peter Steinberger
8dfc031c4d fix: start gateway before control channel 2025-12-28 09:24:43 +00:00
Peter Steinberger
91c9859000 fix: harden heartbeat acks + gateway reconnect 2025-12-27 20:02:27 +00:00
Peter Steinberger
3a485a14a4 fix: skip whatsapp heartbeat when provider inactive 2025-12-27 19:34:10 +00:00
Peter Steinberger
a61c27c4d0 fix: correct beta3 appcast URL 2025-12-27 20:00:08 +01:00
Peter Steinberger
e5cae2a2e4 chore: release 2.0.0-beta4 2025-12-27 19:43:43 +01:00
Peter Steinberger
7f961237f9 chore: harden release checks 2025-12-27 19:35:39 +01:00
Peter Steinberger
69a6538567 docs: note notarytool profile 2025-12-27 19:24:24 +01:00
Peter Steinberger
5b3c18ab84 chore: release 2.0.0-beta3 2025-12-27 19:02:35 +01:00
Peter Steinberger
907371453d fix(macos): soften light mode usage bar track 2025-12-27 14:05:36 +01:00
Peter Steinberger
81abffd145 fix(macos): boost light mode usage bar contrast 2025-12-27 14:03:45 +01:00
Peter Steinberger
44ef8fe5c8 fix(macos): refresh sessions on menu open 2025-12-27 13:49:03 +01:00
Peter Steinberger
cae78b3f91 fix: treat /model status as model list 2025-12-27 12:10:44 +00:00
Peter Steinberger
c0fb814658 fix: normalize imports for lint 2025-12-27 04:02:13 +01:00
Peter Steinberger
7ce0140c81 docs: update changelog 2025-12-27 03:21:25 +01:00
Peter Steinberger
12b3034921 chore(canvas): update a2ui bundle hash 2025-12-27 03:21:20 +01:00
Peter Steinberger
ec482ac867 fix(macos): tighten chat window chrome 2025-12-27 03:21:14 +01:00
Peter Steinberger
ae52fb7a01 fix(macos): relax chat window min size 2025-12-27 02:55:24 +01:00
Peter Steinberger
e8ff08e121 fix(macos): round chat window chrome 2025-12-27 02:51:59 +01:00
Peter Steinberger
cc8e104cd6 fix(macos): enforce chat window default size 2025-12-27 02:43:50 +01:00
Peter Steinberger
5919a277bb fix(macos): stabilize menu width tracking 2025-12-27 02:43:50 +01:00
Peter Steinberger
96911d7790 fix: enqueue system event on model switch 2025-12-27 01:17:12 +00:00
Peter Steinberger
acd3f7dba7 fix(macos): lock menu width on hover 2025-12-27 01:50:25 +01:00
Peter Steinberger
8aff3979db docs: add local lmstudio setup 2025-12-27 00:48:19 +00:00
Peter Steinberger
eafcd862be chore: update protocol models 2025-12-27 01:45:58 +01:00
Peter Steinberger
8826170635 fix: resolve CI lint and android build 2025-12-27 01:41:43 +01:00
Peter Steinberger
c54e4d0900 refactor: node tools and canvas host url 2025-12-27 01:36:29 +01:00
Peter Steinberger
52ca5c4aa2 fix: drop identity emoji response prefix 2025-12-27 00:36:04 +00:00
Peter Steinberger
95f8f80e74 fix: allow empty responsePrefix 2025-12-27 00:33:04 +00:00
Peter Steinberger
7e380bb6f8 fix: enable lmstudio responses and drop think tags 2025-12-27 00:28:52 +00:00
Peter Steinberger
2477ffd860 chore: fix lint/test gating 2025-12-26 23:54:30 +00:00
Peter Steinberger
a3dc46bf9d fix(a2ui): center status overlay 2025-12-27 00:28:38 +01:00
Peter Steinberger
5c8e1b6eef feat: add model aliases + minimax shortlist 2025-12-26 23:26:14 +00:00
Peter Steinberger
ae9a8ce34c fix(a2ui): center status overlay 2025-12-27 00:23:27 +01:00
Peter Steinberger
67b9a675f5 fix(macos): allow http loads in canvas webview 2025-12-27 00:20:58 +01:00
Peter Steinberger
fae11e5a55 fix(gateway): advertise reachable canvas host 2025-12-27 00:07:19 +01:00
Peter Steinberger
4daf75a469 fix(macos): enforce node bridge timeouts 2025-12-27 00:02:41 +01:00
Peter Steinberger
d0293649cd fix(macos): refresh menu sessions without resizing 2025-12-26 22:48:58 +01:00
Peter Steinberger
353366ac54 fix(macos): expand highlighted menu rows to full width 2025-12-26 22:41:29 +01:00
Peter Steinberger
1a8ffebb00 fix(macos): stabilize menu row width 2025-12-26 22:34:18 +01:00
Peter Steinberger
5ffbddcc57 feat(mac): add allow camera toggle 2025-12-26 21:33:22 +00:00
Peter Steinberger
5fbcbe7e52 feat(mac): add discord connections UI 2025-12-26 21:33:22 +00:00
Peter Steinberger
7daa93cf5a fix(macos): expand menu hover highlight width 2025-12-26 22:30:29 +01:00
Peter Steinberger
9e32f29d19 test: organize heartbeat test imports 2025-12-26 21:29:49 +00:00
Peter Steinberger
1f25e38c2d fix(macos): keep menu width stable while open 2025-12-26 22:27:24 +01:00
Peter Steinberger
c10a386d17 fix(macos): detect and reset stale SSH tunnels 2025-12-26 22:12:33 +01:00
Peter Steinberger
a13db82d28 fix(nodes): improve version reporting 2025-12-26 21:45:00 +01:00
Peter Steinberger
ec392dc870 feat(mac): add node ssh and compact versions 2025-12-26 20:42:49 +00:00
Peter Steinberger
90d00fb095 fix(mac): reorder menu toggles 2025-12-26 20:42:45 +00:00
Peter Steinberger
e336b7f27e fix: use final heartbeat payload 2025-12-26 20:39:20 +00:00
Peter Steinberger
7f4c992dd7 fix(mac): move action group below toggles 2025-12-26 20:31:37 +00:00
Peter Steinberger
ba1626a5b9 fix(ios): accept truthy A2UI ready check 2025-12-26 21:17:37 +01:00
Peter Steinberger
ab73c40bfe fix(mac): refine node submenu copy behavior 2025-12-26 20:05:23 +00:00
Peter Steinberger
4016bc2416 fix(a2ui): center empty canvas text 2025-12-26 20:43:45 +01:00
Peter Steinberger
9302daadc1 fix(mac): align node details 2025-12-26 19:32:48 +00:00
Peter Steinberger
de7429e148 fix(mac): show node versions in menu 2025-12-26 19:25:28 +00:00
Peter Steinberger
5892bd45d8 fix(mac): tweak menu icons 2025-12-26 19:23:53 +00:00
Peter Steinberger
9317eccfc8 fix(mac): regroup menubar sections 2025-12-26 19:18:12 +00:00
Peter Steinberger
1236c4dafb refactor: make browser actions ref-only 2025-12-26 19:02:27 +00:00
Peter Steinberger
f50f18f65a feat(mac): refine menubar nodes layout 2025-12-26 19:02:27 +00:00
Peter Steinberger
747cc4daa5 fix: gate libsignal session logs behind verbose 2025-12-26 19:02:27 +00:00
Peter Steinberger
51b6a785e6 fix(canvas): center debug status overlay 2025-12-26 20:01:23 +01:00
Peter Steinberger
f4d41ef254 chore(ios): auto team id fallback 2025-12-26 18:19:48 +01:00
Peter Steinberger
b9d80aa535 chore(ios): add team id helper 2025-12-26 18:16:13 +01:00
Peter Steinberger
2f8213ca9a fix(a2ui): skip bundle when inputs unchanged 2025-12-26 18:11:00 +01:00
Peter Steinberger
541b8cbb6c fix(ios): silence device build warnings 2025-12-26 18:09:44 +01:00
Peter Steinberger
ed2e738ea4 fix: provider startup order and enable flags 2025-12-26 16:54:53 +00:00
Peter Steinberger
17d9ba256b fix(discord): ignore destroy promise 2025-12-26 17:21:32 +01:00
Peter Steinberger
15dbac8193 docs: update beta3 changelog 2025-12-26 17:21:29 +01:00
Peter Steinberger
2119854246 build: skip a2ui bundling in build 2025-12-26 16:00:35 +01:00
Peter Steinberger
034c93fd65 fix: align discord types 2025-12-26 14:47:15 +01:00
Peter Steinberger
ce91aba4de fix: apply biome formatting 2025-12-26 14:38:37 +01:00
Peter Steinberger
e33c09f8d4 fix(tests): align discord + queue changes 2025-12-26 14:32:57 +01:00
Peter Steinberger
a678c3f53e refactor(queue): remove drop mode 2025-12-26 14:29:28 +01:00
Peter Steinberger
3e4fc7ff7f feat(queue): add reset/default directive 2025-12-26 14:24:53 +01:00
Peter Steinberger
8dda07a1e9 feat(queue): add queue modes and discord gating 2025-12-26 13:35:44 +01:00
Peter Steinberger
e9f1851c5d chore: ignore bun build artifacts 2025-12-26 13:20:30 +01:00
Shadow
ac659ff5a7 feat(discord): Discord transport 2025-12-26 13:20:30 +01:00
Peter Steinberger
557f8e5a04 fix: restore build after deps update 2025-12-26 12:17:36 +00:00
Peter Steinberger
54de5ad3fa test: isolate vitest home 2025-12-26 11:45:16 +00:00
Peter Steinberger
0709586e3a fix: support mocked model registry in catalog 2025-12-26 11:53:55 +01:00
Peter Steinberger
82ced33747 fix: align pi model discovery with auth storage 2025-12-26 11:49:13 +01:00
Peter Steinberger
d31c5d7a2c style: format web inbound 2025-12-26 11:39:48 +01:00
Peter Steinberger
2045487d5e fix: extract quoted WhatsApp reply text 2025-12-26 10:51:08 +01:00
Peter Steinberger
4611e799b7 docs: note inbox listener cleanup 2025-12-26 09:37:38 +00:00
Peter Steinberger
ffe9a2435b fix: clean up web inbox listeners on close 2025-12-26 09:27:06 +00:00
Peter Steinberger
f5d8876384 test: expand compaction retry coverage 2025-12-26 10:22:04 +01:00
Peter Steinberger
d28265cfbe fix: handle embedded agent overflow 2025-12-26 10:20:21 +01:00
Peter Steinberger
8059e83c49 chore: bump pi-mono deps 2025-12-26 10:20:21 +01:00
Peter Steinberger
d6f07c9f91 chore: fix lint after logging tweaks 2025-12-26 09:08:37 +00:00
Peter Steinberger
917cb8fa67 fix: brighten gateway model console log 2025-12-26 08:45:15 +00:00
Peter Steinberger
461db9e469 fix: split whatsapp listen hint from subsystem log 2025-12-26 08:41:58 +00:00
Peter Steinberger
112908886c fix: log heartbeat failure reasons 2025-12-26 08:34:42 +00:00
Peter Steinberger
f734801da1 fix: correct heartbeat log formatting 2025-12-26 08:17:29 +00:00
meaningfool
ea6dc7c710 fix: correctly define pnpm workspace and clean up vite build scripts
This change adds the missing 'packages' definition to pnpm-workspace.yaml, allowing pnpm to correctly install dependencies for the 'ui' sub-package. This resolves the 'vite: command not found' error during 'ui:build'. It also reverts the temporary 'pnpm dlx' workarounds in ui/package.json.
2025-12-26 09:13:17 +01:00
Peter Steinberger
cd81348ca5 chore: fix env spread lint 2025-12-26 02:02:49 +00:00
Peter Steinberger
ad91a09b07 ci: avoid macos runner queue 2025-12-26 02:02:49 +00:00
Peter Steinberger
040f73a3f4 docs: clarify heartbeat defaults 2025-12-26 03:02:11 +01:00
Peter Steinberger
0d8e0ddc4f feat: unify gateway heartbeat 2025-12-26 02:35:40 +01:00
Peter Steinberger
8f9d7405ed style: fix biome formatting 2025-12-26 00:50:46 +00:00
Peter Steinberger
72267e97ca docs: note hour durations 2025-12-26 01:36:08 +01:00
Peter Steinberger
19f87f0a89 feat: allow hour durations 2025-12-26 01:34:46 +01:00
Peter Steinberger
9f7b1f0942 feat: move heartbeat config to agent.heartbeat 2025-12-26 01:13:42 +01:00
Peter Steinberger
1ef888ca23 refactor(config): drop agent.provider 2025-12-26 01:13:42 +01:00
Peter Steinberger
8b815bce94 feat(config): allow provider/model shorthand 2025-12-26 01:13:42 +01:00
Peter Steinberger
97539db36d ci: skip ios job 2025-12-26 00:04:46 +00:00
Peter Steinberger
655fa5b8e0 style: fix pi embedded runner lint 2025-12-25 23:58:37 +00:00
Peter Steinberger
9fbd3cc16f ci: ignore ios failures 2025-12-25 23:55:55 +00:00
Rolf Fredheim
2295cbb815 feat(agent): add maxConcurrent config for parallel message handling
Adds `agent.maxConcurrent` config option to control how many agent runs
can execute in parallel across all conversations. Default remains 1
(sequential) for backwards compatibility.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-25 23:55:41 +01:00
Peter Steinberger
198f8ea700 fix(agent): serialize runs per session 2025-12-25 23:50:52 +01:00
Peter Steinberger
9fa9199747 docs: note multi-agent session rule 2025-12-25 23:50:46 +01:00
Peter Steinberger
1cd167a59a ci: run on node 24 2025-12-25 23:05:09 +01:00
Peter Steinberger
2868dc975c chore: require node >=22.12 and fix swiftformat lint 2025-12-25 23:02:31 +01:00
meaningfool
214ab16eb2 fix: correctly define pnpm workspace and clean up vite build scripts
This change adds the missing 'packages' definition to pnpm-workspace.yaml, allowing pnpm to correctly install dependencies for the 'ui' sub-package. This resolves the 'vite: command not found' error during 'ui:build'. It also reverts the temporary 'pnpm dlx' workarounds in ui/package.json.
2025-12-25 22:52:22 +01:00
Peter Steinberger
1c88d9575e fix(webchat): refresh bubbles on theme change 2025-12-25 22:35:46 +01:00
Peter Steinberger
1e4e02ddd3 docs: update beta3 changelog 2025-12-25 21:15:45 +00:00
Peter Steinberger
f6fcddbe0b fix: relax tool typing for bash tools 2025-12-25 20:27:05 +00:00
Peter Steinberger
474180c112 style: fix bash tools lint 2025-12-25 20:20:38 +00:00
Peter Steinberger
c860573f13 style: fix biome formatting 2025-12-25 20:13:48 +00:00
Peter Steinberger
c9c7354009 chore: add gateway:watch 2025-12-25 18:44:23 +00:00
Peter Steinberger
42eb7640f9 feat: add gateway restart tool 2025-12-25 18:05:37 +00:00
Peter Steinberger
aafcd569b1 feat: line-based process logs 2025-12-25 18:03:57 +00:00
Peter Steinberger
b549307ccf docs: add Sparkle HTML release notes 2025-12-25 04:27:20 +01:00
Peter Steinberger
57090d4f8d fix: align chat scroll anchor 2025-12-25 04:10:47 +01:00
Peter Steinberger
764f7586de fix: adjust tool casts for build 2025-12-25 03:36:04 +01:00
Peter Steinberger
d96f2abc4e fix: resolve agent tool typing 2025-12-25 03:33:09 +01:00
Peter Steinberger
92f467e81c fix: clean agent bash lint 2025-12-25 03:29:36 +01:00
Peter Steinberger
2442186a31 fix: silence view warnings 2025-12-25 03:23:31 +01:00
Peter Steinberger
9fb74cb58a test: assert bridge does not add loopback listener 2025-12-25 01:41:09 +00:00
Peter Steinberger
81e11c1d91 fix: bridge tailnet bind also listens on loopback 2025-12-25 01:37:47 +00:00
Peter Steinberger
dc93350e0a docs: add background bash changelog 2025-12-25 00:54:08 +00:00
Peter Steinberger
3c6432da1f feat: add background bash sessions 2025-12-25 00:25:11 +00:00
Peter Steinberger
4eecb6841a docs: add gmail hook quickstart 2025-12-24 22:59:09 +00:00
Peter Steinberger
3b83d3ff3a fix: preserve tool action enums 2025-12-24 22:50:40 +00:00
Peter Steinberger
88b92a9605 style: format gmail hooks and tools 2025-12-24 23:11:14 +01:00
Peter Steinberger
3bb5baa6d2 fix: default tailscale serve in settings 2025-12-24 22:09:23 +00:00
Peter Steinberger
59443d7ec6 style: format reply changes 2025-12-24 23:06:20 +01:00
Peter Steinberger
c1d170e13d docs: note tailscale gmail path behavior 2025-12-24 21:56:21 +00:00
Peter Steinberger
cffac6e11a fix: auto gmail serve path for tailscale 2025-12-24 21:56:17 +00:00
Peter Steinberger
79870472e1 fix: expose union tool parameters 2025-12-24 21:48:22 +00:00
Peter Steinberger
1b69c94f76 docs: clarify reply threading change 2025-12-24 22:37:32 +01:00
Peter Steinberger
cf8d1cf0e7 fix: avoid threaded replies for agent output 2025-12-24 22:36:42 +01:00
Peter Steinberger
009fbeb543 chore: add gmail hook setup notes 2025-12-24 21:20:20 +00:00
Peter Steinberger
9ceb8731d3 chore: clarify gmail serve path 2025-12-24 21:20:20 +00:00
Peter Steinberger
8f934bf817 docs: update file size guidance 2025-12-24 22:19:10 +01:00
Peter Steinberger
88be2701f4 refactor: split utilities 2025-12-24 22:16:06 +01:00
Peter Steinberger
8ee62f0ac8 style: format locator selector 2025-12-24 21:49:31 +01:00
Peter Steinberger
4d4308af78 fix: resolve coverage profile symbol at runtime 2025-12-24 21:43:46 +01:00
Peter Steinberger
f7c5eff35e docs: link webhook docs 2025-12-24 20:07:24 +00:00
Peter Steinberger
3bc1644f34 refactor: split canvas window 2025-12-24 21:04:52 +01:00
Peter Steinberger
27025b71db feat: add selector-based browser actions 2025-12-24 19:52:28 +00:00
Peter Steinberger
523d9ec3c2 feat: add gmail hooks wizard 2025-12-24 19:48:35 +00:00
Peter Steinberger
aeb5455555 feat: add webhook hook mappings
# Conflicts:
#	src/gateway/server.ts
2025-12-24 19:48:05 +00:00
Peter Steinberger
337390b590 fix: allow overlay present access 2025-12-24 20:24:37 +01:00
Peter Steinberger
836d950e05 fix: restore voice wake overlay build 2025-12-24 20:17:01 +01:00
Peter Steinberger
ad096f77fc refactor: split voice wake overlay 2025-12-24 20:09:56 +01:00
Peter Steinberger
3774494f7e test: add ios coverage tests 2025-12-24 20:00:51 +01:00
Peter Steinberger
14fae5af9e test: add ios coverage hooks 2025-12-24 20:00:45 +01:00
Peter Steinberger
65b48561a9 refactor: split critter status label 2025-12-24 19:56:24 +01:00
Peter Steinberger
842dc14c18 style: format port guardian 2025-12-24 19:41:32 +01:00
Peter Steinberger
af1afa7ba6 style: format cron settings 2025-12-24 19:40:11 +01:00
Peter Steinberger
8c4c5e524b refactor: split cron settings 2025-12-24 19:36:10 +01:00
Peter Steinberger
204bd7d2c4 test: add mac coverage helpers 2025-12-24 19:29:44 +01:00
Peter Steinberger
f44014ff00 refactor: split onboarding view 2025-12-24 19:29:27 +01:00
Peter Steinberger
01719b02e2 test: cover bridge settings discovery 2025-12-24 18:07:41 +01:00
Peter Steinberger
4ba86bbe00 test: cover bridge hello defaults 2025-12-24 18:07:38 +01:00
Peter Steinberger
b85503b3b2 fix: guard hook payload strings 2025-12-24 17:49:52 +01:00
Peter Steinberger
131a9aa1ac style: format macos sources 2025-12-24 17:47:35 +01:00
Peter Steinberger
bd223606b1 style: format gateway server 2025-12-24 17:45:39 +01:00
Peter Steinberger
f4fb80e523 test: expand overlay coverage 2025-12-24 17:43:30 +01:00
Peter Steinberger
49e466dd40 test: expand menu and node coverage 2025-12-24 17:43:30 +01:00
Peter Steinberger
deec315f6a test: expand settings coverage 2025-12-24 17:43:30 +01:00
Peter Steinberger
7fafe54e16 test: expand onboarding coverage 2025-12-24 17:43:30 +01:00
Peter Steinberger
bdcbc829a0 test: add coverage flush helper 2025-12-24 17:43:30 +01:00
Peter Steinberger
4a64e86ecb chore: update changelog 2025-12-24 14:39:26 +00:00
Peter Steinberger
1e2946ebc6 test: extend webhook coverage 2025-12-24 14:39:21 +00:00
Peter Steinberger
1ed5ca3fde feat: add gateway webhooks 2025-12-24 14:33:05 +00:00
Peter Steinberger
aa62ac4042 fix: use recognition update segments 2025-12-24 15:27:06 +01:00
Peter Steinberger
e8f24910bd style: swiftformat chat ui 2025-12-24 15:10:31 +01:00
Peter Steinberger
8d34e54dc5 fix: address swiftlint warnings 2025-12-24 15:10:22 +01:00
Peter Steinberger
c5ede3f167 build: align Commander dependency 2025-12-24 14:44:56 +01:00
Peter Steinberger
1cd108e891 fix: clear wake word match warning 2025-12-24 14:44:50 +01:00
Peter Steinberger
8878fd3028 ui: merge tool call results 2025-12-24 14:38:43 +01:00
Peter Steinberger
a22d4e7962 fix: import AnyCodable for tool cards 2025-12-24 14:35:06 +01:00
Peter Steinberger
25d2d7389f ui: render tool call cards 2025-12-24 14:29:40 +01:00
Peter Steinberger
816b784399 ui: constrain typing indicator width 2025-12-24 14:10:32 +01:00
Peter Steinberger
c250f092bb test: cover overlay level throttling 2025-12-24 13:54:03 +01:00
Peter Steinberger
b9c2bdf641 docs: update changelog 2025-12-24 13:52:41 +01:00
Peter Steinberger
5ba90db049 perf: throttle voice overlay updates 2025-12-24 13:51:41 +01:00
Peter Steinberger
88d20c5419 perf: gate idle pulse animations 2025-12-24 13:51:40 +01:00
Peter Steinberger
e158bee95f perf: reduce chat animation churn 2025-12-24 13:51:40 +01:00
Peter Steinberger
0139a77e94 fix: resolve ts build errors 2025-12-24 00:57:11 +00:00
Peter Steinberger
e76d1b899b fix: clean telegram parse error logging 2025-12-24 00:53:27 +00:00
Peter Steinberger
3fcdd6c9d7 feat: enforce final tag parsing for embedded PI 2025-12-24 00:52:33 +00:00
Peter Steinberger
bc916dbf35 feat: require final tag format in system prompt 2025-12-24 00:52:30 +00:00
Peter Steinberger
96da2efb13 style: swiftformat gateway process manager 2025-12-24 00:33:40 +00:00
Peter Steinberger
267cdf20e1 style: fix biome lint 2025-12-24 00:33:35 +00:00
Peter Steinberger
20c7df35c4 docs: note config refactor 2025-12-24 00:24:05 +00:00
Peter Steinberger
0f06e9926b docs: update routing/messages/session config 2025-12-24 00:22:57 +00:00
Peter Steinberger
93af424ce5 refactor: move inbound config 2025-12-24 00:22:52 +00:00
Peter Steinberger
5e07400cd1 refactor: update macOS config paths 2025-12-23 23:45:27 +00:00
Peter Steinberger
364a6a9444 feat: add per-session model selection 2025-12-23 23:45:20 +00:00
Peter Steinberger
b6bfd8e34f fix: anchor typing loop to run 2025-12-23 15:03:05 +00:00
Peter Steinberger
b05981ef27 fix: add reasoning tag hint for local providers 2025-12-23 14:34:56 +00:00
Peter Steinberger
42f1a56832 test: cover system prompt owner numbers 2025-12-23 14:20:09 +00:00
Peter Steinberger
f667d56701 fix: tag owner numbers in system prompt 2025-12-23 14:19:41 +00:00
Peter Steinberger
df5284beaf fix: suppress thinking stream + typing 2025-12-23 14:17:18 +00:00
Peter Steinberger
6d551b0d6e fix: normalize tool schemas for lm studio 2025-12-23 14:09:07 +00:00
Peter Steinberger
25e6339e2e chore: bump pi-mono deps 2025-12-23 14:07:54 +00:00
Peter Steinberger
f70fd30cd3 chore: include runtime info in system prompt 2025-12-23 14:05:43 +00:00
Peter Steinberger
863d26558a fix: delay typing until reply payload 2025-12-23 13:55:01 +00:00
Peter Steinberger
cba12a1abd fix: inject group activation in system prompt 2025-12-23 13:32:07 +00:00
Peter Steinberger
96d57a18ee chore: demote reply chunk logs 2025-12-23 13:25:56 +00:00
Peter Steinberger
e54ed10bc1 fix: honor /new resets with mentions in groups 2025-12-23 13:20:11 +00:00
Peter Steinberger
c8c807adcc refactor: drop PAM auth and require password for funnel 2025-12-23 13:13:09 +00:00
Peter Steinberger
cd6ed79433 fix: honor group requireMention default 2025-12-23 12:53:30 +00:00
Peter Steinberger
ea4b3b74bb chore: log whatsapp identity on start 2025-12-23 12:45:18 +00:00
Peter Steinberger
facfd64787 fix: avoid spawning duplicate gateway when external listener exists 2025-12-23 12:43:51 +00:00
Peter Steinberger
760a83d256 docs: add offline memory system proposal 2025-12-23 13:36:59 +01:00
Peter Steinberger
bbff19698b chore: flatten provider console subsystems 2025-12-23 11:27:14 +00:00
Peter Steinberger
6f38cb162c chore: bump internal version to beta3 2025-12-23 04:28:09 +01:00
Peter Steinberger
af82224f82 fix: relax Sparkle delegate isolation 2025-12-23 03:36:56 +01:00
Peter Steinberger
a938e9473b fix: isolate Sparkle delegate conformance 2025-12-23 03:28:39 +01:00
Peter Steinberger
3e88553d52 fix: isolate updater factory on main actor 2025-12-23 03:16:47 +01:00
Peter Steinberger
56245d5646 fix: strip repeated heartbeat ok tails 2025-12-23 03:12:24 +01:00
Peter Steinberger
4af08b1606 fix: preserve whatsapp group JIDs 2025-12-23 03:05:59 +01:00
Peter Steinberger
fc4a395c88 chore: update gateway protocol models 2025-12-23 03:05:04 +01:00
Peter Steinberger
de1813ab32 docs: add beta3 changelog 2025-12-23 03:02:30 +01:00
Peter Steinberger
89ace66972 style: format macOS sources 2025-12-23 03:02:09 +01:00
Peter Steinberger
63f1857bda docs: add WhatsApp integration guide 2025-12-23 03:00:27 +01:00
Peter Steinberger
279500cba4 fix: resolve build errors 2025-12-23 03:00:04 +01:00
Peter Steinberger
183270b443 fix: correct models config schema 2025-12-23 02:50:26 +01:00
Peter Steinberger
a5f4332f21 style: apply biome formatting 2025-12-23 02:49:49 +01:00
Peter Steinberger
6fad79f581 docs: document custom model providers 2025-12-23 02:48:57 +01:00
Peter Steinberger
dff6274a93 test: cover models config merge 2025-12-23 02:48:54 +01:00
Peter Steinberger
082c872469 feat: support custom model providers 2025-12-23 02:48:48 +01:00
Peter Steinberger
67a3dda53a fix: inject reply context into body 2025-12-23 02:44:38 +01:00
Peter Steinberger
950432eac0 test: update whatsapp reply quote assertions 2025-12-23 02:30:21 +01:00
Peter Steinberger
6550e7d562 fix: add whatsapp reply context 2025-12-23 02:30:21 +01:00
Peter Steinberger
ffe75f3e20 🤖 codex: add telegram reply context
# Conflicts:
#	src/telegram/bot.ts
2025-12-23 02:30:21 +01:00
878 changed files with 98605 additions and 22823 deletions

48
.dockerignore Normal file
View File

@@ -0,0 +1,48 @@
.git
.worktrees
.bun-cache
.bun
.tmp
**/.tmp
.DS_Store
**/.DS_Store
*.png
*.jpg
*.jpeg
*.webp
*.gif
*.mp4
*.mov
*.wav
*.mp3
node_modules
**/node_modules
.pnpm-store
**/.pnpm-store
.turbo
**/.turbo
.cache
**/.cache
.next
**/.next
coverage
**/coverage
*.log
tmp
**/tmp
# build artifacts
dist
**/dist
apps/macos/.build
apps/ios/build
**/*.trace
# large app trees not needed for CLI build
apps/
assets/
Peekaboo/
Swabble/
Core/
Users/
vendor/

View File

@@ -5,12 +5,33 @@ on:
pull_request:
jobs:
build:
checks:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
runtime: [node, bun]
include:
- runtime: node
task: lint
command: pnpm lint
- runtime: node
task: test
command: pnpm test
- runtime: node
task: build
command: pnpm build
- runtime: node
task: protocol
command: pnpm protocol:check
- runtime: bun
task: lint
command: bunx biome check src
- runtime: bun
task: test
command: bunx vitest run
- runtime: bun
task: build
command: bunx tsc -p tsconfig.json
steps:
- name: Checkout
uses: actions/checkout@v4
@@ -34,7 +55,7 @@ jobs:
if: matrix.runtime == 'node'
uses: actions/setup-node@v4
with:
node-version: 22
node-version: 24
check-latest: true
- name: Setup Bun
@@ -49,7 +70,7 @@ jobs:
if: matrix.runtime == 'bun'
uses: actions/setup-node@v4
with:
node-version: 22
node-version: 24
check-latest: true
- name: Runtime versions
@@ -77,36 +98,42 @@ jobs:
pnpm -v
pnpm install --ignore-scripts=false --config.engine-strict=false --config.enable-pre-post-scripts=true || pnpm install --ignore-scripts=false --config.engine-strict=false --config.enable-pre-post-scripts=true
- name: Lint (node)
if: matrix.runtime == 'node'
run: pnpm lint
- name: Test (node)
if: matrix.runtime == 'node'
run: pnpm test
- name: Build (node)
if: matrix.runtime == 'node'
run: pnpm build
- name: Protocol check (node)
if: matrix.runtime == 'node'
run: pnpm protocol:check
- name: Lint (bun)
if: matrix.runtime == 'bun'
run: bunx biome check src
- name: Test (bun)
if: matrix.runtime == 'bun'
run: bunx vitest run
- name: Build (bun)
if: matrix.runtime == 'bun'
run: bunx tsc -p tsconfig.json
- name: Run ${{ matrix.task }} (${{ matrix.runtime }})
run: ${{ matrix.command }}
macos-app:
if: github.event_name == 'pull_request'
runs-on: macos-latest
strategy:
fail-fast: false
matrix:
include:
- task: lint
command: |
swiftlint --config .swiftlint.yml
swiftformat --lint apps/macos/Sources --config .swiftformat
- task: build
command: |
set -euo pipefail
for attempt in 1 2 3; do
if swift build --package-path apps/macos --configuration release; then
exit 0
fi
echo "swift build failed (attempt $attempt/3). Retrying…"
sleep $((attempt * 20))
done
exit 1
- task: test
command: |
set -euo pipefail
for attempt in 1 2 3; do
if swift test --package-path apps/macos --parallel --enable-code-coverage --show-codecov-path; then
exit 0
fi
echo "swift test failed (attempt $attempt/3). Retrying…"
sleep $((attempt * 20))
done
exit 1
steps:
- name: Checkout
uses: actions/checkout@v4
@@ -141,36 +168,10 @@ jobs:
xcodebuild -version
swift --version
- name: SwiftLint
run: swiftlint --config .swiftlint.yml
- name: SwiftFormat (lint mode)
run: swiftformat --lint apps/macos/Sources --config .swiftformat
- name: Swift build (release)
run: |
set -euo pipefail
for attempt in 1 2 3; do
if swift build --package-path apps/macos --configuration release; then
exit 0
fi
echo "swift build failed (attempt $attempt/3). Retrying…"
sleep $((attempt * 20))
done
exit 1
- name: Swift tests (coverage)
run: |
set -euo pipefail
for attempt in 1 2 3; do
if swift test --package-path apps/macos --parallel --enable-code-coverage --show-codecov-path; then
exit 0
fi
echo "swift test failed (attempt $attempt/3). Retrying…"
sleep $((attempt * 20))
done
exit 1
- name: Run ${{ matrix.task }}
run: ${{ matrix.command }}
ios:
if: false # ignore iOS in CI for now
runs-on: macos-latest
steps:
- name: Checkout
@@ -344,6 +345,14 @@ jobs:
android:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- task: test
command: ./gradlew --no-daemon :app:testDebugUnitTest
- task: build
command: ./gradlew --no-daemon :app:assembleDebug
steps:
- name: Checkout
uses: actions/checkout@v4
@@ -383,6 +392,6 @@ jobs:
"platforms;android-36" \
"build-tools;36.0.0"
- name: Android unit tests + debug build
- name: Run Android ${{ matrix.task }}
working-directory: apps/android
run: ./gradlew --no-daemon :app:testDebugUnitTest :app:assembleDebug
run: ${{ matrix.command }}

8
.gitignore vendored
View File

@@ -1,17 +1,22 @@
node_modules
.env
dist
*.bun-build
pnpm-lock.yaml
coverage
.pnpm-store
.worktrees/
.DS_Store
**/.DS_Store
ui/src/ui/__screenshots__/
ui/playwright-report/
ui/test-results/
# Bun build artifacts
*.bun-build
apps/macos/.build/
apps/shared/ClawdisKit/.build/
bin/
bin/clawdis-mac
bin/docs-list
apps/macos/.build-local/
@@ -21,9 +26,11 @@ Core/
apps/ios/*.xcodeproj/
apps/ios/*.xcworkspace/
apps/ios/.swiftpm/
vendor/
# Vendor build artifacts
vendor/a2ui/renderers/lit/dist/
.bundle.hash
# fastlane (iOS)
apps/ios/fastlane/README.md
@@ -39,3 +46,4 @@ apps/ios/*.dSYM.zip
# provisioning profiles (local)
apps/ios/*.mobileprovision
.env

2
.npmrc
View File

@@ -1 +1 @@
allow-build-scripts=@whiskeysockets/baileys,sharp,esbuild,protobufjs,fs-ext
allow-build-scripts=@whiskeysockets/baileys,sharp,esbuild,protobufjs,fs-ext,node-pty

View File

@@ -16,13 +16,14 @@
- Language: TypeScript (ESM). Prefer strict typing; avoid `any`.
- Formatting/linting via Biome; run `pnpm lint` before commits.
- Keep files concise; extract helpers instead of “V2” copies. Use existing patterns for CLI options and dependency injection via `createDefaultDeps`.
- Keep every file ≤ 500 LOC; refactor or split before exceeding and check frequently.
- Aim to keep files under ~700 LOC; guideline only (not a hard guardrail). Split/refactor when it improves clarity or testability.
## Testing Guidelines
- Framework: Vitest with V8 coverage thresholds (70% lines/branches/functions/statements).
- Naming: match source names with `*.test.ts`; e2e in `*.e2e.test.ts`.
- Run `pnpm test` (or `pnpm test:coverage`) before pushing when you touch logic.
- Pure test additions/fixes generally do **not** need a changelog entry unless they alter user-facing behavior or the user asks for one.
- Mobile: before using a simulator, check for connected real devices (iOS + Android) and prefer them when available.
## Commit & Pull Request Guidelines
- Create commits with `scripts/committer "<msg>" <file...>`; avoid manual `git add`/`git commit` so staging stays scoped.
@@ -40,14 +41,20 @@
- macOS logs: use `./scripts/clawlog.sh` (aka `vtlog`) to query unified logs for subsystem `com.steipete.clawdis`; it supports follow/tail/category filters and expects passwordless sudo for `/usr/bin/log`.
- Also read the shared guardrails at `~/Projects/oracle/AGENTS.md` and `~/Projects/agent-scripts/AGENTS.MD` before making changes; align with any cross-repo rules noted there.
- SwiftUI state management (iOS/macOS): prefer the `Observation` framework (`@Observable`, `@Bindable`) over `ObservableObject`/`@StateObject`; dont introduce new `ObservableObject` unless required for compatibility, and migrate existing usages when touching related code.
- Connection providers: when adding a new connection, update every UI surface and docs (macOS app, web UI, mobile if applicable, onboarding/overview docs) and add matching status + configuration forms so provider lists and settings stay in sync.
- **Restart apps:** “restart iOS/Android apps” means rebuild (recompile/install) and relaunch, not just kill/launch.
- **Device checks:** before testing, verify connected real devices (iOS/Android) before reaching for simulators/emulators.
- iOS Team ID lookup: `security find-identity -p codesigning -v` → use Apple Development (…) TEAMID. Fallback: `defaults read com.apple.dt.Xcode IDEProvisioningTeamIdentifiers`.
- A2UI bundle hash: `src/canvas-host/a2ui/.bundle.hash` is auto-generated; regenerate via `pnpm canvas:a2ui:bundle` (or `scripts/bundle-a2ui.sh`) instead of manual conflict resolution.
- Notary key file lives at `~/Library/CloudStorage/Dropbox/Backup/AppStore/AuthKey_NJF3NFGTS3.p8` (Sparkle keys live under `~/Library/CloudStorage/Dropbox/Backup/Sparkle`).
- **Multi-agent safety:** do **not** create/apply/drop `git stash` entries unless Peter explicitly asks (this includes `git pull --rebase --autostash`). Assume other agents may be working; keep unrelated WIP untouched and avoid cross-cutting state changes.
- **Multi-agent safety:** when Peter says "push", you may `git pull --rebase` to integrate latest changes (never discard other agents' work). When Peter says "commit", scope to your changes only. When Peter says "commit all", commit everything in grouped chunks.
- **Multi-agent safety:** do **not** create/remove/modify `git worktree` checkouts (or edit `.worktrees/*`) unless Peter explicitly asks.
- **Multi-agent safety:** do **not** switch branches / check out a different branch unless Peter explicitly asks.
- **Multi-agent safety:** running multiple agents is OK as long as each agent has its own session.
- When asked to open a “session” file, open the Pi session logs under `~/.clawdis/sessions/*.jsonl` (newest unless a specific ID is given), not the default `sessions.json`. If logs are needed from Mac Studio, SSH via Tailscale and read the same path there.
- Menubar dimming + restart flow mirrors Trimmy: use `scripts/restart-mac.sh` (kills all Clawdis variants, runs `swift build`, packages, relaunches). Icon dimming depends on MenuBarExtraAccess wiring in AppMain; keep `appearsDisabled` updates intact when touching the status item.
- Do not rebuild the macOS app over SSH; rebuilds must be run directly on the Mac.
- Never send streaming/partial replies to external messaging surfaces (WhatsApp, Telegram); only final replies should be delivered there. Streaming/tool events may still go to internal UIs/control channel.
- Voice wake forwarding tips:
- Command template should stay `clawdis-mac agent --message "${text}" --thinking low`; `VoiceWakeForwarder` already shell-escapes `${text}`. Dont add extra quotes.

View File

@@ -1,9 +1,392 @@
# Changelog
## Unreleased — 2025-12-23
## Unreleased
### Breaking
- Identifiers: rename bundle IDs and internal domains to `com.clawdis.*` (macOS: `com.clawdis.mac`, iOS: `com.clawdis.ios`, Android: `com.clawdis.android`) and update the gateway LaunchAgent label to `com.clawdis.gateway`.
- Agent tools: drop the `clawdis_` prefix (`browser`, `canvas`, `nodes`, `cron`, `gateway`).
- Bash tool: remove `stdinMode: "pty"`/node-pty support; use the tmux skill for real TTYs.
- Sessions: primary session key is fixed to `main` (or `global` for global scope); `session.mainKey` is ignored.
### Features
- Highlight: agent-to-agent ping-pong (reply-back loop) with `REPLY_SKIP` plus target announce step with `ANNOUNCE_SKIP` (max turns configurable, 05).
- Gateway: support `gateway.port` + `CLAWDIS_GATEWAY_PORT` across CLI, TUI, and macOS app.
- Gateway: add config hot reload with hybrid restart strategy (`gateway.reload`) and per-section reload handling.
- UI: centralize tool display metadata and show action/detail summaries across Web Chat, SwiftUI, Android, and the TUI.
- Control UI: support configurable base paths (`gateway.controlUi.basePath`, default unchanged) for hosting under URL prefixes.
- Onboarding: shared wizard engine powering CLI + macOS via gateway wizard RPC.
- Config: expose schema + UI hints for generic config forms (Web UI + future clients).
- Browser: add multi-profile browser control with per-profile remote CDP URLs — thanks @jamesgroat.
- Skills: add blogwatcher skill for RSS/Atom monitoring — thanks @Hyaxia.
- Skills: add Notion API skill — thanks @scald.
- Discord: emit system events for reaction add/remove with per-guild reaction notifications (off|own|all|allowlist) (#140) — thanks @thewilloftheshadow.
- Agent: add optional per-session Docker sandbox for tool execution (`agent.sandbox`) with allow/deny policy and auto-pruning.
- Agent: add sandboxed Chromium browser (CDP + optional noVNC observer) for sandboxed sessions.
- Nodes: add `location.get` with Always/Precise settings on macOS/iOS/Android plus CLI/tool support.
- Android nodes: add `sms.send` with permission-gated capability refresh (#172) — thanks @vsabavat.
### Fixes
- Telegram/WhatsApp: native replies now target the original inbound message; reply context is captured in `ReplyTo*` fields for templates. (Thanks @joshp123 for the PR and follow-up question.)
- macOS: improve Swift 6 strict concurrency compatibility (#166) — thanks @Nachx639.
- CI: fix lint ordering after merge cleanup (#156) — thanks @steipete.
- CI: consolidate checks to avoid redundant installs (#144) — thanks @thewilloftheshadow.
- WhatsApp: support `gifPlayback` for MP4 GIF sends via CLI/gateway.
- Gateway: log config hot reloads for dynamic-read changes without restarts.
- Sessions: prevent `sessions_send` timeouts by running nested agent turns on a separate lane.
- Sessions: use per-send run IDs for gateway agent calls to avoid wait collisions.
- Auto-reply: drop final payloads when block streaming to avoid duplicate Discord sends.
- Auto-reply: fix typing TTL to 2 minutes and log TTL with s/m units.
- Bash tool: default auto-background delay to 10s.
- Telegram: chunk block-stream replies to avoid “message is too long” errors (#124) — thanks @mukhtharcm.
- Block streaming: default to text_end and suppress duplicate block sends while in-flight.
- Block streaming: avoid duplicate block chunks when providers repeat full content on text_end.
- Block streaming: drop final payloads after soft chunking to keep Discord order intact.
- Gmail hooks: resolve gcloud Python to a real executable when PATH uses mise shims — thanks @joargp.
- Control UI: generate UUIDs when `crypto.randomUUID()` is unavailable over HTTP — thanks @ratulsarna.
- Control UI: stream live tool output cards in Chat (agent events include sessionKey).
- Chat UI: render assistant `<think>`/`<final>` markup as italic thinking text in history + streaming instead of showing raw tags.
- Agent: add soft block-stream chunking (8001200 chars default) with paragraph/newline preference.
- Agent: route embedded run lifecycle logs through subsystem console formatting and reduce log noise.
- Agent tools: scope the Discord tool to Discord surface runs.
- Agent tools: format verbose tool summaries without brackets, with unique emojis and `tool: detail` style.
- Agent tools: emit verbose tool summaries at tool start (no debounce).
- Gateway: split server helpers/tests into hooks/session-utils/ws-log/net modules for better isolation; add unit coverage for hooks/session utils/ws log.
- Gateway: extract WS method handling + HTTP/provider/constant helpers to shrink server wiring and improve testability.
- Gateway: prevent deleting the main session and abort active runs before deleting other sessions.
- Onboarding: fix Control UI basePath usage when showing/opening gateway URLs.
- Onboarding: clarify provider requirements (WhatsApp/Signal phone numbers, iMessage Apple ID guidance) in the provider picker.
- macOS Connections: move to sidebar + detail layout with structured sections and header actions.
- macOS onboarding: increase window height so the permissions page fits without scrolling.
- Thinking: default to low for reasoning-capable models when no /think or config default is set.
- Logging: decouple file log levels from console verbosity; verbose-only details are captured when `logging.level` is debug/trace.
- Build: fix regex literal in tool-meta path detection (watch build error).
- Build: require AVX2 Bun for x86_64 relay packaging (reject baseline builds).
- Build: drop stale ClawdisCLI product from macOS build-and-run script.
- Auto-reply: add run-level telemetry + typing TTL guardrails to diagnose stuck replies.
- WhatsApp: honor per-group mention gating overrides when group ids are stored as session keys.
- Dependencies: bump pi-mono packages to 0.32.3.
### Docs
- Skills: add Sheets/Docs examples to gog skill (#128) — thanks @mbelinky.
- Skills: clarify bear-notes token + callback usage (#120) — thanks @tylerwince.
- Skills: document Discord `sendMessage` media attachments and `to` format clarification.
- Skills: expand peekaboo skill examples + common parameters.
- Skills: add tmux skill + interactive coding guidance in coding-agent.
- Gateway: document port configuration + multi-instance isolation.
- Gateway: document config hot reload + reload matrix.
- Onboarding/Config: add protocol notes for wizard + schema RPC.
- Queue: clarify steer-backlog behavior with inline commands and update examples for streaming surfaces.
- Sandbox: document per-session agent sandbox setup, browser image, and Docker build.
- macOS: clarify menu bar uses sessionKey from agent events.
- Sessions: document agent-to-agent reply loop (`REPLY_SKIP`) and announce step (`ANNOUNCE_SKIP`).
- Skills: clarify wacli third-party messaging scope and JID format examples.
## 2.0.0-beta5 — 2026-01-03
### Fixed
- Media: preserve GIF animation when uploading to Discord/other providers (skip JPEG optimization for image/gif).
- Agent runtime: update pi-mono dependencies to 0.31.1 (agent-core split).
- Dependencies: bump to latest compatible versions (TypeBox, grammY, Zod, Rolldown, oxlint-tsgolint).
- Tests: cover read tool image metadata + text output.
- Tests: add queue mode coverage (collect/followup + directive parsing).
### Breaking
- Skills config schema moved under `skills.*`:
- `skillsLoad.extraDirs``skills.load.extraDirs`
- `skillsInstall.*``skills.install.*`
- per-skill config map moved to `skills.entries` (e.g. `skills.peekaboo.enabled``skills.entries.peekaboo.enabled`)
- new optional bundled allowlist: `skills.allowBundled` (only affects bundled skills)
- Sessions: group keys now use `surface:group:<id>` / `surface:channel:<id>`; legacy `group:*` keys migrate on next message; `groupdm` keys are no longer recognized.
- Discord: remove legacy `discord.allowFrom`, `discord.guildAllowFrom`, and `discord.requireMention`; use `discord.dm` + `discord.guilds`.
- Providers: Discord/Telegram no longer auto-start from env tokens alone; add `discord: { enabled: true }` / `telegram: { enabled: true }` to your config when using `DISCORD_BOT_TOKEN` / `TELEGRAM_BOT_TOKEN`.
- Config: remove `routing.allowFrom`; use `whatsapp.allowFrom` instead (run `clawdis doctor` to migrate).
- Config: remove `routing.groupChat.requireMention` + `telegram.requireMention`; use `whatsapp.groups`, `imessage.groups`, and `telegram.groups` defaults instead (run `clawdis doctor` to migrate).
### Features
- Discord: expand `discord` tool actions (reactions, stickers, polls, threads, search, moderation gates) (#115) — thanks @thewilloftheshadow.
- Discord/Telegram: add reply tags (`[[reply_to_current]]`, `[[reply_to:<id>]]`) with per-provider `replyToMode` (off|first|all) for native threaded replies.
- Talk mode: continuous speech conversations (macOS/iOS/Android) with ElevenLabs TTS, reply directives, and optional interrupt-on-speech.
- Auto-reply: expand queue modes (steer/followup/collect/steer-backlog) with debounce/cap/drop options and followup backlog handling.
- UI: add optional `ui.seamColor` accent to tint the Talk Mode side bubble (macOS/iOS/Android).
- Nix mode: opt-in declarative config + read-only settings UI when `CLAWDIS_NIX_MODE=1` (thanks @joshp123 for the persistence — earned my trust; I'll merge these going forward).
- CLI: add Google Antigravity OAuth auth option for Claude Opus 4.5/Gemini 3 (#88) — thanks @mukhtharcm.
- Agent runtime: accept legacy `Z_AI_API_KEY` for Z.AI provider auth (maps to `ZAI_API_KEY`).
- Groups: add per-group mention gating defaults/overrides for Telegram/WhatsApp/iMessage via `*.groups` with `"*"` defaults; Discord now supports `discord.guilds."*"` as a default.
- Discord: add user-installed slash command handling with per-user sessions and auto-registration (#94) — thanks @thewilloftheshadow.
- Discord: add DM enable/allowlist plus guild channel/user/guild allowlists with id/name matching.
- Signal: add `signal-cli` JSON-RPC support for send/receive via the Signal provider.
- iMessage: add imsg JSON-RPC integration (stdio), chat_id routing, and group chat support.
- Chat UI: add recent-session dropdown switcher (main first) in macOS/iOS/Android + Control UI.
- UI: add Discord/Signal/iMessage connection panels in macOS + Control UI (thanks @thewilloftheshadow).
- Discord: allow agent-triggered reactions via `clawdis_discord` when enabled, and surface message ids in context.
- Discord: revamp guild routing config with per-guild/channel rules and slugged display names; add optional group DM support (default off).
- Discord: remove legacy guild/channel ignore lists in favor of per-guild allowlists (and proposed per-guild ignore lists).
- Skills: add Trello skill for board/list/card management (thanks @clawd).
- Docker: add containerized gateway/CLI setup via Dockerfile, compose, and setup script (thanks @dan-dr).
- Tests: add a Z.AI live test gate for smoke validation when keys are present.
- macOS Debug: add app log verbosity and rolling file log toggle for swift-log-backed app logs.
- CLI: add onboarding wizard (gateway + workspace + skills) with daemon installers and Anthropic/Minimax setup paths.
- CLI: add ASCII banner header to wizard entry points.
- CLI: add `configure`, `doctor`, and `update` wizards for ongoing setup, health checks, and modernization.
- CLI: add Signal CLI auto-install from GitHub releases in the wizard and persist wizard run metadata in config.
- CLI: add remote gateway client config (gateway.remote.*) with Bonjour-assisted discovery.
- CLI: enhance `clawdis tui` with model/session pickers, tool cards, and slash commands (local or remote).
- Gateway: allow `sessions.patch` to set per-session model overrides (used by the TUI `/model` flow).
- Skills: allow `bun` as a node manager for skill installs.
- Skills: add `things-mac` (Things 3 CLI) for read/search plus add/update via URL scheme.
- Skills: add Apple Notes + Reminders skills via memo CLI (thanks @tylerwince).
- Tests: add a Docker-based onboarding E2E harness.
- Tests: harden wizard E2E flows for reset, providers, skills, and remote non-interactive runs.
- Browser tools: add remote CDP URL support, Linux launcher options (`executablePath`, `noSandbox`), and surface `cdpUrl` in status.
- Skills: add tmux-first coding-agent skill + `requires.anyBins` gate for multi-CLI setup (thanks @sreekaransrinath).
### Fixes
- macOS codesign: make ad-hoc signing opt-in with loud warnings and document TCC permission fragility — thanks @mcinteerj.
- Gog calendar: format date ranges as RFC 3339 with timezone to satisfy Google Calendar API (thanks @jayhickey).
- macOS onboarding: add scrollable page gutter for overflowing content (#105) — thanks @thewilloftheshadow.
- Chat UI: keep the chat scrolled to the latest message after switching sessions.
- Chat UI: show rich session display names in Web Chat + SwiftUI + Android.
- Auto-reply: stream completed reply blocks as soon as they finish (configurable default + break); skip empty tool-only blocks unless verbose.
- Discord: avoid duplicate sends when block streaming is enabled (race with typing hook).
- Providers: make outbound text chunk limits configurable via `*.textChunkLimit` (defaults remain 4000/Discord 2000).
- CLI onboarding: persist gateway token in config so local CLI auth works; recommend auth Off unless you need multi-machine access.
- Control UI: accept a `?token=` URL param to auto-fill Gateway auth; onboarding now opens the dashboard with token auth when configured.
- Agent prompt: remove hardcoded user name in system prompt example.
- Chat UI: add extra top padding before the first message bubble in Web Chat (macOS/iOS/Android).
- Control UI: refine Web Chat session selector styling (chevron spacing + background).
- WebChat: stream live updates for sessions even when runs start outside the chat UI.
- Gateway CLI: read `CLAWDIS_GATEWAY_PASSWORD` from environment in `callGateway()` — allows `doctor`/`health` commands to auth without explicit `--password` flag.
- Gateway: add password auth support for remote gateway connections (thanks @jeffersonwarrior).
- Auto-reply: strip stray leading/trailing `HEARTBEAT_OK` from normal replies; drop short (≤ 30 chars) heartbeat acks.
- WhatsApp auto-reply: default to self-only when no config is present.
- Logging: trim provider prefix duplication in Discord/Signal/Telegram runtime log lines.
- Logging/Signal: treat signal-cli "Failed …" lines as errors in gateway logs.
- Discord: include recent guild context when replying to mentions and add `discord.historyLimit` to tune how many messages are captured.
- Discord: include author tag + id in group context `[from:]` lines for ping-ready replies (thanks @thewilloftheshadow).
- Discord: include replied-to message context when a Discord message references another message (thanks @thewilloftheshadow).
- Discord: preserve newlines when stripping reply tags from agent output.
- Gateway: fix TypeScript build by aligning hook mapping `channel` types and removing a dead Group DM branch in Discord monitor.
- Skills: switch imsg installer to brew tap formula.
- Skills: gate macOS-only skills by OS and surface block reasons in the Skills UI.
- Onboarding: show skill descriptions in the macOS setup flow and surface clearer Gateway/skills error messages.
- Onboarding: auto-verify Claude OAuth tokens, show “verified” when detected working, and avoid re-auth prompts unless verification fails.
- CLI onboarding: include exit code + a useful one-line summary when skill dependency installs fail.
- CLI onboarding: explain Tailscale exposure options (Off/Serve/Funnel) and colorize provider status (linked/configured/needs setup).
- CLI onboarding: add provider primers (WhatsApp/Telegram/Discord/Signal) incl. Discord bot token setup steps.
- CLI onboarding: allow skipping the “install missing skill dependencies” selection without canceling the wizard.
- CLI onboarding: always prompt for WhatsApp `whatsapp.allowFrom` and print (optionally open) the Control UI URL when done.
- CLI onboarding: detect gateway reachability and annotate Local/Remote choices (helps pick the right mode).
- macOS settings: colorize provider status subtitles to distinguish healthy vs degraded states.
- macOS: keep config writes on the main actor to satisfy Swift concurrency rules.
- macOS menu: show multi-line gateway error details, add an always-visible gateway row, avoid duplicate gateway status rows, suppress transient `cancelled` device refresh errors, and auto-recover the control channel on disconnect.
- macOS menu: show session last-used timestamps in the list and add recent-message previews in session submenus.
- macOS menu: tighten session row padding and time out session preview loading with cached fallback.
- macOS: log health refresh failures and recovery to make gateway issues easier to diagnose.
- macOS codesign: skip hardened runtime for ad-hoc signing and avoid empty options args (#70) — thanks @petter-b
- macOS codesign: include camera entitlement so permission prompts work in the menu bar app.
- Agent tools: bash tool supports real TTY via `stdinMode: "pty"` with node-pty, warning + fallback on load/start failure.
- Agent tools: map `camera.snap` JPEG payloads to `image/jpeg` to avoid MIME mismatch errors.
- Tests: cover `camera.snap` MIME mapping to prevent image/png vs image/jpeg mismatches.
- macOS camera: wait for exposure/white balance to settle before capturing a snap to avoid dark images.
- Camera snap: add `delayMs` parameter (default 2000ms on macOS) to improve exposure reliability.
- Camera: add `camera.list` and optional `deviceId` selection for snaps/clips.
- Tests: cover camera device selection params in CLI + agent tools.
- macOS packaging: move rpath config into swift build for reliability (#69) — thanks @petter-b
- macOS: prioritize main bundle for device resources to prevent crash (#73) — thanks @petter-b
- macOS remote: route settings through gateway config and avoid local config reads in remote mode.
- Telegram: align token resolution for cron/agent/CLI sends (env/config/tokenFile) to prevent isolated delivery failures (#76).
- Telegram: honor per-group mention gating defaults/overrides via `telegram.groups` and `"*"` defaults (thanks @joshp123).
- Chat UI: clear composer input immediately and allow clear while editing to prevent duplicate sends (#72) — thanks @hrdwdmrbl
- Restart: use systemd on Linux (and report actual restart method) instead of always launchctl.
- Gateway relay: detect Bun binaries via execPath to resolve packaged assets on macOS.
- Cron: prevent `every` schedules without an anchor from firing in a tight loop (thanks @jamesgroat).
- Docs: add manual OAuth setup for remote/headless deployments (#67) — thanks @wstock
- Docs/agent tools: clarify that browser `wait` should be avoided by default and used only in exceptional cases.
- Docs: clarify self-chat mode and group mention gating config (#111) — thanks @rafaelreis-r.
- Browser tools: `upload` supports auto-click refs, direct `inputRef`/`element` file inputs, and emits input/change after `setFiles` so JS-heavy sites pick up attachments.
- Browser tools: harden CDP readiness (HTTP + WS), retry CDP connects, and auto-restart the clawd browser when the socket handshake stalls.
- Browser CLI: add `clawdis browser reset-profile` to move the clawd profile to Trash when it gets wedged.
- Signal: fix daemon startup race (wait for `/api/v1/check`) and normalize JSON-RPC `version` probe parsing.
- Docs/Signal: clarify bot-number vs personal-account setup (self-chat loop protection) and add a quickstart config snippet.
- Docs: refresh the CLI wizard guide and highlight onboarding in the README.
- CLI: tighten onboarding prompt typing to keep bun builds green.
- macOS: Voice Wake now fully tears down the Speech pipeline when disabled (cancel pending restarts, drop stale callbacks) to avoid high CPU in the background.
- macOS menu: add a Talk Mode action alongside the Open Dashboard/Chat/Canvas entries.
- macOS Debug: hide “Restart Gateway” when the app wont start a local gateway (remote mode / attach-only).
- macOS Debug: add an icon for the App Logging submenu.
- macOS Talk Mode: orb overlay refresh, ElevenLabs request logging, API key status in settings, and auto-select first voice when none is configured.
- macOS Talk Mode: add hard timeout around ElevenLabs TTS synthesis to avoid getting stuck “speaking” forever on hung requests.
- macOS Talk Mode: avoid stuck playback when the audio player never starts (fail-fast + watchdog).
- macOS Talk Mode: fix audio stop ordering so disabling Talk Mode always stops in-flight playback.
- macOS Talk Mode: throttle audio-level updates (avoid per-buffer task creation) to reduce CPU/task churn.
- macOS Talk Mode: increase overlay window size so wave rings dont clip; close button is hover-only and closer to the orb.
- WebChat: preserve chat run ordering per session so concurrent runs dont strand the typing indicator.
- Talk Mode: fall back to system TTS when ElevenLabs is unavailable, returns non-audio, or playback fails (macOS/iOS/Android).
- Talk Mode: stream PCM on macOS/iOS for lower latency (incremental playback); Android continues MP3 streaming.
- Talk Mode: validate ElevenLabs v3 stability and latency tier directives before sending requests.
- iOS/Android Talk Mode: auto-select the first ElevenLabs voice when none is configured.
- ElevenLabs: add retry/backoff for 429/5xx and include content-type in errors for debugging.
- Talk Mode: align to the gateways main session key and fall back to history polling when chat events drop (prevents stuck “thinking” / missing messages).
- Talk Mode: treat history timestamps as seconds or milliseconds to avoid stale assistant picks (macOS/iOS/Android).
- Chat UI: clear streaming/tool bubbles when external runs finish, preventing duplicate assistant bubbles.
- Chat UI: user bubbles use `ui.seamColor` (fallback to a calmer default blue).
- Android Chat UI: use `onPrimary` for user bubble text to preserve contrast (thanks @Syhids).
- Control UI: sync sidebar navigation with the URL for deep-linking, and auto-scroll chat to the latest message.
- Control UI: disable Web Chat + Talk when no iOS/Android node is connected; refreshed Web Chat styling and keyboard send.
- Control UI: keep chat pinned to the latest message while typing/sending and restore drafts on send failures.
- Control UI: soften chat bubble text opacity for calmer readability.
- macOS Web Chat: improve empty/error states, focus message field on open, keep pill/send inside the input field, and make the composer pill edge-to-edge with square top corners.
- macOS: bundle Control UI assets into the app relay so the packaged app can serve them (thanks @mbelinky).
- Talk Mode: wait for chat history to surface the assistant reply before starting TTS (macOS/iOS/Android).
- iOS Talk Mode: fix chat completion wait to time out even if no events arrive (prevents “Thinking…” hangs).
- iOS Talk Mode: keep recognition running during playback to support interrupt-on-speech.
- iOS Talk Mode: preserve directive voice/model overrides across config reloads and add ElevenLabs request timeouts.
- iOS/Android Talk Mode: explicitly `chat.subscribe` when Talk Mode is active, so completion events arrive even if the Chat UI isnt open.
- Chat UI: refresh history when another client finishes a run in the same session, so Talk Mode + Voice Wake transcripts appear consistently.
- Gateway: `voice.transcript` now also maps agent bus output to `chat` events, ensuring chat UIs refresh for voice-triggered runs.
- Gateway: auto-migrate legacy config on startup (non-Nix); Nix mode hard-fails with a clear error when legacy keys are present.
- iOS/Android: show a centered Talk Mode orb overlay while Talk Mode is enabled.
- Gateway config: inject `talk.apiKey` from `ELEVENLABS_API_KEY`/shell profile so nodes can fetch it on demand.
- Canvas A2UI: tag requests with `platform=android|ios|macos` and boost Android canvas background contrast.
- iOS/Android nodes: enable scrolling for loaded web pages in the Canvas WebView (default scaffold stays touch-first).
- macOS menu: device list now uses `node.list` (devices only; no agent/tool presence entries).
- macOS menu: device list now shows connected nodes only.
- macOS menu: device rows now pack platform/version on the first line, and command lists wrap in submenus.
- macOS menu: split device platform/version across first and second rows for better fit.
- macOS Canvas: show remote control status in the debug overlay and log A2UI auto-nav decisions.
- Canvas A2UI: polish the debug status HUD styling.
- iOS node: fix ReplayKit screen recording crash caused by queue isolation assertions during capture.
- iOS Talk Mode: avoid audio tap queue assertions when starting recognition.
- macOS: use $HOME/Library/pnpm for SSH PATH exports (thanks @mbelinky).
- macOS remote: harden SSH tunnel recovery/logging, honor `gateway.remote.url` port when forwarding, clarify gateway disconnect status, and add Debug menu tunnel reset.
- iOS/Android nodes: bridge auto-connect refreshes stale tokens and settings now show richer bridge/device details.
- macOS: bundle device model resources to prevent Instances crashes (thanks @mbelinky).
- iOS/Android nodes: status pill now surfaces camera activity instead of overlay toasts.
- iOS/Android/macOS nodes: camera snaps recompress to keep base64 payloads under 5 MB.
- iOS/Android nodes: status pill now surfaces pairing, screen recording, voice wake, and foreground-required states.
- iOS/Android nodes: avoid duplicating “Gateway reconnecting…” when the bridge is already connecting.
- iOS/Android nodes: Talk Mode now lives on a side bubble (with an iOS toggle to hide it), and Android settings no longer show the Talk Mode switch.
- macOS menu: top status line now shows pending node pairing approvals (incl. repairs).
- CLI: avoid spurious gateway close errors after successful request/response cycles.
- Agent runtime: clamp tool-result images to the 5MB Anthropic limit to avoid hard request rejections.
- Agent runtime: write v2 session headers so Pi session branching stays in the Clawdis sessions dir.
- Tests: add Swift Testing coverage for camera errors and Kotest coverage for Android bridge endpoints.
## 2.0.0-beta4 — 2025-12-27
### Fixes
- Package contents: include Discord/hooks build outputs in the npm tarball to avoid missing module errors.
- Heartbeat replies now drop any output containing `HEARTBEAT_OK`, preventing stray emoji/text from being delivered.
- macOS menu now refreshes the control channel after the gateway starts and shows “Connecting to gateway…” while the gateway is coming up.
- macOS local mode now waits for the gateway to be ready before configuring the control channel, avoiding false “no connection” flashes.
- WhatsApp watchdog now forces a reconnect even if the socket close event stalls (force-close to unblock reconnect loop).
- Gateway presence now reports macOS product version (via `sw_vers`) instead of Darwin kernel version.
## 2.0.0-beta3 — 2025-12-27
### Highlights
- First-class Clawdis tools (browser, canvas, nodes, cron) replace the old `clawdis-*` skills; tool schemas are now injected directly into the agent runtime.
- Per-session model selection + custom model providers: `models.providers` merges into `~/.clawdis/agent/models.json` (merge/replace modes) for LiteLLM, local OpenAI-compatible servers, Anthropic proxies, etc.
- Group chat activation modes: per-group `/activation mention|always` command with status visibility.
- Discord bot transport for DMs and guild text channels, with allowlists + mention gating.
- Gateway webhooks: external `wake` and isolated `agent` hooks with dedicated token auth.
- Hook mappings + Gmail Pub/Sub helper (`clawdis hooks gmail setup/run`) with auto-renew + Tailscale Funnel support.
- Command queue modes + per-session overrides (`/queue ...`) and new `agent.maxConcurrent` cap for safe parallelism across sessions.
- Background bash tasks: `bash` auto-yields after 20s (or on demand) with a `process` tool to list/poll/log/write/kill sessions.
- Gateway in-process restart: `gateway` tool action triggers a SIGUSR1 restart without needing a supervisor.
### Breaking
- Config refactor: `inbound.*` removed; use top-level `routing` (allowlists + group rules + transcription), `messages` (prefixes/timestamps), and `session` (scoping/store/mainKey). No legacy keys read.
- Heartbeat config moved to `agent.heartbeat`: set `every: "30m"` (duration string) and optional `model`. `agent.heartbeatMinutes` is removed, and heartbeats are disabled unless `agent.heartbeat.every` is set.
- Heartbeats now run via the gateway runner (main session) and deliver to the last used channel by default. WhatsApp reply-heartbeat behavior is removed; use `agent.heartbeat.target`/`to` (or `target: "none"`) to control delivery.
- Browser `act` no longer accepts CSS `selector`; use `snapshot` refs (default `ai`) or `evaluate` as an escape hatch.
### Fixes
- Heartbeat replies now strip repeated `HEARTBEAT_OK` tails to avoid accidental “OK OK” spam.
- Heartbeat delivery now uses the last non-empty payload, preventing tool preambles from swallowing the final reply.
- Heartbeats now skip WhatsApp delivery when the web provider is inactive or unlinked (instead of logging “no active gateway listener”).
- Heartbeat failure logs now include the error reason instead of `[object Object]`.
- Duration strings now accept `h` (hours) where durations are parsed (e.g., heartbeat intervals).
- WhatsApp inbound now normalizes more wrapper types so quoted reply bodies are extracted reliably.
- WhatsApp send now preserves existing JIDs (including group `@g.us`) instead of coercing to `@s.whatsapp.net`. (Thanks @arun-8687.)
- Telegram/WhatsApp: reply context stays in `Body`/`ReplyTo*`, but outbound replies no longer thread to the original message. (Thanks @joshp123 for the PR and follow-up question.)
- Suppressed libsignal session cleanup spam from console logs unless verbose mode is enabled.
- WhatsApp web creds persistence hardened; credentials are restored before auth checks and QR login auto-restarts if it stalls.
- Group chats now honor `routing.groupChat.requireMention=false` as the default activation when no per-group override exists.
- Gateway auth no longer supports PAM/system mode; use token or shared password.
- Tailscale Funnel now requires password auth (no token-only public exposure).
- Group `/new` resets now work with @mentions so activation guidance appears on fresh sessions.
- Group chat activation context is now injected into the system prompt at session start (and after activation changes), including /new greetings.
- Typing indicators now start only once a reply payload is produced (no "thinking" typing for silent runs).
- WhatsApp group typing now starts immediately only when the bot is mentioned; otherwise it waits until real output exists.
- Streamed `<think>` segments are stripped before partial replies are emitted.
- System prompt now tags allowlisted owner numbers as the user identity to avoid mistaken “friend” assumptions.
- LM Studio/Ollama replies now require <final> tags; streaming ignores content until <final> begins.
- LM Studio responses API: tools payloads no longer include `strict: null`, and LM Studio no longer gets forced `<think>/<final>` tags.
- Identity emoji no longer auto-prefixes replies (set `messages.responsePrefix` explicitly if desired).
- Model switches now enqueue a system event so the next run knows the active model.
- `/model status` now lists available models (same as `/model`).
- `process log` pagination is now line-based (omit `offset` to grab the last N lines).
- macOS WebChat: assistant bubbles now update correctly when toggling light/dark mode.
- macOS: avoid spawning a duplicate gateway process when an external listener already exists.
- Node bridge: when binding to a non-loopback host (e.g. Tailnet IP), also listens on `127.0.0.1` for local connections (without creating duplicate loopback listeners for `0.0.0.0`/`127.0.0.1` binds).
- UI perf: pause repeat animations when scenes are inactive (typing dots, onboarding glow, iOS status pulse), throttle voice overlay level updates, and reduce overlay focus churn.
- Canvas defaults/A2UI auto-nav aligned; debug status overlay centered; redundant await removed in `CanvasManager`.
- Gateway launchd loop fixed by removing redundant `kickstart -k`.
- CLI now hints when Peekaboo is unauthorized.
- WhatsApp web inbox listeners now clean up on close to avoid duplicate handlers.
- Gateway startup now brings up browser control before external providers; WhatsApp/Telegram/Discord auto-start can be disabled with `web.enabled`, `telegram.enabled`, or `discord.enabled`.
### Providers & Routing
- New Discord provider for DMs + guild text channels with allowlists and mention-gated replies by default.
- `routing.queue` now controls queue vs interrupt behavior globally + per surface (defaults: WhatsApp/Telegram interrupt, Discord/WebChat queue).
- `/queue <mode>` supports one-shot or per-session overrides; `/queue reset|default` clears overrides.
- `agent.maxConcurrent` caps global parallel runs while keeping per-session serialization.
### macOS app
- Update-ready state surfaced in the menu; menu sections regrouped with session submenus.
- Menu bar now shows a dedicated Nodes section under Context with inline rows, overflow submenu, and iconized actions.
- Nodes now expose consistent inline details with per-node submenus for quick copy of key fields.
- Node rows now show compact app versions (build numbers moved to submenus) and offer SSH launch from Bonjour when available.
- Menu actions are grouped below toggles; Open Canvas hides when disabled and Voice Wake now anchors the mic picker.
- Connections now include Discord provider status + configuration UI.
- Menu bar gains an Allow Camera toggle alongside Canvas.
- Session list polish: sleeping/disconnected/error states, usage bar restored, padding + bar sizing tuned, syncing menu removed, header hidden when disconnected.
- Chat UI polish: tool call cards + merged tool results, glass background, tighter composer spacing, visual effect host tweaks.
- OAuth storage moved; legacy session syncing metadata removed.
- Remote SSH tunnels now get health checks; Debug → Ports highlights unhealthy tunnels and offers Reset SSH tunnel.
- Menu bar session/node sections no longer reflow while open, keeping hover highlights aligned.
- Menu hover highlights now span the full width (including submenu arrows).
- Menu session rows now refresh while open without width changes (no more stuck “Loading sessions…”).
- Menu width no longer grows on hover when moving the mouse across rows.
- Context usage bars now have higher contrast in light mode.
- macOS node timeouts now share a single async timeout helper for consistent behavior.
- WebChat window defaults tightened (narrower width, edge-to-edge layout) and the SwiftUI tag removed from the title.
### Nodes & Canvas
- Debug status overlay gated and toggleable on macOS/iOS/Android nodes.
- Gateway now derives the canvas host URL via a shared helper for bridge + WS handshakes (avoids loopback pitfalls).
- `canvas a2ui push` validates JSONL with line errors, rejects v0.9 payloads, and supports `--text` quick renders.
- `nodes rename` lets you override paired node display names without editing JSON.
- Android scaffold asset cleanup; iOS canvas/voice wake adjustments.
### Logging & Observability
- New subsystem console formatter with color modes, shortened prefixes, and TTY detection; browser/gateway logs route through the subsystem logger.
- WhatsApp console output streamlined; chalk/tslog typing fixes.
### Web UI
- Chat is now the dashboard landing view; health status simplified; initial scroll animation removed.
### Build, Dev, Docs
- Notarization flow added for macOS release artifacts; packaging scripts updated.
- macOS signing auto-selects Developer ID → Apple Distribution → Apple Development; no ad-hoc fallback.
- Added type-aware oxlint; docs list resolves from cwd; formatting/lint cleanup and dependency bumps (Peekaboo).
- Docs refreshed for tools, custom model providers, Discord, queue/routing, group activation commands, logging, restart semantics, release notes, GitHub pages CTAs, and npm pitfalls.
- `pnpm build` now skips A2UI bundling for faster builds (run `pnpm canvas:a2ui:bundle` when needed).
### Tests
- Coverage added for models config merging, WhatsApp reply context, QR login flows, auto-reply behavior, and gateway SIGTERM timeouts.
- Added gateway webhook coverage (auth, validation, and summary posting).
- Vitest now isolates HOME/XDG config roots so tests never touch a real `~/.clawdis` install.
## 2.0.0-beta2 — 2025-12-21
@@ -45,7 +428,7 @@ First Clawdis release post rebrand. This is a semver-major because we dropped le
### Breaking
- Renamed to **Clawdis**: defaults now live under `~/.clawdis` (sessions in `~/.clawdis/sessions/`, IPC at `~/.clawdis/clawdis.sock`, logs in `/tmp/clawdis`). Launchd labels and config filenames follow the new name; legacy stores are copied forward on first run.
- Pi only: `inbound.reply.agent.kind` accepts only `"pi"`, and the agent CLI/CLI flags for Claude/Codex/Gemini were removed. The Pi CLI runs in RPC mode with a persistent worker.
- Pi only: only the embedded Pi runtime remains, and the agent CLI/CLI flags for Claude/Codex/Gemini were removed. The Pi CLI runs in RPC mode with a persistent worker.
- WhatsApp Web is the only transport; Twilio support and related CLI flags/tests were removed.
- Direct chats now collapse into a single `main` session by default (no config needed); groups stay isolated as `group:<jid>`.
- Gateway is now a loopback-only WebSocket daemon (`ws://127.0.0.1:18789`) that owns all providers/state; clients (CLI, WebChat, macOS app, nodes) connect to it. Start it explicitly (`clawdis gateway …`) or via Clawdis.app; helper subcommands no longer auto-spawn a gateway.
@@ -93,7 +476,7 @@ First Clawdis release post rebrand. This is a semver-major because we dropped le
## 1.5.0 — 2025-12-05
### Breaking
- Dropped all non-Pi agents (Claude, Codex, Gemini, Opencode); `inbound.reply.agent.kind` now only accepts `"pi"` and related CLI helpers have been removed.
- Dropped all non-Pi agents (Claude, Codex, Gemini, Opencode); only the embedded Pi runtime remains and related CLI helpers have been removed.
- Removed Twilio support and all related commands/options (webhook/up/provider flags/wait-poll); CLAWDIS is Baileys Web-only.
### Changes
@@ -120,7 +503,7 @@ First Clawdis release post rebrand. This is a semver-major because we dropped le
## 1.4.0 — 2025-12-03
### Highlights
- **Thinking directives & state:** `/t|/think|/thinking <level>` (aliases off|minimal|low|medium|high|max/highest). Inline applies to that message; directive-only message pins the level for the session; `/think:off` clears. Resolution: inline > session override > `inbound.reply.thinkingDefault` > off. Pi gets `--thinking <level>` (except off); other agents append cue words (`think``think hard``think harder``ultrathink`). Heartbeat probe uses `HEARTBEAT /think:high`.
- **Thinking directives & state:** `/t|/think|/thinking <level>` (aliases off|minimal|low|medium|high|max/highest). Inline applies to that message; directive-only message pins the level for the session; `/think:off` clears. Resolution: inline > session override > `agent.thinkingDefault` > off. Pi gets `--thinking <level>` (except off); other agents append cue words (`think``think hard``think harder``ultrathink`). Heartbeat probe uses `HEARTBEAT /think:high`.
- **Group chats (web provider):** Clawdis now fully supports WhatsApp groups: mention-gated triggers (including image-only @ mentions), recent group history injection, per-group sessions, sender attribution, and a first-turn primer with group subject/member roster; heartbeats are skipped for groups.
- **Group session primer:** The first turn of a group session now tells the agent it is in a WhatsApp group and lists known members/subject so it can address the right speaker.
- **Media failures are surfaced:** When a web auto-reply media fetch/send fails (e.g., HTTP 404), we now append a warning to the fallback text so you know the attachment was skipped.
@@ -162,7 +545,7 @@ First Clawdis release post rebrand. This is a semver-major because we dropped le
## 1.3.0 — 2025-12-02
### Highlights
- **Pluggable agents (Claude, Pi, Codex, Opencode):** `inbound.reply.agent` selects CLI/parser; per-agent argv builders and NDJSON parsers enable swapping without template changes.
- **Pluggable agents (Claude, Pi, Codex, Opencode):** agent selection via config/CLI plus per-agent argv builders and NDJSON parsers enable swapping without template changes.
- **Safety stop words:** `stop|esc|abort|wait|exit` immediately reply “Agent was aborted.” and mark the session so the next prompt is prefixed with an abort reminder.
- **Agent session reliability:** Only Claude returns a stable `session_id`; others may reset between runs.
@@ -179,7 +562,7 @@ First Clawdis release post rebrand. This is a semver-major because we dropped le
- Batched inbound messages with timestamps; typing indicator after sends.
- Watchdog restarts WhatsApp after long inactivity; heartbeat logging includes minutes since last message.
- Early `allowFrom` filtering before decryption.
- Same-phone mode with echo detection and optional `inbound.samePhoneMarker`.
- Same-phone mode with echo detection and optional message prefix marker.
## 1.2.2 — 2025-11-28
@@ -209,10 +592,10 @@ First Clawdis release post rebrand. This is a semver-major because we dropped le
## 1.1.0 — 2025-11-26
### Changes
- Web auto-replies resize/recompress media and honor `inbound.reply.mediaMaxMb`.
- Web auto-replies resize/recompress media and honor `agent.mediaMaxMb`.
- Detect media kind, enforce provider caps (images ≤6MB, audio/video ≤16MB, docs ≤100MB).
- `session.sendSystemOnce` and optional `sessionIntro`.
- Typing indicator refresh during commands; configurable via `inbound.reply.typingIntervalSeconds`.
- Typing indicator refresh during commands; configurable via `agent.typingIntervalSeconds`.
- Optional audio transcription via external CLI.
- Command replies return structured payload/meta; respect `mediaMaxMb`; log Claude metadata; include `cwd` in timeout messages.
- Web provider refactor; logout command; web-only gateway start helper.

42
CONTRIBUTING.md Normal file
View File

@@ -0,0 +1,42 @@
# Contributing to Clawdis
Welcome to the lobster tank! 🦞
## Quick Links
- **GitHub:** https://github.com/steipete/clawdis
- **Discord:** https://discord.gg/qkhbAGHRBT
- **X/Twitter:** [@steipete](https://x.com/steipete) / [@clawdbot](https://x.com/clawdbot)
## Maintainers
- **Peter Steinberger** - Benevolent Dictator
- GitHub: [@steipete](https://github.com/steipete) · X: [@steipete](https://x.com/steipete)
- **Shadow** - Discord subsystem
- GitHub: [@4shadowed](https://github.com/4shadowed) · X: [@4shad0wed](https://x.com/4shad0wed)
- **Jos** - Telegram, API, Nix mode
- GitHub: [@joshp123](https://github.com/joshp123) · X: [@jjpcodes](https://x.com/jjpcodes)
## How to Contribute
1. **Bugs & small fixes** → Open a PR!
2. **New features / architecture** → Start a [GitHub Discussion](https://github.com/steipete/clawdis/discussions) or ask in Discord first
3. **Questions** → Discord #setup-help
## Before You PR
- Test locally with your Clawdis instance
- Run linter: `npm run lint`
- Keep PRs focused (one thing per PR)
- Describe what & why
## AI/Vibe-Coded PRs Welcome! 🤖
Built with Codex, Claude, or other AI tools? **Awesome - just mark it!**
Please include in your PR:
- [ ] Mark as AI-assisted in the PR title or description
- [ ] Note the degree of testing (untested / lightly tested / fully tested)
- [ ] Include prompts or session logs if possible (super helpful!)
- [ ] Confirm you understand what the code does
AI PRs are first-class citizens here. We just want transparency so reviewers know what to look for.

16
Dockerfile Normal file
View File

@@ -0,0 +1,16 @@
FROM node:22-bookworm
RUN corepack enable
WORKDIR /app
COPY . .
RUN pnpm install --frozen-lockfile
RUN pnpm build
RUN pnpm ui:install
RUN pnpm ui:build
ENV NODE_ENV=production
CMD ["node", "dist/index.js"]

16
Dockerfile.sandbox Normal file
View File

@@ -0,0 +1,16 @@
FROM debian:bookworm-slim
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
bash \
ca-certificates \
curl \
git \
jq \
python3 \
ripgrep \
&& rm -rf /var/lib/apt/lists/*
CMD ["sleep", "infinity"]

View File

@@ -0,0 +1,27 @@
FROM debian:bookworm-slim
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
bash \
ca-certificates \
chromium \
curl \
fonts-liberation \
fonts-noto-color-emoji \
git \
jq \
novnc \
python3 \
websockify \
x11vnc \
xvfb \
&& rm -rf /var/lib/apt/lists/*
COPY scripts/sandbox-browser-entrypoint.sh /usr/local/bin/clawdis-sandbox-browser
RUN chmod +x /usr/local/bin/clawdis-sandbox-browser
EXPOSE 9222 5900 6080
CMD ["clawdis-sandbox-browser"]

View File

@@ -11,13 +11,20 @@
<p align="center">
<a href="https://github.com/steipete/clawdis/actions/workflows/ci.yml?branch=main"><img src="https://img.shields.io/github/actions/workflow/status/steipete/clawdis/ci.yml?branch=main&style=for-the-badge" alt="CI status"></a>
<a href="https://github.com/steipete/clawdis/releases"><img src="https://img.shields.io/github/v/release/steipete/clawdis?include_prereleases&style=for-the-badge" alt="GitHub release"></a>
<a href="https://discord.gg/clawd"><img src="https://img.shields.io/discord/1456350064065904867?label=Discord&logo=discord&logoColor=white&color=5865F2&style=for-the-badge" alt="Discord"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-blue.svg?style=for-the-badge" alt="MIT License"></a>
</p>
**Clawdis** is a *personal AI assistant* you run on your own devices.
It answers you on the surfaces you already use (WhatsApp, Telegram, WebChat), can speak and listen on macOS/iOS, and can render a live Canvas you control. The Gateway is just the control plane — the product is the assistant.
It answers you on the surfaces you already use (WhatsApp, Telegram, Discord, iMessage, WebChat), can speak and listen on macOS/iOS, and can render a live Canvas you control. The Gateway is just the control plane — the product is the assistant.
If you want a private, single-user assistant that feels local, fast, and always-on, this is it.
If you want a personal, single-user assistant that feels local, fast, and always-on, this is it.
Website: https://clawd.me · Docs: [`docs/index.md`](docs/index.md) · FAQ: [`docs/faq.md`](docs/faq.md) · Wizard: [`docs/wizard.md`](docs/wizard.md) · Docker (optional): [`docs/docker.md`](docs/docker.md) · Discord: https://discord.gg/clawd
Preferred setup: run the onboarding wizard (`clawdis onboard`). It walks through gateway, workspace, providers, and skills. The CLI wizard is the recommended path and works on **macOS, Windows, and Linux**.
Using Claude Pro/Max subscription? See `docs/onboarding.md` for the Anthropic OAuth setup.
```
Your surfaces
@@ -38,12 +45,13 @@ Your surfaces
## What Clawdis does
- **Personal assistant** — one user, one identity, one memory surface.
- **Multi-surface inbox** — WhatsApp, Telegram, WebChat, macOS, iOS.
- **Multi-surface inbox** — WhatsApp, Telegram, Discord, iMessage, WebChat, macOS, iOS. Signal support via `signal-cli` (see `docs/signal.md`). iMessage uses `imsg` (see `docs/imessage.md`).
- **Voice wake + push-to-talk** — local speech recognition on macOS/iOS.
- **Canvas** — a live visual workspace you can drive from the agent.
- **Automation-ready** — browser control, media handling, and tool streaming.
- **Local-first control plane** — the Gateway owns state, everything else connects.
- **Group chats** — mention-based by default, `/activation always|mention` per group (owner-only).
- **Nix mode** — opt-in declarative config + read-only UI when `CLAWDIS_NIX_MODE=1`.
## How it works (short)
@@ -61,16 +69,22 @@ pnpm install
pnpm build
pnpm ui:build
# Recommended: run the onboarding wizard
pnpm clawdis onboard
# Link WhatsApp (stores creds in ~/.clawdis/credentials)
pnpm clawdis login
# Start the gateway
pnpm clawdis gateway --port 18789 --verbose
# Dev loop (auto-reload on TS changes)
pnpm gateway:watch
# Send a message
pnpm clawdis send --to +1234567890 --message "Hello from Clawdis"
# Talk to the assistant (optionally deliver back to WhatsApp/Telegram)
# Talk to the assistant (optionally deliver back to WhatsApp/Telegram/Discord)
pnpm clawdis agent --message "Ship checklist" --thinking high
```
@@ -95,7 +109,7 @@ Send these in WhatsApp/Telegram/WebChat (group commands are owner-only):
- **Events + snapshot**: handshake returns a snapshot (presence/health) and declares event types; runtime events include `agent`, `chat`, `presence`, `tick`, `health`, `heartbeat`, `cron`, `node.pair.*`, `voicewake.changed`, `shutdown`.
- **Idempotency & safety**: `send`/`agent`/`chat.send` require idempotency keys with a TTL cache (5 min, cap 1000) to avoid doublesends on reconnects; payload sizes are capped per connection.
- **Bridge for nodes**: optional TCP bridge (`src/infra/bridge/server.ts`) is newlinedelimited JSON frames (`hello`, pairing, RPC, `invoke`); node connect/disconnect is surfaced into presence.
- **Control UI + Canvas Host**: HTTP serves `/ui` assets (if built) and can host a livereload Canvas host for nodes (`src/canvas-host/server.ts`), injecting the A2UI postMessage bridge.
- **Control UI + Canvas Host**: HTTP serves Control UI assets (default `/`, optional base path) and can host a livereload Canvas host for nodes (`src/canvas-host/server.ts`), injecting the A2UI postMessage bridge.
### iOS app (apps/ios)
- **Discovery + pairing**: Bonjour discovery via `BridgeDiscoveryModel` (NWBrowser). `BridgeConnectionController` autoconnects using Keychain token or allows manual host/port.
@@ -133,7 +147,7 @@ Runbook: `docs/ios/connect.md`.
## Agent workspace + skills
- Workspace root: `~/clawd` (configurable via `inbound.workspace`).
- Workspace root: `~/clawd` (configurable via `agent.workspace`).
- Injected prompt files: `AGENTS.md`, `SOUL.md`, `TOOLS.md`.
- Skills: `~/clawd/skills/<skill>/SKILL.md`.
@@ -143,7 +157,7 @@ Minimal `~/.clawdis/clawdis.json`:
```json5
{
inbound: {
whatsapp: {
allowFrom: ["+1234567890"]
}
}
@@ -152,12 +166,12 @@ Minimal `~/.clawdis/clawdis.json`:
### WhatsApp
- Link the device: `pnpm clawdis login` (stores creds in `~/.clawdis/credentials`).
- Allowlist who can talk to the assistant via `inbound.allowFrom`.
- Allowlist who can talk to the assistant via `whatsapp.allowFrom`.
### Telegram
- Set `TELEGRAM_BOT_TOKEN` or `telegram.botToken` (env wins).
- Optional: set `telegram.requireMention`, `telegram.allowFrom`, or `telegram.webhookUrl` as needed.
- Optional: set `telegram.groups` (with `telegram.groups."*".requireMention`), `telegram.allowFrom`, or `telegram.webhookUrl` as needed.
```json5
{
@@ -167,6 +181,19 @@ Minimal `~/.clawdis/clawdis.json`:
}
```
### Discord
- Set `DISCORD_BOT_TOKEN` or `discord.token` (env wins).
- Optional: set `discord.slashCommand`, `discord.dm.allowFrom`, `discord.guilds`, or `discord.mediaMaxMb` as needed.
```json5
{
discord: {
token: "1234abcd"
}
}
```
Browser control (optional):
```json5
@@ -188,11 +215,28 @@ Browser control (optional):
- [`docs/web.md`](docs/web.md)
- [`docs/discovery.md`](docs/discovery.md)
- [`docs/agent.md`](docs/agent.md)
- [`docs/discord.md`](docs/discord.md)
- [`docs/wizard.md`](docs/wizard.md)
- Webhooks + external triggers: [`docs/webhook.md`](docs/webhook.md)
- Gmail hooks (email → wake): [`docs/gmail-pubsub.md`](docs/gmail-pubsub.md)
## Email hooks (Gmail)
```bash
clawdis hooks gmail setup --account you@gmail.com
clawdis hooks gmail run
```
- [`docs/security.md`](docs/security.md)
- [`docs/troubleshooting.md`](docs/troubleshooting.md)
- [`docs/ios/connect.md`](docs/ios/connect.md)
- [`docs/clawdis-mac.md`](docs/clawdis-mac.md)
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines, maintainers, and how to submit PRs.
AI/vibe-coded PRs welcome! 🤖
## Clawd
Clawdis was built for **Clawd**, a space lobster AI assistant.

View File

@@ -1,13 +1,13 @@
{
"originHash" : "3018b2c8c183d55b57ad0c4526b2380ac3b957d13a3a86e1b2845e81323c443a",
"originHash" : "5d29ee82825e0764775562242cfa1ff4dc79584797dd638f76c9876545454748",
"pins" : [
{
"identity" : "commander",
"identity" : "elevenlabskit",
"kind" : "remoteSourceControl",
"location" : "https://github.com/steipete/Commander.git",
"location" : "https://github.com/steipete/ElevenLabsKit",
"state" : {
"revision" : "8b8cb4f34315ce9e5307b3a2bcd77ff73f586a02",
"version" : "0.2.0"
"revision" : "c8679fbd37416a8780fe43be88a497ff16209e2d",
"version" : "0.1.0"
}
},
{

View File

@@ -13,7 +13,7 @@ let package = Package(
.executable(name: "swabble", targets: ["SwabbleCLI"]),
],
dependencies: [
.package(url: "https://github.com/steipete/Commander.git", from: "0.2.0"),
.package(path: "../Peekaboo/Commander"),
.package(url: "https://github.com/apple/swift-testing", from: "0.99.0"),
],
targets: [

View File

@@ -68,21 +68,13 @@ public enum WakeWordGate {
let tokens = self.normalizeSegments(segments)
guard !tokens.isEmpty else { return nil }
var bestIndex: Int?
var bestTriggerEnd: TimeInterval = 0
var bestGap: TimeInterval = 0
var best: (index: Int, triggerEnd: TimeInterval, gap: TimeInterval)?
for trigger in triggerTokens {
let count = trigger.tokens.count
guard count > 0, tokens.count > count else { continue }
for i in 0...(tokens.count - count - 1) {
var matched = true
for t in 0..<count {
if tokens[i + t].normalized != trigger.tokens[t] {
matched = false
break
}
}
let matched = (0..<count).allSatisfy { tokens[i + $0].normalized == trigger.tokens[$0] }
if !matched { continue }
let triggerEnd = tokens[i + count - 1].end
@@ -90,19 +82,17 @@ public enum WakeWordGate {
let gap = nextToken.start - triggerEnd
if gap < config.minPostTriggerGap { continue }
if let bestIndex, i <= bestIndex { continue }
if let best, i <= best.index { continue }
bestIndex = i
bestTriggerEnd = triggerEnd
bestGap = gap
best = (i, triggerEnd, gap)
}
}
guard let bestIndex else { return nil }
let command = self.commandText(transcript: transcript, segments: segments, triggerEndTime: bestTriggerEnd)
guard let best else { return nil }
let command = self.commandText(transcript: transcript, segments: segments, triggerEndTime: best.triggerEnd)
.trimmingCharacters(in: Self.whitespaceAndPunctuation)
guard command.count >= config.minCommandLength else { return nil }
return WakeWordGateMatch(triggerEndTime: bestTriggerEnd, postGap: bestGap, command: command)
return WakeWordGateMatch(triggerEndTime: best.triggerEnd, postGap: best.gap, command: command)
}
public static func commandText(

View File

@@ -1,32 +1,212 @@
<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0"
xmlns:sparkle="http://www.andymatuschak.org/xml-namespaces/sparkle"
xmlns:dc="http://purl.org/dc/elements/1.1/">
<channel>
<title>Clawdis Updates</title>
<link>https://raw.githubusercontent.com/steipete/clawdis/main/appcast.xml</link>
<description>Signed update feed for the Clawdis macOS companion app.</description>
<item>
<title>Clawdis 2.0.0-beta2</title>
<sparkle:releaseNotesLink>https://github.com/steipete/clawdis/releases/tag/v2.0.0-beta2</sparkle:releaseNotesLink>
<pubDate>Sun, 21 Dec 2025 02:25:39 +0000</pubDate>
<enclosure url="https://github.com/steipete/clawdis/releases/download/v2.0.0-beta2/Clawdis-2.0.0-beta2.zip"
sparkle:edSignature="voRWLh2Cbg/i2KtUV6ci/MW3b7hK/u1ZPoiryKs+S36ua3xnc51R97JGwmIaToCfTHg2mgFWF7M6qppfe7YsAw=="
sparkle:version="2.0.0-beta2"
sparkle:shortVersionString="2.0.0-beta2"
length="67435891"
type="application/octet-stream" />
</item>
<item>
<title>Clawdis 2.0.0-beta1</title>
<sparkle:releaseNotesLink>https://github.com/steipete/clawdis/releases/tag/v2.0.0-beta1</sparkle:releaseNotesLink>
<pubDate>Fri, 19 Dec 2025 17:19:50 +0000</pubDate>
<enclosure url="https://github.com/steipete/clawdis/releases/download/v2.0.0-beta1/Clawdis-2.0.0-beta1.zip"
sparkle:edSignature="oEpGD46U4ZyBBSY9/piUIFDJU+KlFB751JIWOW2yS0sRNHKszyG5khDHg9o9bV9Zo8DOCNF/HOi88jmtHJAaCQ=="
sparkle:version="2.0.0-beta1"
sparkle:shortVersionString="2.0.0-beta1"
length="72410016"
type="application/octet-stream" />
</item>
</channel>
</rss>
<?xml version="1.0" standalone="yes"?>
<rss xmlns:sparkle="http://www.andymatuschak.org/xml-namespaces/sparkle" version="2.0">
<channel>
<title>Clawdis</title>
<item>
<title>2.0.0-beta5</title>
<pubDate>Sat, 03 Jan 2026 07:15:16 +0100</pubDate>
<link>https://raw.githubusercontent.com/steipete/clawdis/main/appcast.xml</link>
<sparkle:version>2765</sparkle:version>
<sparkle:shortVersionString>2.0.0-beta5</sparkle:shortVersionString>
<sparkle:minimumSystemVersion>15.0</sparkle:minimumSystemVersion>
<description><![CDATA[<h2>Clawdis 2.0.0-beta5</h2>
<h3>Fixed</h3>
<ul>
<li>Media: preserve GIF animation when uploading to Discord/other providers (skip JPEG optimization for image/gif).</li>
<li>Agent runtime: update pi-mono dependencies to 0.31.1 (agent-core split).</li>
<li>Dependencies: bump to latest compatible versions (TypeBox, grammY, Zod, Rolldown, oxlint-tsgolint).</li>
<li>Tests: cover read tool image metadata + text output.</li>
<li>Tests: add queue mode coverage (collect/followup + directive parsing).</li>
</ul>
<h3>Breaking</h3>
<ul>
<li>Skills config schema moved under <code>skills.*</code>:</li>
</ul>
- <code>skillsLoad.extraDirs</code> → <code>skills.load.extraDirs</code>
- <code>skillsInstall.*</code> → <code>skills.install.*</code>
- per-skill config map moved to <code>skills.entries</code> (e.g. <code>skills.peekaboo.enabled</code> → <code>skills.entries.peekaboo.enabled</code>)
- new optional bundled allowlist: <code>skills.allowBundled</code> (only affects bundled skills)
<ul>
<li>Sessions: group keys now use <code>surface:group:<id></code> / <code>surface:channel:<id></code>; legacy <code>group:*</code> keys migrate on next message; <code>groupdm</code> keys are no longer recognized.</li>
<li>Discord: remove legacy <code>discord.allowFrom</code>, <code>discord.guildAllowFrom</code>, and <code>discord.requireMention</code>; use <code>discord.dm</code> + <code>discord.guilds</code>.</li>
<li>Providers: Discord/Telegram no longer auto-start from env tokens alone; add <code>discord: { enabled: true }</code> / <code>telegram: { enabled: true }</code> to your config when using <code>DISCORD_BOT_TOKEN</code> / <code>TELEGRAM_BOT_TOKEN</code>.</li>
<li>Config: remove <code>routing.allowFrom</code>; use <code>whatsapp.allowFrom</code> instead (run <code>clawdis doctor</code> to migrate).</li>
<li>Config: remove <code>routing.groupChat.requireMention</code> + <code>telegram.requireMention</code>; use <code>whatsapp.groups</code>, <code>imessage.groups</code>, and <code>telegram.groups</code> defaults instead (run <code>clawdis doctor</code> to migrate).</li>
</ul>
<h3>Features</h3>
<ul>
<li>Discord: expand <code>discord</code> tool actions (reactions, stickers, polls, threads, search, moderation gates) (#115) — thanks @thewilloftheshadow.</li>
<li>Discord/Telegram: add reply tags (<code>[[reply_to_current]]</code>, <code>[[reply_to:<id>]]</code>) with per-provider <code>replyToMode</code> (off|first|all) for native threaded replies.</li>
<li>Talk mode: continuous speech conversations (macOS/iOS/Android) with ElevenLabs TTS, reply directives, and optional interrupt-on-speech.</li>
<li>Auto-reply: expand queue modes (steer/followup/collect/steer-backlog) with debounce/cap/drop options and followup backlog handling.</li>
<li>UI: add optional <code>ui.seamColor</code> accent to tint the Talk Mode side bubble (macOS/iOS/Android).</li>
<li>Nix mode: opt-in declarative config + read-only settings UI when <code>CLAWDIS_NIX_MODE=1</code> (thanks @joshp123 for the persistence — earned my trust; I'll merge these going forward).</li>
<li>CLI: add Google Antigravity OAuth auth option for Claude Opus 4.5/Gemini 3 (#88) — thanks @mukhtharcm.</li>
<li>Agent runtime: accept legacy <code>Z_AI_API_KEY</code> for Z.AI provider auth (maps to <code>ZAI_API_KEY</code>).</li>
<li>Groups: add per-group mention gating defaults/overrides for Telegram/WhatsApp/iMessage via <code>*.groups</code> with <code>"*"</code> defaults; Discord now supports <code>discord.guilds."*"</code> as a default.</li>
<li>Discord: add user-installed slash command handling with per-user sessions and auto-registration (#94) — thanks @thewilloftheshadow.</li>
<li>Discord: add DM enable/allowlist plus guild channel/user/guild allowlists with id/name matching.</li>
<li>Signal: add <code>signal-cli</code> JSON-RPC support for send/receive via the Signal provider.</li>
<li>iMessage: add imsg JSON-RPC integration (stdio), chat_id routing, and group chat support.</li>
<li>Chat UI: add recent-session dropdown switcher (main first) in macOS/iOS/Android + Control UI.</li>
<li>UI: add Discord/Signal/iMessage connection panels in macOS + Control UI (thanks @thewilloftheshadow).</li>
<li>Discord: allow agent-triggered reactions via <code>clawdis_discord</code> when enabled, and surface message ids in context.</li>
<li>Discord: revamp guild routing config with per-guild/channel rules and slugged display names; add optional group DM support (default off).</li>
<li>Discord: remove legacy guild/channel ignore lists in favor of per-guild allowlists (and proposed per-guild ignore lists).</li>
<li>Skills: add Trello skill for board/list/card management (thanks @clawd).</li>
<li>Docker: add containerized gateway/CLI setup via Dockerfile, compose, and setup script (thanks @dan-dr).</li>
<li>Tests: add a Z.AI live test gate for smoke validation when keys are present.</li>
<li>macOS Debug: add app log verbosity and rolling file log toggle for swift-log-backed app logs.</li>
<li>CLI: add onboarding wizard (gateway + workspace + skills) with daemon installers and Anthropic/Minimax setup paths.</li>
<li>CLI: add ASCII banner header to wizard entry points.</li>
<li>CLI: add <code>configure</code>, <code>doctor</code>, and <code>update</code> wizards for ongoing setup, health checks, and modernization.</li>
<li>CLI: add Signal CLI auto-install from GitHub releases in the wizard and persist wizard run metadata in config.</li>
<li>CLI: add remote gateway client config (gateway.remote.*) with Bonjour-assisted discovery.</li>
<li>CLI: enhance <code>clawdis tui</code> with model/session pickers, tool cards, and slash commands (local or remote).</li>
<li>Gateway: allow <code>sessions.patch</code> to set per-session model overrides (used by the TUI <code>/model</code> flow).</li>
<li>Skills: allow <code>bun</code> as a node manager for skill installs.</li>
<li>Skills: add <code>things-mac</code> (Things 3 CLI) for read/search plus add/update via URL scheme.</li>
<li>Skills: add Apple Notes + Reminders skills via memo CLI (thanks @tylerwince).</li>
<li>Tests: add a Docker-based onboarding E2E harness.</li>
<li>Tests: harden wizard E2E flows for reset, providers, skills, and remote non-interactive runs.</li>
<li>Browser tools: add remote CDP URL support, Linux launcher options (<code>executablePath</code>, <code>noSandbox</code>), and surface <code>cdpUrl</code> in status.</li>
<li>Skills: add tmux-first coding-agent skill + <code>requires.anyBins</code> gate for multi-CLI setup (thanks @sreekaransrinath).</li>
</ul>
<h3>Fixes</h3>
<ul>
<li>Gog calendar: format date ranges as RFC 3339 with timezone to satisfy Google Calendar API (thanks @jayhickey).</li>
<li>macOS onboarding: add scrollable page gutter for overflowing content (#105) — thanks @thewilloftheshadow.</li>
<li>Chat UI: keep the chat scrolled to the latest message after switching sessions.</li>
<li>Chat UI: show rich session display names in Web Chat + SwiftUI + Android.</li>
<li>Auto-reply: stream completed reply blocks as soon as they finish (configurable default + break); skip empty tool-only blocks unless verbose.</li>
<li>Discord: avoid duplicate sends when block streaming is enabled (race with typing hook).</li>
<li>Providers: make outbound text chunk limits configurable via <code>*.textChunkLimit</code> (defaults remain 4000/Discord 2000).</li>
<li>CLI onboarding: persist gateway token in config so local CLI auth works; recommend auth Off unless you need multi-machine access.</li>
<li>Control UI: accept a <code>?token=</code> URL param to auto-fill Gateway auth; onboarding now opens the dashboard with token auth when configured.</li>
<li>Agent prompt: remove hardcoded user name in system prompt example.</li>
<li>Chat UI: add extra top padding before the first message bubble in Web Chat (macOS/iOS/Android).</li>
<li>Control UI: refine Web Chat session selector styling (chevron spacing + background).</li>
<li>WebChat: stream live updates for sessions even when runs start outside the chat UI.</li>
<li>Gateway CLI: read <code>CLAWDIS_GATEWAY_PASSWORD</code> from environment in <code>callGateway()</code> — allows <code>doctor</code>/<code>health</code> commands to auth without explicit <code>--password</code> flag.</li>
<li>Gateway: add password auth support for remote gateway connections (thanks @jeffersonwarrior).</li>
<li>Auto-reply: strip stray leading/trailing <code>HEARTBEAT_OK</code> from normal replies; drop short (≤ 30 chars) heartbeat acks.</li>
<li>WhatsApp auto-reply: default to self-only when no config is present.</li>
<li>Logging: trim provider prefix duplication in Discord/Signal/Telegram runtime log lines.</li>
<li>Logging/Signal: treat signal-cli "Failed …" lines as errors in gateway logs.</li>
<li>Discord: include recent guild context when replying to mentions and add <code>discord.historyLimit</code> to tune how many messages are captured.</li>
<li>Discord: include author tag + id in group context <code>[from:]</code> lines for ping-ready replies (thanks @thewilloftheshadow).</li>
<li>Discord: include replied-to message context when a Discord message references another message (thanks @thewilloftheshadow).</li>
<li>Discord: preserve newlines when stripping reply tags from agent output.</li>
<li>Gateway: fix TypeScript build by aligning hook mapping <code>channel</code> types and removing a dead Group DM branch in Discord monitor.</li>
<li>Skills: switch imsg installer to brew tap formula.</li>
<li>Skills: gate macOS-only skills by OS and surface block reasons in the Skills UI.</li>
<li>Onboarding: show skill descriptions in the macOS setup flow and surface clearer Gateway/skills error messages.</li>
<li>Onboarding: auto-verify Claude OAuth tokens, show “verified” when detected working, and avoid re-auth prompts unless verification fails.</li>
<li>CLI onboarding: include exit code + a useful one-line summary when skill dependency installs fail.</li>
<li>CLI onboarding: explain Tailscale exposure options (Off/Serve/Funnel) and colorize provider status (linked/configured/needs setup).</li>
<li>CLI onboarding: add provider primers (WhatsApp/Telegram/Discord/Signal) incl. Discord bot token setup steps.</li>
<li>CLI onboarding: allow skipping the “install missing skill dependencies” selection without canceling the wizard.</li>
<li>CLI onboarding: always prompt for WhatsApp <code>whatsapp.allowFrom</code> and print (optionally open) the Control UI URL when done.</li>
<li>CLI onboarding: detect gateway reachability and annotate Local/Remote choices (helps pick the right mode).</li>
<li>macOS settings: colorize provider status subtitles to distinguish healthy vs degraded states.</li>
<li>macOS: keep config writes on the main actor to satisfy Swift concurrency rules.</li>
<li>macOS menu: show multi-line gateway error details, add an always-visible gateway row, avoid duplicate gateway status rows, suppress transient <code>cancelled</code> device refresh errors, and auto-recover the control channel on disconnect.</li>
<li>macOS menu: show session last-used timestamps in the list and add recent-message previews in session submenus.</li>
<li>macOS menu: tighten session row padding and time out session preview loading with cached fallback.</li>
<li>macOS: log health refresh failures and recovery to make gateway issues easier to diagnose.</li>
<li>macOS codesign: skip hardened runtime for ad-hoc signing and avoid empty options args (#70) — thanks @petter-b</li>
<li>macOS codesign: include camera entitlement so permission prompts work in the menu bar app.</li>
<li>Agent tools: bash tool supports real TTY via <code>stdinMode: "pty"</code> with node-pty, warning + fallback on load/start failure.</li>
<li>Agent tools: map <code>camera.snap</code> JPEG payloads to <code>image/jpeg</code> to avoid MIME mismatch errors.</li>
<li>Tests: cover <code>camera.snap</code> MIME mapping to prevent image/png vs image/jpeg mismatches.</li>
<li>macOS camera: wait for exposure/white balance to settle before capturing a snap to avoid dark images.</li>
<li>Camera snap: add <code>delayMs</code> parameter (default 2000ms on macOS) to improve exposure reliability.</li>
<li>Camera: add <code>camera.list</code> and optional <code>deviceId</code> selection for snaps/clips.</li>
<li>Tests: cover camera device selection params in CLI + agent tools.</li>
<li>macOS packaging: move rpath config into swift build for reliability (#69) — thanks @petter-b</li>
<li>macOS: prioritize main bundle for device resources to prevent crash (#73) — thanks @petter-b</li>
<li>macOS remote: route settings through gateway config and avoid local config reads in remote mode.</li>
<li>Telegram: align token resolution for cron/agent/CLI sends (env/config/tokenFile) to prevent isolated delivery failures (#76).</li>
<li>Telegram: honor per-group mention gating defaults/overrides via <code>telegram.groups</code> and <code>"*"</code> defaults (thanks @joshp123).</li>
<li>Chat UI: clear composer input immediately and allow clear while editing to prevent duplicate sends (#72) — thanks @hrdwdmrbl</li>
<li>Restart: use systemd on Linux (and report actual restart method) instead of always launchctl.</li>
<li>Gateway relay: detect Bun binaries via execPath to resolve packaged assets on macOS.</li>
<li>Cron: prevent <code>every</code> schedules without an anchor from firing in a tight loop (thanks @jamesgroat).</li>
<li>Docs: add manual OAuth setup for remote/headless deployments (#67) — thanks @wstock</li>
<li>Docs/agent tools: clarify that browser <code>wait</code> should be avoided by default and used only in exceptional cases.</li>
<li>Docs: clarify self-chat mode and group mention gating config (#111) — thanks @rafaelreis-r.</li>
<li>Browser tools: <code>upload</code> supports auto-click refs, direct <code>inputRef</code>/<code>element</code> file inputs, and emits input/change after <code>setFiles</code> so JS-heavy sites pick up attachments.</li>
<li>Browser tools: harden CDP readiness (HTTP + WS), retry CDP connects, and auto-restart the clawd browser when the socket handshake stalls.</li>
<li>Browser CLI: add <code>clawdis browser reset-profile</code> to move the clawd profile to Trash when it gets wedged.</li>
<li>Signal: fix daemon startup race (wait for <code>/api/v1/check</code>) and normalize JSON-RPC <code>version</code> probe parsing.</li>
<li>Docs/Signal: clarify bot-number vs personal-account setup (self-chat loop protection) and add a quickstart config snippet.</li>
<li>Docs: refresh the CLI wizard guide and highlight onboarding in the README.</li>
<li>CLI: tighten onboarding prompt typing to keep bun builds green.</li>
<li>macOS: Voice Wake now fully tears down the Speech pipeline when disabled (cancel pending restarts, drop stale callbacks) to avoid high CPU in the background.</li>
<li>macOS menu: add a Talk Mode action alongside the Open Dashboard/Chat/Canvas entries.</li>
<li>macOS Debug: hide “Restart Gateway” when the app wont start a local gateway (remote mode / attach-only).</li>
<li>macOS Debug: add an icon for the App Logging submenu.</li>
<li>macOS Talk Mode: orb overlay refresh, ElevenLabs request logging, API key status in settings, and auto-select first voice when none is configured.</li>
<li>macOS Talk Mode: add hard timeout around ElevenLabs TTS synthesis to avoid getting stuck “speaking” forever on hung requests.</li>
<li>macOS Talk Mode: avoid stuck playback when the audio player never starts (fail-fast + watchdog).</li>
<li>macOS Talk Mode: fix audio stop ordering so disabling Talk Mode always stops in-flight playback.</li>
<li>macOS Talk Mode: throttle audio-level updates (avoid per-buffer task creation) to reduce CPU/task churn.</li>
<li>macOS Talk Mode: increase overlay window size so wave rings dont clip; close button is hover-only and closer to the orb.</li>
<li>WebChat: preserve chat run ordering per session so concurrent runs dont strand the typing indicator.</li>
<li>Talk Mode: fall back to system TTS when ElevenLabs is unavailable, returns non-audio, or playback fails (macOS/iOS/Android).</li>
<li>Talk Mode: stream PCM on macOS/iOS for lower latency (incremental playback); Android continues MP3 streaming.</li>
<li>Talk Mode: validate ElevenLabs v3 stability and latency tier directives before sending requests.</li>
<li>iOS/Android Talk Mode: auto-select the first ElevenLabs voice when none is configured.</li>
<li>ElevenLabs: add retry/backoff for 429/5xx and include content-type in errors for debugging.</li>
<li>Talk Mode: align to the gateways main session key and fall back to history polling when chat events drop (prevents stuck “thinking” / missing messages).</li>
<li>Talk Mode: treat history timestamps as seconds or milliseconds to avoid stale assistant picks (macOS/iOS/Android).</li>
<li>Chat UI: clear streaming/tool bubbles when external runs finish, preventing duplicate assistant bubbles.</li>
<li>Chat UI: user bubbles use <code>ui.seamColor</code> (fallback to a calmer default blue).</li>
<li>Android Chat UI: use <code>onPrimary</code> for user bubble text to preserve contrast (thanks @Syhids).</li>
<li>Control UI: sync sidebar navigation with the URL for deep-linking, and auto-scroll chat to the latest message.</li>
<li>Control UI: disable Web Chat + Talk when no iOS/Android node is connected; refreshed Web Chat styling and keyboard send.</li>
<li>Control UI: keep chat pinned to the latest message while typing/sending and restore drafts on send failures.</li>
<li>Control UI: soften chat bubble text opacity for calmer readability.</li>
<li>macOS Web Chat: improve empty/error states, focus message field on open, keep pill/send inside the input field, and make the composer pill edge-to-edge with square top corners.</li>
<li>macOS: bundle Control UI assets into the app relay so the packaged app can serve them (thanks @mbelinky).</li>
<li>Talk Mode: wait for chat history to surface the assistant reply before starting TTS (macOS/iOS/Android).</li>
<li>iOS Talk Mode: fix chat completion wait to time out even if no events arrive (prevents “Thinking…” hangs).</li>
<li>iOS Talk Mode: keep recognition running during playback to support interrupt-on-speech.</li>
<li>iOS Talk Mode: preserve directive voice/model overrides across config reloads and add ElevenLabs request timeouts.</li>
<li>iOS/Android Talk Mode: explicitly <code>chat.subscribe</code> when Talk Mode is active, so completion events arrive even if the Chat UI isnt open.</li>
<li>Chat UI: refresh history when another client finishes a run in the same session, so Talk Mode + Voice Wake transcripts appear consistently.</li>
<li>Gateway: <code>voice.transcript</code> now also maps agent bus output to <code>chat</code> events, ensuring chat UIs refresh for voice-triggered runs.</li>
<li>Gateway: auto-migrate legacy config on startup (non-Nix); Nix mode hard-fails with a clear error when legacy keys are present.</li>
<li>iOS/Android: show a centered Talk Mode orb overlay while Talk Mode is enabled.</li>
<li>Gateway config: inject <code>talk.apiKey</code> from <code>ELEVENLABS_API_KEY</code>/shell profile so nodes can fetch it on demand.</li>
<li>Canvas A2UI: tag requests with <code>platform=android|ios|macos</code> and boost Android canvas background contrast.</li>
<li>iOS/Android nodes: enable scrolling for loaded web pages in the Canvas WebView (default scaffold stays touch-first).</li>
<li>macOS menu: device list now uses <code>node.list</code> (devices only; no agent/tool presence entries).</li>
<li>macOS menu: device list now shows connected nodes only.</li>
<li>macOS menu: device rows now pack platform/version on the first line, and command lists wrap in submenus.</li>
<li>macOS menu: split device platform/version across first and second rows for better fit.</li>
<li>macOS Canvas: show remote control status in the debug overlay and log A2UI auto-nav decisions.</li>
<li>Canvas A2UI: polish the debug status HUD styling.</li>
<li>iOS node: fix ReplayKit screen recording crash caused by queue isolation assertions during capture.</li>
<li>iOS Talk Mode: avoid audio tap queue assertions when starting recognition.</li>
<li>macOS: use $HOME/Library/pnpm for SSH PATH exports (thanks @mbelinky).</li>
<li>macOS remote: harden SSH tunnel recovery/logging, honor <code>gateway.remote.url</code> port when forwarding, clarify gateway disconnect status, and add Debug menu tunnel reset.</li>
<li>iOS/Android nodes: bridge auto-connect refreshes stale tokens and settings now show richer bridge/device details.</li>
<li>macOS: bundle device model resources to prevent Instances crashes (thanks @mbelinky).</li>
<li>iOS/Android nodes: status pill now surfaces camera activity instead of overlay toasts.</li>
<li>iOS/Android/macOS nodes: camera snaps recompress to keep base64 payloads under 5 MB.</li>
<li>iOS/Android nodes: status pill now surfaces pairing, screen recording, voice wake, and foreground-required states.</li>
<li>iOS/Android nodes: avoid duplicating “Gateway reconnecting…” when the bridge is already connecting.</li>
<li>iOS/Android nodes: Talk Mode now lives on a side bubble (with an iOS toggle to hide it), and Android settings no longer show the Talk Mode switch.</li>
<li>macOS menu: top status line now shows pending node pairing approvals (incl. repairs).</li>
<li>CLI: avoid spurious gateway close errors after successful request/response cycles.</li>
<li>Agent runtime: clamp tool-result images to the 5MB Anthropic limit to avoid hard request rejections.</li>
<li>Agent runtime: write v2 session headers so Pi session branching stays in the Clawdis sessions dir.</li>
<li>Tests: add Swift Testing coverage for camera errors and Kotest coverage for Android bridge endpoints.</li>
</ul>
<p><a href="https://github.com/steipete/clawdis/blob/main/CHANGELOG.md">View full changelog</a></p>
]]></description>
<enclosure url="https://github.com/steipete/clawdis/releases/download/v2.0.0-beta5/Clawdis-2.0.0-beta5.zip" length="145432870" type="application/octet-stream" sparkle:edSignature="qKPcmSx2pAaIYz9NqFp0TY63KrcDlpctUHnNpRs6Q60qQqBWtQycLIhhvhxmGnHupaiEXJfspb/Ad9RgODIzAw=="/>
</item>
</channel>
</rss>

View File

@@ -6,7 +6,7 @@ plugins {
}
android {
namespace = "com.steipete.clawdis.node"
namespace = "com.clawdis.android"
compileSdk = 36
sourceSets {
@@ -16,11 +16,11 @@ android {
}
defaultConfig {
applicationId = "com.steipete.clawdis.node"
applicationId = "com.clawdis.android"
minSdk = 31
targetSdk = 36
versionCode = 1
versionName = "0.1"
versionName = "2.0.0-beta3"
}
buildTypes {
@@ -31,6 +31,7 @@ android {
buildFeatures {
compose = true
buildConfig = true
}
compileOptions {
@@ -63,6 +64,7 @@ dependencies {
implementation("androidx.core:core-ktx:1.17.0")
implementation("androidx.lifecycle:lifecycle-runtime-ktx:2.10.0")
implementation("androidx.activity:activity-compose:1.12.2")
implementation("androidx.webkit:webkit:1.14.0")
implementation("androidx.compose.ui:ui")
implementation("androidx.compose.ui:ui-tooling-preview")
@@ -92,4 +94,11 @@ dependencies {
testImplementation("junit:junit:4.13.2")
testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.10.2")
testImplementation("io.kotest:kotest-runner-junit5-jvm:6.0.7")
testImplementation("io.kotest:kotest-assertions-core-jvm:6.0.7")
testRuntimeOnly("org.junit.vintage:junit-vintage-engine:5.13.3")
}
tasks.withType<Test>().configureEach {
useJUnitPlatform()
}

View File

@@ -9,17 +9,18 @@
<uses-permission
android:name="android.permission.NEARBY_WIFI_DEVICES"
android:usesPermissionFlags="neverForLocation" />
<uses-permission
android:name="android.permission.ACCESS_FINE_LOCATION"
android:maxSdkVersion="32" />
<uses-permission
android:name="android.permission.ACCESS_COARSE_LOCATION"
android:maxSdkVersion="32" />
<uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" />
<uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION" />
<uses-permission android:name="android.permission.ACCESS_BACKGROUND_LOCATION" />
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.SEND_SMS" />
<uses-feature
android:name="android.hardware.camera"
android:required="false" />
<uses-feature
android:name="android.hardware.telephony"
android:required="false" />
<application
android:name=".NodeApp"

View File

@@ -0,0 +1,197 @@
{
"version": 1,
"fallback": {
"emoji": "🧩",
"detailKeys": [
"command",
"path",
"url",
"targetUrl",
"targetId",
"ref",
"element",
"node",
"nodeId",
"jobId",
"requestId",
"to",
"channelId",
"guildId",
"userId",
"name",
"query",
"pattern",
"messageId"
]
},
"tools": {
"bash": {
"emoji": "🛠️",
"title": "Bash",
"detailKeys": ["command"]
},
"process": {
"emoji": "🧰",
"title": "Process",
"detailKeys": ["sessionId"]
},
"read": {
"emoji": "📖",
"title": "Read",
"detailKeys": ["path"]
},
"write": {
"emoji": "✍️",
"title": "Write",
"detailKeys": ["path"]
},
"edit": {
"emoji": "📝",
"title": "Edit",
"detailKeys": ["path"]
},
"attach": {
"emoji": "📎",
"title": "Attach",
"detailKeys": ["path", "url", "fileName"]
},
"browser": {
"emoji": "🌐",
"title": "Browser",
"actions": {
"status": { "label": "status" },
"start": { "label": "start" },
"stop": { "label": "stop" },
"tabs": { "label": "tabs" },
"open": { "label": "open", "detailKeys": ["targetUrl"] },
"focus": { "label": "focus", "detailKeys": ["targetId"] },
"close": { "label": "close", "detailKeys": ["targetId"] },
"snapshot": {
"label": "snapshot",
"detailKeys": ["targetUrl", "targetId", "ref", "element", "format"]
},
"screenshot": {
"label": "screenshot",
"detailKeys": ["targetUrl", "targetId", "ref", "element"]
},
"navigate": {
"label": "navigate",
"detailKeys": ["targetUrl", "targetId"]
},
"console": { "label": "console", "detailKeys": ["level", "targetId"] },
"pdf": { "label": "pdf", "detailKeys": ["targetId"] },
"upload": {
"label": "upload",
"detailKeys": ["paths", "ref", "inputRef", "element", "targetId"]
},
"dialog": {
"label": "dialog",
"detailKeys": ["accept", "promptText", "targetId"]
},
"act": {
"label": "act",
"detailKeys": ["request.kind", "request.ref", "request.selector", "request.text", "request.value"]
}
}
},
"canvas": {
"emoji": "🖼️",
"title": "Canvas",
"actions": {
"present": { "label": "present", "detailKeys": ["target", "node", "nodeId"] },
"hide": { "label": "hide", "detailKeys": ["node", "nodeId"] },
"navigate": { "label": "navigate", "detailKeys": ["url", "node", "nodeId"] },
"eval": { "label": "eval", "detailKeys": ["javaScript", "node", "nodeId"] },
"snapshot": { "label": "snapshot", "detailKeys": ["format", "node", "nodeId"] },
"a2ui_push": { "label": "A2UI push", "detailKeys": ["jsonlPath", "node", "nodeId"] },
"a2ui_reset": { "label": "A2UI reset", "detailKeys": ["node", "nodeId"] }
}
},
"nodes": {
"emoji": "📱",
"title": "Nodes",
"actions": {
"status": { "label": "status" },
"describe": { "label": "describe", "detailKeys": ["node", "nodeId"] },
"pending": { "label": "pending" },
"approve": { "label": "approve", "detailKeys": ["requestId"] },
"reject": { "label": "reject", "detailKeys": ["requestId"] },
"notify": { "label": "notify", "detailKeys": ["node", "nodeId", "title", "body"] },
"camera_snap": { "label": "camera snap", "detailKeys": ["node", "nodeId", "facing", "deviceId"] },
"camera_list": { "label": "camera list", "detailKeys": ["node", "nodeId"] },
"camera_clip": { "label": "camera clip", "detailKeys": ["node", "nodeId", "facing", "duration", "durationMs"] },
"screen_record": {
"label": "screen record",
"detailKeys": ["node", "nodeId", "duration", "durationMs", "fps", "screenIndex"]
}
}
},
"cron": {
"emoji": "⏰",
"title": "Cron",
"actions": {
"status": { "label": "status" },
"list": { "label": "list" },
"add": {
"label": "add",
"detailKeys": ["job.name", "job.id", "job.schedule", "job.cron"]
},
"update": { "label": "update", "detailKeys": ["jobId"] },
"remove": { "label": "remove", "detailKeys": ["jobId"] },
"run": { "label": "run", "detailKeys": ["jobId"] },
"runs": { "label": "runs", "detailKeys": ["jobId"] },
"wake": { "label": "wake", "detailKeys": ["text", "mode"] }
}
},
"gateway": {
"emoji": "🔌",
"title": "Gateway",
"actions": {
"restart": { "label": "restart", "detailKeys": ["reason", "delayMs"] }
}
},
"whatsapp_login": {
"emoji": "🟢",
"title": "WhatsApp Login",
"actions": {
"start": { "label": "start" },
"wait": { "label": "wait" }
}
},
"discord": {
"emoji": "💬",
"title": "Discord",
"actions": {
"react": { "label": "react", "detailKeys": ["channelId", "messageId", "emoji"] },
"reactions": { "label": "reactions", "detailKeys": ["channelId", "messageId"] },
"sticker": { "label": "sticker", "detailKeys": ["to", "stickerIds"] },
"poll": { "label": "poll", "detailKeys": ["question", "to"] },
"permissions": { "label": "permissions", "detailKeys": ["channelId"] },
"readMessages": { "label": "read messages", "detailKeys": ["channelId", "limit"] },
"sendMessage": { "label": "send", "detailKeys": ["to", "content"] },
"editMessage": { "label": "edit", "detailKeys": ["channelId", "messageId"] },
"deleteMessage": { "label": "delete", "detailKeys": ["channelId", "messageId"] },
"threadCreate": { "label": "thread create", "detailKeys": ["channelId", "name"] },
"threadList": { "label": "thread list", "detailKeys": ["guildId", "channelId"] },
"threadReply": { "label": "thread reply", "detailKeys": ["channelId", "content"] },
"pinMessage": { "label": "pin", "detailKeys": ["channelId", "messageId"] },
"unpinMessage": { "label": "unpin", "detailKeys": ["channelId", "messageId"] },
"listPins": { "label": "list pins", "detailKeys": ["channelId"] },
"searchMessages": { "label": "search", "detailKeys": ["guildId", "content"] },
"memberInfo": { "label": "member", "detailKeys": ["guildId", "userId"] },
"roleInfo": { "label": "roles", "detailKeys": ["guildId"] },
"emojiList": { "label": "emoji list", "detailKeys": ["guildId"] },
"roleAdd": { "label": "role add", "detailKeys": ["guildId", "userId", "roleId"] },
"roleRemove": { "label": "role remove", "detailKeys": ["guildId", "userId", "roleId"] },
"channelInfo": { "label": "channel", "detailKeys": ["channelId"] },
"channelList": { "label": "channels", "detailKeys": ["guildId"] },
"voiceStatus": { "label": "voice", "detailKeys": ["guildId", "userId"] },
"eventList": { "label": "events", "detailKeys": ["guildId"] },
"eventCreate": { "label": "event create", "detailKeys": ["guildId", "name"] },
"timeout": { "label": "timeout", "detailKeys": ["guildId", "userId"] },
"kick": { "label": "kick", "detailKeys": ["guildId", "userId"] },
"ban": { "label": "ban", "detailKeys": ["guildId", "userId"] }
}
}
}
}

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node
package com.clawdis.android
enum class CameraHudKind {
Photo,

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node
package com.clawdis.android
import android.content.Context
import android.os.Build

View File

@@ -0,0 +1,15 @@
package com.clawdis.android
enum class LocationMode(val rawValue: String) {
Off("off"),
WhileUsing("whileUsing"),
Always("always"),
;
companion object {
fun fromRawValue(raw: String?): LocationMode {
val normalized = raw?.trim()?.lowercase()
return entries.firstOrNull { it.rawValue.lowercase() == normalized } ?: Off
}
}
}

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node
package com.clawdis.android
import android.Manifest
import android.content.pm.ApplicationInfo
@@ -18,8 +18,8 @@ import androidx.core.view.WindowInsetsControllerCompat
import androidx.lifecycle.Lifecycle
import androidx.lifecycle.lifecycleScope
import androidx.lifecycle.repeatOnLifecycle
import com.steipete.clawdis.node.ui.RootScreen
import com.steipete.clawdis.node.ui.ClawdisTheme
import com.clawdis.android.ui.RootScreen
import com.clawdis.android.ui.ClawdisTheme
import kotlinx.coroutines.launch
class MainActivity : ComponentActivity() {
@@ -39,6 +39,7 @@ class MainActivity : ComponentActivity() {
screenCaptureRequester = ScreenCaptureRequester(this)
viewModel.camera.attachLifecycleOwner(this)
viewModel.camera.attachPermissionRequester(permissionRequester)
viewModel.sms.attachPermissionRequester(permissionRequester)
viewModel.screenRecorder.attachScreenCaptureRequester(screenCaptureRequester)
viewModel.screenRecorder.attachPermissionRequester(permissionRequester)

View File

@@ -1,12 +1,13 @@
package com.steipete.clawdis.node
package com.clawdis.android
import android.app.Application
import androidx.lifecycle.AndroidViewModel
import com.steipete.clawdis.node.bridge.BridgeEndpoint
import com.steipete.clawdis.node.chat.OutgoingAttachment
import com.steipete.clawdis.node.node.CameraCaptureManager
import com.steipete.clawdis.node.node.CanvasController
import com.steipete.clawdis.node.node.ScreenRecordManager
import com.clawdis.android.bridge.BridgeEndpoint
import com.clawdis.android.chat.OutgoingAttachment
import com.clawdis.android.node.CameraCaptureManager
import com.clawdis.android.node.CanvasController
import com.clawdis.android.node.ScreenRecordManager
import com.clawdis.android.node.SmsManager
import kotlinx.coroutines.flow.StateFlow
class MainViewModel(app: Application) : AndroidViewModel(app) {
@@ -15,6 +16,7 @@ class MainViewModel(app: Application) : AndroidViewModel(app) {
val canvas: CanvasController = runtime.canvas
val camera: CameraCaptureManager = runtime.camera
val screenRecorder: ScreenRecordManager = runtime.screenRecorder
val sms: SmsManager = runtime.sms
val bridges: StateFlow<List<BridgeEndpoint>> = runtime.bridges
val discoveryStatusText: StateFlow<String> = runtime.discoveryStatusText
@@ -23,18 +25,27 @@ class MainViewModel(app: Application) : AndroidViewModel(app) {
val statusText: StateFlow<String> = runtime.statusText
val serverName: StateFlow<String?> = runtime.serverName
val remoteAddress: StateFlow<String?> = runtime.remoteAddress
val isForeground: StateFlow<Boolean> = runtime.isForeground
val seamColorArgb: StateFlow<Long> = runtime.seamColorArgb
val cameraHud: StateFlow<CameraHudState?> = runtime.cameraHud
val cameraFlashToken: StateFlow<Long> = runtime.cameraFlashToken
val screenRecordActive: StateFlow<Boolean> = runtime.screenRecordActive
val instanceId: StateFlow<String> = runtime.instanceId
val displayName: StateFlow<String> = runtime.displayName
val cameraEnabled: StateFlow<Boolean> = runtime.cameraEnabled
val locationMode: StateFlow<LocationMode> = runtime.locationMode
val locationPreciseEnabled: StateFlow<Boolean> = runtime.locationPreciseEnabled
val preventSleep: StateFlow<Boolean> = runtime.preventSleep
val wakeWords: StateFlow<List<String>> = runtime.wakeWords
val voiceWakeMode: StateFlow<VoiceWakeMode> = runtime.voiceWakeMode
val voiceWakeStatusText: StateFlow<String> = runtime.voiceWakeStatusText
val voiceWakeIsListening: StateFlow<Boolean> = runtime.voiceWakeIsListening
val talkEnabled: StateFlow<Boolean> = runtime.talkEnabled
val talkStatusText: StateFlow<String> = runtime.talkStatusText
val talkIsListening: StateFlow<Boolean> = runtime.talkIsListening
val talkIsSpeaking: StateFlow<Boolean> = runtime.talkIsSpeaking
val manualEnabled: StateFlow<Boolean> = runtime.manualEnabled
val manualHost: StateFlow<String> = runtime.manualHost
val manualPort: StateFlow<Int> = runtime.manualPort
@@ -63,6 +74,14 @@ class MainViewModel(app: Application) : AndroidViewModel(app) {
runtime.setCameraEnabled(value)
}
fun setLocationMode(mode: LocationMode) {
runtime.setLocationMode(mode)
}
fun setLocationPreciseEnabled(value: Boolean) {
runtime.setLocationPreciseEnabled(value)
}
fun setPreventSleep(value: Boolean) {
runtime.setPreventSleep(value)
}
@@ -95,6 +114,14 @@ class MainViewModel(app: Application) : AndroidViewModel(app) {
runtime.setVoiceWakeMode(mode)
}
fun setTalkEnabled(enabled: Boolean) {
runtime.setTalkEnabled(enabled)
}
fun refreshBridgeHello() {
runtime.refreshBridgeHello()
}
fun connect(endpoint: BridgeEndpoint) {
runtime.connect(endpoint)
}

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node
package com.clawdis.android
import android.app.Application

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node
package com.clawdis.android
import android.app.Notification
import android.app.NotificationChannel
@@ -146,7 +146,7 @@ class NodeForegroundService : Service() {
private const val CHANNEL_ID = "connection"
private const val NOTIFICATION_ID = 1
private const val ACTION_STOP = "com.steipete.clawdis.node.action.STOP"
private const val ACTION_STOP = "com.clawdis.android.action.STOP"
fun start(context: Context) {
val intent = Intent(context, NodeForegroundService::class.java)

View File

@@ -1,34 +1,42 @@
package com.steipete.clawdis.node
package com.clawdis.android
import android.Manifest
import android.content.Context
import android.content.pm.PackageManager
import android.location.LocationManager
import android.os.Build
import android.os.SystemClock
import androidx.core.content.ContextCompat
import com.steipete.clawdis.node.chat.ChatController
import com.steipete.clawdis.node.chat.ChatMessage
import com.steipete.clawdis.node.chat.ChatPendingToolCall
import com.steipete.clawdis.node.chat.ChatSessionEntry
import com.steipete.clawdis.node.chat.OutgoingAttachment
import com.steipete.clawdis.node.bridge.BridgeDiscovery
import com.steipete.clawdis.node.bridge.BridgeEndpoint
import com.steipete.clawdis.node.bridge.BridgePairingClient
import com.steipete.clawdis.node.bridge.BridgeSession
import com.steipete.clawdis.node.node.CameraCaptureManager
import com.steipete.clawdis.node.node.CanvasController
import com.steipete.clawdis.node.node.ScreenRecordManager
import com.steipete.clawdis.node.protocol.ClawdisCapability
import com.steipete.clawdis.node.protocol.ClawdisCameraCommand
import com.steipete.clawdis.node.protocol.ClawdisCanvasA2UIAction
import com.steipete.clawdis.node.protocol.ClawdisCanvasA2UICommand
import com.steipete.clawdis.node.protocol.ClawdisCanvasCommand
import com.steipete.clawdis.node.protocol.ClawdisScreenCommand
import com.steipete.clawdis.node.voice.VoiceWakeManager
import com.clawdis.android.chat.ChatController
import com.clawdis.android.chat.ChatMessage
import com.clawdis.android.chat.ChatPendingToolCall
import com.clawdis.android.chat.ChatSessionEntry
import com.clawdis.android.chat.OutgoingAttachment
import com.clawdis.android.bridge.BridgeDiscovery
import com.clawdis.android.bridge.BridgeEndpoint
import com.clawdis.android.bridge.BridgePairingClient
import com.clawdis.android.bridge.BridgeSession
import com.clawdis.android.node.CameraCaptureManager
import com.clawdis.android.node.LocationCaptureManager
import com.clawdis.android.BuildConfig
import com.clawdis.android.node.CanvasController
import com.clawdis.android.node.ScreenRecordManager
import com.clawdis.android.node.SmsManager
import com.clawdis.android.protocol.ClawdisCapability
import com.clawdis.android.protocol.ClawdisCameraCommand
import com.clawdis.android.protocol.ClawdisCanvasA2UIAction
import com.clawdis.android.protocol.ClawdisCanvasA2UICommand
import com.clawdis.android.protocol.ClawdisCanvasCommand
import com.clawdis.android.protocol.ClawdisScreenCommand
import com.clawdis.android.protocol.ClawdisLocationCommand
import com.clawdis.android.protocol.ClawdisSmsCommand
import com.clawdis.android.voice.TalkModeManager
import com.clawdis.android.voice.VoiceWakeManager
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.Job
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.TimeoutCancellationException
import kotlinx.coroutines.delay
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
@@ -53,7 +61,9 @@ class NodeRuntime(context: Context) {
val prefs = SecurePrefs(appContext)
val canvas = CanvasController()
val camera = CameraCaptureManager(appContext)
val location = LocationCaptureManager(appContext)
val screenRecorder = ScreenRecordManager(appContext)
val sms = SmsManager(appContext)
private val json = Json { ignoreUnknownKeys = true }
private val externalAudioCaptureActive = MutableStateFlow(false)
@@ -68,7 +78,7 @@ class NodeRuntime(context: Context) {
payloadJson =
buildJsonObject {
put("message", JsonPrimitive(command))
put("sessionKey", JsonPrimitive("main"))
put("sessionKey", JsonPrimitive(mainSessionKey.value))
put("thinking", JsonPrimitive(chatThinkingLevel.value))
put("deliver", JsonPrimitive(false))
}.toString(),
@@ -83,6 +93,15 @@ class NodeRuntime(context: Context) {
val voiceWakeStatusText: StateFlow<String>
get() = voiceWake.statusText
val talkStatusText: StateFlow<String>
get() = talkMode.statusText
val talkIsListening: StateFlow<Boolean>
get() = talkMode.isListening
val talkIsSpeaking: StateFlow<Boolean>
get() = talkMode.isSpeaking
private val discovery = BridgeDiscovery(appContext, scope = scope)
val bridges: StateFlow<List<BridgeEndpoint>> = discovery.bridges
val discoveryStatusText: StateFlow<String> = discovery.statusText
@@ -93,6 +112,9 @@ class NodeRuntime(context: Context) {
private val _statusText = MutableStateFlow("Offline")
val statusText: StateFlow<String> = _statusText.asStateFlow()
private val _mainSessionKey = MutableStateFlow("main")
val mainSessionKey: StateFlow<String> = _mainSessionKey.asStateFlow()
private val cameraHudSeq = AtomicLong(0)
private val _cameraHud = MutableStateFlow<CameraHudState?>(null)
val cameraHud: StateFlow<CameraHudState?> = _cameraHud.asStateFlow()
@@ -100,12 +122,18 @@ class NodeRuntime(context: Context) {
private val _cameraFlashToken = MutableStateFlow(0L)
val cameraFlashToken: StateFlow<Long> = _cameraFlashToken.asStateFlow()
private val _screenRecordActive = MutableStateFlow(false)
val screenRecordActive: StateFlow<Boolean> = _screenRecordActive.asStateFlow()
private val _serverName = MutableStateFlow<String?>(null)
val serverName: StateFlow<String?> = _serverName.asStateFlow()
private val _remoteAddress = MutableStateFlow<String?>(null)
val remoteAddress: StateFlow<String?> = _remoteAddress.asStateFlow()
private val _seamColorArgb = MutableStateFlow(DEFAULT_SEAM_COLOR_ARGB)
val seamColorArgb: StateFlow<Long> = _seamColorArgb.asStateFlow()
private val _isForeground = MutableStateFlow(true)
val isForeground: StateFlow<Boolean> = _isForeground.asStateFlow()
@@ -119,6 +147,8 @@ class NodeRuntime(context: Context) {
_serverName.value = name
_remoteAddress.value = remote
_isConnected.value = true
_seamColorArgb.value = DEFAULT_SEAM_COLOR_ARGB
scope.launch { refreshBrandingFromGateway() }
scope.launch { refreshWakeWordsFromGateway() }
maybeNavigateToA2uiOnConnect()
},
@@ -132,12 +162,17 @@ class NodeRuntime(context: Context) {
)
private val chat = ChatController(scope = scope, session = session, json = json)
private val talkMode: TalkModeManager by lazy {
TalkModeManager(context = appContext, scope = scope).also { it.attachSession(session) }
}
private fun handleSessionDisconnected(message: String) {
_statusText.value = message
_serverName.value = null
_remoteAddress.value = null
_isConnected.value = false
_seamColorArgb.value = DEFAULT_SEAM_COLOR_ARGB
_mainSessionKey.value = "main"
chat.onDisconnected(message)
showLocalCanvasOnDisconnect()
}
@@ -159,9 +194,12 @@ class NodeRuntime(context: Context) {
val instanceId: StateFlow<String> = prefs.instanceId
val displayName: StateFlow<String> = prefs.displayName
val cameraEnabled: StateFlow<Boolean> = prefs.cameraEnabled
val locationMode: StateFlow<LocationMode> = prefs.locationMode
val locationPreciseEnabled: StateFlow<Boolean> = prefs.locationPreciseEnabled
val preventSleep: StateFlow<Boolean> = prefs.preventSleep
val wakeWords: StateFlow<List<String>> = prefs.wakeWords
val voiceWakeMode: StateFlow<VoiceWakeMode> = prefs.voiceWakeMode
val talkEnabled: StateFlow<Boolean> = prefs.talkEnabled
val manualEnabled: StateFlow<Boolean> = prefs.manualEnabled
val manualHost: StateFlow<String> = prefs.manualHost
val manualPort: StateFlow<Int> = prefs.manualPort
@@ -217,6 +255,13 @@ class NodeRuntime(context: Context) {
}
}
scope.launch {
talkEnabled.collect { enabled ->
talkMode.setEnabled(enabled)
externalAudioCaptureActive.value = enabled
}
}
scope.launch(Dispatchers.Default) {
bridges.collect { list ->
if (list.isNotEmpty()) {
@@ -277,6 +322,14 @@ class NodeRuntime(context: Context) {
prefs.setCameraEnabled(value)
}
fun setLocationMode(mode: LocationMode) {
prefs.setLocationMode(mode)
}
fun setLocationPreciseEnabled(value: Boolean) {
prefs.setLocationPreciseEnabled(value)
}
fun setPreventSleep(value: Boolean) {
prefs.setPreventSleep(value)
}
@@ -310,63 +363,124 @@ class NodeRuntime(context: Context) {
prefs.setVoiceWakeMode(mode)
}
fun setTalkEnabled(value: Boolean) {
prefs.setTalkEnabled(value)
}
private fun buildInvokeCommands(): List<String> =
buildList {
add(ClawdisCanvasCommand.Present.rawValue)
add(ClawdisCanvasCommand.Hide.rawValue)
add(ClawdisCanvasCommand.Navigate.rawValue)
add(ClawdisCanvasCommand.Eval.rawValue)
add(ClawdisCanvasCommand.Snapshot.rawValue)
add(ClawdisCanvasA2UICommand.Push.rawValue)
add(ClawdisCanvasA2UICommand.PushJSONL.rawValue)
add(ClawdisCanvasA2UICommand.Reset.rawValue)
add(ClawdisScreenCommand.Record.rawValue)
if (cameraEnabled.value) {
add(ClawdisCameraCommand.Snap.rawValue)
add(ClawdisCameraCommand.Clip.rawValue)
}
if (locationMode.value != LocationMode.Off) {
add(ClawdisLocationCommand.Get.rawValue)
}
if (sms.canSendSms()) {
add(ClawdisSmsCommand.Send.rawValue)
}
}
private fun buildCapabilities(): List<String> =
buildList {
add(ClawdisCapability.Canvas.rawValue)
add(ClawdisCapability.Screen.rawValue)
if (cameraEnabled.value) add(ClawdisCapability.Camera.rawValue)
if (sms.canSendSms()) add(ClawdisCapability.Sms.rawValue)
if (voiceWakeMode.value != VoiceWakeMode.Off && hasRecordAudioPermission()) {
add(ClawdisCapability.VoiceWake.rawValue)
}
if (locationMode.value != LocationMode.Off) {
add(ClawdisCapability.Location.rawValue)
}
}
private fun buildPairingHello(token: String?): BridgePairingClient.Hello {
val modelIdentifier = listOfNotNull(Build.MANUFACTURER, Build.MODEL)
.joinToString(" ")
.trim()
.ifEmpty { null }
val versionName = BuildConfig.VERSION_NAME.trim().ifEmpty { "dev" }
val advertisedVersion =
if (BuildConfig.DEBUG && !versionName.contains("dev", ignoreCase = true)) {
"$versionName-dev"
} else {
versionName
}
return BridgePairingClient.Hello(
nodeId = instanceId.value,
displayName = displayName.value,
token = token,
platform = "Android",
version = advertisedVersion,
deviceFamily = "Android",
modelIdentifier = modelIdentifier,
caps = buildCapabilities(),
commands = buildInvokeCommands(),
)
}
private fun buildSessionHello(token: String?): BridgeSession.Hello {
val modelIdentifier = listOfNotNull(Build.MANUFACTURER, Build.MODEL)
.joinToString(" ")
.trim()
.ifEmpty { null }
val versionName = BuildConfig.VERSION_NAME.trim().ifEmpty { "dev" }
val advertisedVersion =
if (BuildConfig.DEBUG && !versionName.contains("dev", ignoreCase = true)) {
"$versionName-dev"
} else {
versionName
}
return BridgeSession.Hello(
nodeId = instanceId.value,
displayName = displayName.value,
token = token,
platform = "Android",
version = advertisedVersion,
deviceFamily = "Android",
modelIdentifier = modelIdentifier,
caps = buildCapabilities(),
commands = buildInvokeCommands(),
)
}
fun refreshBridgeHello() {
scope.launch {
if (!_isConnected.value) return@launch
val token = prefs.loadBridgeToken()
if (token.isNullOrBlank()) return@launch
session.updateHello(buildSessionHello(token))
}
}
fun connect(endpoint: BridgeEndpoint) {
scope.launch {
_statusText.value = "Connecting…"
val storedToken = prefs.loadBridgeToken()
val modelIdentifier = listOfNotNull(Build.MANUFACTURER, Build.MODEL)
.joinToString(" ")
.trim()
.ifEmpty { null }
val invokeCommands =
buildList {
add(ClawdisCanvasCommand.Present.rawValue)
add(ClawdisCanvasCommand.Hide.rawValue)
add(ClawdisCanvasCommand.Navigate.rawValue)
add(ClawdisCanvasCommand.Eval.rawValue)
add(ClawdisCanvasCommand.Snapshot.rawValue)
add(ClawdisCanvasA2UICommand.Push.rawValue)
add(ClawdisCanvasA2UICommand.PushJSONL.rawValue)
add(ClawdisCanvasA2UICommand.Reset.rawValue)
add(ClawdisScreenCommand.Record.rawValue)
if (cameraEnabled.value) {
add(ClawdisCameraCommand.Snap.rawValue)
add(ClawdisCameraCommand.Clip.rawValue)
}
}
val resolved =
if (storedToken.isNullOrBlank()) {
_statusText.value = "Pairing…"
val caps = buildList {
add(ClawdisCapability.Canvas.rawValue)
add(ClawdisCapability.Screen.rawValue)
if (cameraEnabled.value) add(ClawdisCapability.Camera.rawValue)
if (voiceWakeMode.value != VoiceWakeMode.Off && hasRecordAudioPermission()) {
add(ClawdisCapability.VoiceWake.rawValue)
}
}
BridgePairingClient().pairAndHello(
endpoint = endpoint,
hello =
BridgePairingClient.Hello(
nodeId = instanceId.value,
displayName = displayName.value,
token = null,
platform = "Android",
version = "dev",
deviceFamily = "Android",
modelIdentifier = modelIdentifier,
caps = caps,
commands = invokeCommands,
),
hello = buildPairingHello(token = null),
)
} else {
BridgePairingClient.PairResult(ok = true, token = storedToken.trim())
}
if (!resolved.ok || resolved.token.isNullOrBlank()) {
_statusText.value = "Failed: pairing required"
val errorMessage = resolved.error?.trim().orEmpty().ifEmpty { "pairing required" }
_statusText.value = "Failed: $errorMessage"
return@launch
}
@@ -374,26 +488,7 @@ class NodeRuntime(context: Context) {
prefs.saveBridgeToken(authToken)
session.connect(
endpoint = endpoint,
hello =
BridgeSession.Hello(
nodeId = instanceId.value,
displayName = displayName.value,
token = authToken,
platform = "Android",
version = "dev",
deviceFamily = "Android",
modelIdentifier = modelIdentifier,
caps =
buildList {
add(ClawdisCapability.Canvas.rawValue)
add(ClawdisCapability.Screen.rawValue)
if (cameraEnabled.value) add(ClawdisCapability.Camera.rawValue)
if (voiceWakeMode.value != VoiceWakeMode.Off && hasRecordAudioPermission()) {
add(ClawdisCapability.VoiceWake.rawValue)
}
},
commands = invokeCommands,
),
hello = buildSessionHello(token = authToken),
)
}
}
@@ -405,6 +500,28 @@ class NodeRuntime(context: Context) {
)
}
private fun hasFineLocationPermission(): Boolean {
return (
ContextCompat.checkSelfPermission(appContext, Manifest.permission.ACCESS_FINE_LOCATION) ==
PackageManager.PERMISSION_GRANTED
)
}
private fun hasCoarseLocationPermission(): Boolean {
return (
ContextCompat.checkSelfPermission(appContext, Manifest.permission.ACCESS_COARSE_LOCATION) ==
PackageManager.PERMISSION_GRANTED
)
}
private fun hasBackgroundLocationPermission(): Boolean {
if (Build.VERSION.SDK_INT < Build.VERSION_CODES.Q) return true
return (
ContextCompat.checkSelfPermission(appContext, Manifest.permission.ACCESS_BACKGROUND_LOCATION) ==
PackageManager.PERMISSION_GRANTED
)
}
fun connectManual() {
val host = manualHost.value.trim()
val port = manualPort.value
@@ -533,6 +650,7 @@ class NodeRuntime(context: Context) {
return
}
talkMode.handleBridgeEvent(event, payloadJson)
chat.handleBridgeEvent(event, payloadJson)
}
@@ -574,6 +692,25 @@ class NodeRuntime(context: Context) {
}
}
private suspend fun refreshBrandingFromGateway() {
if (!_isConnected.value) return
try {
val res = session.request("config.get", "{}")
val root = json.parseToJsonElement(res).asObjectOrNull()
val config = root?.get("config").asObjectOrNull()
val ui = config?.get("ui").asObjectOrNull()
val raw = ui?.get("seamColor").asStringOrNull()?.trim()
val sessionCfg = config?.get("session").asObjectOrNull()
val rawMainKey = sessionCfg?.get("mainKey").asStringOrNull()?.trim()
_mainSessionKey.value = rawMainKey?.takeIf { it.isNotEmpty() } ?: "main"
val parsed = parseHexColorArgb(raw)
_seamColorArgb.value = parsed ?: DEFAULT_SEAM_COLOR_ARGB
} catch (_: Throwable) {
// ignore
}
}
private suspend fun handleInvoke(command: String, paramsJson: String?): BridgeSession.InvokeResult {
if (
command.startsWith(ClawdisCanvasCommand.NamespacePrefix) ||
@@ -594,6 +731,14 @@ class NodeRuntime(context: Context) {
message = "CAMERA_DISABLED: enable Camera in Settings",
)
}
if (command.startsWith(ClawdisLocationCommand.NamespacePrefix) &&
locationMode.value == LocationMode.Off
) {
return BridgeSession.InvokeResult.error(
code = "LOCATION_DISABLED",
message = "LOCATION_DISABLED: enable Location in Settings",
)
}
return when (command) {
ClawdisCanvasCommand.Present.rawValue -> {
@@ -714,15 +859,85 @@ class NodeRuntime(context: Context) {
if (includeAudio) externalAudioCaptureActive.value = false
}
}
ClawdisScreenCommand.Record.rawValue -> {
val res =
try {
screenRecorder.record(paramsJson)
} catch (err: Throwable) {
val (code, message) = invokeErrorFromThrowable(err)
return BridgeSession.InvokeResult.error(code = code, message = message)
ClawdisLocationCommand.Get.rawValue -> {
val mode = locationMode.value
if (!isForeground.value && mode != LocationMode.Always) {
return BridgeSession.InvokeResult.error(
code = "LOCATION_BACKGROUND_UNAVAILABLE",
message = "LOCATION_BACKGROUND_UNAVAILABLE: background location requires Always",
)
}
if (!hasFineLocationPermission() && !hasCoarseLocationPermission()) {
return BridgeSession.InvokeResult.error(
code = "LOCATION_PERMISSION_REQUIRED",
message = "LOCATION_PERMISSION_REQUIRED: grant Location permission",
)
}
if (!isForeground.value && mode == LocationMode.Always && !hasBackgroundLocationPermission()) {
return BridgeSession.InvokeResult.error(
code = "LOCATION_PERMISSION_REQUIRED",
message = "LOCATION_PERMISSION_REQUIRED: enable Always in system Settings",
)
}
val (maxAgeMs, timeoutMs, desiredAccuracy) = parseLocationParams(paramsJson)
val preciseEnabled = locationPreciseEnabled.value
val accuracy =
when (desiredAccuracy) {
"precise" -> if (preciseEnabled && hasFineLocationPermission()) "precise" else "balanced"
"coarse" -> "coarse"
else -> if (preciseEnabled && hasFineLocationPermission()) "precise" else "balanced"
}
BridgeSession.InvokeResult.ok(res.payloadJson)
val providers =
when (accuracy) {
"precise" -> listOf(LocationManager.GPS_PROVIDER, LocationManager.NETWORK_PROVIDER)
"coarse" -> listOf(LocationManager.NETWORK_PROVIDER, LocationManager.GPS_PROVIDER)
else -> listOf(LocationManager.NETWORK_PROVIDER, LocationManager.GPS_PROVIDER)
}
try {
val payload =
location.getLocation(
desiredProviders = providers,
maxAgeMs = maxAgeMs,
timeoutMs = timeoutMs,
isPrecise = accuracy == "precise",
)
BridgeSession.InvokeResult.ok(payload.payloadJson)
} catch (err: TimeoutCancellationException) {
BridgeSession.InvokeResult.error(
code = "LOCATION_TIMEOUT",
message = "LOCATION_TIMEOUT: no fix in time",
)
} catch (err: Throwable) {
val message = err.message ?: "LOCATION_UNAVAILABLE: no fix"
BridgeSession.InvokeResult.error(code = "LOCATION_UNAVAILABLE", message = message)
}
}
ClawdisScreenCommand.Record.rawValue -> {
// Status pill mirrors screen recording state so it stays visible without overlay stacking.
_screenRecordActive.value = true
try {
val res =
try {
screenRecorder.record(paramsJson)
} catch (err: Throwable) {
val (code, message) = invokeErrorFromThrowable(err)
return BridgeSession.InvokeResult.error(code = code, message = message)
}
BridgeSession.InvokeResult.ok(res.payloadJson)
} finally {
_screenRecordActive.value = false
}
}
ClawdisSmsCommand.Send.rawValue -> {
val res = sms.send(paramsJson)
if (res.ok) {
BridgeSession.InvokeResult.ok(res.payloadJson)
} else {
val error = res.error ?: "SMS_SEND_FAILED"
val idx = error.indexOf(':')
val code = if (idx > 0) error.substring(0, idx).trim() else "SMS_SEND_FAILED"
BridgeSession.InvokeResult.error(code = code, message = error)
}
}
else ->
BridgeSession.InvokeResult.error(
@@ -761,11 +976,30 @@ class NodeRuntime(context: Context) {
return code to "$code: $message"
}
private fun parseLocationParams(paramsJson: String?): Triple<Long?, Long, String?> {
if (paramsJson.isNullOrBlank()) {
return Triple(null, 10_000L, null)
}
val root =
try {
json.parseToJsonElement(paramsJson).asObjectOrNull()
} catch (_: Throwable) {
null
}
val maxAgeMs = (root?.get("maxAgeMs") as? JsonPrimitive)?.content?.toLongOrNull()
val timeoutMs =
(root?.get("timeoutMs") as? JsonPrimitive)?.content?.toLongOrNull()?.coerceIn(1_000L, 60_000L)
?: 10_000L
val desiredAccuracy =
(root?.get("desiredAccuracy") as? JsonPrimitive)?.content?.trim()?.lowercase()
return Triple(maxAgeMs, timeoutMs, desiredAccuracy)
}
private fun resolveA2uiHostUrl(): String? {
val raw = session.currentCanvasHostUrl()?.trim().orEmpty()
if (raw.isBlank()) return null
val base = raw.trimEnd('/')
return "${base}/__clawdis__/a2ui/"
return "${base}/__clawdis__/a2ui/?platform=android"
}
private suspend fun ensureA2uiReady(a2uiUrl: String): Boolean {
@@ -851,6 +1085,8 @@ class NodeRuntime(context: Context) {
private data class Quad<A, B, C, D>(val first: A, val second: B, val third: C, val fourth: D)
private const val DEFAULT_SEAM_COLOR_ARGB: Long = 0xFF4F7A9A
private const val a2uiReadyCheckJS: String =
"""
(() => {
@@ -905,3 +1141,12 @@ private fun JsonElement?.asStringOrNull(): String? =
is JsonPrimitive -> content
else -> null
}
private fun parseHexColorArgb(raw: String?): Long? {
val trimmed = raw?.trim().orEmpty()
if (trimmed.isEmpty()) return null
val hex = if (trimmed.startsWith("#")) trimmed.drop(1) else trimmed
if (hex.length != 6) return null
val rgb = hex.toLongOrNull(16) ?: return null
return 0xFF000000L or rgb
}

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node
package com.clawdis.android
import android.content.pm.PackageManager
import android.content.Intent
@@ -115,7 +115,7 @@ class PermissionRequester(private val activity: ComponentActivity) {
private fun buildRationaleMessage(permissions: List<String>): String {
val labels = permissions.map { permissionLabel(it) }
return "Clawdis needs ${labels.joinToString(", ")} to capture camera media."
return "Clawdis needs ${labels.joinToString(", ")} permissions to continue."
}
private fun buildSettingsMessage(permissions: List<String>): String {
@@ -127,6 +127,7 @@ class PermissionRequester(private val activity: ComponentActivity) {
when (permission) {
Manifest.permission.CAMERA -> "Camera"
Manifest.permission.RECORD_AUDIO -> "Microphone"
Manifest.permission.SEND_SMS -> "SMS"
else -> permission
}
}

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node
package com.clawdis.android
import android.app.Activity
import android.content.Context

View File

@@ -1,6 +1,6 @@
@file:Suppress("DEPRECATION")
package com.steipete.clawdis.node
package com.clawdis.android
import android.content.Context
import androidx.core.content.edit
@@ -47,6 +47,14 @@ class SecurePrefs(context: Context) {
private val _cameraEnabled = MutableStateFlow(prefs.getBoolean("camera.enabled", true))
val cameraEnabled: StateFlow<Boolean> = _cameraEnabled
private val _locationMode =
MutableStateFlow(LocationMode.fromRawValue(prefs.getString("location.enabledMode", "off")))
val locationMode: StateFlow<LocationMode> = _locationMode
private val _locationPreciseEnabled =
MutableStateFlow(prefs.getBoolean("location.preciseEnabled", true))
val locationPreciseEnabled: StateFlow<Boolean> = _locationPreciseEnabled
private val _preventSleep = MutableStateFlow(prefs.getBoolean("screen.preventSleep", true))
val preventSleep: StateFlow<Boolean> = _preventSleep
@@ -73,6 +81,9 @@ class SecurePrefs(context: Context) {
private val _voiceWakeMode = MutableStateFlow(loadVoiceWakeMode())
val voiceWakeMode: StateFlow<VoiceWakeMode> = _voiceWakeMode
private val _talkEnabled = MutableStateFlow(prefs.getBoolean("talk.enabled", false))
val talkEnabled: StateFlow<Boolean> = _talkEnabled
fun setLastDiscoveredStableId(value: String) {
val trimmed = value.trim()
prefs.edit { putString("bridge.lastDiscoveredStableId", trimmed) }
@@ -90,6 +101,16 @@ class SecurePrefs(context: Context) {
_cameraEnabled.value = value
}
fun setLocationMode(mode: LocationMode) {
prefs.edit { putString("location.enabledMode", mode.rawValue) }
_locationMode.value = mode
}
fun setLocationPreciseEnabled(value: Boolean) {
prefs.edit { putBoolean("location.preciseEnabled", value) }
_locationPreciseEnabled.value = value
}
fun setPreventSleep(value: Boolean) {
prefs.edit { putBoolean("screen.preventSleep", value) }
_preventSleep.value = value
@@ -158,6 +179,11 @@ class SecurePrefs(context: Context) {
_voiceWakeMode.value = mode
}
fun setTalkEnabled(value: Boolean) {
prefs.edit { putBoolean("talk.enabled", value) }
_talkEnabled.value = value
}
private fun loadVoiceWakeMode(): VoiceWakeMode {
val raw = prefs.getString(voiceWakeModeKey, null)
val resolved = VoiceWakeMode.fromRawValue(raw)

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node
package com.clawdis.android
enum class VoiceWakeMode(val rawValue: String) {
Off("off"),

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node
package com.clawdis.android
object WakeWords {
const val maxWords: Int = 32

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.bridge
package com.clawdis.android.bridge
object BonjourEscapes {
fun decode(input: String): String {

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.bridge
package com.clawdis.android.bridge
import android.content.Context
import android.net.ConnectivityManager
@@ -130,20 +130,36 @@ class BridgeDiscovery(
object : NsdManager.ResolveListener {
override fun onResolveFailed(serviceInfo: NsdServiceInfo, errorCode: Int) {}
override fun onServiceResolved(resolved: NsdServiceInfo) {
val host = resolved.host?.hostAddress ?: return
val port = resolved.port
if (port <= 0) return
override fun onServiceResolved(resolved: NsdServiceInfo) {
val host = resolved.host?.hostAddress ?: return
val port = resolved.port
if (port <= 0) return
val rawServiceName = resolved.serviceName
val serviceName = BonjourEscapes.decode(rawServiceName)
val displayName = BonjourEscapes.decode(txt(resolved, "displayName") ?: serviceName)
val id = stableId(serviceName, "local.")
localById[id] = BridgeEndpoint(stableId = id, name = displayName, host = host, port = port)
publish()
}
},
)
val rawServiceName = resolved.serviceName
val serviceName = BonjourEscapes.decode(rawServiceName)
val displayName = BonjourEscapes.decode(txt(resolved, "displayName") ?: serviceName)
val lanHost = txt(resolved, "lanHost")
val tailnetDns = txt(resolved, "tailnetDns")
val gatewayPort = txtInt(resolved, "gatewayPort")
val bridgePort = txtInt(resolved, "bridgePort")
val canvasPort = txtInt(resolved, "canvasPort")
val id = stableId(serviceName, "local.")
localById[id] =
BridgeEndpoint(
stableId = id,
name = displayName,
host = host,
port = port,
lanHost = lanHost,
tailnetDns = tailnetDns,
gatewayPort = gatewayPort,
bridgePort = bridgePort,
canvasPort = canvasPort,
)
publish()
}
},
)
}
private fun publish() {
@@ -189,6 +205,10 @@ class BridgeDiscovery(
}
}
private fun txtInt(info: NsdServiceInfo, key: String): Int? {
return txt(info, key)?.toIntOrNull()
}
private suspend fun refreshUnicast(domain: String) {
val ptrName = "${serviceType}${domain}"
val ptrMsg = lookupUnicastMessage(ptrName, Type.PTR) ?: return
@@ -227,8 +247,24 @@ class BridgeDiscovery(
}
val instanceName = BonjourEscapes.decode(decodeInstanceName(instanceFqdn, domain))
val displayName = BonjourEscapes.decode(txtValue(txt, "displayName") ?: instanceName)
val lanHost = txtValue(txt, "lanHost")
val tailnetDns = txtValue(txt, "tailnetDns")
val gatewayPort = txtIntValue(txt, "gatewayPort")
val bridgePort = txtIntValue(txt, "bridgePort")
val canvasPort = txtIntValue(txt, "canvasPort")
val id = stableId(instanceName, domain)
next[id] = BridgeEndpoint(stableId = id, name = displayName, host = host, port = port)
next[id] =
BridgeEndpoint(
stableId = id,
name = displayName,
host = host,
port = port,
lanHost = lanHost,
tailnetDns = tailnetDns,
gatewayPort = gatewayPort,
bridgePort = bridgePort,
canvasPort = canvasPort,
)
}
unicastById.clear()
@@ -434,6 +470,10 @@ class BridgeDiscovery(
return null
}
private fun txtIntValue(records: List<TXTRecord>, key: String): Int? {
return txtValue(records, key)?.toIntOrNull()
}
private fun decodeDnsTxtString(raw: String): String {
// dnsjava treats TXT as opaque bytes and decodes as ISO-8859-1 to preserve bytes.
// Our TXT payload is UTF-8 (written by the gateway), so re-decode when possible.

View File

@@ -1,10 +1,15 @@
package com.steipete.clawdis.node.bridge
package com.clawdis.android.bridge
data class BridgeEndpoint(
val stableId: String,
val name: String,
val host: String,
val port: Int,
val lanHost: String? = null,
val tailnetDns: String? = null,
val gatewayPort: Int? = null,
val bridgePort: Int? = null,
val canvasPort: Int? = null,
) {
companion object {
fun manual(host: String, port: Int): BridgeEndpoint =
@@ -16,4 +21,3 @@ data class BridgeEndpoint(
)
}
}

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.bridge
package com.clawdis.android.bridge
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
@@ -37,21 +37,21 @@ class BridgePairingClient {
withContext(Dispatchers.IO) {
val socket = Socket()
socket.tcpNoDelay = true
socket.connect(InetSocketAddress(endpoint.host, endpoint.port), 8_000)
socket.soTimeout = 60_000
val reader = BufferedReader(InputStreamReader(socket.getInputStream(), Charsets.UTF_8))
val writer = BufferedWriter(OutputStreamWriter(socket.getOutputStream(), Charsets.UTF_8))
fun send(line: String) {
writer.write(line)
writer.write("\n")
writer.flush()
}
fun sendJson(obj: JsonObject) = send(obj.toString())
try {
socket.connect(InetSocketAddress(endpoint.host, endpoint.port), 8_000)
socket.soTimeout = 60_000
val reader = BufferedReader(InputStreamReader(socket.getInputStream(), Charsets.UTF_8))
val writer = BufferedWriter(OutputStreamWriter(socket.getOutputStream(), Charsets.UTF_8))
fun send(line: String) {
writer.write(line)
writer.write("\n")
writer.flush()
}
fun sendJson(obj: JsonObject) = send(obj.toString())
sendJson(
buildJsonObject {
put("type", JsonPrimitive("hello"))
@@ -111,6 +111,9 @@ class BridgePairingClient {
}
else -> PairResult(ok = false, token = null, error = "unexpected bridge response")
}
} catch (e: Exception) {
val message = e.message?.trim().orEmpty().ifEmpty { "gateway unreachable" }
PairResult(ok = false, token = null, error = message)
} finally {
try {
socket.close()

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.bridge
package com.clawdis.android.bridge
import kotlinx.coroutines.CompletableDeferred
import kotlinx.coroutines.CoroutineScope
@@ -11,6 +11,7 @@ import kotlinx.coroutines.launch
import kotlinx.coroutines.sync.Mutex
import kotlinx.coroutines.sync.withLock
import kotlinx.coroutines.withContext
import com.clawdis.android.BuildConfig
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonArray
import kotlinx.serialization.json.JsonObject
@@ -23,6 +24,7 @@ import java.io.BufferedWriter
import java.io.InputStreamReader
import java.io.OutputStreamWriter
import java.net.InetSocketAddress
import java.net.URI
import java.net.Socket
import java.util.UUID
import java.util.concurrent.ConcurrentHashMap
@@ -73,8 +75,17 @@ class BridgeSession(
}
}
suspend fun updateHello(hello: Hello) {
val target = desired ?: return
desired = target.first to hello
val conn = currentConnection ?: return
conn.sendJson(buildHelloJson(hello))
}
fun disconnect() {
desired = null
// Unblock connectOnce() read loop. Coroutine cancellation alone won't interrupt BufferedReader.readLine().
currentConnection?.closeQuietly()
scope.launch(Dispatchers.IO) {
job?.cancelAndJoin()
job = null
@@ -192,20 +203,7 @@ class BridgeSession(
currentConnection = conn
try {
conn.sendJson(
buildJsonObject {
put("type", JsonPrimitive("hello"))
put("nodeId", JsonPrimitive(hello.nodeId))
hello.displayName?.let { put("displayName", JsonPrimitive(it)) }
hello.token?.let { put("token", JsonPrimitive(it)) }
hello.platform?.let { put("platform", JsonPrimitive(it)) }
hello.version?.let { put("version", JsonPrimitive(it)) }
hello.deviceFamily?.let { put("deviceFamily", JsonPrimitive(it)) }
hello.modelIdentifier?.let { put("modelIdentifier", JsonPrimitive(it)) }
hello.caps?.let { put("caps", JsonArray(it.map(::JsonPrimitive))) }
hello.commands?.let { put("commands", JsonArray(it.map(::JsonPrimitive))) }
},
)
conn.sendJson(buildHelloJson(hello))
val firstLine = reader.readLine() ?: throw IllegalStateException("bridge closed connection")
val first = json.parseToJsonElement(firstLine).asObjectOrNull()
@@ -213,7 +211,17 @@ class BridgeSession(
when (first["type"].asStringOrNull()) {
"hello-ok" -> {
val name = first["serverName"].asStringOrNull() ?: "Bridge"
canvasHostUrl = first["canvasHostUrl"].asStringOrNull()?.trim()?.ifEmpty { null }
val rawCanvasUrl = first["canvasHostUrl"].asStringOrNull()?.trim()?.ifEmpty { null }
canvasHostUrl = normalizeCanvasHostUrl(rawCanvasUrl, endpoint)
if (BuildConfig.DEBUG) {
// Local JVM unit tests use android.jar stubs; Log.d can throw "not mocked".
runCatching {
android.util.Log.d(
"ClawdisBridge",
"canvasHostUrl resolved=${canvasHostUrl ?: "none"} (raw=${rawCanvasUrl ?: "none"})",
)
}
}
onConnected(name, conn.remoteAddress)
}
"error" -> {
@@ -292,6 +300,51 @@ class BridgeSession(
conn.closeQuietly()
}
}
private fun buildHelloJson(hello: Hello): JsonObject =
buildJsonObject {
put("type", JsonPrimitive("hello"))
put("nodeId", JsonPrimitive(hello.nodeId))
hello.displayName?.let { put("displayName", JsonPrimitive(it)) }
hello.token?.let { put("token", JsonPrimitive(it)) }
hello.platform?.let { put("platform", JsonPrimitive(it)) }
hello.version?.let { put("version", JsonPrimitive(it)) }
hello.deviceFamily?.let { put("deviceFamily", JsonPrimitive(it)) }
hello.modelIdentifier?.let { put("modelIdentifier", JsonPrimitive(it)) }
hello.caps?.let { put("caps", JsonArray(it.map(::JsonPrimitive))) }
hello.commands?.let { put("commands", JsonArray(it.map(::JsonPrimitive))) }
}
private fun normalizeCanvasHostUrl(raw: String?, endpoint: BridgeEndpoint): String? {
val trimmed = raw?.trim().orEmpty()
val parsed = trimmed.takeIf { it.isNotBlank() }?.let { runCatching { URI(it) }.getOrNull() }
val host = parsed?.host?.trim().orEmpty()
val port = parsed?.port ?: -1
val scheme = parsed?.scheme?.trim().orEmpty().ifBlank { "http" }
if (trimmed.isNotBlank() && !isLoopbackHost(host)) {
return trimmed
}
val fallbackHost =
endpoint.tailnetDns?.trim().takeIf { !it.isNullOrEmpty() }
?: endpoint.lanHost?.trim().takeIf { !it.isNullOrEmpty() }
?: endpoint.host.trim()
if (fallbackHost.isEmpty()) return trimmed.ifBlank { null }
val fallbackPort = endpoint.canvasPort ?: if (port > 0) port else 18793
val formattedHost = if (fallbackHost.contains(":")) "[${fallbackHost}]" else fallbackHost
return "$scheme://$formattedHost:$fallbackPort"
}
private fun isLoopbackHost(raw: String?): Boolean {
val host = raw?.trim()?.lowercase().orEmpty()
if (host.isEmpty()) return false
if (host == "localhost") return true
if (host == "::1") return true
if (host == "0.0.0.0" || host == "::") return true
return host.startsWith("127.")
}
}
private fun JsonElement?.asObjectOrNull(): JsonObject? = this as? JsonObject

View File

@@ -1,6 +1,6 @@
package com.steipete.clawdis.node.chat
package com.clawdis.android.chat
import com.steipete.clawdis.node.bridge.BridgeSession
import com.clawdis.android.bridge.BridgeSession
import java.util.UUID
import java.util.concurrent.ConcurrentHashMap
import kotlinx.coroutines.CoroutineScope
@@ -361,10 +361,12 @@ class ChatController(
val ts = payload["ts"].asLongOrNull() ?: System.currentTimeMillis()
if (phase == "start") {
val args = data?.get("args").asObjectOrNull()
pendingToolCallsById[toolCallId] =
ChatPendingToolCall(
toolCallId = toolCallId,
name = name,
args = args,
startedAtMs = ts,
isError = null,
)
@@ -469,7 +471,8 @@ class ChatController(
val key = obj["key"].asStringOrNull()?.trim().orEmpty()
if (key.isEmpty()) return@mapNotNull null
val updatedAt = obj["updatedAt"].asLongOrNull()
ChatSessionEntry(key = key, updatedAtMs = updatedAt)
val displayName = obj["displayName"].asStringOrNull()?.trim()
ChatSessionEntry(key = key, updatedAtMs = updatedAt, displayName = displayName)
}
}

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.chat
package com.clawdis.android.chat
data class ChatMessage(
val id: String,
@@ -18,6 +18,7 @@ data class ChatMessageContent(
data class ChatPendingToolCall(
val toolCallId: String,
val name: String,
val args: kotlinx.serialization.json.JsonObject? = null,
val startedAtMs: Long,
val isError: Boolean? = null,
)
@@ -25,6 +26,7 @@ data class ChatPendingToolCall(
data class ChatSessionEntry(
val key: String,
val updatedAtMs: Long?,
val displayName: String? = null,
)
data class ChatHistory(

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.node
package com.clawdis.android.node
import android.Manifest
import android.content.Context
@@ -20,7 +20,7 @@ import androidx.camera.video.VideoRecordEvent
import androidx.core.content.ContextCompat
import androidx.core.content.ContextCompat.checkSelfPermission
import androidx.core.graphics.scale
import com.steipete.clawdis.node.PermissionRequester
import com.clawdis.android.PermissionRequester
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.suspendCancellableCoroutine
import kotlinx.coroutines.withTimeout
@@ -28,6 +28,7 @@ import kotlinx.coroutines.withContext
import java.io.ByteArrayOutputStream
import java.io.File
import java.util.concurrent.Executor
import kotlin.math.roundToInt
import kotlin.coroutines.resume
import kotlin.coroutines.resumeWithException
@@ -99,14 +100,36 @@ class CameraCaptureManager(private val context: Context) {
decoded
}
val out = ByteArrayOutputStream()
val jpegQuality = (quality * 100.0).toInt().coerceIn(10, 100)
if (!scaled.compress(Bitmap.CompressFormat.JPEG, jpegQuality, out)) {
throw IllegalStateException("UNAVAILABLE: failed to encode JPEG")
}
val base64 = Base64.encodeToString(out.toByteArray(), Base64.NO_WRAP)
val maxPayloadBytes = 5 * 1024 * 1024
// Base64 inflates payloads by ~4/3; cap encoded bytes so the payload stays under 5MB (API limit).
val maxEncodedBytes = (maxPayloadBytes / 4) * 3
val result =
JpegSizeLimiter.compressToLimit(
initialWidth = scaled.width,
initialHeight = scaled.height,
startQuality = (quality * 100.0).roundToInt().coerceIn(10, 100),
maxBytes = maxEncodedBytes,
encode = { width, height, q ->
val bitmap =
if (width == scaled.width && height == scaled.height) {
scaled
} else {
scaled.scale(width, height)
}
val out = ByteArrayOutputStream()
if (!bitmap.compress(Bitmap.CompressFormat.JPEG, q, out)) {
if (bitmap !== scaled) bitmap.recycle()
throw IllegalStateException("UNAVAILABLE: failed to encode JPEG")
}
if (bitmap !== scaled) {
bitmap.recycle()
}
out.toByteArray()
},
)
val base64 = Base64.encodeToString(result.bytes, Base64.NO_WRAP)
Payload(
"""{"format":"jpg","base64":"$base64","width":${scaled.width},"height":${scaled.height}}""",
"""{"format":"jpg","base64":"$base64","width":${result.width},"height":${result.height}}""",
)
}

View File

@@ -1,8 +1,9 @@
package com.steipete.clawdis.node.node
package com.clawdis.android.node
import android.graphics.Bitmap
import android.graphics.Canvas
import android.os.Looper
import android.util.Log
import android.webkit.WebView
import androidx.core.graphics.createBitmap
import androidx.core.graphics.scale
@@ -16,6 +17,7 @@ import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonElement
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import com.clawdis.android.BuildConfig
import kotlin.coroutines.resume
class CanvasController {
@@ -81,8 +83,14 @@ class CanvasController {
val currentUrl = url
withWebViewOnMain { wv ->
if (currentUrl == null) {
if (BuildConfig.DEBUG) {
Log.d("ClawdisCanvas", "load scaffold: $scaffoldAssetUrl")
}
wv.loadUrl(scaffoldAssetUrl)
} else {
if (BuildConfig.DEBUG) {
Log.d("ClawdisCanvas", "load url: $currentUrl")
}
wv.loadUrl(currentUrl)
}
}

View File

@@ -0,0 +1,61 @@
package com.clawdis.android.node
import kotlin.math.max
import kotlin.math.min
import kotlin.math.roundToInt
internal data class JpegSizeLimiterResult(
val bytes: ByteArray,
val width: Int,
val height: Int,
val quality: Int,
)
internal object JpegSizeLimiter {
fun compressToLimit(
initialWidth: Int,
initialHeight: Int,
startQuality: Int,
maxBytes: Int,
minQuality: Int = 20,
minSize: Int = 256,
scaleStep: Double = 0.85,
maxScaleAttempts: Int = 6,
maxQualityAttempts: Int = 6,
encode: (width: Int, height: Int, quality: Int) -> ByteArray,
): JpegSizeLimiterResult {
require(initialWidth > 0 && initialHeight > 0) { "Invalid image size" }
require(maxBytes > 0) { "Invalid maxBytes" }
var width = initialWidth
var height = initialHeight
val clampedStartQuality = startQuality.coerceIn(minQuality, 100)
var best = JpegSizeLimiterResult(bytes = encode(width, height, clampedStartQuality), width = width, height = height, quality = clampedStartQuality)
if (best.bytes.size <= maxBytes) return best
repeat(maxScaleAttempts) {
var quality = clampedStartQuality
repeat(maxQualityAttempts) {
val bytes = encode(width, height, quality)
best = JpegSizeLimiterResult(bytes = bytes, width = width, height = height, quality = quality)
if (bytes.size <= maxBytes) return best
if (quality <= minQuality) return@repeat
quality = max(minQuality, (quality * 0.75).roundToInt())
}
val minScale = (minSize.toDouble() / min(width, height).toDouble()).coerceAtMost(1.0)
val nextScale = max(scaleStep, minScale)
val nextWidth = max(minSize, (width * nextScale).roundToInt())
val nextHeight = max(minSize, (height * nextScale).roundToInt())
if (nextWidth == width && nextHeight == height) return@repeat
width = min(nextWidth, width)
height = min(nextHeight, height)
}
if (best.bytes.size > maxBytes) {
throw IllegalStateException("CAMERA_TOO_LARGE: ${best.bytes.size} bytes > $maxBytes bytes")
}
return best
}
}

View File

@@ -0,0 +1,98 @@
package com.clawdis.android.node
import android.annotation.SuppressLint
import android.content.Context
import android.location.Location
import android.location.LocationManager
import android.os.CancellationSignal
import java.time.Instant
import java.time.format.DateTimeFormatter
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import kotlinx.coroutines.withTimeout
import kotlin.coroutines.resume
import kotlin.coroutines.resumeWithException
import kotlinx.coroutines.suspendCancellableCoroutine
class LocationCaptureManager(private val context: Context) {
data class Payload(val payloadJson: String)
suspend fun getLocation(
desiredProviders: List<String>,
maxAgeMs: Long?,
timeoutMs: Long,
isPrecise: Boolean,
): Payload =
withContext(Dispatchers.Main) {
val manager = context.getSystemService(Context.LOCATION_SERVICE) as LocationManager
if (!manager.isProviderEnabled(LocationManager.GPS_PROVIDER) &&
!manager.isProviderEnabled(LocationManager.NETWORK_PROVIDER)
) {
throw IllegalStateException("LOCATION_UNAVAILABLE: no location providers enabled")
}
val cached = bestLastKnown(manager, desiredProviders, maxAgeMs)
val location =
cached ?: requestCurrent(manager, desiredProviders, timeoutMs)
val timestamp = DateTimeFormatter.ISO_INSTANT.format(Instant.ofEpochMilli(location.time))
val source = location.provider
val altitudeMeters = if (location.hasAltitude()) location.altitude else null
val speedMps = if (location.hasSpeed()) location.speed.toDouble() else null
val headingDeg = if (location.hasBearing()) location.bearing.toDouble() else null
Payload(
buildString {
append("{\"lat\":")
append(location.latitude)
append(",\"lon\":")
append(location.longitude)
append(",\"accuracyMeters\":")
append(location.accuracy.toDouble())
if (altitudeMeters != null) append(",\"altitudeMeters\":").append(altitudeMeters)
if (speedMps != null) append(",\"speedMps\":").append(speedMps)
if (headingDeg != null) append(",\"headingDeg\":").append(headingDeg)
append(",\"timestamp\":\"").append(timestamp).append('"')
append(",\"isPrecise\":").append(isPrecise)
append(",\"source\":\"").append(source).append('"')
append('}')
},
)
}
private fun bestLastKnown(
manager: LocationManager,
providers: List<String>,
maxAgeMs: Long?,
): Location? {
val now = System.currentTimeMillis()
val candidates =
providers.mapNotNull { provider -> manager.getLastKnownLocation(provider) }
val freshest = candidates.maxByOrNull { it.time } ?: return null
if (maxAgeMs != null && now - freshest.time > maxAgeMs) return null
return freshest
}
@SuppressLint("MissingPermission")
private suspend fun requestCurrent(
manager: LocationManager,
providers: List<String>,
timeoutMs: Long,
): Location {
val resolved =
providers.firstOrNull { manager.isProviderEnabled(it) }
?: throw IllegalStateException("LOCATION_UNAVAILABLE: no providers available")
return withTimeout(timeoutMs.coerceAtLeast(1)) {
suspendCancellableCoroutine { cont ->
val signal = CancellationSignal()
cont.invokeOnCancellation { signal.cancel() }
manager.getCurrentLocation(resolved, signal, context.mainExecutor) { location ->
if (location != null) {
cont.resume(location)
} else {
cont.resumeWithException(IllegalStateException("LOCATION_UNAVAILABLE: no fix"))
}
}
}
}
}
}

View File

@@ -1,11 +1,11 @@
package com.steipete.clawdis.node.node
package com.clawdis.android.node
import android.content.Context
import android.hardware.display.DisplayManager
import android.media.MediaRecorder
import android.media.projection.MediaProjectionManager
import android.util.Base64
import com.steipete.clawdis.node.ScreenCaptureRequester
import com.clawdis.android.ScreenCaptureRequester
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.delay
import kotlinx.coroutines.withContext
@@ -16,13 +16,13 @@ class ScreenRecordManager(private val context: Context) {
data class Payload(val payloadJson: String)
@Volatile private var screenCaptureRequester: ScreenCaptureRequester? = null
@Volatile private var permissionRequester: com.steipete.clawdis.node.PermissionRequester? = null
@Volatile private var permissionRequester: com.clawdis.android.PermissionRequester? = null
fun attachScreenCaptureRequester(requester: ScreenCaptureRequester) {
screenCaptureRequester = requester
}
fun attachPermissionRequester(requester: com.steipete.clawdis.node.PermissionRequester) {
fun attachPermissionRequester(requester: com.clawdis.android.PermissionRequester) {
permissionRequester = requester
}

View File

@@ -0,0 +1,230 @@
package com.clawdis.android.node
import android.Manifest
import android.content.Context
import android.content.pm.PackageManager
import android.telephony.SmsManager as AndroidSmsManager
import androidx.core.content.ContextCompat
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonElement
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.jsonObject
import kotlinx.serialization.encodeToString
import com.clawdis.android.PermissionRequester
/**
* Sends SMS messages via the Android SMS API.
* Requires SEND_SMS permission to be granted.
*/
class SmsManager(private val context: Context) {
private val json = JsonConfig
@Volatile private var permissionRequester: PermissionRequester? = null
data class SendResult(
val ok: Boolean,
val to: String,
val message: String?,
val error: String? = null,
val payloadJson: String,
)
internal data class ParsedParams(
val to: String,
val message: String,
)
internal sealed class ParseResult {
data class Ok(val params: ParsedParams) : ParseResult()
data class Error(
val error: String,
val to: String = "",
val message: String? = null,
) : ParseResult()
}
internal data class SendPlan(
val parts: List<String>,
val useMultipart: Boolean,
)
companion object {
internal val JsonConfig = Json { ignoreUnknownKeys = true }
internal fun parseParams(paramsJson: String?, json: Json = JsonConfig): ParseResult {
val params = paramsJson?.trim().orEmpty()
if (params.isEmpty()) {
return ParseResult.Error(error = "INVALID_REQUEST: paramsJSON required")
}
val obj = try {
json.parseToJsonElement(params).jsonObject
} catch (_: Throwable) {
null
}
if (obj == null) {
return ParseResult.Error(error = "INVALID_REQUEST: expected JSON object")
}
val to = (obj["to"] as? JsonPrimitive)?.content?.trim().orEmpty()
val message = (obj["message"] as? JsonPrimitive)?.content.orEmpty()
if (to.isEmpty()) {
return ParseResult.Error(
error = "INVALID_REQUEST: 'to' phone number required",
message = message,
)
}
if (message.isEmpty()) {
return ParseResult.Error(
error = "INVALID_REQUEST: 'message' text required",
to = to,
)
}
return ParseResult.Ok(ParsedParams(to = to, message = message))
}
internal fun buildSendPlan(
message: String,
divider: (String) -> List<String>,
): SendPlan {
val parts = divider(message).ifEmpty { listOf(message) }
return SendPlan(parts = parts, useMultipart = parts.size > 1)
}
internal fun buildPayloadJson(
json: Json = JsonConfig,
ok: Boolean,
to: String,
error: String?,
): String {
val payload =
mutableMapOf<String, JsonElement>(
"ok" to JsonPrimitive(ok),
"to" to JsonPrimitive(to),
)
if (!ok) {
payload["error"] = JsonPrimitive(error ?: "SMS_SEND_FAILED")
}
return json.encodeToString(JsonObject.serializer(), JsonObject(payload))
}
}
fun hasSmsPermission(): Boolean {
return ContextCompat.checkSelfPermission(
context,
Manifest.permission.SEND_SMS
) == PackageManager.PERMISSION_GRANTED
}
fun canSendSms(): Boolean {
return hasSmsPermission() && hasTelephonyFeature()
}
fun hasTelephonyFeature(): Boolean {
return context.packageManager?.hasSystemFeature(PackageManager.FEATURE_TELEPHONY) == true
}
fun attachPermissionRequester(requester: PermissionRequester) {
permissionRequester = requester
}
/**
* Send an SMS message.
*
* @param paramsJson JSON with "to" (phone number) and "message" (text) fields
* @return SendResult indicating success or failure
*/
suspend fun send(paramsJson: String?): SendResult {
if (!hasTelephonyFeature()) {
return errorResult(
error = "SMS_UNAVAILABLE: telephony not available",
)
}
if (!ensureSmsPermission()) {
return errorResult(
error = "SMS_PERMISSION_REQUIRED: grant SMS permission",
)
}
val parseResult = parseParams(paramsJson, json)
if (parseResult is ParseResult.Error) {
return errorResult(
error = parseResult.error,
to = parseResult.to,
message = parseResult.message,
)
}
val params = (parseResult as ParseResult.Ok).params
return try {
val smsManager = context.getSystemService(AndroidSmsManager::class.java)
?: throw IllegalStateException("SMS_UNAVAILABLE: SmsManager not available")
val plan = buildSendPlan(params.message) { smsManager.divideMessage(it) }
if (plan.useMultipart) {
smsManager.sendMultipartTextMessage(
params.to, // destination
null, // service center (null = default)
ArrayList(plan.parts), // message parts
null, // sent intents
null, // delivery intents
)
} else {
smsManager.sendTextMessage(
params.to, // destination
null, // service center (null = default)
params.message,// message
null, // sent intent
null, // delivery intent
)
}
okResult(to = params.to, message = params.message)
} catch (e: SecurityException) {
errorResult(
error = "SMS_PERMISSION_REQUIRED: ${e.message}",
to = params.to,
message = params.message,
)
} catch (e: Throwable) {
errorResult(
error = "SMS_SEND_FAILED: ${e.message ?: "unknown error"}",
to = params.to,
message = params.message,
)
}
}
private suspend fun ensureSmsPermission(): Boolean {
if (hasSmsPermission()) return true
val requester = permissionRequester ?: return false
val results = requester.requestIfMissing(listOf(Manifest.permission.SEND_SMS))
return results[Manifest.permission.SEND_SMS] == true
}
private fun okResult(to: String, message: String): SendResult {
return SendResult(
ok = true,
to = to,
message = message,
error = null,
payloadJson = buildPayloadJson(json = json, ok = true, to = to, error = null),
)
}
private fun errorResult(error: String, to: String = "", message: String? = null): SendResult {
return SendResult(
ok = false,
to = to,
message = message,
error = error,
payloadJson = buildPayloadJson(json = json, ok = false, to = to, error = error),
)
}
}

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.protocol
package com.clawdis.android.protocol
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive

View File

@@ -1,10 +1,12 @@
package com.steipete.clawdis.node.protocol
package com.clawdis.android.protocol
enum class ClawdisCapability(val rawValue: String) {
Canvas("canvas"),
Camera("camera"),
Screen("screen"),
Sms("sms"),
VoiceWake("voiceWake"),
Location("location"),
}
enum class ClawdisCanvasCommand(val rawValue: String) {
@@ -49,3 +51,21 @@ enum class ClawdisScreenCommand(val rawValue: String) {
const val NamespacePrefix: String = "screen."
}
}
enum class ClawdisSmsCommand(val rawValue: String) {
Send("sms.send"),
;
companion object {
const val NamespacePrefix: String = "sms."
}
}
enum class ClawdisLocationCommand(val rawValue: String) {
Get("location.get"),
;
companion object {
const val NamespacePrefix: String = "location."
}
}

View File

@@ -0,0 +1,222 @@
package com.clawdis.android.tools
import android.content.Context
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonArray
import kotlinx.serialization.json.JsonElement
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
import kotlinx.serialization.json.contentOrNull
@Serializable
private data class ToolDisplayActionSpec(
val label: String? = null,
val detailKeys: List<String>? = null,
)
@Serializable
private data class ToolDisplaySpec(
val emoji: String? = null,
val title: String? = null,
val label: String? = null,
val detailKeys: List<String>? = null,
val actions: Map<String, ToolDisplayActionSpec>? = null,
)
@Serializable
private data class ToolDisplayConfig(
val version: Int? = null,
val fallback: ToolDisplaySpec? = null,
val tools: Map<String, ToolDisplaySpec>? = null,
)
data class ToolDisplaySummary(
val name: String,
val emoji: String,
val title: String,
val label: String,
val verb: String?,
val detail: String?,
) {
val detailLine: String?
get() {
val parts = mutableListOf<String>()
if (!verb.isNullOrBlank()) parts.add(verb)
if (!detail.isNullOrBlank()) parts.add(detail)
return if (parts.isEmpty()) null else parts.joinToString(" · ")
}
val summaryLine: String
get() = if (detailLine != null) "${emoji} ${label}: ${detailLine}" else "${emoji} ${label}"
}
object ToolDisplayRegistry {
private const val CONFIG_ASSET = "tool-display.json"
private val json = Json { ignoreUnknownKeys = true }
@Volatile private var cachedConfig: ToolDisplayConfig? = null
fun resolve(
context: Context,
name: String?,
args: JsonObject?,
meta: String? = null,
): ToolDisplaySummary {
val trimmedName = name?.trim().orEmpty().ifEmpty { "tool" }
val key = trimmedName.lowercase()
val config = loadConfig(context)
val spec = config.tools?.get(key)
val fallback = config.fallback
val emoji = spec?.emoji ?: fallback?.emoji ?: "🧩"
val title = spec?.title ?: titleFromName(trimmedName)
val label = spec?.label ?: trimmedName
val actionRaw = args?.get("action")?.asStringOrNull()?.trim()
val action = actionRaw?.takeIf { it.isNotEmpty() }
val actionSpec = action?.let { spec?.actions?.get(it) }
val verb = normalizeVerb(actionSpec?.label ?: action)
var detail: String? = null
if (key == "read") {
detail = readDetail(args)
} else if (key == "write" || key == "edit" || key == "attach") {
detail = pathDetail(args)
}
val detailKeys = actionSpec?.detailKeys ?: spec?.detailKeys ?: fallback?.detailKeys ?: emptyList()
if (detail == null) {
detail = firstValue(args, detailKeys)
}
if (detail == null) {
detail = meta
}
if (detail != null) {
detail = shortenHomeInString(detail)
}
return ToolDisplaySummary(
name = trimmedName,
emoji = emoji,
title = title,
label = label,
verb = verb,
detail = detail,
)
}
private fun loadConfig(context: Context): ToolDisplayConfig {
val existing = cachedConfig
if (existing != null) return existing
return try {
val jsonString = context.assets.open(CONFIG_ASSET).bufferedReader().use { it.readText() }
val decoded = json.decodeFromString(ToolDisplayConfig.serializer(), jsonString)
cachedConfig = decoded
decoded
} catch (_: Throwable) {
val fallback = ToolDisplayConfig()
cachedConfig = fallback
fallback
}
}
private fun titleFromName(name: String): String {
val cleaned = name.replace("_", " ").trim()
if (cleaned.isEmpty()) return "Tool"
return cleaned
.split(Regex("\\s+"))
.joinToString(" ") { part ->
val upper = part.uppercase()
if (part.length <= 2 && part == upper) part
else upper.firstOrNull()?.toString().orEmpty() + part.lowercase().drop(1)
}
}
private fun normalizeVerb(value: String?): String? {
val trimmed = value?.trim().orEmpty()
if (trimmed.isEmpty()) return null
return trimmed.replace("_", " ")
}
private fun readDetail(args: JsonObject?): String? {
val path = args?.get("path")?.asStringOrNull() ?: return null
val offset = args["offset"].asNumberOrNull()
val limit = args["limit"].asNumberOrNull()
return if (offset != null && limit != null) {
val end = offset + limit
"${path}:${offset.toInt()}-${end.toInt()}"
} else {
path
}
}
private fun pathDetail(args: JsonObject?): String? {
return args?.get("path")?.asStringOrNull()
}
private fun firstValue(args: JsonObject?, keys: List<String>): String? {
for (key in keys) {
val value = valueForPath(args, key)
val rendered = renderValue(value)
if (!rendered.isNullOrBlank()) return rendered
}
return null
}
private fun valueForPath(args: JsonObject?, path: String): JsonElement? {
var current: JsonElement? = args
for (segment in path.split(".")) {
if (segment.isBlank()) return null
val obj = current as? JsonObject ?: return null
current = obj[segment]
}
return current
}
private fun renderValue(value: JsonElement?): String? {
if (value == null) return null
if (value is JsonPrimitive) {
if (value.isString) {
val trimmed = value.contentOrNull?.trim().orEmpty()
if (trimmed.isEmpty()) return null
val firstLine = trimmed.lineSequence().firstOrNull()?.trim().orEmpty()
if (firstLine.isEmpty()) return null
return if (firstLine.length > 160) "${firstLine.take(157)}" else firstLine
}
val raw = value.contentOrNull?.trim().orEmpty()
raw.toBooleanStrictOrNull()?.let { return it.toString() }
raw.toLongOrNull()?.let { return it.toString() }
raw.toDoubleOrNull()?.let { return it.toString() }
}
if (value is JsonArray) {
val items = value.mapNotNull { renderValue(it) }
if (items.isEmpty()) return null
val preview = items.take(3).joinToString(", ")
return if (items.size > 3) "${preview}" else preview
}
return null
}
private fun shortenHomeInString(value: String): String {
val home = System.getProperty("user.home")?.takeIf { it.isNotBlank() }
?: System.getenv("HOME")?.takeIf { it.isNotBlank() }
if (home.isNullOrEmpty()) return value
return value.replace(home, "~")
.replace(Regex("/Users/[^/]+"), "~")
.replace(Regex("/home/[^/]+"), "~")
}
private fun JsonElement?.asStringOrNull(): String? {
val primitive = this as? JsonPrimitive ?: return null
return if (primitive.isString) primitive.contentOrNull else primitive.toString()
}
private fun JsonElement?.asNumberOrNull(): Double? {
val primitive = this as? JsonPrimitive ?: return null
val raw = primitive.contentOrNull ?: return null
return raw.toDoubleOrNull()
}
}

View File

@@ -0,0 +1,44 @@
package com.clawdis.android.ui
import androidx.compose.foundation.background
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableFloatStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.Modifier
import androidx.compose.ui.draw.alpha
import androidx.compose.ui.graphics.Color
import kotlinx.coroutines.delay
@Composable
fun CameraFlashOverlay(
token: Long,
modifier: Modifier = Modifier,
) {
Box(modifier = modifier.fillMaxSize()) {
CameraFlash(token = token)
}
}
@Composable
private fun CameraFlash(token: Long) {
var alpha by remember { mutableFloatStateOf(0f) }
LaunchedEffect(token) {
if (token == 0L) return@LaunchedEffect
alpha = 0.85f
delay(110)
alpha = 0f
}
Box(
modifier =
Modifier
.fillMaxSize()
.alpha(alpha)
.background(Color.White),
)
}

View File

@@ -1,8 +1,8 @@
package com.steipete.clawdis.node.ui
package com.clawdis.android.ui
import androidx.compose.runtime.Composable
import com.steipete.clawdis.node.MainViewModel
import com.steipete.clawdis.node.ui.chat.ChatSheetContent
import com.clawdis.android.MainViewModel
import com.clawdis.android.ui.chat.ChatSheetContent
@Composable
fun ChatSheet(viewModel: MainViewModel) {

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.ui
package com.clawdis.android.ui
import androidx.compose.foundation.isSystemInDarkTheme
import androidx.compose.material3.MaterialTheme

View File

@@ -0,0 +1,444 @@
package com.clawdis.android.ui
import android.annotation.SuppressLint
import android.Manifest
import android.content.pm.PackageManager
import android.graphics.Color
import android.util.Log
import android.view.View
import android.webkit.JavascriptInterface
import android.webkit.ConsoleMessage
import android.webkit.WebChromeClient
import android.webkit.WebView
import android.webkit.WebSettings
import android.webkit.WebResourceError
import android.webkit.WebResourceRequest
import android.webkit.WebResourceResponse
import android.webkit.WebViewClient
import androidx.activity.compose.rememberLauncherForActivityResult
import androidx.activity.result.contract.ActivityResultContracts
import androidx.webkit.WebSettingsCompat
import androidx.webkit.WebViewFeature
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.WindowInsets
import androidx.compose.foundation.layout.WindowInsetsSides
import androidx.compose.foundation.layout.only
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.safeDrawing
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.layout.windowInsetsPadding
import androidx.compose.material3.ExperimentalMaterial3Api
import androidx.compose.material3.FilledTonalIconButton
import androidx.compose.material3.Icon
import androidx.compose.material3.IconButtonDefaults
import androidx.compose.material3.LocalContentColor
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.ModalBottomSheet
import androidx.compose.material3.rememberModalBottomSheetState
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.ChatBubble
import androidx.compose.material.icons.filled.CheckCircle
import androidx.compose.material.icons.filled.Error
import androidx.compose.material.icons.filled.FiberManualRecord
import androidx.compose.material.icons.filled.PhotoCamera
import androidx.compose.material.icons.filled.RecordVoiceOver
import androidx.compose.material.icons.filled.Refresh
import androidx.compose.material.icons.filled.Report
import androidx.compose.material.icons.filled.ScreenShare
import androidx.compose.material.icons.filled.Settings
import androidx.compose.runtime.Composable
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color as ComposeColor
import androidx.compose.ui.graphics.lerp
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.unit.dp
import androidx.compose.ui.viewinterop.AndroidView
import androidx.compose.ui.window.Popup
import androidx.compose.ui.window.PopupProperties
import androidx.core.content.ContextCompat
import com.clawdis.android.CameraHudKind
import com.clawdis.android.MainViewModel
@OptIn(ExperimentalMaterial3Api::class)
@Composable
fun RootScreen(viewModel: MainViewModel) {
var sheet by remember { mutableStateOf<Sheet?>(null) }
val sheetState = rememberModalBottomSheetState(skipPartiallyExpanded = true)
val safeOverlayInsets = WindowInsets.safeDrawing.only(WindowInsetsSides.Top + WindowInsetsSides.Horizontal)
val context = LocalContext.current
val serverName by viewModel.serverName.collectAsState()
val statusText by viewModel.statusText.collectAsState()
val cameraHud by viewModel.cameraHud.collectAsState()
val cameraFlashToken by viewModel.cameraFlashToken.collectAsState()
val screenRecordActive by viewModel.screenRecordActive.collectAsState()
val isForeground by viewModel.isForeground.collectAsState()
val voiceWakeStatusText by viewModel.voiceWakeStatusText.collectAsState()
val talkEnabled by viewModel.talkEnabled.collectAsState()
val talkStatusText by viewModel.talkStatusText.collectAsState()
val talkIsListening by viewModel.talkIsListening.collectAsState()
val talkIsSpeaking by viewModel.talkIsSpeaking.collectAsState()
val seamColorArgb by viewModel.seamColorArgb.collectAsState()
val seamColor = remember(seamColorArgb) { ComposeColor(seamColorArgb) }
val audioPermissionLauncher =
rememberLauncherForActivityResult(ActivityResultContracts.RequestPermission()) { granted ->
if (granted) viewModel.setTalkEnabled(true)
}
val activity =
remember(cameraHud, screenRecordActive, isForeground, statusText, voiceWakeStatusText) {
// Status pill owns transient activity state so it doesn't overlap the connection indicator.
if (!isForeground) {
return@remember StatusActivity(
title = "Foreground required",
icon = Icons.Default.Report,
contentDescription = "Foreground required",
)
}
val lowerStatus = statusText.lowercase()
if (lowerStatus.contains("repair")) {
return@remember StatusActivity(
title = "Repairing…",
icon = Icons.Default.Refresh,
contentDescription = "Repairing",
)
}
if (lowerStatus.contains("pairing") || lowerStatus.contains("approval")) {
return@remember StatusActivity(
title = "Approval pending",
icon = Icons.Default.RecordVoiceOver,
contentDescription = "Approval pending",
)
}
// Avoid duplicating the primary bridge status ("Connecting…") in the activity slot.
if (screenRecordActive) {
return@remember StatusActivity(
title = "Recording screen…",
icon = Icons.Default.ScreenShare,
contentDescription = "Recording screen",
tint = androidx.compose.ui.graphics.Color.Red,
)
}
cameraHud?.let { hud ->
return@remember when (hud.kind) {
CameraHudKind.Photo ->
StatusActivity(
title = hud.message,
icon = Icons.Default.PhotoCamera,
contentDescription = "Taking photo",
)
CameraHudKind.Recording ->
StatusActivity(
title = hud.message,
icon = Icons.Default.FiberManualRecord,
contentDescription = "Recording",
tint = androidx.compose.ui.graphics.Color.Red,
)
CameraHudKind.Success ->
StatusActivity(
title = hud.message,
icon = Icons.Default.CheckCircle,
contentDescription = "Capture finished",
)
CameraHudKind.Error ->
StatusActivity(
title = hud.message,
icon = Icons.Default.Error,
contentDescription = "Capture failed",
tint = androidx.compose.ui.graphics.Color.Red,
)
}
}
if (voiceWakeStatusText.contains("Microphone permission", ignoreCase = true)) {
return@remember StatusActivity(
title = "Mic permission",
icon = Icons.Default.Error,
contentDescription = "Mic permission required",
)
}
if (voiceWakeStatusText == "Paused") {
val suffix = if (!isForeground) " (background)" else ""
return@remember StatusActivity(
title = "Voice Wake paused$suffix",
icon = Icons.Default.RecordVoiceOver,
contentDescription = "Voice Wake paused",
)
}
null
}
val bridgeState =
remember(serverName, statusText) {
when {
serverName != null -> BridgeState.Connected
statusText.contains("connecting", ignoreCase = true) ||
statusText.contains("reconnecting", ignoreCase = true) -> BridgeState.Connecting
statusText.contains("error", ignoreCase = true) -> BridgeState.Error
else -> BridgeState.Disconnected
}
}
val voiceEnabled =
ContextCompat.checkSelfPermission(context, Manifest.permission.RECORD_AUDIO) ==
PackageManager.PERMISSION_GRANTED
Box(modifier = Modifier.fillMaxSize()) {
CanvasView(viewModel = viewModel, modifier = Modifier.fillMaxSize())
}
// Camera flash must be in a Popup to render above the WebView.
Popup(alignment = Alignment.Center, properties = PopupProperties(focusable = false)) {
CameraFlashOverlay(token = cameraFlashToken, modifier = Modifier.fillMaxSize())
}
// Keep the overlay buttons above the WebView canvas (AndroidView), otherwise they may not receive touches.
Popup(alignment = Alignment.TopStart, properties = PopupProperties(focusable = false)) {
StatusPill(
bridge = bridgeState,
voiceEnabled = voiceEnabled,
activity = activity,
onClick = { sheet = Sheet.Settings },
modifier = Modifier.windowInsetsPadding(safeOverlayInsets).padding(start = 12.dp, top = 12.dp),
)
}
Popup(alignment = Alignment.TopEnd, properties = PopupProperties(focusable = false)) {
Column(
modifier = Modifier.windowInsetsPadding(safeOverlayInsets).padding(end = 12.dp, top = 12.dp),
verticalArrangement = Arrangement.spacedBy(10.dp),
horizontalAlignment = Alignment.End,
) {
OverlayIconButton(
onClick = { sheet = Sheet.Chat },
icon = { Icon(Icons.Default.ChatBubble, contentDescription = "Chat") },
)
// Talk mode gets a dedicated side bubble instead of burying it in settings.
val baseOverlay = overlayContainerColor()
val talkContainer =
lerp(
baseOverlay,
seamColor.copy(alpha = baseOverlay.alpha),
if (talkEnabled) 0.35f else 0.22f,
)
val talkContent = if (talkEnabled) seamColor else overlayIconColor()
OverlayIconButton(
onClick = {
val next = !talkEnabled
if (next) {
val micOk =
ContextCompat.checkSelfPermission(context, Manifest.permission.RECORD_AUDIO) ==
PackageManager.PERMISSION_GRANTED
if (!micOk) audioPermissionLauncher.launch(Manifest.permission.RECORD_AUDIO)
viewModel.setTalkEnabled(true)
} else {
viewModel.setTalkEnabled(false)
}
},
containerColor = talkContainer,
contentColor = talkContent,
icon = {
Icon(
Icons.Default.RecordVoiceOver,
contentDescription = "Talk Mode",
)
},
)
OverlayIconButton(
onClick = { sheet = Sheet.Settings },
icon = { Icon(Icons.Default.Settings, contentDescription = "Settings") },
)
}
}
if (talkEnabled) {
Popup(alignment = Alignment.Center, properties = PopupProperties(focusable = false)) {
TalkOrbOverlay(
seamColor = seamColor,
statusText = talkStatusText,
isListening = talkIsListening,
isSpeaking = talkIsSpeaking,
)
}
}
val currentSheet = sheet
if (currentSheet != null) {
ModalBottomSheet(
onDismissRequest = { sheet = null },
sheetState = sheetState,
) {
when (currentSheet) {
Sheet.Chat -> ChatSheet(viewModel = viewModel)
Sheet.Settings -> SettingsSheet(viewModel = viewModel)
}
}
}
}
private enum class Sheet {
Chat,
Settings,
}
@Composable
private fun OverlayIconButton(
onClick: () -> Unit,
icon: @Composable () -> Unit,
containerColor: ComposeColor? = null,
contentColor: ComposeColor? = null,
) {
FilledTonalIconButton(
onClick = onClick,
modifier = Modifier.size(44.dp),
colors =
IconButtonDefaults.filledTonalIconButtonColors(
containerColor = containerColor ?: overlayContainerColor(),
contentColor = contentColor ?: overlayIconColor(),
),
) {
icon()
}
}
@SuppressLint("SetJavaScriptEnabled")
@Composable
private fun CanvasView(viewModel: MainViewModel, modifier: Modifier = Modifier) {
val context = LocalContext.current
val isDebuggable = (context.applicationInfo.flags and android.content.pm.ApplicationInfo.FLAG_DEBUGGABLE) != 0
AndroidView(
modifier = modifier,
factory = {
WebView(context).apply {
settings.javaScriptEnabled = true
// Some embedded web UIs (incl. the "background website") use localStorage/sessionStorage.
settings.domStorageEnabled = true
settings.mixedContentMode = WebSettings.MIXED_CONTENT_COMPATIBILITY_MODE
if (WebViewFeature.isFeatureSupported(WebViewFeature.FORCE_DARK)) {
WebSettingsCompat.setForceDark(settings, WebSettingsCompat.FORCE_DARK_OFF)
}
if (WebViewFeature.isFeatureSupported(WebViewFeature.ALGORITHMIC_DARKENING)) {
WebSettingsCompat.setAlgorithmicDarkeningAllowed(settings, false)
}
if (isDebuggable) {
Log.d("ClawdisWebView", "userAgent: ${settings.userAgentString}")
}
isScrollContainer = true
overScrollMode = View.OVER_SCROLL_IF_CONTENT_SCROLLS
isVerticalScrollBarEnabled = true
isHorizontalScrollBarEnabled = true
webViewClient =
object : WebViewClient() {
override fun onReceivedError(
view: WebView,
request: WebResourceRequest,
error: WebResourceError,
) {
if (!isDebuggable) return
if (!request.isForMainFrame) return
Log.e("ClawdisWebView", "onReceivedError: ${error.errorCode} ${error.description} ${request.url}")
}
override fun onReceivedHttpError(
view: WebView,
request: WebResourceRequest,
errorResponse: WebResourceResponse,
) {
if (!isDebuggable) return
if (!request.isForMainFrame) return
Log.e(
"ClawdisWebView",
"onReceivedHttpError: ${errorResponse.statusCode} ${errorResponse.reasonPhrase} ${request.url}",
)
}
override fun onPageFinished(view: WebView, url: String?) {
if (isDebuggable) {
Log.d("ClawdisWebView", "onPageFinished: $url")
}
viewModel.canvas.onPageFinished()
}
override fun onRenderProcessGone(
view: WebView,
detail: android.webkit.RenderProcessGoneDetail,
): Boolean {
if (isDebuggable) {
Log.e(
"ClawdisWebView",
"onRenderProcessGone didCrash=${detail.didCrash()} priorityAtExit=${detail.rendererPriorityAtExit()}",
)
}
return true
}
}
webChromeClient =
object : WebChromeClient() {
override fun onConsoleMessage(consoleMessage: ConsoleMessage?): Boolean {
if (!isDebuggable) return false
val msg = consoleMessage ?: return false
Log.d(
"ClawdisWebView",
"console ${msg.messageLevel()} @ ${msg.sourceId()}:${msg.lineNumber()} ${msg.message()}",
)
return false
}
}
// Use default layer/background; avoid forcing a black fill over WebView content.
val a2uiBridge =
CanvasA2UIActionBridge { payload ->
viewModel.handleCanvasA2UIActionFromWebView(payload)
}
addJavascriptInterface(a2uiBridge, CanvasA2UIActionBridge.interfaceName)
addJavascriptInterface(
CanvasA2UIActionLegacyBridge(a2uiBridge),
CanvasA2UIActionLegacyBridge.interfaceName,
)
viewModel.canvas.attach(this)
}
},
)
}
private class CanvasA2UIActionBridge(private val onMessage: (String) -> Unit) {
@JavascriptInterface
fun postMessage(payload: String?) {
val msg = payload?.trim().orEmpty()
if (msg.isEmpty()) return
onMessage(msg)
}
companion object {
const val interfaceName: String = "clawdisCanvasA2UIAction"
}
}
private class CanvasA2UIActionLegacyBridge(private val bridge: CanvasA2UIActionBridge) {
@JavascriptInterface
fun canvasAction(payload: String?) {
bridge.postMessage(payload)
}
@JavascriptInterface
fun postMessage(payload: String?) {
bridge.postMessage(payload)
}
companion object {
const val interfaceName: String = "Android"
}
}

View File

@@ -1,7 +1,12 @@
package com.steipete.clawdis.node.ui
package com.clawdis.android.ui
import android.Manifest
import android.content.Context
import android.content.Intent
import android.content.pm.PackageManager
import android.net.Uri
import android.os.Build
import android.provider.Settings
import androidx.activity.compose.rememberLauncherForActivityResult
import androidx.activity.result.contract.ActivityResultContracts
import androidx.compose.animation.AnimatedVisibility
@@ -41,14 +46,17 @@ import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.Modifier
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.text.style.TextAlign
import androidx.compose.ui.unit.dp
import androidx.core.content.ContextCompat
import com.steipete.clawdis.node.MainViewModel
import com.steipete.clawdis.node.NodeForegroundService
import com.steipete.clawdis.node.VoiceWakeMode
import com.clawdis.android.BuildConfig
import com.clawdis.android.LocationMode
import com.clawdis.android.MainViewModel
import com.clawdis.android.NodeForegroundService
import com.clawdis.android.VoiceWakeMode
@Composable
fun SettingsSheet(viewModel: MainViewModel) {
@@ -56,6 +64,8 @@ fun SettingsSheet(viewModel: MainViewModel) {
val instanceId by viewModel.instanceId.collectAsState()
val displayName by viewModel.displayName.collectAsState()
val cameraEnabled by viewModel.cameraEnabled.collectAsState()
val locationMode by viewModel.locationMode.collectAsState()
val locationPreciseEnabled by viewModel.locationPreciseEnabled.collectAsState()
val preventSleep by viewModel.preventSleep.collectAsState()
val wakeWords by viewModel.wakeWords.collectAsState()
val voiceWakeMode by viewModel.voiceWakeMode.collectAsState()
@@ -74,6 +84,22 @@ fun SettingsSheet(viewModel: MainViewModel) {
val listState = rememberLazyListState()
val (wakeWordsText, setWakeWordsText) = remember { mutableStateOf("") }
val (advancedExpanded, setAdvancedExpanded) = remember { mutableStateOf(false) }
val deviceModel =
remember {
listOfNotNull(Build.MANUFACTURER, Build.MODEL)
.joinToString(" ")
.trim()
.ifEmpty { "Android" }
}
val appVersion =
remember {
val versionName = BuildConfig.VERSION_NAME.trim().ifEmpty { "dev" }
if (BuildConfig.DEBUG && !versionName.contains("dev", ignoreCase = true)) {
"$versionName-dev"
} else {
versionName
}
}
LaunchedEffect(wakeWords) { setWakeWordsText(wakeWords.joinToString(", ")) }
@@ -83,11 +109,63 @@ fun SettingsSheet(viewModel: MainViewModel) {
viewModel.setCameraEnabled(cameraOk)
}
var pendingLocationMode by remember { mutableStateOf<LocationMode?>(null) }
var pendingPreciseToggle by remember { mutableStateOf(false) }
val locationPermissionLauncher =
rememberLauncherForActivityResult(ActivityResultContracts.RequestMultiplePermissions()) { perms ->
val fineOk = perms[Manifest.permission.ACCESS_FINE_LOCATION] == true
val coarseOk = perms[Manifest.permission.ACCESS_COARSE_LOCATION] == true
val granted = fineOk || coarseOk
val requestedMode = pendingLocationMode
pendingLocationMode = null
if (pendingPreciseToggle) {
pendingPreciseToggle = false
viewModel.setLocationPreciseEnabled(fineOk)
return@rememberLauncherForActivityResult
}
if (!granted) {
viewModel.setLocationMode(LocationMode.Off)
return@rememberLauncherForActivityResult
}
if (requestedMode != null) {
viewModel.setLocationMode(requestedMode)
if (requestedMode == LocationMode.Always && Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q) {
val backgroundOk =
ContextCompat.checkSelfPermission(context, Manifest.permission.ACCESS_BACKGROUND_LOCATION) ==
PackageManager.PERMISSION_GRANTED
if (!backgroundOk) {
openAppSettings(context)
}
}
}
}
val audioPermissionLauncher =
rememberLauncherForActivityResult(ActivityResultContracts.RequestPermission()) { _ ->
// Status text is handled by NodeRuntime.
}
val smsPermissionAvailable =
remember {
context.packageManager?.hasSystemFeature(PackageManager.FEATURE_TELEPHONY) == true
}
var smsPermissionGranted by
remember {
mutableStateOf(
ContextCompat.checkSelfPermission(context, Manifest.permission.SEND_SMS) ==
PackageManager.PERMISSION_GRANTED,
)
}
val smsPermissionLauncher =
rememberLauncherForActivityResult(ActivityResultContracts.RequestPermission()) { granted ->
smsPermissionGranted = granted
viewModel.refreshBridgeHello()
}
fun setCameraEnabledChecked(checked: Boolean) {
if (!checked) {
viewModel.setCameraEnabled(false)
@@ -104,6 +182,47 @@ fun SettingsSheet(viewModel: MainViewModel) {
}
}
fun requestLocationPermissions(targetMode: LocationMode) {
val fineOk =
ContextCompat.checkSelfPermission(context, Manifest.permission.ACCESS_FINE_LOCATION) ==
PackageManager.PERMISSION_GRANTED
val coarseOk =
ContextCompat.checkSelfPermission(context, Manifest.permission.ACCESS_COARSE_LOCATION) ==
PackageManager.PERMISSION_GRANTED
if (fineOk || coarseOk) {
viewModel.setLocationMode(targetMode)
if (targetMode == LocationMode.Always && Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q) {
val backgroundOk =
ContextCompat.checkSelfPermission(context, Manifest.permission.ACCESS_BACKGROUND_LOCATION) ==
PackageManager.PERMISSION_GRANTED
if (!backgroundOk) {
openAppSettings(context)
}
}
} else {
pendingLocationMode = targetMode
locationPermissionLauncher.launch(
arrayOf(Manifest.permission.ACCESS_FINE_LOCATION, Manifest.permission.ACCESS_COARSE_LOCATION),
)
}
}
fun setPreciseLocationChecked(checked: Boolean) {
if (!checked) {
viewModel.setLocationPreciseEnabled(false)
return
}
val fineOk =
ContextCompat.checkSelfPermission(context, Manifest.permission.ACCESS_FINE_LOCATION) ==
PackageManager.PERMISSION_GRANTED
if (fineOk) {
viewModel.setLocationPreciseEnabled(true)
} else {
pendingPreciseToggle = true
locationPermissionLauncher.launch(arrayOf(Manifest.permission.ACCESS_FINE_LOCATION))
}
}
val visibleBridges =
if (isConnected && remoteAddress != null) {
bridges.filterNot { "${it.host}:${it.port}" == remoteAddress }
@@ -131,7 +250,7 @@ fun SettingsSheet(viewModel: MainViewModel) {
contentPadding = PaddingValues(16.dp),
verticalArrangement = Arrangement.spacedBy(6.dp),
) {
// Order parity: Node → Bridge → Voice → Camera → Screen.
// Order parity: Node → Bridge → Voice → Camera → Messaging → Location → Screen.
item { Text("Node", style = MaterialTheme.typography.titleSmall) }
item {
OutlinedTextField(
@@ -142,6 +261,8 @@ fun SettingsSheet(viewModel: MainViewModel) {
)
}
item { Text("Instance ID: $instanceId", color = MaterialTheme.colorScheme.onSurfaceVariant) }
item { Text("Device: $deviceModel", color = MaterialTheme.colorScheme.onSurfaceVariant) }
item { Text("Version: $appVersion", color = MaterialTheme.colorScheme.onSurfaceVariant) }
item { HorizontalDivider() }
@@ -181,9 +302,27 @@ fun SettingsSheet(viewModel: MainViewModel) {
item { Text("No bridges found yet.", color = MaterialTheme.colorScheme.onSurfaceVariant) }
} else {
items(items = visibleBridges, key = { it.stableId }) { bridge ->
val detailLines =
buildList {
add("IP: ${bridge.host}:${bridge.port}")
bridge.lanHost?.let { add("LAN: $it") }
bridge.tailnetDns?.let { add("Tailnet: $it") }
if (bridge.gatewayPort != null || bridge.bridgePort != null || bridge.canvasPort != null) {
val gw = bridge.gatewayPort?.toString() ?: ""
val br = (bridge.bridgePort ?: bridge.port).toString()
val canvas = bridge.canvasPort?.toString() ?: ""
add("Ports: gw $gw · bridge $br · canvas $canvas")
}
}
ListItem(
headlineContent = { Text(bridge.name) },
supportingContent = { Text("${bridge.host}:${bridge.port}") },
supportingContent = {
Column(verticalArrangement = Arrangement.spacedBy(2.dp)) {
detailLines.forEach { line ->
Text(line, color = MaterialTheme.colorScheme.onSurfaceVariant)
}
}
},
trailingContent = {
Button(
onClick = {
@@ -343,7 +482,7 @@ fun SettingsSheet(viewModel: MainViewModel) {
Row(horizontalArrangement = Arrangement.spacedBy(12.dp)) {
Button(
onClick = {
val parsed = com.steipete.clawdis.node.WakeWords.parseCommaSeparated(wakeWordsText)
val parsed = com.clawdis.android.WakeWords.parseCommaSeparated(wakeWordsText)
viewModel.setWakeWords(parsed)
},
enabled = isConnected,
@@ -385,6 +524,104 @@ fun SettingsSheet(viewModel: MainViewModel) {
item { HorizontalDivider() }
// Messaging
item { Text("Messaging", style = MaterialTheme.typography.titleSmall) }
item {
val buttonLabel =
when {
!smsPermissionAvailable -> "Unavailable"
smsPermissionGranted -> "Manage"
else -> "Grant"
}
ListItem(
headlineContent = { Text("SMS Permission") },
supportingContent = {
Text(
if (smsPermissionAvailable) {
"Allow the bridge to send SMS from this device."
} else {
"SMS requires a device with telephony hardware."
},
)
},
trailingContent = {
Button(
onClick = {
if (!smsPermissionAvailable) return@Button
if (smsPermissionGranted) {
openAppSettings(context)
} else {
smsPermissionLauncher.launch(Manifest.permission.SEND_SMS)
}
},
enabled = smsPermissionAvailable,
) {
Text(buttonLabel)
}
},
)
}
item { HorizontalDivider() }
// Location
item { Text("Location", style = MaterialTheme.typography.titleSmall) }
item {
Column(verticalArrangement = Arrangement.spacedBy(6.dp), modifier = Modifier.fillMaxWidth()) {
ListItem(
headlineContent = { Text("Off") },
supportingContent = { Text("Disable location sharing.") },
trailingContent = {
RadioButton(
selected = locationMode == LocationMode.Off,
onClick = { viewModel.setLocationMode(LocationMode.Off) },
)
},
)
ListItem(
headlineContent = { Text("While Using") },
supportingContent = { Text("Only while Clawdis is open.") },
trailingContent = {
RadioButton(
selected = locationMode == LocationMode.WhileUsing,
onClick = { requestLocationPermissions(LocationMode.WhileUsing) },
)
},
)
ListItem(
headlineContent = { Text("Always") },
supportingContent = { Text("Allow background location (requires system permission).") },
trailingContent = {
RadioButton(
selected = locationMode == LocationMode.Always,
onClick = { requestLocationPermissions(LocationMode.Always) },
)
},
)
}
}
item {
ListItem(
headlineContent = { Text("Precise Location") },
supportingContent = { Text("Use precise GPS when available.") },
trailingContent = {
Switch(
checked = locationPreciseEnabled,
onCheckedChange = ::setPreciseLocationChecked,
enabled = locationMode != LocationMode.Off,
)
},
)
}
item {
Text(
"Always may require Android Settings to allow background location.",
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
}
item { HorizontalDivider() }
// Screen
item { Text("Screen", style = MaterialTheme.typography.titleSmall) }
item {
@@ -415,3 +652,12 @@ fun SettingsSheet(viewModel: MainViewModel) {
item { Spacer(modifier = Modifier.height(20.dp)) }
}
}
private fun openAppSettings(context: Context) {
val intent =
Intent(
Settings.ACTION_APPLICATION_DETAILS_SETTINGS,
Uri.fromParts("package", context.packageName, null),
)
context.startActivity(intent)
}

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.ui
package com.clawdis.android.ui
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Row
@@ -28,6 +28,7 @@ import androidx.compose.ui.unit.dp
fun StatusPill(
bridge: BridgeState,
voiceEnabled: Boolean,
activity: StatusActivity? = null,
onClick: () -> Unit,
modifier: Modifier = Modifier,
) {
@@ -62,23 +63,49 @@ fun StatusPill(
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
Icon(
imageVector = if (voiceEnabled) Icons.Default.Mic else Icons.Default.MicOff,
contentDescription = if (voiceEnabled) "Voice enabled" else "Voice disabled",
tint =
if (voiceEnabled) {
overlayIconColor()
} else {
MaterialTheme.colorScheme.onSurfaceVariant
},
modifier = Modifier.size(18.dp),
)
if (activity != null) {
Row(
horizontalArrangement = Arrangement.spacedBy(6.dp),
verticalAlignment = Alignment.CenterVertically,
) {
Icon(
imageVector = activity.icon,
contentDescription = activity.contentDescription,
tint = activity.tint ?: overlayIconColor(),
modifier = Modifier.size(18.dp),
)
Text(
text = activity.title,
style = MaterialTheme.typography.labelLarge,
maxLines = 1,
)
}
} else {
Icon(
imageVector = if (voiceEnabled) Icons.Default.Mic else Icons.Default.MicOff,
contentDescription = if (voiceEnabled) "Voice enabled" else "Voice disabled",
tint =
if (voiceEnabled) {
overlayIconColor()
} else {
MaterialTheme.colorScheme.onSurfaceVariant
},
modifier = Modifier.size(18.dp),
)
}
Spacer(modifier = Modifier.width(2.dp))
}
}
}
data class StatusActivity(
val title: String,
val icon: androidx.compose.ui.graphics.vector.ImageVector,
val contentDescription: String,
val tint: Color? = null,
)
enum class BridgeState(val title: String, val color: Color) {
Connected("Connected", Color(0xFF2ECC71)),
Connecting("Connecting…", Color(0xFFF1C40F)),

View File

@@ -0,0 +1,134 @@
package com.clawdis.android.ui
import androidx.compose.animation.core.LinearEasing
import androidx.compose.animation.core.RepeatMode
import androidx.compose.animation.core.animateFloat
import androidx.compose.animation.core.infiniteRepeatable
import androidx.compose.animation.core.rememberInfiniteTransition
import androidx.compose.animation.core.tween
import androidx.compose.foundation.Canvas
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.shape.CircleShape
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.getValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Brush
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.graphics.drawscope.Stroke
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.unit.dp
@Composable
fun TalkOrbOverlay(
seamColor: Color,
statusText: String,
isListening: Boolean,
isSpeaking: Boolean,
modifier: Modifier = Modifier,
) {
val transition = rememberInfiniteTransition(label = "talk-orb")
val t by
transition.animateFloat(
initialValue = 0f,
targetValue = 1f,
animationSpec =
infiniteRepeatable(
animation = tween(durationMillis = 1500, easing = LinearEasing),
repeatMode = RepeatMode.Restart,
),
label = "pulse",
)
val trimmed = statusText.trim()
val showStatus = trimmed.isNotEmpty() && trimmed != "Off"
val phase =
when {
isSpeaking -> "Speaking"
isListening -> "Listening"
else -> "Thinking"
}
Column(
modifier = modifier.padding(24.dp),
horizontalAlignment = Alignment.CenterHorizontally,
verticalArrangement = Arrangement.spacedBy(12.dp),
) {
Box(contentAlignment = Alignment.Center) {
Canvas(modifier = Modifier.size(360.dp)) {
val center = this.center
val baseRadius = size.minDimension * 0.30f
val ring1 = 1.05f + (t * 0.25f)
val ring2 = 1.20f + (t * 0.55f)
val ringAlpha1 = (1f - t) * 0.34f
val ringAlpha2 = (1f - t) * 0.22f
drawCircle(
color = seamColor.copy(alpha = ringAlpha1),
radius = baseRadius * ring1,
center = center,
style = Stroke(width = 3.dp.toPx()),
)
drawCircle(
color = seamColor.copy(alpha = ringAlpha2),
radius = baseRadius * ring2,
center = center,
style = Stroke(width = 3.dp.toPx()),
)
drawCircle(
brush =
Brush.radialGradient(
colors =
listOf(
seamColor.copy(alpha = 0.92f),
seamColor.copy(alpha = 0.40f),
Color.Black.copy(alpha = 0.56f),
),
center = center,
radius = baseRadius * 1.35f,
),
radius = baseRadius,
center = center,
)
drawCircle(
color = seamColor.copy(alpha = 0.34f),
radius = baseRadius,
center = center,
style = Stroke(width = 1.dp.toPx()),
)
}
}
if (showStatus) {
Surface(
color = Color.Black.copy(alpha = 0.40f),
shape = CircleShape,
) {
Text(
text = trimmed,
modifier = Modifier.padding(horizontal = 14.dp, vertical = 8.dp),
color = Color.White.copy(alpha = 0.92f),
style = MaterialTheme.typography.labelLarge,
fontWeight = FontWeight.SemiBold,
)
}
} else {
Text(
text = phase,
color = Color.White.copy(alpha = 0.80f),
style = MaterialTheme.typography.labelLarge,
fontWeight = FontWeight.SemiBold,
)
}
}
}

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.ui.chat
package com.clawdis.android.ui.chat
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
@@ -15,7 +15,6 @@ import androidx.compose.foundation.horizontalScroll
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.ArrowUpward
import androidx.compose.material.icons.filled.AttachFile
import androidx.compose.material.icons.filled.FolderOpen
import androidx.compose.material.icons.filled.Refresh
import androidx.compose.material.icons.filled.Stop
import androidx.compose.material3.ButtonDefaults
@@ -39,10 +38,12 @@ import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.unit.dp
import com.clawdis.android.chat.ChatSessionEntry
@Composable
fun ChatComposer(
sessionKey: String,
sessions: List<ChatSessionEntry>,
healthOk: Boolean,
thinkingLevel: String,
pendingRunCount: Int,
@@ -51,13 +52,18 @@ fun ChatComposer(
onPickImages: () -> Unit,
onRemoveAttachment: (id: String) -> Unit,
onSetThinkingLevel: (level: String) -> Unit,
onShowSessions: () -> Unit,
onSelectSession: (sessionKey: String) -> Unit,
onRefresh: () -> Unit,
onAbort: () -> Unit,
onSend: (text: String) -> Unit,
) {
var input by rememberSaveable { mutableStateOf("") }
var showThinkingMenu by remember { mutableStateOf(false) }
var showSessionMenu by remember { mutableStateOf(false) }
val sessionOptions = resolveSessionChoices(sessionKey, sessions)
val currentSessionLabel =
sessionOptions.firstOrNull { it.key == sessionKey }?.displayName ?: sessionKey
val canSend = pendingRunCount == 0 && (input.trim().isNotEmpty() || attachments.isNotEmpty()) && healthOk
@@ -73,6 +79,34 @@ fun ChatComposer(
horizontalArrangement = Arrangement.spacedBy(8.dp),
verticalAlignment = Alignment.CenterVertically,
) {
Box {
FilledTonalButton(
onClick = { showSessionMenu = true },
contentPadding = ButtonDefaults.ContentPadding,
) {
Text("Session: $currentSessionLabel")
}
DropdownMenu(expanded = showSessionMenu, onDismissRequest = { showSessionMenu = false }) {
for (entry in sessionOptions) {
DropdownMenuItem(
text = { Text(entry.displayName ?: entry.key) },
onClick = {
onSelectSession(entry.key)
showSessionMenu = false
},
trailingIcon = {
if (entry.key == sessionKey) {
Text("")
} else {
Spacer(modifier = Modifier.width(10.dp))
}
},
)
}
}
}
Box {
FilledTonalButton(
onClick = { showThinkingMenu = true },
@@ -91,10 +125,6 @@ fun ChatComposer(
Spacer(modifier = Modifier.weight(1f))
FilledTonalIconButton(onClick = onShowSessions, modifier = Modifier.size(42.dp)) {
Icon(Icons.Default.FolderOpen, contentDescription = "Sessions")
}
FilledTonalIconButton(onClick = onRefresh, modifier = Modifier.size(42.dp)) {
Icon(Icons.Default.Refresh, contentDescription = "Refresh")
}
@@ -118,7 +148,7 @@ fun ChatComposer(
)
Row(modifier = Modifier.fillMaxWidth(), verticalAlignment = Alignment.CenterVertically) {
ConnectionPill(sessionKey = sessionKey, healthOk = healthOk)
ConnectionPill(sessionLabel = currentSessionLabel, healthOk = healthOk)
Spacer(modifier = Modifier.weight(1f))
if (pendingRunCount > 0) {
@@ -156,7 +186,7 @@ fun ChatComposer(
}
@Composable
private fun ConnectionPill(sessionKey: String, healthOk: Boolean) {
private fun ConnectionPill(sessionLabel: String, healthOk: Boolean) {
Surface(
shape = RoundedCornerShape(999.dp),
color = MaterialTheme.colorScheme.surfaceContainerHighest,
@@ -171,7 +201,7 @@ private fun ConnectionPill(sessionKey: String, healthOk: Boolean) {
shape = androidx.compose.foundation.shape.CircleShape,
color = if (healthOk) Color(0xFF2ECC71) else Color(0xFFF39C12),
) {}
Text(sessionKey, style = MaterialTheme.typography.labelSmall)
Text(sessionLabel, style = MaterialTheme.typography.labelSmall)
Text(
if (healthOk) "Connected" else "Connecting…",
style = MaterialTheme.typography.labelSmall,

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.ui.chat
package com.clawdis.android.ui.chat
import android.graphics.BitmapFactory
import android.util.Base64
@@ -17,6 +17,7 @@ import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.graphics.asImageBitmap
import androidx.compose.ui.layout.ContentScale
import androidx.compose.ui.text.AnnotatedString
@@ -31,7 +32,7 @@ import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
@Composable
fun ChatMarkdown(text: String) {
fun ChatMarkdown(text: String, textColor: Color) {
val blocks = remember(text) { splitMarkdown(text) }
val inlineCodeBg = MaterialTheme.colorScheme.surfaceContainerLow
@@ -44,7 +45,7 @@ fun ChatMarkdown(text: String) {
Text(
text = parseInlineMarkdown(trimmed, inlineCodeBg = inlineCodeBg),
style = MaterialTheme.typography.bodyMedium,
color = MaterialTheme.colorScheme.onSurface,
color = textColor,
)
}
is ChatMarkdownBlock.Code -> {

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.ui.chat
package com.clawdis.android.ui.chat
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
@@ -20,8 +20,8 @@ import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.draw.alpha
import androidx.compose.ui.unit.dp
import com.steipete.clawdis.node.chat.ChatMessage
import com.steipete.clawdis.node.chat.ChatPendingToolCall
import com.clawdis.android.chat.ChatMessage
import com.clawdis.android.chat.ChatPendingToolCall
@Composable
fun ChatMessageListCard(

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.ui.chat
package com.clawdis.android.ui.chat
import android.graphics.BitmapFactory
import android.util.Base64
@@ -7,11 +7,9 @@ import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.layout.width
import androidx.compose.foundation.shape.CircleShape
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.material3.MaterialTheme
@@ -33,11 +31,13 @@ import androidx.compose.ui.layout.ContentScale
import androidx.compose.ui.text.font.FontFamily
import androidx.compose.ui.unit.dp
import androidx.compose.foundation.Image
import com.steipete.clawdis.node.chat.ChatMessage
import com.steipete.clawdis.node.chat.ChatMessageContent
import com.steipete.clawdis.node.chat.ChatPendingToolCall
import com.clawdis.android.chat.ChatMessage
import com.clawdis.android.chat.ChatMessageContent
import com.clawdis.android.chat.ChatPendingToolCall
import com.clawdis.android.tools.ToolDisplayRegistry
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import androidx.compose.ui.platform.LocalContext
@Composable
fun ChatMessageBubble(message: ChatMessage) {
@@ -60,20 +60,21 @@ fun ChatMessageBubble(message: ChatMessage) {
.background(bubbleBackground(isUser))
.padding(horizontal = 12.dp, vertical = 10.dp),
) {
ChatMessageBody(content = message.content)
val textColor = textColorOverBubble(isUser)
ChatMessageBody(content = message.content, textColor = textColor)
}
}
}
}
@Composable
private fun ChatMessageBody(content: List<ChatMessageContent>) {
private fun ChatMessageBody(content: List<ChatMessageContent>, textColor: Color) {
Column(verticalArrangement = Arrangement.spacedBy(10.dp)) {
for (part in content) {
when (part.type) {
"text" -> {
val text = part.text ?: continue
ChatMarkdown(text = text)
ChatMarkdown(text = text, textColor = textColor)
}
else -> {
val b64 = part.base64 ?: continue
@@ -105,18 +106,42 @@ fun ChatTypingIndicatorBubble() {
@Composable
fun ChatPendingToolsBubble(toolCalls: List<ChatPendingToolCall>) {
val context = LocalContext.current
val displays =
remember(toolCalls, context) {
toolCalls.map { ToolDisplayRegistry.resolve(context, it.name, it.args) }
}
Row(modifier = Modifier.fillMaxWidth(), horizontalArrangement = Arrangement.Start) {
Surface(
shape = RoundedCornerShape(16.dp),
color = MaterialTheme.colorScheme.surfaceContainer,
) {
Column(modifier = Modifier.padding(horizontal = 12.dp, vertical = 10.dp), verticalArrangement = Arrangement.spacedBy(6.dp)) {
Text("Tools", style = MaterialTheme.typography.labelLarge, color = MaterialTheme.colorScheme.onSurface)
for (t in toolCalls.take(6)) {
Text("· ${t.name}", style = MaterialTheme.typography.bodyMedium, color = MaterialTheme.colorScheme.onSurfaceVariant)
Text("Running tools", style = MaterialTheme.typography.labelLarge, color = MaterialTheme.colorScheme.onSurface)
for (display in displays.take(6)) {
Column(verticalArrangement = Arrangement.spacedBy(2.dp)) {
Text(
"${display.emoji} ${display.label}",
style = MaterialTheme.typography.bodyMedium,
color = MaterialTheme.colorScheme.onSurfaceVariant,
fontFamily = FontFamily.Monospace,
)
display.detailLine?.let { detail ->
Text(
detail,
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
fontFamily = FontFamily.Monospace,
)
}
}
}
if (toolCalls.size > 6) {
Text("… +${toolCalls.size - 6} more", style = MaterialTheme.typography.bodySmall, color = MaterialTheme.colorScheme.onSurfaceVariant)
Text(
"… +${toolCalls.size - 6} more",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
}
}
}
@@ -131,7 +156,7 @@ fun ChatStreamingAssistantBubble(text: String) {
color = MaterialTheme.colorScheme.surfaceContainer,
) {
Box(modifier = Modifier.padding(horizontal = 12.dp, vertical = 10.dp)) {
ChatMarkdown(text = text)
ChatMarkdown(text = text, textColor = MaterialTheme.colorScheme.onSurface)
}
}
}
@@ -150,6 +175,15 @@ private fun bubbleBackground(isUser: Boolean): Brush {
}
}
@Composable
private fun textColorOverBubble(isUser: Boolean): Color {
return if (isUser) {
MaterialTheme.colorScheme.onPrimary
} else {
MaterialTheme.colorScheme.onSurface
}
}
@Composable
private fun ChatBase64Image(base64: String, mimeType: String?) {
var image by remember(base64) { mutableStateOf<androidx.compose.ui.graphics.ImageBitmap?>(null) }

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.ui.chat
package com.clawdis.android.ui.chat
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Column
@@ -20,7 +20,7 @@ import androidx.compose.runtime.Composable
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.unit.dp
import com.steipete.clawdis.node.chat.ChatSessionEntry
import com.clawdis.android.chat.ChatSessionEntry
@Composable
fun ChatSessionsDialog(
@@ -82,7 +82,7 @@ private fun SessionRow(
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy(10.dp),
) {
Text(entry.key, style = MaterialTheme.typography.bodyMedium)
Text(entry.displayName ?: entry.key, style = MaterialTheme.typography.bodyMedium)
Spacer(modifier = Modifier.weight(1f))
if (isCurrent) {
Text("Current", style = MaterialTheme.typography.labelSmall, color = MaterialTheme.colorScheme.onSurfaceVariant)
@@ -90,4 +90,3 @@ private fun SessionRow(
}
}
}

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.ui.chat
package com.clawdis.android.ui.chat
import android.content.ContentResolver
import android.net.Uri
@@ -14,15 +14,13 @@ import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateListOf
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.rememberCoroutineScope
import androidx.compose.runtime.setValue
import androidx.compose.ui.Modifier
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.unit.dp
import com.steipete.clawdis.node.MainViewModel
import com.steipete.clawdis.node.chat.OutgoingAttachment
import com.clawdis.android.MainViewModel
import com.clawdis.android.chat.OutgoingAttachment
import java.io.ByteArrayOutputStream
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.launch
@@ -40,10 +38,9 @@ fun ChatSheetContent(viewModel: MainViewModel) {
val pendingToolCalls by viewModel.chatPendingToolCalls.collectAsState()
val sessions by viewModel.chatSessions.collectAsState()
var showSessions by remember { mutableStateOf(false) }
LaunchedEffect(Unit) {
viewModel.loadChat("main")
viewModel.refreshChatSessions(limit = 200)
}
val context = LocalContext.current
@@ -87,6 +84,7 @@ fun ChatSheetContent(viewModel: MainViewModel) {
ChatComposer(
sessionKey = sessionKey,
sessions = sessions,
healthOk = healthOk,
thinkingLevel = thinkingLevel,
pendingRunCount = pendingRunCount,
@@ -95,8 +93,11 @@ fun ChatSheetContent(viewModel: MainViewModel) {
onPickImages = { pickImages.launch("image/*") },
onRemoveAttachment = { id -> attachments.removeAll { it.id == id } },
onSetThinkingLevel = { level -> viewModel.setChatThinkingLevel(level) },
onShowSessions = { showSessions = true },
onRefresh = { viewModel.refreshChat() },
onSelectSession = { key -> viewModel.switchChatSession(key) },
onRefresh = {
viewModel.refreshChat()
viewModel.refreshChatSessions(limit = 200)
},
onAbort = { viewModel.abortChat() },
onSend = { text ->
val outgoing =
@@ -113,19 +114,6 @@ fun ChatSheetContent(viewModel: MainViewModel) {
},
)
}
if (showSessions) {
ChatSessionsDialog(
currentSessionKey = sessionKey,
sessions = sessions,
onDismiss = { showSessions = false },
onRefresh = { viewModel.refreshChatSessions(limit = 50) },
onSelect = { key ->
viewModel.switchChatSession(key)
showSessions = false
},
)
}
}
data class PendingImageAttachment(

View File

@@ -0,0 +1,46 @@
package com.clawdis.android.ui.chat
import com.clawdis.android.chat.ChatSessionEntry
private const val MAIN_SESSION_KEY = "main"
private const val RECENT_WINDOW_MS = 24 * 60 * 60 * 1000L
fun resolveSessionChoices(
currentSessionKey: String,
sessions: List<ChatSessionEntry>,
nowMs: Long = System.currentTimeMillis(),
): List<ChatSessionEntry> {
val current = currentSessionKey.trim()
val cutoff = nowMs - RECENT_WINDOW_MS
val sorted = sessions.sortedByDescending { it.updatedAtMs ?: 0L }
val recent = mutableListOf<ChatSessionEntry>()
val seen = mutableSetOf<String>()
for (entry in sorted) {
if (!seen.add(entry.key)) continue
if ((entry.updatedAtMs ?: 0L) < cutoff) continue
recent.add(entry)
}
val result = mutableListOf<ChatSessionEntry>()
val included = mutableSetOf<String>()
val mainEntry = sorted.firstOrNull { it.key == MAIN_SESSION_KEY }
if (mainEntry != null) {
result.add(mainEntry)
included.add(MAIN_SESSION_KEY)
} else if (current == MAIN_SESSION_KEY) {
result.add(ChatSessionEntry(key = MAIN_SESSION_KEY, updatedAtMs = null))
included.add(MAIN_SESSION_KEY)
}
for (entry in recent) {
if (included.add(entry.key)) {
result.add(entry)
}
}
if (current.isNotEmpty() && !included.contains(current)) {
result.add(ChatSessionEntry(key = current, updatedAtMs = null))
}
return result
}

View File

@@ -0,0 +1,98 @@
package com.clawdis.android.voice
import android.media.MediaDataSource
import kotlin.math.min
internal class StreamingMediaDataSource : MediaDataSource() {
private data class Chunk(val start: Long, val data: ByteArray)
private val lock = Object()
private val chunks = ArrayList<Chunk>()
private var totalSize: Long = 0
private var closed = false
private var finished = false
private var lastReadIndex = 0
fun append(data: ByteArray) {
if (data.isEmpty()) return
synchronized(lock) {
if (closed || finished) return
val chunk = Chunk(totalSize, data)
chunks.add(chunk)
totalSize += data.size.toLong()
lock.notifyAll()
}
}
fun finish() {
synchronized(lock) {
if (closed) return
finished = true
lock.notifyAll()
}
}
fun fail() {
synchronized(lock) {
closed = true
lock.notifyAll()
}
}
override fun readAt(position: Long, buffer: ByteArray, offset: Int, size: Int): Int {
if (position < 0) return -1
synchronized(lock) {
while (!closed && !finished && position >= totalSize) {
lock.wait()
}
if (closed) return -1
if (position >= totalSize && finished) return -1
val available = (totalSize - position).toInt()
val toRead = min(size, available)
var remaining = toRead
var destOffset = offset
var pos = position
var index = findChunkIndex(pos)
while (remaining > 0 && index < chunks.size) {
val chunk = chunks[index]
val inChunkOffset = (pos - chunk.start).toInt()
if (inChunkOffset >= chunk.data.size) {
index++
continue
}
val copyLen = min(remaining, chunk.data.size - inChunkOffset)
System.arraycopy(chunk.data, inChunkOffset, buffer, destOffset, copyLen)
remaining -= copyLen
destOffset += copyLen
pos += copyLen
if (inChunkOffset + copyLen >= chunk.data.size) {
index++
}
}
return toRead - remaining
}
}
override fun getSize(): Long = -1
override fun close() {
synchronized(lock) {
closed = true
lock.notifyAll()
}
}
private fun findChunkIndex(position: Long): Int {
var index = lastReadIndex
while (index < chunks.size) {
val chunk = chunks[index]
if (position < chunk.start + chunk.data.size) break
index++
}
lastReadIndex = index
return index
}
}

View File

@@ -0,0 +1,191 @@
package com.clawdis.android.voice
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonElement
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.JsonPrimitive
private val directiveJson = Json { ignoreUnknownKeys = true }
data class TalkDirective(
val voiceId: String? = null,
val modelId: String? = null,
val speed: Double? = null,
val rateWpm: Int? = null,
val stability: Double? = null,
val similarity: Double? = null,
val style: Double? = null,
val speakerBoost: Boolean? = null,
val seed: Long? = null,
val normalize: String? = null,
val language: String? = null,
val outputFormat: String? = null,
val latencyTier: Int? = null,
val once: Boolean? = null,
)
data class TalkDirectiveParseResult(
val directive: TalkDirective?,
val stripped: String,
val unknownKeys: List<String>,
)
object TalkDirectiveParser {
fun parse(text: String): TalkDirectiveParseResult {
val normalized = text.replace("\r\n", "\n")
val lines = normalized.split("\n").toMutableList()
if (lines.isEmpty()) return TalkDirectiveParseResult(null, text, emptyList())
val firstNonEmpty = lines.indexOfFirst { it.trim().isNotEmpty() }
if (firstNonEmpty == -1) return TalkDirectiveParseResult(null, text, emptyList())
val head = lines[firstNonEmpty].trim()
if (!head.startsWith("{") || !head.endsWith("}")) {
return TalkDirectiveParseResult(null, text, emptyList())
}
val obj = parseJsonObject(head) ?: return TalkDirectiveParseResult(null, text, emptyList())
val speakerBoost =
boolValue(obj, listOf("speaker_boost", "speakerBoost"))
?: boolValue(obj, listOf("no_speaker_boost", "noSpeakerBoost"))?.not()
val directive = TalkDirective(
voiceId = stringValue(obj, listOf("voice", "voice_id", "voiceId")),
modelId = stringValue(obj, listOf("model", "model_id", "modelId")),
speed = doubleValue(obj, listOf("speed")),
rateWpm = intValue(obj, listOf("rate", "wpm")),
stability = doubleValue(obj, listOf("stability")),
similarity = doubleValue(obj, listOf("similarity", "similarity_boost", "similarityBoost")),
style = doubleValue(obj, listOf("style")),
speakerBoost = speakerBoost,
seed = longValue(obj, listOf("seed")),
normalize = stringValue(obj, listOf("normalize", "apply_text_normalization")),
language = stringValue(obj, listOf("lang", "language_code", "language")),
outputFormat = stringValue(obj, listOf("output_format", "format")),
latencyTier = intValue(obj, listOf("latency", "latency_tier", "latencyTier")),
once = boolValue(obj, listOf("once")),
)
val hasDirective = listOf(
directive.voiceId,
directive.modelId,
directive.speed,
directive.rateWpm,
directive.stability,
directive.similarity,
directive.style,
directive.speakerBoost,
directive.seed,
directive.normalize,
directive.language,
directive.outputFormat,
directive.latencyTier,
directive.once,
).any { it != null }
if (!hasDirective) return TalkDirectiveParseResult(null, text, emptyList())
val knownKeys = setOf(
"voice", "voice_id", "voiceid",
"model", "model_id", "modelid",
"speed", "rate", "wpm",
"stability", "similarity", "similarity_boost", "similarityboost",
"style",
"speaker_boost", "speakerboost",
"no_speaker_boost", "nospeakerboost",
"seed",
"normalize", "apply_text_normalization",
"lang", "language_code", "language",
"output_format", "format",
"latency", "latency_tier", "latencytier",
"once",
)
val unknownKeys = obj.keys.filter { !knownKeys.contains(it.lowercase()) }.sorted()
lines.removeAt(firstNonEmpty)
if (firstNonEmpty < lines.size) {
if (lines[firstNonEmpty].trim().isEmpty()) {
lines.removeAt(firstNonEmpty)
}
}
return TalkDirectiveParseResult(directive, lines.joinToString("\n"), unknownKeys)
}
private fun parseJsonObject(line: String): JsonObject? {
return try {
directiveJson.parseToJsonElement(line) as? JsonObject
} catch (_: Throwable) {
null
}
}
private fun stringValue(obj: JsonObject, keys: List<String>): String? {
for (key in keys) {
val value = obj[key].asStringOrNull()?.trim()
if (!value.isNullOrEmpty()) return value
}
return null
}
private fun doubleValue(obj: JsonObject, keys: List<String>): Double? {
for (key in keys) {
val value = obj[key].asDoubleOrNull()
if (value != null) return value
}
return null
}
private fun intValue(obj: JsonObject, keys: List<String>): Int? {
for (key in keys) {
val value = obj[key].asIntOrNull()
if (value != null) return value
}
return null
}
private fun longValue(obj: JsonObject, keys: List<String>): Long? {
for (key in keys) {
val value = obj[key].asLongOrNull()
if (value != null) return value
}
return null
}
private fun boolValue(obj: JsonObject, keys: List<String>): Boolean? {
for (key in keys) {
val value = obj[key].asBooleanOrNull()
if (value != null) return value
}
return null
}
}
private fun JsonElement?.asStringOrNull(): String? =
(this as? JsonPrimitive)?.takeIf { it.isString }?.content
private fun JsonElement?.asDoubleOrNull(): Double? {
val primitive = this as? JsonPrimitive ?: return null
return primitive.content.toDoubleOrNull()
}
private fun JsonElement?.asIntOrNull(): Int? {
val primitive = this as? JsonPrimitive ?: return null
return primitive.content.toIntOrNull()
}
private fun JsonElement?.asLongOrNull(): Long? {
val primitive = this as? JsonPrimitive ?: return null
return primitive.content.toLongOrNull()
}
private fun JsonElement?.asBooleanOrNull(): Boolean? {
val primitive = this as? JsonPrimitive ?: return null
val content = primitive.content.trim().lowercase()
return when (content) {
"true", "yes", "1" -> true
"false", "no", "0" -> false
else -> null
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.voice
package com.clawdis.android.voice
object VoiceWakeCommandExtractor {
fun extractCommand(text: String, triggerWords: List<String>): String? {

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.voice
package com.clawdis.android.voice
import android.content.Context
import android.content.Intent

View File

@@ -1,123 +0,0 @@
package com.steipete.clawdis.node.ui
import androidx.compose.animation.AnimatedVisibility
import androidx.compose.animation.fadeIn
import androidx.compose.animation.fadeOut
import androidx.compose.animation.slideInVertically
import androidx.compose.animation.slideOutVertically
import androidx.compose.foundation.background
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.layout.statusBarsPadding
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.CheckCircle
import androidx.compose.material.icons.filled.Error
import androidx.compose.material.icons.filled.FiberManualRecord
import androidx.compose.material.icons.filled.PhotoCamera
import androidx.compose.material3.CircularProgressIndicator
import androidx.compose.material3.Icon
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableFloatStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.draw.alpha
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.text.style.TextOverflow
import androidx.compose.ui.unit.dp
import com.steipete.clawdis.node.CameraHudKind
import com.steipete.clawdis.node.CameraHudState
import kotlinx.coroutines.delay
@Composable
fun CameraHudOverlay(
hud: CameraHudState?,
flashToken: Long,
modifier: Modifier = Modifier,
) {
Box(modifier = modifier.fillMaxSize()) {
CameraFlash(token = flashToken)
AnimatedVisibility(
visible = hud != null,
enter = slideInVertically(initialOffsetY = { -it / 2 }) + fadeIn(),
exit = slideOutVertically(targetOffsetY = { -it / 2 }) + fadeOut(),
modifier = Modifier.align(Alignment.TopStart).statusBarsPadding().padding(start = 12.dp, top = 58.dp),
) {
if (hud != null) {
Toast(hud = hud)
}
}
}
}
@Composable
private fun CameraFlash(token: Long) {
var alpha by remember { mutableFloatStateOf(0f) }
LaunchedEffect(token) {
if (token == 0L) return@LaunchedEffect
alpha = 0.85f
delay(110)
alpha = 0f
}
Box(
modifier =
Modifier
.fillMaxSize()
.alpha(alpha)
.background(Color.White),
)
}
@Composable
private fun Toast(hud: CameraHudState) {
Surface(
shape = RoundedCornerShape(14.dp),
color = MaterialTheme.colorScheme.surface.copy(alpha = 0.85f),
tonalElevation = 2.dp,
shadowElevation = 8.dp,
) {
Row(
modifier = Modifier.padding(vertical = 10.dp, horizontal = 12.dp),
verticalAlignment = Alignment.CenterVertically,
) {
when (hud.kind) {
CameraHudKind.Photo -> {
Icon(Icons.Default.PhotoCamera, contentDescription = null)
Spacer(Modifier.size(10.dp))
CircularProgressIndicator(modifier = Modifier.size(14.dp), strokeWidth = 2.dp)
}
CameraHudKind.Recording -> {
Icon(Icons.Default.FiberManualRecord, contentDescription = null, tint = Color.Red)
}
CameraHudKind.Success -> {
Icon(Icons.Default.CheckCircle, contentDescription = null)
}
CameraHudKind.Error -> {
Icon(Icons.Default.Error, contentDescription = null)
}
}
Spacer(Modifier.size(10.dp))
Text(
text = hud.message,
style = MaterialTheme.typography.bodyMedium,
maxLines = 1,
overflow = TextOverflow.Ellipsis,
)
}
}
}

View File

@@ -1,240 +0,0 @@
package com.steipete.clawdis.node.ui
import android.annotation.SuppressLint
import android.Manifest
import android.content.pm.PackageManager
import android.graphics.Color
import android.util.Log
import android.view.View
import android.webkit.JavascriptInterface
import android.webkit.WebView
import android.webkit.WebSettings
import android.webkit.WebResourceError
import android.webkit.WebResourceRequest
import android.webkit.WebResourceResponse
import android.webkit.WebViewClient
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.WindowInsets
import androidx.compose.foundation.layout.WindowInsetsSides
import androidx.compose.foundation.layout.only
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.safeDrawing
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.layout.windowInsetsPadding
import androidx.compose.material3.ExperimentalMaterial3Api
import androidx.compose.material3.FilledTonalIconButton
import androidx.compose.material3.Icon
import androidx.compose.material3.IconButtonDefaults
import androidx.compose.material3.ModalBottomSheet
import androidx.compose.material3.rememberModalBottomSheetState
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.ChatBubble
import androidx.compose.material.icons.filled.Settings
import androidx.compose.runtime.Composable
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.unit.dp
import androidx.compose.ui.viewinterop.AndroidView
import androidx.compose.ui.window.Popup
import androidx.compose.ui.window.PopupProperties
import androidx.core.content.ContextCompat
import com.steipete.clawdis.node.MainViewModel
@OptIn(ExperimentalMaterial3Api::class)
@Composable
fun RootScreen(viewModel: MainViewModel) {
var sheet by remember { mutableStateOf<Sheet?>(null) }
val sheetState = rememberModalBottomSheetState(skipPartiallyExpanded = true)
val safeOverlayInsets = WindowInsets.safeDrawing.only(WindowInsetsSides.Top + WindowInsetsSides.Horizontal)
val context = LocalContext.current
val serverName by viewModel.serverName.collectAsState()
val statusText by viewModel.statusText.collectAsState()
val cameraHud by viewModel.cameraHud.collectAsState()
val cameraFlashToken by viewModel.cameraFlashToken.collectAsState()
val bridgeState =
remember(serverName, statusText) {
when {
serverName != null -> BridgeState.Connected
statusText.contains("connecting", ignoreCase = true) ||
statusText.contains("reconnecting", ignoreCase = true) -> BridgeState.Connecting
statusText.contains("error", ignoreCase = true) -> BridgeState.Error
else -> BridgeState.Disconnected
}
}
val voiceEnabled =
ContextCompat.checkSelfPermission(context, Manifest.permission.RECORD_AUDIO) ==
PackageManager.PERMISSION_GRANTED
Box(modifier = Modifier.fillMaxSize()) {
CanvasView(viewModel = viewModel, modifier = Modifier.fillMaxSize())
}
// Camera HUD (flash + toast) must be in a Popup to render above the WebView.
Popup(alignment = Alignment.Center, properties = PopupProperties(focusable = false)) {
CameraHudOverlay(hud = cameraHud, flashToken = cameraFlashToken, modifier = Modifier.fillMaxSize())
}
// Keep the overlay buttons above the WebView canvas (AndroidView), otherwise they may not receive touches.
Popup(alignment = Alignment.TopStart, properties = PopupProperties(focusable = false)) {
StatusPill(
bridge = bridgeState,
voiceEnabled = voiceEnabled,
onClick = { sheet = Sheet.Settings },
modifier = Modifier.windowInsetsPadding(safeOverlayInsets).padding(start = 12.dp, top = 12.dp),
)
}
Popup(alignment = Alignment.TopEnd, properties = PopupProperties(focusable = false)) {
Column(
modifier = Modifier.windowInsetsPadding(safeOverlayInsets).padding(end = 12.dp, top = 12.dp),
verticalArrangement = Arrangement.spacedBy(10.dp),
horizontalAlignment = Alignment.End,
) {
OverlayIconButton(
onClick = { sheet = Sheet.Chat },
icon = { Icon(Icons.Default.ChatBubble, contentDescription = "Chat") },
)
OverlayIconButton(
onClick = { sheet = Sheet.Settings },
icon = { Icon(Icons.Default.Settings, contentDescription = "Settings") },
)
}
}
val currentSheet = sheet
if (currentSheet != null) {
ModalBottomSheet(
onDismissRequest = { sheet = null },
sheetState = sheetState,
) {
when (currentSheet) {
Sheet.Chat -> ChatSheet(viewModel = viewModel)
Sheet.Settings -> SettingsSheet(viewModel = viewModel)
}
}
}
}
private enum class Sheet {
Chat,
Settings,
}
@Composable
private fun OverlayIconButton(
onClick: () -> Unit,
icon: @Composable () -> Unit,
) {
FilledTonalIconButton(
onClick = onClick,
modifier = Modifier.size(44.dp),
colors =
IconButtonDefaults.filledTonalIconButtonColors(
containerColor = overlayContainerColor(),
contentColor = overlayIconColor(),
),
) {
icon()
}
}
@SuppressLint("SetJavaScriptEnabled")
@Composable
private fun CanvasView(viewModel: MainViewModel, modifier: Modifier = Modifier) {
val context = LocalContext.current
val isDebuggable = (context.applicationInfo.flags and android.content.pm.ApplicationInfo.FLAG_DEBUGGABLE) != 0
AndroidView(
modifier = modifier,
factory = {
WebView(context).apply {
settings.javaScriptEnabled = true
// Some embedded web UIs (incl. the "background website") use localStorage/sessionStorage.
settings.domStorageEnabled = true
settings.mixedContentMode = WebSettings.MIXED_CONTENT_COMPATIBILITY_MODE
webViewClient =
object : WebViewClient() {
override fun onReceivedError(
view: WebView,
request: WebResourceRequest,
error: WebResourceError,
) {
if (!isDebuggable) return
if (!request.isForMainFrame) return
Log.e("ClawdisWebView", "onReceivedError: ${error.errorCode} ${error.description} ${request.url}")
}
override fun onReceivedHttpError(
view: WebView,
request: WebResourceRequest,
errorResponse: WebResourceResponse,
) {
if (!isDebuggable) return
if (!request.isForMainFrame) return
Log.e(
"ClawdisWebView",
"onReceivedHttpError: ${errorResponse.statusCode} ${errorResponse.reasonPhrase} ${request.url}",
)
}
override fun onPageFinished(view: WebView, url: String?) {
viewModel.canvas.onPageFinished()
}
}
setBackgroundColor(Color.BLACK)
setLayerType(View.LAYER_TYPE_HARDWARE, null)
val a2uiBridge =
CanvasA2UIActionBridge { payload ->
viewModel.handleCanvasA2UIActionFromWebView(payload)
}
addJavascriptInterface(a2uiBridge, CanvasA2UIActionBridge.interfaceName)
addJavascriptInterface(
CanvasA2UIActionLegacyBridge(a2uiBridge),
CanvasA2UIActionLegacyBridge.interfaceName,
)
viewModel.canvas.attach(this)
}
},
)
}
private class CanvasA2UIActionBridge(private val onMessage: (String) -> Unit) {
@JavascriptInterface
fun postMessage(payload: String?) {
val msg = payload?.trim().orEmpty()
if (msg.isEmpty()) return
onMessage(msg)
}
companion object {
const val interfaceName: String = "clawdisCanvasA2UIAction"
}
}
private class CanvasA2UIActionLegacyBridge(private val bridge: CanvasA2UIActionBridge) {
@JavascriptInterface
fun canvasAction(payload: String?) {
bridge.postMessage(payload)
}
@JavascriptInterface
fun postMessage(payload: String?) {
bridge.postMessage(payload)
}
companion object {
const val interfaceName: String = "Android"
}
}

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node
package com.clawdis.android
import org.junit.Assert.assertEquals
import org.junit.Test

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.bridge
package com.clawdis.android.bridge
import org.junit.Assert.assertEquals
import org.junit.Test

View File

@@ -0,0 +1,14 @@
package com.clawdis.android.bridge
import io.kotest.core.spec.style.StringSpec
import io.kotest.matchers.shouldBe
class BridgeEndpointKotestTest : StringSpec({
"manual endpoint builds stable id + name" {
val endpoint = BridgeEndpoint.manual("10.0.0.5", 18790)
endpoint.stableId shouldBe "manual|10.0.0.5|18790"
endpoint.name shouldBe "10.0.0.5:18790"
endpoint.host shouldBe "10.0.0.5"
endpoint.port shouldBe 18790
}
})

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.bridge
package com.clawdis.android.bridge
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.async

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.bridge
package com.clawdis.android.bridge
import kotlinx.coroutines.CompletableDeferred
import kotlinx.coroutines.CoroutineScope

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.node
package com.clawdis.android.node
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNull

View File

@@ -0,0 +1,47 @@
package com.clawdis.android.node
import org.junit.Assert.assertEquals
import org.junit.Assert.assertTrue
import org.junit.Test
import kotlin.math.min
class JpegSizeLimiterTest {
@Test
fun compressesLargePayloadsUnderLimit() {
val maxBytes = 5 * 1024 * 1024
val result =
JpegSizeLimiter.compressToLimit(
initialWidth = 4000,
initialHeight = 3000,
startQuality = 95,
maxBytes = maxBytes,
encode = { width, height, quality ->
val estimated = (width.toLong() * height.toLong() * quality.toLong()) / 100
val size = min(maxBytes.toLong() * 2, estimated).toInt()
ByteArray(size)
},
)
assertTrue(result.bytes.size <= maxBytes)
assertTrue(result.width <= 4000)
assertTrue(result.height <= 3000)
assertTrue(result.quality <= 95)
}
@Test
fun keepsSmallPayloadsAsIs() {
val maxBytes = 5 * 1024 * 1024
val result =
JpegSizeLimiter.compressToLimit(
initialWidth = 800,
initialHeight = 600,
startQuality = 90,
maxBytes = maxBytes,
encode = { _, _, _ -> ByteArray(120_000) },
)
assertEquals(800, result.width)
assertEquals(600, result.height)
assertEquals(90, result.quality)
}
}

View File

@@ -0,0 +1,91 @@
package com.clawdis.android.node
import kotlinx.serialization.json.jsonObject
import kotlinx.serialization.json.jsonPrimitive
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Test
class SmsManagerTest {
private val json = SmsManager.JsonConfig
@Test
fun parseParamsRejectsEmptyPayload() {
val result = SmsManager.parseParams("", json)
assertTrue(result is SmsManager.ParseResult.Error)
val error = result as SmsManager.ParseResult.Error
assertEquals("INVALID_REQUEST: paramsJSON required", error.error)
}
@Test
fun parseParamsRejectsInvalidJson() {
val result = SmsManager.parseParams("not-json", json)
assertTrue(result is SmsManager.ParseResult.Error)
val error = result as SmsManager.ParseResult.Error
assertEquals("INVALID_REQUEST: expected JSON object", error.error)
}
@Test
fun parseParamsRejectsNonObjectJson() {
val result = SmsManager.parseParams("[]", json)
assertTrue(result is SmsManager.ParseResult.Error)
val error = result as SmsManager.ParseResult.Error
assertEquals("INVALID_REQUEST: expected JSON object", error.error)
}
@Test
fun parseParamsRejectsMissingTo() {
val result = SmsManager.parseParams("{\"message\":\"Hi\"}", json)
assertTrue(result is SmsManager.ParseResult.Error)
val error = result as SmsManager.ParseResult.Error
assertEquals("INVALID_REQUEST: 'to' phone number required", error.error)
assertEquals("Hi", error.message)
}
@Test
fun parseParamsRejectsMissingMessage() {
val result = SmsManager.parseParams("{\"to\":\"+1234\"}", json)
assertTrue(result is SmsManager.ParseResult.Error)
val error = result as SmsManager.ParseResult.Error
assertEquals("INVALID_REQUEST: 'message' text required", error.error)
assertEquals("+1234", error.to)
}
@Test
fun parseParamsTrimsToField() {
val result = SmsManager.parseParams("{\"to\":\" +1555 \",\"message\":\"Hello\"}", json)
assertTrue(result is SmsManager.ParseResult.Ok)
val ok = result as SmsManager.ParseResult.Ok
assertEquals("+1555", ok.params.to)
assertEquals("Hello", ok.params.message)
}
@Test
fun buildPayloadJsonEscapesFields() {
val payload = SmsManager.buildPayloadJson(
json = json,
ok = false,
to = "+1\"23",
error = "SMS_SEND_FAILED: \"nope\"",
)
val parsed = json.parseToJsonElement(payload).jsonObject
assertEquals("false", parsed["ok"]?.jsonPrimitive?.content)
assertEquals("+1\"23", parsed["to"]?.jsonPrimitive?.content)
assertEquals("SMS_SEND_FAILED: \"nope\"", parsed["error"]?.jsonPrimitive?.content)
}
@Test
fun buildSendPlanUsesMultipartWhenMultipleParts() {
val plan = SmsManager.buildSendPlan("hello") { listOf("a", "b") }
assertTrue(plan.useMultipart)
assertEquals(listOf("a", "b"), plan.parts)
}
@Test
fun buildSendPlanFallsBackToSinglePartWhenDividerEmpty() {
val plan = SmsManager.buildSendPlan("hello") { emptyList() }
assertFalse(plan.useMultipart)
assertEquals(listOf("hello"), plan.parts)
}
}

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.protocol
package com.clawdis.android.protocol
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.jsonObject

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.protocol
package com.clawdis.android.protocol
import org.junit.Assert.assertEquals
import org.junit.Test

View File

@@ -0,0 +1,35 @@
package com.clawdis.android.ui.chat
import com.clawdis.android.chat.ChatSessionEntry
import org.junit.Assert.assertEquals
import org.junit.Test
class SessionFiltersTest {
@Test
fun sessionChoicesPreferMainAndRecent() {
val now = 1_700_000_000_000L
val recent1 = now - 2 * 60 * 60 * 1000L
val recent2 = now - 5 * 60 * 60 * 1000L
val stale = now - 26 * 60 * 60 * 1000L
val sessions =
listOf(
ChatSessionEntry(key = "recent-1", updatedAtMs = recent1),
ChatSessionEntry(key = "main", updatedAtMs = stale),
ChatSessionEntry(key = "old-1", updatedAtMs = stale),
ChatSessionEntry(key = "recent-2", updatedAtMs = recent2),
)
val result = resolveSessionChoices("main", sessions, nowMs = now).map { it.key }
assertEquals(listOf("main", "recent-1", "recent-2"), result)
}
@Test
fun sessionChoicesIncludeCurrentWhenMissing() {
val now = 1_700_000_000_000L
val recent = now - 10 * 60 * 1000L
val sessions = listOf(ChatSessionEntry(key = "main", updatedAtMs = recent))
val result = resolveSessionChoices("custom", sessions, nowMs = now).map { it.key }
assertEquals(listOf("main", "custom"), result)
}
}

View File

@@ -0,0 +1,55 @@
package com.clawdis.android.voice
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNull
import org.junit.Assert.assertTrue
import org.junit.Test
class TalkDirectiveParserTest {
@Test
fun parsesDirectiveAndStripsHeader() {
val input = """
{"voice":"voice-123","once":true}
Hello from talk mode.
""".trimIndent()
val result = TalkDirectiveParser.parse(input)
assertEquals("voice-123", result.directive?.voiceId)
assertEquals(true, result.directive?.once)
assertEquals("Hello from talk mode.", result.stripped.trim())
}
@Test
fun ignoresUnknownKeysButReportsThem() {
val input = """
{"voice":"abc","foo":1,"bar":"baz"}
Hi there.
""".trimIndent()
val result = TalkDirectiveParser.parse(input)
assertEquals("abc", result.directive?.voiceId)
assertTrue(result.unknownKeys.containsAll(listOf("bar", "foo")))
}
@Test
fun parsesAlternateKeys() {
val input = """
{"model_id":"eleven_v3","similarity_boost":0.4,"no_speaker_boost":true,"rate":200}
Speak.
""".trimIndent()
val result = TalkDirectiveParser.parse(input)
assertEquals("eleven_v3", result.directive?.modelId)
assertEquals(0.4, result.directive?.similarity)
assertEquals(false, result.directive?.speakerBoost)
assertEquals(200, result.directive?.rateWpm)
}
@Test
fun returnsNullWhenNoDirectivePresent() {
val input = """
{}
Hello.
""".trimIndent()
val result = TalkDirectiveParser.parse(input)
assertNull(result.directive)
assertEquals(input, result.stripped)
}
}

View File

@@ -1,4 +1,4 @@
package com.steipete.clawdis.node.voice
package com.clawdis.android.voice
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNull

View File

@@ -14,7 +14,7 @@ actor BridgeClient {
{
self.lineBuffer = Data()
let connection = NWConnection(to: endpoint, using: .tcp)
let queue = DispatchQueue(label: "com.steipete.clawdis.ios.bridge-client")
let queue = DispatchQueue(label: "com.clawdis.ios.bridge-client")
defer { connection.cancel() }
try await self.withTimeout(seconds: 8, purpose: "connect") {
try await self.startAndWaitForReady(connection, queue: queue)

View File

@@ -6,6 +6,15 @@ import Observation
import SwiftUI
import UIKit
protocol BridgePairingClient: Sendable {
func pairAndHello(
endpoint: NWEndpoint,
hello: BridgeHello,
onStatus: (@Sendable (String) -> Void)?) async throws -> String
}
extension BridgeClient: BridgePairingClient {}
@MainActor
@Observable
final class BridgeConnectionController {
@@ -16,10 +25,16 @@ final class BridgeConnectionController {
private let discovery = BridgeDiscoveryModel()
private weak var appModel: NodeAppModel?
private var didAutoConnect = false
private var seenStableIDs = Set<String>()
init(appModel: NodeAppModel, startDiscovery: Bool = true) {
private let bridgeClientFactory: @Sendable () -> any BridgePairingClient
init(
appModel: NodeAppModel,
startDiscovery: Bool = true,
bridgeClientFactory: @escaping @Sendable () -> any BridgePairingClient = { BridgeClient() })
{
self.appModel = appModel
self.bridgeClientFactory = bridgeClientFactory
BridgeSettingsStore.bootstrapPersistence()
let defaults = UserDefaults.standard
@@ -84,8 +99,8 @@ final class BridgeConnectionController {
guard !instanceId.isEmpty else { return }
let token = KeychainStore.loadString(
service: "com.steipete.clawdis.bridge",
account: "bridge-token.\(instanceId)")?
service: "com.clawdis.bridge",
account: self.keychainAccount(instanceId: instanceId))?
.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
guard !token.isEmpty else { return }
@@ -99,28 +114,40 @@ final class BridgeConnectionController {
guard let port = NWEndpoint.Port(rawValue: UInt16(resolvedPort)) else { return }
self.didAutoConnect = true
appModel.connectToBridge(
endpoint: .hostPort(host: NWEndpoint.Host(manualHost), port: port),
hello: self.makeHello(token: token))
let endpoint = NWEndpoint.hostPort(host: NWEndpoint.Host(manualHost), port: port)
self.startAutoConnect(endpoint: endpoint, token: token, instanceId: instanceId)
return
}
let targetStableID = defaults.string(forKey: "bridge.lastDiscoveredStableID")?
let preferredStableID = defaults.string(forKey: "bridge.preferredStableID")?
.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
guard !targetStableID.isEmpty else { return }
let lastDiscoveredStableID = defaults.string(forKey: "bridge.lastDiscoveredStableID")?
.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
let candidates = [preferredStableID, lastDiscoveredStableID].filter { !$0.isEmpty }
guard let targetStableID = candidates.first(where: { id in
self.bridges.contains(where: { $0.stableID == id })
}) else { return }
guard let target = self.bridges.first(where: { $0.stableID == targetStableID }) else { return }
self.didAutoConnect = true
appModel.connectToBridge(endpoint: target.endpoint, hello: self.makeHello(token: token))
self.startAutoConnect(endpoint: target.endpoint, token: token, instanceId: instanceId)
}
private func updateLastDiscoveredBridge(from bridges: [BridgeDiscoveryModel.DiscoveredBridge]) {
let newlyDiscovered = bridges.filter { self.seenStableIDs.insert($0.stableID).inserted }
guard let last = newlyDiscovered.last else { return }
let defaults = UserDefaults.standard
let preferred = defaults.string(forKey: "bridge.preferredStableID")?
.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
let existingLast = defaults.string(forKey: "bridge.lastDiscoveredStableID")?
.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
UserDefaults.standard.set(last.stableID, forKey: "bridge.lastDiscoveredStableID")
BridgeSettingsStore.saveLastDiscoveredBridgeStableID(last.stableID)
// Avoid overriding user intent (preferred/lastDiscovered are also set on manual Connect).
guard preferred.isEmpty, existingLast.isEmpty else { return }
guard let first = bridges.first else { return }
defaults.set(first.stableID, forKey: "bridge.lastDiscoveredStableID")
BridgeSettingsStore.saveLastDiscoveredBridgeStableID(first.stableID)
}
private func makeHello(token: String) -> BridgeHello {
@@ -140,6 +167,40 @@ final class BridgeConnectionController {
commands: self.currentCommands())
}
private func keychainAccount(instanceId: String) -> String {
"bridge-token.\(instanceId)"
}
private func startAutoConnect(endpoint: NWEndpoint, token: String, instanceId: String) {
guard let appModel else { return }
Task { [weak self] in
guard let self else { return }
do {
let hello = self.makeHello(token: token)
let refreshed = try await self.bridgeClientFactory().pairAndHello(
endpoint: endpoint,
hello: hello,
onStatus: { status in
Task { @MainActor in
appModel.bridgeStatusText = status
}
})
let resolvedToken = refreshed.isEmpty ? token : refreshed
if !refreshed.isEmpty, refreshed != token {
_ = KeychainStore.saveString(
refreshed,
service: "com.clawdis.bridge",
account: self.keychainAccount(instanceId: instanceId))
}
appModel.connectToBridge(endpoint: endpoint, hello: self.makeHello(token: resolvedToken))
} catch {
await MainActor.run {
appModel.bridgeStatusText = "Bridge error: \(error.localizedDescription)"
}
}
}
}
private func resolvedDisplayName(defaults: UserDefaults) -> String {
let key = "node.displayName"
let existing = defaults.string(forKey: key)?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
@@ -168,6 +229,10 @@ final class BridgeConnectionController {
let voiceWakeEnabled = UserDefaults.standard.bool(forKey: VoiceWakePreferences.enabledKey)
if voiceWakeEnabled { caps.append(ClawdisCapability.voiceWake.rawValue) }
let locationModeRaw = UserDefaults.standard.string(forKey: "location.enabledMode") ?? "off"
let locationMode = ClawdisLocationMode(rawValue: locationModeRaw) ?? .off
if locationMode != .off { caps.append(ClawdisCapability.location.rawValue) }
return caps
}
@@ -186,9 +251,13 @@ final class BridgeConnectionController {
let caps = Set(self.currentCaps())
if caps.contains(ClawdisCapability.camera.rawValue) {
commands.append(ClawdisCameraCommand.list.rawValue)
commands.append(ClawdisCameraCommand.snap.rawValue)
commands.append(ClawdisCameraCommand.clip.rawValue)
}
if caps.contains(ClawdisCapability.location.rawValue) {
commands.append(ClawdisLocationCommand.get.rawValue)
}
return commands
}
@@ -231,3 +300,47 @@ final class BridgeConnectionController {
Bundle.main.infoDictionary?["CFBundleShortVersionString"] as? String ?? "dev"
}
}
#if DEBUG
extension BridgeConnectionController {
func _test_makeHello(token: String) -> BridgeHello {
self.makeHello(token: token)
}
func _test_resolvedDisplayName(defaults: UserDefaults) -> String {
self.resolvedDisplayName(defaults: defaults)
}
func _test_currentCaps() -> [String] {
self.currentCaps()
}
func _test_currentCommands() -> [String] {
self.currentCommands()
}
func _test_platformString() -> String {
self.platformString()
}
func _test_deviceFamily() -> String {
self.deviceFamily()
}
func _test_modelIdentifier() -> String {
self.modelIdentifier()
}
func _test_appVersion() -> String {
self.appVersion()
}
func _test_setBridges(_ bridges: [BridgeDiscoveryModel.DiscoveredBridge]) {
self.bridges = bridges
}
func _test_triggerAutoConnect() {
self.maybeAutoConnect()
}
}
#endif

View File

@@ -18,6 +18,12 @@ final class BridgeDiscoveryModel {
var endpoint: NWEndpoint
var stableID: String
var debugID: String
var lanHost: String?
var tailnetDns: String?
var gatewayPort: Int?
var bridgePort: Int?
var canvasPort: Int?
var cliPath: String?
}
var bridges: [DiscoveredBridge] = []
@@ -68,7 +74,8 @@ final class BridgeDiscoveryModel {
switch result.endpoint {
case let .service(name, _, _, _):
let decodedName = BonjourEscapes.decode(name)
let advertisedName = result.endpoint.txtRecord?.dictionary["displayName"]
let txt = result.endpoint.txtRecord?.dictionary ?? [:]
let advertisedName = txt["displayName"]
let prettyAdvertised = advertisedName
.map(Self.prettifyInstanceName)
.flatMap { $0.isEmpty ? nil : $0 }
@@ -77,7 +84,13 @@ final class BridgeDiscoveryModel {
name: prettyName,
endpoint: result.endpoint,
stableID: BridgeEndpointID.stableID(result.endpoint),
debugID: BridgeEndpointID.prettyDescription(result.endpoint))
debugID: BridgeEndpointID.prettyDescription(result.endpoint),
lanHost: Self.txtValue(txt, key: "lanHost"),
tailnetDns: Self.txtValue(txt, key: "tailnetDns"),
gatewayPort: Self.txtIntValue(txt, key: "gatewayPort"),
bridgePort: Self.txtIntValue(txt, key: "bridgePort"),
canvasPort: Self.txtIntValue(txt, key: "canvasPort"),
cliPath: Self.txtValue(txt, key: "cliPath"))
default:
return nil
}
@@ -89,7 +102,7 @@ final class BridgeDiscoveryModel {
}
self.browsers[domain] = browser
browser.start(queue: DispatchQueue(label: "com.steipete.clawdis.ios.bridge-discovery.\(domain)"))
browser.start(queue: DispatchQueue(label: "com.clawdis.ios.bridge-discovery.\(domain)"))
}
}
@@ -191,4 +204,14 @@ final class BridgeDiscoveryModel {
.replacingOccurrences(of: #"\s+\(\d+\)$"#, with: "", options: .regularExpression)
return stripped.trimmingCharacters(in: .whitespacesAndNewlines)
}
private static func txtValue(_ dict: [String: String], key: String) -> String? {
let raw = dict[key]?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
return raw.isEmpty ? nil : raw
}
private static func txtIntValue(_ dict: [String: String], key: String) -> Int? {
guard let raw = self.txtValue(dict, key: key) else { return nil }
return Int(raw)
}
}

View File

@@ -78,7 +78,7 @@ actor BridgeSession {
let params = NWParameters.tcp
params.includePeerToPeer = true
let connection = NWConnection(to: endpoint, using: params)
let queue = DispatchQueue(label: "com.steipete.clawdis.ios.bridge-session")
let queue = DispatchQueue(label: "com.clawdis.ios.bridge-session")
self.connection = connection
self.queue = queue

View File

@@ -1,8 +1,8 @@
import Foundation
enum BridgeSettingsStore {
private static let bridgeService = "com.steipete.clawdis.bridge"
private static let nodeService = "com.steipete.clawdis.node"
private static let bridgeService = "com.clawdis.bridge"
private static let nodeService = "com.clawdis.node"
private static let instanceIdDefaultsKey = "node.instanceId"
private static let preferredBridgeStableIDDefaultsKey = "bridge.preferredStableID"

View File

@@ -3,6 +3,13 @@ import ClawdisKit
import Foundation
actor CameraController {
struct CameraDeviceInfo: Codable, Sendable {
var id: String
var name: String
var position: String
var deviceType: String
}
enum CameraError: LocalizedError, Sendable {
case cameraUnavailable
case microphoneUnavailable
@@ -41,13 +48,14 @@ actor CameraController {
// If you need the full-res photo, explicitly request a larger maxWidth.
let maxWidth = params.maxWidth.flatMap { $0 > 0 ? $0 : nil } ?? 1600
let quality = Self.clampQuality(params.quality)
let delayMs = max(0, params.delayMs ?? 0)
try await self.ensureAccess(for: .video)
let session = AVCaptureSession()
session.sessionPreset = .photo
guard let device = Self.pickCamera(facing: facing) else {
guard let device = Self.pickCamera(facing: facing, deviceId: params.deviceId) else {
throw CameraError.cameraUnavailable
}
@@ -67,6 +75,7 @@ actor CameraController {
session.startRunning()
defer { session.stopRunning() }
await Self.warmUpCaptureSession()
await Self.sleepDelayMs(delayMs)
let settings: AVCapturePhotoSettings = {
if output.availablePhotoCodecTypes.contains(.jpeg) {
@@ -84,10 +93,14 @@ actor CameraController {
}
withExtendedLifetime(delegate) {}
let maxPayloadBytes = 5 * 1024 * 1024
// Base64 inflates payloads by ~4/3; cap encoded bytes so the payload stays under 5MB (API limit).
let maxEncodedBytes = (maxPayloadBytes / 4) * 3
let res = try JPEGTranscoder.transcodeToJPEG(
imageData: rawData,
maxWidthPx: maxWidth,
quality: quality)
quality: quality,
maxBytes: maxEncodedBytes)
return (
format: format.rawValue,
@@ -115,7 +128,7 @@ actor CameraController {
let session = AVCaptureSession()
session.sessionPreset = .high
guard let camera = Self.pickCamera(facing: facing) else {
guard let camera = Self.pickCamera(facing: facing, deviceId: params.deviceId) else {
throw CameraError.cameraUnavailable
}
let cameraInput = try AVCaptureDeviceInput(device: camera)
@@ -176,6 +189,23 @@ actor CameraController {
hasAudio: includeAudio)
}
func listDevices() -> [CameraDeviceInfo] {
let types: [AVCaptureDevice.DeviceType] = [
.builtInWideAngleCamera,
]
let session = AVCaptureDevice.DiscoverySession(
deviceTypes: types,
mediaType: .video,
position: .unspecified)
return session.devices.map { device in
CameraDeviceInfo(
id: device.uniqueID,
name: device.localizedName,
position: Self.positionLabel(device.position),
deviceType: device.deviceType.rawValue)
}
}
private func ensureAccess(for mediaType: AVMediaType) async throws {
let status = AVCaptureDevice.authorizationStatus(for: mediaType)
switch status {
@@ -197,7 +227,15 @@ actor CameraController {
}
}
private nonisolated static func pickCamera(facing: ClawdisCameraFacing) -> AVCaptureDevice? {
private nonisolated static func pickCamera(
facing: ClawdisCameraFacing,
deviceId: String?) -> AVCaptureDevice?
{
if let deviceId, !deviceId.isEmpty {
if let match = AVCaptureDevice.devices(for: .video).first(where: { $0.uniqueID == deviceId }) {
return match
}
}
let position: AVCaptureDevice.Position = (facing == .front) ? .front : .back
if let device = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: position) {
return device
@@ -206,6 +244,14 @@ actor CameraController {
return AVCaptureDevice.default(for: .video)
}
private nonisolated static func positionLabel(_ position: AVCaptureDevice.Position) -> String {
switch position {
case .front: "front"
case .back: "back"
default: "unspecified"
}
}
nonisolated static func clampQuality(_ quality: Double?) -> Double {
let q = quality ?? 0.9
return min(1.0, max(0.05, q))
@@ -258,6 +304,13 @@ actor CameraController {
// A short delay after `startRunning()` significantly reduces "blank first frame" captures on some devices.
try? await Task.sleep(nanoseconds: 150_000_000) // 150ms
}
private nonisolated static func sleepDelayMs(_ delayMs: Int) async {
guard delayMs > 0 else { return }
let maxDelayMs = 10 * 1000
let ns = UInt64(min(delayMs, maxDelayMs)) * UInt64(NSEC_PER_MSEC)
try? await Task.sleep(nanoseconds: ns)
}
}
private final class PhotoCaptureDelegate: NSObject, AVCapturePhotoCaptureDelegate {

View File

@@ -4,18 +4,23 @@ import SwiftUI
struct ChatSheet: View {
@Environment(\.dismiss) private var dismiss
@State private var viewModel: ClawdisChatViewModel
private let userAccent: Color?
init(bridge: BridgeSession, sessionKey: String = "main") {
init(bridge: BridgeSession, sessionKey: String = "main", userAccent: Color? = nil) {
let transport = IOSBridgeChatTransport(bridge: bridge)
self._viewModel = State(
initialValue: ClawdisChatViewModel(
sessionKey: sessionKey,
transport: transport))
self.userAccent = userAccent
}
var body: some View {
NavigationStack {
ClawdisChatView(viewModel: self.viewModel)
ClawdisChatView(
viewModel: self.viewModel,
showsSessionSwitcher: true,
userAccent: self.userAccent)
.navigationTitle("Chat")
.navigationBarTitleDisplayMode(.inline)
.toolbar {

View File

@@ -35,6 +35,10 @@
<string>Clawdis can capture photos or short video clips when requested via the bridge.</string>
<key>NSLocalNetworkUsageDescription</key>
<string>Clawdis discovers and connects to your Clawdis bridge on the local network.</string>
<key>NSLocationWhenInUseUsageDescription</key>
<string>Clawdis uses your location when you allow location sharing.</string>
<key>NSLocationAlwaysAndWhenInUseUsageDescription</key>
<string>Clawdis can share your location in the background when you enable Always.</string>
<key>NSMicrophoneUsageDescription</key>
<string>Clawdis needs microphone access for voice wake.</string>
<key>NSSpeechRecognitionUsageDescription</key>
@@ -50,5 +54,19 @@
</array>
<key>UILaunchScreen</key>
<dict/>
<key>UISupportedInterfaceOrientations</key>
<array>
<string>UIInterfaceOrientationPortrait</string>
<string>UIInterfaceOrientationPortraitUpsideDown</string>
<string>UIInterfaceOrientationLandscapeLeft</string>
<string>UIInterfaceOrientationLandscapeRight</string>
</array>
<key>UISupportedInterfaceOrientations~ipad</key>
<array>
<string>UIInterfaceOrientationPortrait</string>
<string>UIInterfaceOrientationPortraitUpsideDown</string>
<string>UIInterfaceOrientationLandscapeLeft</string>
<string>UIInterfaceOrientationLandscapeRight</string>
</array>
</dict>
</plist>

View File

@@ -0,0 +1,142 @@
import ClawdisKit
import CoreLocation
import Foundation
@MainActor
final class LocationService: NSObject, CLLocationManagerDelegate {
enum Error: Swift.Error {
case timeout
case unavailable
}
private let manager = CLLocationManager()
private var authContinuation: CheckedContinuation<CLAuthorizationStatus, Never>?
private var locationContinuation: CheckedContinuation<CLLocation, Swift.Error>?
override init() {
super.init()
self.manager.delegate = self
self.manager.desiredAccuracy = kCLLocationAccuracyBest
}
func authorizationStatus() -> CLAuthorizationStatus {
self.manager.authorizationStatus
}
func accuracyAuthorization() -> CLAccuracyAuthorization {
if #available(iOS 14.0, *) {
return self.manager.accuracyAuthorization
}
return .fullAccuracy
}
func ensureAuthorization(mode: ClawdisLocationMode) async -> CLAuthorizationStatus {
guard CLLocationManager.locationServicesEnabled() else { return .denied }
let status = self.manager.authorizationStatus
if status == .notDetermined {
self.manager.requestWhenInUseAuthorization()
let updated = await self.awaitAuthorizationChange()
if mode != .always { return updated }
}
if mode == .always {
let current = self.manager.authorizationStatus
if current == .authorizedWhenInUse {
self.manager.requestAlwaysAuthorization()
return await self.awaitAuthorizationChange()
}
return current
}
return self.manager.authorizationStatus
}
func currentLocation(
params: ClawdisLocationGetParams,
desiredAccuracy: ClawdisLocationAccuracy,
maxAgeMs: Int?,
timeoutMs: Int?) async throws -> CLLocation
{
let now = Date()
if let maxAgeMs,
let cached = self.manager.location,
now.timeIntervalSince(cached.timestamp) * 1000 <= Double(maxAgeMs)
{
return cached
}
self.manager.desiredAccuracy = Self.accuracyValue(desiredAccuracy)
let timeout = max(0, timeoutMs ?? 10_000)
return try await self.withTimeout(timeoutMs: timeout) {
try await self.requestLocation()
}
}
private func requestLocation() async throws -> CLLocation {
try await withCheckedThrowingContinuation { cont in
self.locationContinuation = cont
self.manager.requestLocation()
}
}
private func awaitAuthorizationChange() async -> CLAuthorizationStatus {
await withCheckedContinuation { cont in
self.authContinuation = cont
}
}
private func withTimeout<T>(
timeoutMs: Int,
operation: @escaping () async throws -> T) async throws -> T
{
if timeoutMs == 0 {
return try await operation()
}
return try await withThrowingTaskGroup(of: T.self) { group in
group.addTask { try await operation() }
group.addTask {
try await Task.sleep(nanoseconds: UInt64(timeoutMs) * 1_000_000)
throw Error.timeout
}
let result = try await group.next()!
group.cancelAll()
return result
}
}
private static func accuracyValue(_ accuracy: ClawdisLocationAccuracy) -> CLLocationAccuracy {
switch accuracy {
case .coarse:
return kCLLocationAccuracyKilometer
case .balanced:
return kCLLocationAccuracyHundredMeters
case .precise:
return kCLLocationAccuracyBest
}
}
func locationManagerDidChangeAuthorization(_ manager: CLLocationManager) {
if let cont = self.authContinuation {
self.authContinuation = nil
cont.resume(returning: manager.authorizationStatus)
}
}
func locationManager(_ manager: CLLocationManager, didUpdateLocations locations: [CLLocation]) {
guard let cont = self.locationContinuation else { return }
self.locationContinuation = nil
if let latest = locations.last {
cont.resume(returning: latest)
} else {
cont.resume(throwing: Error.unavailable)
}
}
func locationManager(_ manager: CLLocationManager, didFailWithError error: Swift.Error) {
guard let cont = self.locationContinuation else { return }
self.locationContinuation = nil
cont.resume(throwing: error)
}
}

View File

@@ -22,12 +22,16 @@ final class NodeAppModel {
var bridgeServerName: String?
var bridgeRemoteAddress: String?
var connectedBridgeID: String?
var seamColorHex: String?
var mainSessionKey: String = "main"
private let bridge = BridgeSession()
private var bridgeTask: Task<Void, Never>?
private var voiceWakeSyncTask: Task<Void, Never>?
@ObservationIgnored private var cameraHUDDismissTask: Task<Void, Never>?
let voiceWake = VoiceWakeManager()
let talkMode = TalkModeManager()
private let locationService = LocationService()
private var lastAutoA2uiURL: String?
var bridgeSession: BridgeSession { self.bridge }
@@ -35,11 +39,12 @@ final class NodeAppModel {
var cameraHUDText: String?
var cameraHUDKind: CameraHUDKind?
var cameraFlashNonce: Int = 0
var screenRecordActive: Bool = false
init() {
self.voiceWake.configure { [weak self] cmd in
guard let self else { return }
let sessionKey = "main"
let sessionKey = await MainActor.run { self.mainSessionKey }
do {
try await self.sendVoiceTranscript(text: cmd, sessionKey: sessionKey)
} catch {
@@ -49,6 +54,9 @@ final class NodeAppModel {
let enabled = UserDefaults.standard.bool(forKey: "voiceWake.enabled")
self.voiceWake.setEnabled(enabled)
self.talkMode.attachBridge(self.bridge)
let talkEnabled = UserDefaults.standard.bool(forKey: "talk.enabled")
self.talkMode.setEnabled(talkEnabled)
// Wire up deep links from canvas taps
self.screen.onDeepLink = { [weak self] url in
@@ -145,7 +153,7 @@ final class NodeAppModel {
guard let raw = await self.bridge.currentCanvasHostUrl() else { return nil }
let trimmed = raw.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty, let base = URL(string: trimmed) else { return nil }
return base.appendingPathComponent("__clawdis__/a2ui/").absoluteString
return base.appendingPathComponent("__clawdis__/a2ui/").absoluteString + "?platform=ios"
}
private func showA2UIOnConnectIfNeeded() async {
@@ -177,6 +185,23 @@ final class NodeAppModel {
self.voiceWake.setEnabled(enabled)
}
func setTalkEnabled(_ enabled: Bool) {
self.talkMode.setEnabled(enabled)
}
func requestLocationPermissions(mode: ClawdisLocationMode) async -> Bool {
guard mode != .off else { return true }
let status = await self.locationService.ensureAuthorization(mode: mode)
switch status {
case .authorizedAlways:
return true
case .authorizedWhenInUse:
return mode != .always
default:
return false
}
}
func connectToBridge(
endpoint: NWEndpoint,
hello: BridgeHello)
@@ -216,6 +241,7 @@ final class NodeAppModel {
self.bridgeRemoteAddress = addr
}
}
await self.refreshBrandingFromGateway()
await self.startVoiceWakeSync()
await self.showA2UIOnConnectIfNeeded()
},
@@ -255,6 +281,8 @@ final class NodeAppModel {
self.bridgeServerName = nil
self.bridgeRemoteAddress = nil
self.connectedBridgeID = nil
self.seamColorHex = nil
self.mainSessionKey = "main"
self.showLocalCanvasOnDisconnect()
}
}
@@ -270,9 +298,47 @@ final class NodeAppModel {
self.bridgeServerName = nil
self.bridgeRemoteAddress = nil
self.connectedBridgeID = nil
self.seamColorHex = nil
self.mainSessionKey = "main"
self.showLocalCanvasOnDisconnect()
}
var seamColor: Color {
Self.color(fromHex: self.seamColorHex) ?? Self.defaultSeamColor
}
private static let defaultSeamColor = Color(red: 79 / 255.0, green: 122 / 255.0, blue: 154 / 255.0)
private static func color(fromHex raw: String?) -> Color? {
let trimmed = (raw ?? "").trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty else { return nil }
let hex = trimmed.hasPrefix("#") ? String(trimmed.dropFirst()) : trimmed
guard hex.count == 6, let value = Int(hex, radix: 16) else { return nil }
let r = Double((value >> 16) & 0xFF) / 255.0
let g = Double((value >> 8) & 0xFF) / 255.0
let b = Double(value & 0xFF) / 255.0
return Color(red: r, green: g, blue: b)
}
private func refreshBrandingFromGateway() async {
do {
let res = try await self.bridge.request(method: "config.get", paramsJSON: "{}", timeoutSeconds: 8)
guard let json = try JSONSerialization.jsonObject(with: res) as? [String: Any] else { return }
guard let config = json["config"] as? [String: Any] else { return }
let ui = config["ui"] as? [String: Any]
let raw = (ui?["seamColor"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
let session = config["session"] as? [String: Any]
let rawMainKey = (session?["mainKey"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
let mainKey = rawMainKey.isEmpty ? "main" : rawMainKey
await MainActor.run {
self.seamColorHex = raw.isEmpty ? nil : raw
self.mainSessionKey = mainKey
}
} catch {
// ignore
}
}
func setGlobalWakeWords(_ words: [String]) async {
let sanitized = VoiceWakePreferences.sanitizeTriggerWords(words)
@@ -414,6 +480,64 @@ final class NodeAppModel {
do {
switch command {
case ClawdisLocationCommand.get.rawValue:
let mode = self.locationMode()
guard mode != .off else {
return BridgeInvokeResponse(
id: req.id,
ok: false,
error: ClawdisNodeError(
code: .unavailable,
message: "LOCATION_DISABLED: enable Location in Settings"))
}
if self.isBackgrounded, mode != .always {
return BridgeInvokeResponse(
id: req.id,
ok: false,
error: ClawdisNodeError(
code: .backgroundUnavailable,
message: "LOCATION_BACKGROUND_UNAVAILABLE: background location requires Always"))
}
let params = (try? Self.decodeParams(ClawdisLocationGetParams.self, from: req.paramsJSON)) ??
ClawdisLocationGetParams()
let desired = params.desiredAccuracy ??
(self.isLocationPreciseEnabled() ? .precise : .balanced)
let status = self.locationService.authorizationStatus()
if status != .authorizedAlways && status != .authorizedWhenInUse {
return BridgeInvokeResponse(
id: req.id,
ok: false,
error: ClawdisNodeError(
code: .unavailable,
message: "LOCATION_PERMISSION_REQUIRED: grant Location permission"))
}
if self.isBackgrounded && status != .authorizedAlways {
return BridgeInvokeResponse(
id: req.id,
ok: false,
error: ClawdisNodeError(
code: .unavailable,
message: "LOCATION_PERMISSION_REQUIRED: enable Always for background access"))
}
let location = try await self.locationService.currentLocation(
params: params,
desiredAccuracy: desired,
maxAgeMs: params.maxAgeMs,
timeoutMs: params.timeoutMs)
let isPrecise = self.locationService.accuracyAuthorization() == .fullAccuracy
let payload = ClawdisLocationPayload(
lat: location.coordinate.latitude,
lon: location.coordinate.longitude,
accuracyMeters: location.horizontalAccuracy,
altitudeMeters: location.verticalAccuracy >= 0 ? location.altitude : nil,
speedMps: location.speed >= 0 ? location.speed : nil,
headingDeg: location.course >= 0 ? location.course : nil,
timestamp: ISO8601DateFormatter().string(from: location.timestamp),
isPrecise: isPrecise,
source: nil)
let json = try Self.encodePayload(payload)
return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: json)
case ClawdisCanvasCommand.present.rawValue:
let params = (try? Self.decodeParams(ClawdisCanvasPresentParams.self, from: req.paramsJSON)) ??
ClawdisCanvasPresentParams()
@@ -537,6 +661,14 @@ final class NodeAppModel {
let resultJSON = try await self.screen.eval(javaScript: js)
return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: resultJSON)
case ClawdisCameraCommand.list.rawValue:
let devices = await self.camera.listDevices()
struct Payload: Codable {
var devices: [CameraController.CameraDeviceInfo]
}
let payload = try Self.encodePayload(Payload(devices: devices))
return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: payload)
case ClawdisCameraCommand.snap.rawValue:
self.showCameraHUD(text: "Taking photo…", kind: .photo)
self.triggerCameraFlash()
@@ -590,6 +722,9 @@ final class NodeAppModel {
NSLocalizedDescriptionKey: "INVALID_REQUEST: screen format must be mp4",
])
}
// Status pill mirrors screen recording state so it stays visible without overlay stacking.
self.screenRecordActive = true
defer { self.screenRecordActive = false }
let path = try await self.screenRecorder.record(
screenIndex: params.screenIndex,
durationMs: params.durationMs,
@@ -633,6 +768,16 @@ final class NodeAppModel {
}
}
private func locationMode() -> ClawdisLocationMode {
let raw = UserDefaults.standard.string(forKey: "location.enabledMode") ?? "off"
return ClawdisLocationMode(rawValue: raw) ?? .off
}
private func isLocationPreciseEnabled() -> Bool {
if UserDefaults.standard.object(forKey: "location.preciseEnabled") == nil { return true }
return UserDefaults.standard.bool(forKey: "location.preciseEnabled")
}
private static func decodeParams<T: Decodable>(_ type: T.Type, from json: String?) throws -> T {
guard let json, let data = json.data(using: .utf8) else {
throw NSError(domain: "Bridge", code: 20, userInfo: [
@@ -680,3 +825,43 @@ final class NodeAppModel {
}
}
}
#if DEBUG
extension NodeAppModel {
func _test_handleInvoke(_ req: BridgeInvokeRequest) async -> BridgeInvokeResponse {
await self.handleInvoke(req)
}
static func _test_decodeParams<T: Decodable>(_ type: T.Type, from json: String?) throws -> T {
try self.decodeParams(type, from: json)
}
static func _test_encodePayload(_ obj: some Encodable) throws -> String {
try self.encodePayload(obj)
}
func _test_isCameraEnabled() -> Bool {
self.isCameraEnabled()
}
func _test_triggerCameraFlash() {
self.triggerCameraFlash()
}
func _test_showCameraHUD(text: String, kind: CameraHUDKind, autoHideSeconds: Double? = nil) {
self.showCameraHUD(text: text, kind: kind, autoHideSeconds: autoHideSeconds)
}
func _test_handleCanvasA2UIAction(body: [String: Any]) async {
await self.handleCanvasA2UIAction(body: body)
}
func _test_resolveA2UIHostURL() async -> String? {
await self.resolveA2UIHostURL()
}
func _test_showLocalCanvasOnDisconnect() {
self.showLocalCanvasOnDisconnect()
}
}
#endif

View File

@@ -51,7 +51,10 @@ struct RootCanvas: View {
case .settings:
SettingsTab()
case .chat:
ChatSheet(bridge: self.appModel.bridgeSession)
ChatSheet(
bridge: self.appModel.bridgeSession,
sessionKey: self.appModel.mainSessionKey,
userAccent: self.appModel.seamColor)
}
}
.onAppear { self.updateIdleTimer() }
@@ -119,6 +122,9 @@ struct RootCanvas: View {
}
private struct CanvasContent: View {
@Environment(NodeAppModel.self) private var appModel
@AppStorage("talk.enabled") private var talkEnabled: Bool = false
@AppStorage("talk.button.enabled") private var talkButtonEnabled: Bool = true
var systemColorScheme: ColorScheme
var bridgeStatus: StatusPill.BridgeState
var voiceWakeEnabled: Bool
@@ -140,6 +146,21 @@ private struct CanvasContent: View {
}
.accessibilityLabel("Chat")
if self.talkButtonEnabled {
// Talk mode lives on a side bubble so it doesn't get buried in settings.
OverlayButton(
systemImage: self.appModel.talkMode.isEnabled ? "waveform.circle.fill" : "waveform.circle",
brighten: self.brightenButtons,
tint: self.appModel.seamColor,
isActive: self.appModel.talkMode.isEnabled)
{
let next = !self.appModel.talkMode.isEnabled
self.talkEnabled = next
self.appModel.setTalkEnabled(next)
}
.accessibilityLabel("Talk Mode")
}
OverlayButton(systemImage: "gearshape.fill", brighten: self.brightenButtons) {
self.openSettings()
}
@@ -148,10 +169,17 @@ private struct CanvasContent: View {
.padding(.top, 10)
.padding(.trailing, 10)
}
.overlay(alignment: .center) {
if self.appModel.talkMode.isEnabled {
TalkOrbOverlay()
.transition(.opacity)
}
}
.overlay(alignment: .topLeading) {
StatusPill(
bridge: self.bridgeStatus,
voiceWakeEnabled: self.voiceWakeEnabled,
activity: self.statusActivity,
brighten: self.brightenButtons,
onTap: {
self.openSettings()
@@ -169,45 +197,78 @@ private struct CanvasContent: View {
.transition(.move(edge: .top).combined(with: .opacity))
}
}
.overlay(alignment: .topLeading) {
if let cameraHUDText, !cameraHUDText.isEmpty, let cameraHUDKind {
CameraCaptureToast(
text: cameraHUDText,
kind: self.mapCameraKind(cameraHUDKind),
brighten: self.brightenButtons)
.padding(SwiftUI.Edge.Set.leading, 10)
.safeAreaPadding(SwiftUI.Edge.Set.top, 106)
.transition(
AnyTransition.move(edge: SwiftUI.Edge.top)
.combined(with: AnyTransition.opacity))
}
}
}
private func mapCameraKind(_ kind: NodeAppModel.CameraHUDKind) -> CameraCaptureToast.Kind {
switch kind {
case .photo:
.photo
case .recording:
.recording
case .success:
.success
case .error:
.error
private var statusActivity: StatusPill.Activity? {
// Status pill owns transient activity state so it doesn't overlap the connection indicator.
if self.appModel.isBackgrounded {
return StatusPill.Activity(
title: "Foreground required",
systemImage: "exclamationmark.triangle.fill",
tint: .orange)
}
let bridgeStatus = self.appModel.bridgeStatusText.trimmingCharacters(in: .whitespacesAndNewlines)
let bridgeLower = bridgeStatus.lowercased()
if bridgeLower.contains("repair") {
return StatusPill.Activity(title: "Repairing…", systemImage: "wrench.and.screwdriver", tint: .orange)
}
if bridgeLower.contains("approval") || bridgeLower.contains("pairing") {
return StatusPill.Activity(title: "Approval pending", systemImage: "person.crop.circle.badge.clock")
}
// Avoid duplicating the primary bridge status ("Connecting") in the activity slot.
if self.appModel.screenRecordActive {
return StatusPill.Activity(title: "Recording screen…", systemImage: "record.circle.fill", tint: .red)
}
if let cameraHUDText, !cameraHUDText.isEmpty, let cameraHUDKind {
let systemImage: String
let tint: Color?
switch cameraHUDKind {
case .photo:
systemImage = "camera.fill"
tint = nil
case .recording:
systemImage = "video.fill"
tint = .red
case .success:
systemImage = "checkmark.circle.fill"
tint = .green
case .error:
systemImage = "exclamationmark.triangle.fill"
tint = .red
}
return StatusPill.Activity(title: cameraHUDText, systemImage: systemImage, tint: tint)
}
if self.voiceWakeEnabled {
let voiceStatus = self.appModel.voiceWake.statusText
if voiceStatus.localizedCaseInsensitiveContains("microphone permission") {
return StatusPill.Activity(title: "Mic permission", systemImage: "mic.slash", tint: .orange)
}
if voiceStatus == "Paused" {
let suffix = self.appModel.isBackgrounded ? " (background)" : ""
return StatusPill.Activity(title: "Voice Wake paused\(suffix)", systemImage: "pause.circle.fill")
}
}
return nil
}
}
private struct OverlayButton: View {
let systemImage: String
let brighten: Bool
var tint: Color?
var isActive: Bool = false
let action: () -> Void
var body: some View {
Button(action: self.action) {
Image(systemName: self.systemImage)
.font(.system(size: 16, weight: .semibold))
.foregroundStyle(.primary)
.foregroundStyle(self.isActive ? (self.tint ?? .primary) : .primary)
.padding(10)
.background {
RoundedRectangle(cornerRadius: 12, style: .continuous)
@@ -225,9 +286,26 @@ private struct OverlayButton: View {
endPoint: .bottomTrailing))
.blendMode(.overlay)
}
.overlay {
if let tint {
RoundedRectangle(cornerRadius: 12, style: .continuous)
.fill(
LinearGradient(
colors: [
tint.opacity(self.isActive ? 0.22 : 0.14),
tint.opacity(self.isActive ? 0.10 : 0.06),
.clear,
],
startPoint: .topLeading,
endPoint: .bottomTrailing))
.blendMode(.overlay)
}
}
.overlay {
RoundedRectangle(cornerRadius: 12, style: .continuous)
.strokeBorder(.white.opacity(self.brighten ? 0.24 : 0.18), lineWidth: 0.5)
.strokeBorder(
(self.tint ?? .white).opacity(self.isActive ? 0.34 : (self.brighten ? 0.24 : 0.18)),
lineWidth: self.isActive ? 0.7 : 0.5)
}
.shadow(color: .black.opacity(0.35), radius: 12, y: 6)
}
@@ -261,59 +339,3 @@ private struct CameraFlashOverlay: View {
}
}
}
private struct CameraCaptureToast: View {
enum Kind {
case photo
case recording
case success
case error
}
var text: String
var kind: Kind
var brighten: Bool = false
var body: some View {
HStack(spacing: 10) {
self.icon
.font(.system(size: 14, weight: .semibold))
.foregroundStyle(.primary)
Text(self.text)
.font(.system(size: 14, weight: .semibold))
.foregroundStyle(.primary)
.lineLimit(1)
.truncationMode(.tail)
}
.padding(.vertical, 10)
.padding(.horizontal, 12)
.background {
RoundedRectangle(cornerRadius: 14, style: .continuous)
.fill(.ultraThinMaterial)
.overlay {
RoundedRectangle(cornerRadius: 14, style: .continuous)
.strokeBorder(.white.opacity(self.brighten ? 0.24 : 0.18), lineWidth: 0.5)
}
.shadow(color: .black.opacity(0.25), radius: 12, y: 6)
}
.accessibilityLabel("Camera")
.accessibilityValue(self.text)
}
@ViewBuilder
private var icon: some View {
switch self.kind {
case .photo:
Image(systemName: "camera.fill")
case .recording:
Image(systemName: "record.circle.fill")
.symbolRenderingMode(.palette)
.foregroundStyle(.red, .primary)
case .success:
Image(systemName: "checkmark.circle.fill")
case .error:
Image(systemName: "exclamationmark.triangle.fill")
}
}
}

View File

@@ -26,6 +26,7 @@ struct RootTabs: View {
StatusPill(
bridge: self.bridgeStatus,
voiceWakeEnabled: self.voiceWakeEnabled,
activity: self.statusActivity,
onTap: { self.selectedTab = 2 })
.padding(.leading, 10)
.safeAreaPadding(.top, 10)
@@ -79,4 +80,64 @@ struct RootTabs: View {
return .disconnected
}
private var statusActivity: StatusPill.Activity? {
// Keep the top pill consistent across tabs (camera + voice wake + pairing states).
if self.appModel.isBackgrounded {
return StatusPill.Activity(
title: "Foreground required",
systemImage: "exclamationmark.triangle.fill",
tint: .orange)
}
let bridgeStatus = self.appModel.bridgeStatusText.trimmingCharacters(in: .whitespacesAndNewlines)
let bridgeLower = bridgeStatus.lowercased()
if bridgeLower.contains("repair") {
return StatusPill.Activity(title: "Repairing…", systemImage: "wrench.and.screwdriver", tint: .orange)
}
if bridgeLower.contains("approval") || bridgeLower.contains("pairing") {
return StatusPill.Activity(title: "Approval pending", systemImage: "person.crop.circle.badge.clock")
}
// Avoid duplicating the primary bridge status ("Connecting") in the activity slot.
if self.appModel.screenRecordActive {
return StatusPill.Activity(title: "Recording screen…", systemImage: "record.circle.fill", tint: .red)
}
if let cameraHUDText = self.appModel.cameraHUDText,
let cameraHUDKind = self.appModel.cameraHUDKind,
!cameraHUDText.isEmpty
{
let systemImage: String
let tint: Color?
switch cameraHUDKind {
case .photo:
systemImage = "camera.fill"
tint = nil
case .recording:
systemImage = "video.fill"
tint = .red
case .success:
systemImage = "checkmark.circle.fill"
tint = .green
case .error:
systemImage = "exclamationmark.triangle.fill"
tint = .red
}
return StatusPill.Activity(title: cameraHUDText, systemImage: systemImage, tint: tint)
}
if self.voiceWakeEnabled {
let voiceStatus = self.appModel.voiceWake.statusText
if voiceStatus.localizedCaseInsensitiveContains("microphone permission") {
return StatusPill.Activity(title: "Mic permission", systemImage: "mic.slash", tint: .orange)
}
if voiceStatus == "Paused" {
let suffix = self.appModel.isBackgrounded ? " (background)" : ""
return StatusPill.Activity(title: "Voice Wake paused\(suffix)", systemImage: "pause.circle.fill")
}
}
return nil
}
}

View File

@@ -43,9 +43,7 @@ final class ScreenController {
self.webView.scrollView.contentInset = .zero
self.webView.scrollView.scrollIndicatorInsets = .zero
self.webView.scrollView.automaticallyAdjustsScrollIndicatorInsets = false
// Disable scroll to allow touch events to pass through to canvas
self.webView.scrollView.isScrollEnabled = false
self.webView.scrollView.bounces = false
self.applyScrollBehavior()
self.webView.navigationDelegate = self.navigationDelegate
self.navigationDelegate.controller = self
a2uiActionHandler.controller = self
@@ -60,6 +58,7 @@ final class ScreenController {
func reload() {
let trimmed = self.urlString.trimmingCharacters(in: .whitespacesAndNewlines)
self.applyScrollBehavior()
if trimmed.isEmpty {
guard let url = Self.canvasScaffoldURL else { return }
self.errorText = nil
@@ -129,7 +128,8 @@ final class ScreenController {
} catch (_) { return false; }
})()
""")
if res == "true" { return true }
let trimmed = res.trimmingCharacters(in: .whitespacesAndNewlines).lowercased()
if trimmed == "true" || trimmed == "1" { return true }
} catch {
// ignore; page likely still loading
}
@@ -249,6 +249,15 @@ final class ScreenController {
return false
}
private func applyScrollBehavior() {
let trimmed = self.urlString.trimmingCharacters(in: .whitespacesAndNewlines)
let allowScroll = !trimmed.isEmpty
let scrollView = self.webView.scrollView
// Default canvas needs raw touch events; external pages should scroll.
scrollView.isScrollEnabled = allowScroll
scrollView.bounces = allowScroll
}
private static func jsValue(_ value: String?) -> String {
guard let value else { return "null" }
if let data = try? JSONSerialization.data(withJSONObject: [value]),

View File

@@ -1,12 +1,28 @@
import AVFoundation
import ReplayKit
@MainActor
final class ScreenRecordService {
final class ScreenRecordService: @unchecked Sendable {
private struct UncheckedSendableBox<T>: @unchecked Sendable {
let value: T
}
private final class CaptureState: @unchecked Sendable {
private let lock = NSLock()
var writer: AVAssetWriter?
var videoInput: AVAssetWriterInput?
var audioInput: AVAssetWriterInput?
var started = false
var sawVideo = false
var lastVideoTime: CMTime?
var handlerError: Error?
func withLock<T>(_ body: (CaptureState) -> T) -> T {
self.lock.lock()
defer { lock.unlock() }
return body(self)
}
}
enum ScreenRecordError: LocalizedError {
case invalidScreenIndex(Int)
case captureFailed(String)
@@ -51,126 +67,158 @@ final class ScreenRecordService {
}()
try? FileManager.default.removeItem(at: outURL)
let recorder = RPScreenRecorder.shared()
recorder.isMicrophoneEnabled = includeAudio
var writer: AVAssetWriter?
var videoInput: AVAssetWriterInput?
var audioInput: AVAssetWriterInput?
var started = false
var sawVideo = false
var lastVideoTime: CMTime?
var handlerError: Error?
let lock = NSLock()
func setHandlerError(_ error: Error) {
lock.lock()
defer { lock.unlock() }
if handlerError == nil { handlerError = error }
}
let state = CaptureState()
let recordQueue = DispatchQueue(label: "com.clawdis.screenrecord")
try await withCheckedThrowingContinuation { (cont: CheckedContinuation<Void, Error>) in
recorder.startCapture(handler: { sample, type, error in
if let error {
setHandlerError(error)
return
}
guard CMSampleBufferDataIsReady(sample) else { return }
switch type {
case .video:
let pts = CMSampleBufferGetPresentationTimeStamp(sample)
if let lastVideoTime {
let delta = CMTimeSubtract(pts, lastVideoTime)
if delta.seconds < (1.0 / fpsValue) { return }
}
if writer == nil {
guard let imageBuffer = CMSampleBufferGetImageBuffer(sample) else {
setHandlerError(ScreenRecordError.captureFailed("Missing image buffer"))
return
let handler: @Sendable (CMSampleBuffer, RPSampleBufferType, Error?) -> Void = { sample, type, error in
// ReplayKit can call the capture handler on a background queue.
// Serialize writes to avoid queue asserts.
recordQueue.async {
if let error {
state.withLock { state in
if state.handlerError == nil { state.handlerError = error }
}
let width = CVPixelBufferGetWidth(imageBuffer)
let height = CVPixelBufferGetHeight(imageBuffer)
do {
let w = try AVAssetWriter(outputURL: outURL, fileType: .mp4)
let settings: [String: Any] = [
AVVideoCodecKey: AVVideoCodecType.h264,
AVVideoWidthKey: width,
AVVideoHeightKey: height,
]
let vInput = AVAssetWriterInput(mediaType: .video, outputSettings: settings)
vInput.expectsMediaDataInRealTime = true
guard w.canAdd(vInput) else {
throw ScreenRecordError.writeFailed("Cannot add video input")
}
w.add(vInput)
return
}
guard CMSampleBufferDataIsReady(sample) else { return }
if includeAudio {
let aInput = AVAssetWriterInput(mediaType: .audio, outputSettings: nil)
aInput.expectsMediaDataInRealTime = true
if w.canAdd(aInput) {
w.add(aInput)
audioInput = aInput
switch type {
case .video:
let pts = CMSampleBufferGetPresentationTimeStamp(sample)
let shouldSkip = state.withLock { state in
if let lastVideoTime = state.lastVideoTime {
let delta = CMTimeSubtract(pts, lastVideoTime)
return delta.seconds < (1.0 / fpsValue)
}
return false
}
if shouldSkip { return }
if state.withLock({ $0.writer == nil }) {
guard let imageBuffer = CMSampleBufferGetImageBuffer(sample) else {
state.withLock { state in
if state.handlerError == nil {
state.handlerError = ScreenRecordError.captureFailed("Missing image buffer")
}
}
return
}
let width = CVPixelBufferGetWidth(imageBuffer)
let height = CVPixelBufferGetHeight(imageBuffer)
do {
let w = try AVAssetWriter(outputURL: outURL, fileType: .mp4)
let settings: [String: Any] = [
AVVideoCodecKey: AVVideoCodecType.h264,
AVVideoWidthKey: width,
AVVideoHeightKey: height,
]
let vInput = AVAssetWriterInput(mediaType: .video, outputSettings: settings)
vInput.expectsMediaDataInRealTime = true
guard w.canAdd(vInput) else {
throw ScreenRecordError.writeFailed("Cannot add video input")
}
w.add(vInput)
if includeAudio {
let aInput = AVAssetWriterInput(mediaType: .audio, outputSettings: nil)
aInput.expectsMediaDataInRealTime = true
if w.canAdd(aInput) {
w.add(aInput)
state.withLock { state in
state.audioInput = aInput
}
}
}
guard w.startWriting() else {
throw ScreenRecordError
.writeFailed(w.error?.localizedDescription ?? "Failed to start writer")
}
w.startSession(atSourceTime: pts)
state.withLock { state in
state.writer = w
state.videoInput = vInput
state.started = true
}
} catch {
state.withLock { state in
if state.handlerError == nil { state.handlerError = error }
}
return
}
}
let vInput = state.withLock { $0.videoInput }
let isStarted = state.withLock { $0.started }
guard let vInput, isStarted else { return }
if vInput.isReadyForMoreMediaData {
if vInput.append(sample) {
state.withLock { state in
state.sawVideo = true
state.lastVideoTime = pts
}
} else {
let err = state.withLock { $0.writer?.error }
if let err {
state.withLock { state in
if state.handlerError == nil {
state.handlerError = ScreenRecordError.writeFailed(err.localizedDescription)
}
}
}
}
guard w.startWriting() else {
throw ScreenRecordError
.writeFailed(w.error?.localizedDescription ?? "Failed to start writer")
}
w.startSession(atSourceTime: pts)
writer = w
videoInput = vInput
started = true
} catch {
setHandlerError(error)
return
}
}
guard let vInput = videoInput, started else { return }
if vInput.isReadyForMoreMediaData {
if vInput.append(sample) {
sawVideo = true
lastVideoTime = pts
} else {
if let err = writer?.error {
setHandlerError(ScreenRecordError.writeFailed(err.localizedDescription))
}
case .audioApp, .audioMic:
let aInput = state.withLock { $0.audioInput }
let isStarted = state.withLock { $0.started }
guard includeAudio, let aInput, isStarted else { return }
if aInput.isReadyForMoreMediaData {
_ = aInput.append(sample)
}
}
case .audioApp, .audioMic:
guard includeAudio, let aInput = audioInput, started else { return }
if aInput.isReadyForMoreMediaData {
_ = aInput.append(sample)
@unknown default:
break
}
@unknown default:
break
}
}, completionHandler: { error in
}
let completion: @Sendable (Error?) -> Void = { error in
if let error { cont.resume(throwing: error) } else { cont.resume() }
})
}
Task { @MainActor in
startReplayKitCapture(
includeAudio: includeAudio,
handler: handler,
completion: completion)
}
}
try await Task.sleep(nanoseconds: UInt64(durationMs) * 1_000_000)
let stopError = await withCheckedContinuation { cont in
recorder.stopCapture { error in cont.resume(returning: error) }
Task { @MainActor in
stopReplayKitCapture { error in cont.resume(returning: error) }
}
}
if let stopError { throw stopError }
if let handlerError { throw handlerError }
guard let writer, let videoInput, sawVideo else {
let handlerErrorSnapshot = state.withLock { $0.handlerError }
if let handlerErrorSnapshot { throw handlerErrorSnapshot }
let writerSnapshot = state.withLock { $0.writer }
let videoInputSnapshot = state.withLock { $0.videoInput }
let audioInputSnapshot = state.withLock { $0.audioInput }
let sawVideoSnapshot = state.withLock { $0.sawVideo }
guard let writerSnapshot, let videoInputSnapshot, sawVideoSnapshot else {
throw ScreenRecordError.captureFailed("No frames captured")
}
videoInput.markAsFinished()
audioInput?.markAsFinished()
videoInputSnapshot.markAsFinished()
audioInputSnapshot?.markAsFinished()
let writerBox = UncheckedSendableBox(value: writer)
let writerBox = UncheckedSendableBox(value: writerSnapshot)
try await withCheckedThrowingContinuation { (cont: CheckedContinuation<Void, Error>) in
writerBox.value.finishWriting {
let writer = writerBox.value
@@ -198,3 +246,31 @@ final class ScreenRecordService {
return min(30, max(1, v))
}
}
@MainActor
private func startReplayKitCapture(
includeAudio: Bool,
handler: @escaping @Sendable (CMSampleBuffer, RPSampleBufferType, Error?) -> Void,
completion: @escaping @Sendable (Error?) -> Void)
{
let recorder = RPScreenRecorder.shared()
recorder.isMicrophoneEnabled = includeAudio
recorder.startCapture(handler: handler, completionHandler: completion)
}
@MainActor
private func stopReplayKitCapture(_ completion: @escaping @Sendable (Error?) -> Void) {
RPScreenRecorder.shared().stopCapture { error in completion(error) }
}
#if DEBUG
extension ScreenRecordService {
nonisolated static func _test_clampDurationMs(_ ms: Int?) -> Int {
self.clampDurationMs(ms)
}
nonisolated static func _test_clampFps(_ fps: Double?) -> Double {
self.clampFps(fps)
}
}
#endif

View File

@@ -20,7 +20,11 @@ struct SettingsTab: View {
@AppStorage("node.displayName") private var displayName: String = "iOS Node"
@AppStorage("node.instanceId") private var instanceId: String = UUID().uuidString
@AppStorage("voiceWake.enabled") private var voiceWakeEnabled: Bool = false
@AppStorage("talk.enabled") private var talkEnabled: Bool = false
@AppStorage("talk.button.enabled") private var talkButtonEnabled: Bool = true
@AppStorage("camera.enabled") private var cameraEnabled: Bool = true
@AppStorage("location.enabledMode") private var locationEnabledModeRaw: String = ClawdisLocationMode.off.rawValue
@AppStorage("location.preciseEnabled") private var locationPreciseEnabled: Bool = true
@AppStorage("screen.preventSleep") private var preventSleep: Bool = true
@AppStorage("bridge.preferredStableID") private var preferredBridgeStableID: String = ""
@AppStorage("bridge.lastDiscoveredStableID") private var lastDiscoveredBridgeStableID: String = ""
@@ -32,6 +36,7 @@ struct SettingsTab: View {
@State private var connectStatus = ConnectStatusStore()
@State private var connectingBridgeID: String?
@State private var localIPAddress: String?
@State private var lastLocationModeRaw: String = ClawdisLocationMode.off.rawValue
var body: some View {
NavigationStack {
@@ -51,6 +56,9 @@ struct SettingsTab: View {
}
}
}
LabeledContent("Platform", value: self.platformString())
LabeledContent("Version", value: self.appVersion())
LabeledContent("Model", value: self.modelIdentifier())
}
Section("Bridge") {
@@ -153,6 +161,12 @@ struct SettingsTab: View {
.onChange(of: self.voiceWakeEnabled) { _, newValue in
self.appModel.setVoiceWakeEnabled(newValue)
}
Toggle("Talk Mode", isOn: self.$talkEnabled)
.onChange(of: self.talkEnabled) { _, newValue in
self.appModel.setTalkEnabled(newValue)
}
// Keep this separate so users can hide the side bubble without disabling Talk Mode.
Toggle("Show Talk Button", isOn: self.$talkButtonEnabled)
NavigationLink {
VoiceWakeWordsSettingsView()
@@ -170,6 +184,22 @@ struct SettingsTab: View {
.foregroundStyle(.secondary)
}
Section("Location") {
Picker("Location Access", selection: self.$locationEnabledModeRaw) {
Text("Off").tag(ClawdisLocationMode.off.rawValue)
Text("While Using").tag(ClawdisLocationMode.whileUsing.rawValue)
Text("Always").tag(ClawdisLocationMode.always.rawValue)
}
.pickerStyle(.segmented)
Toggle("Precise Location", isOn: self.$locationPreciseEnabled)
.disabled(self.locationMode == .off)
Text("Always requires system permission and may prompt to open Settings.")
.font(.footnote)
.foregroundStyle(.secondary)
}
Section("Screen") {
Toggle("Prevent Sleep", isOn: self.$preventSleep)
Text("Keeps the screen awake while Clawdis is open.")
@@ -190,6 +220,7 @@ struct SettingsTab: View {
}
.onAppear {
self.localIPAddress = Self.primaryIPv4Address()
self.lastLocationModeRaw = self.locationEnabledModeRaw
}
.onChange(of: self.preferredBridgeStableID) { _, newValue in
let trimmed = newValue.trimmingCharacters(in: .whitespacesAndNewlines)
@@ -199,6 +230,20 @@ struct SettingsTab: View {
.onChange(of: self.appModel.bridgeServerName) { _, _ in
self.connectStatus.text = nil
}
.onChange(of: self.locationEnabledModeRaw) { _, newValue in
let previous = self.lastLocationModeRaw
self.lastLocationModeRaw = newValue
guard let mode = ClawdisLocationMode(rawValue: newValue) else { return }
Task {
let granted = await self.appModel.requestLocationPermissions(mode: mode)
if !granted {
await MainActor.run {
self.locationEnabledModeRaw = previous
self.lastLocationModeRaw = previous
}
}
}
}
}
}
@@ -227,6 +272,12 @@ struct SettingsTab: View {
HStack {
VStack(alignment: .leading, spacing: 2) {
Text(bridge.name)
let detailLines = self.bridgeDetailLines(bridge)
ForEach(detailLines, id: \.self) { line in
Text(line)
.font(.footnote)
.foregroundStyle(.secondary)
}
}
Spacer()
@@ -261,6 +312,10 @@ struct SettingsTab: View {
return "iOS \(v.majorVersion).\(v.minorVersion).\(v.patchVersion)"
}
private var locationMode: ClawdisLocationMode {
ClawdisLocationMode(rawValue: self.locationEnabledModeRaw) ?? .off
}
private func appVersion() -> String {
Bundle.main.infoDictionary?["CFBundleShortVersionString"] as? String ?? "dev"
}
@@ -316,6 +371,7 @@ struct SettingsTab: View {
let caps = Set(self.currentCaps())
if caps.contains(ClawdisCapability.camera.rawValue) {
commands.append(ClawdisCameraCommand.list.rawValue)
commands.append(ClawdisCameraCommand.snap.rawValue)
commands.append(ClawdisCameraCommand.clip.rawValue)
}
@@ -335,7 +391,7 @@ struct SettingsTab: View {
do {
let statusStore = self.connectStatus
let existing = KeychainStore.loadString(
service: "com.steipete.clawdis.bridge",
service: "com.clawdis.bridge",
account: self.keychainAccount())
let existingToken = (existing?.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty == false) ?
existing :
@@ -363,7 +419,7 @@ struct SettingsTab: View {
if !token.isEmpty, token != existingToken {
_ = KeychainStore.saveString(
token,
service: "com.steipete.clawdis.bridge",
service: "com.clawdis.bridge",
account: self.keychainAccount())
}
@@ -409,7 +465,7 @@ struct SettingsTab: View {
do {
let statusStore = self.connectStatus
let existing = KeychainStore.loadString(
service: "com.steipete.clawdis.bridge",
service: "com.clawdis.bridge",
account: self.keychainAccount())
let existingToken = (existing?.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty == false) ?
existing :
@@ -437,7 +493,7 @@ struct SettingsTab: View {
if !token.isEmpty, token != existingToken {
_ = KeychainStore.saveString(
token,
service: "com.steipete.clawdis.bridge",
service: "com.clawdis.bridge",
account: self.keychainAccount())
}
@@ -504,4 +560,26 @@ struct SettingsTab: View {
private static func httpURLString(host: String?, port: Int?, fallback: String) -> String {
SettingsNetworkingHelpers.httpURLString(host: host, port: port, fallback: fallback)
}
private func bridgeDetailLines(_ bridge: BridgeDiscoveryModel.DiscoveredBridge) -> [String] {
var lines: [String] = []
if let lanHost = bridge.lanHost { lines.append("LAN: \(lanHost)") }
if let tailnet = bridge.tailnetDns { lines.append("Tailnet: \(tailnet)") }
let gatewayPort = bridge.gatewayPort
let bridgePort = bridge.bridgePort
let canvasPort = bridge.canvasPort
if gatewayPort != nil || bridgePort != nil || canvasPort != nil {
let gw = gatewayPort.map(String.init) ?? ""
let br = bridgePort.map(String.init) ?? ""
let canvas = canvasPort.map(String.init) ?? ""
lines.append("Ports: gw \(gw) · bridge \(br) · canvas \(canvas)")
}
if lines.isEmpty {
lines.append(bridge.debugID)
}
return lines
}
}

View File

@@ -1,6 +1,8 @@
import SwiftUI
struct StatusPill: View {
@Environment(\.scenePhase) private var scenePhase
enum BridgeState: Equatable {
case connected
case connecting
@@ -26,8 +28,15 @@ struct StatusPill: View {
}
}
struct Activity: Equatable {
var title: String
var systemImage: String
var tint: Color?
}
var bridge: BridgeState
var voiceWakeEnabled: Bool
var activity: Activity?
var brighten: Bool = false
var onTap: () -> Void
@@ -52,10 +61,24 @@ struct StatusPill: View {
.frame(height: 14)
.opacity(0.35)
Image(systemName: self.voiceWakeEnabled ? "mic.fill" : "mic.slash")
.font(.system(size: 13, weight: .semibold))
.foregroundStyle(self.voiceWakeEnabled ? .primary : .secondary)
.accessibilityLabel(self.voiceWakeEnabled ? "Voice Wake enabled" : "Voice Wake disabled")
if let activity {
HStack(spacing: 6) {
Image(systemName: activity.systemImage)
.font(.system(size: 13, weight: .semibold))
.foregroundStyle(activity.tint ?? .primary)
Text(activity.title)
.font(.system(size: 13, weight: .semibold))
.foregroundStyle(.primary)
.lineLimit(1)
}
.transition(.opacity.combined(with: .move(edge: .top)))
} else {
Image(systemName: self.voiceWakeEnabled ? "mic.fill" : "mic.slash")
.font(.system(size: 13, weight: .semibold))
.foregroundStyle(self.voiceWakeEnabled ? .primary : .secondary)
.accessibilityLabel(self.voiceWakeEnabled ? "Voice Wake enabled" : "Voice Wake disabled")
.transition(.opacity.combined(with: .move(edge: .top)))
}
}
.padding(.vertical, 8)
.padding(.horizontal, 12)
@@ -71,15 +94,27 @@ struct StatusPill: View {
}
.buttonStyle(.plain)
.accessibilityLabel("Status")
.accessibilityValue("\(self.bridge.title), Voice Wake \(self.voiceWakeEnabled ? "enabled" : "disabled")")
.onAppear { self.updatePulse(for: self.bridge) }
.accessibilityValue(self.accessibilityValue)
.onAppear { self.updatePulse(for: self.bridge, scenePhase: self.scenePhase) }
.onDisappear { self.pulse = false }
.onChange(of: self.bridge) { _, newValue in
self.updatePulse(for: newValue)
self.updatePulse(for: newValue, scenePhase: self.scenePhase)
}
.onChange(of: self.scenePhase) { _, newValue in
self.updatePulse(for: self.bridge, scenePhase: newValue)
}
.animation(.easeInOut(duration: 0.18), value: self.activity?.title)
}
private func updatePulse(for bridge: BridgeState) {
guard bridge == .connecting else {
private var accessibilityValue: String {
if let activity {
return "\(self.bridge.title), \(activity.title)"
}
return "\(self.bridge.title), Voice Wake \(self.voiceWakeEnabled ? "enabled" : "disabled")"
}
private func updatePulse(for bridge: BridgeState, scenePhase: ScenePhase) {
guard bridge == .connecting, scenePhase == .active else {
withAnimation(.easeOut(duration: 0.2)) { self.pulse = false }
return
}

View File

@@ -0,0 +1,713 @@
import AVFAudio
import ClawdisKit
import Foundation
import Observation
import OSLog
import Speech
@MainActor
@Observable
final class TalkModeManager: NSObject {
private typealias SpeechRequest = SFSpeechAudioBufferRecognitionRequest
private static let defaultModelIdFallback = "eleven_v3"
var isEnabled: Bool = false
var isListening: Bool = false
var isSpeaking: Bool = false
var statusText: String = "Off"
private let audioEngine = AVAudioEngine()
private var speechRecognizer: SFSpeechRecognizer?
private var recognitionRequest: SFSpeechAudioBufferRecognitionRequest?
private var recognitionTask: SFSpeechRecognitionTask?
private var silenceTask: Task<Void, Never>?
private var lastHeard: Date?
private var lastTranscript: String = ""
private var lastSpokenText: String?
private var lastInterruptedAtSeconds: Double?
private var defaultVoiceId: String?
private var currentVoiceId: String?
private var defaultModelId: String?
private var currentModelId: String?
private var voiceOverrideActive = false
private var modelOverrideActive = false
private var defaultOutputFormat: String?
private var apiKey: String?
private var voiceAliases: [String: String] = [:]
private var interruptOnSpeech: Bool = true
private var mainSessionKey: String = "main"
private var fallbackVoiceId: String?
private var lastPlaybackWasPCM: Bool = false
var pcmPlayer: PCMStreamingAudioPlaying = PCMStreamingAudioPlayer.shared
var mp3Player: StreamingAudioPlaying = StreamingAudioPlayer.shared
private var bridge: BridgeSession?
private let silenceWindow: TimeInterval = 0.7
private var chatSubscribedSessionKeys = Set<String>()
private let logger = Logger(subsystem: "com.clawdis", category: "TalkMode")
func attachBridge(_ bridge: BridgeSession) {
self.bridge = bridge
}
func setEnabled(_ enabled: Bool) {
self.isEnabled = enabled
if enabled {
self.logger.info("enabled")
Task { await self.start() }
} else {
self.logger.info("disabled")
self.stop()
}
}
func start() async {
guard self.isEnabled else { return }
if self.isListening { return }
self.logger.info("start")
self.statusText = "Requesting permissions…"
let micOk = await Self.requestMicrophonePermission()
guard micOk else {
self.logger.warning("start blocked: microphone permission denied")
self.statusText = "Microphone permission denied"
return
}
let speechOk = await Self.requestSpeechPermission()
guard speechOk else {
self.logger.warning("start blocked: speech permission denied")
self.statusText = "Speech recognition permission denied"
return
}
await self.reloadConfig()
do {
try Self.configureAudioSession()
try self.startRecognition()
self.isListening = true
self.statusText = "Listening"
self.startSilenceMonitor()
await self.subscribeChatIfNeeded(sessionKey: self.mainSessionKey)
self.logger.info("listening")
} catch {
self.isListening = false
self.statusText = "Start failed: \(error.localizedDescription)"
self.logger.error("start failed: \(error.localizedDescription, privacy: .public)")
}
}
func stop() {
self.isEnabled = false
self.isListening = false
self.statusText = "Off"
self.lastTranscript = ""
self.lastHeard = nil
self.silenceTask?.cancel()
self.silenceTask = nil
self.stopRecognition()
self.stopSpeaking()
self.lastInterruptedAtSeconds = nil
TalkSystemSpeechSynthesizer.shared.stop()
do {
try AVAudioSession.sharedInstance().setActive(false, options: [.notifyOthersOnDeactivation])
} catch {
self.logger.warning("audio session deactivate failed: \(error.localizedDescription, privacy: .public)")
}
Task { await self.unsubscribeAllChats() }
}
func userTappedOrb() {
self.stopSpeaking()
}
private func startRecognition() throws {
self.stopRecognition()
self.speechRecognizer = SFSpeechRecognizer()
guard let recognizer = self.speechRecognizer else {
throw NSError(domain: "TalkMode", code: 1, userInfo: [
NSLocalizedDescriptionKey: "Speech recognizer unavailable",
])
}
self.recognitionRequest = SFSpeechAudioBufferRecognitionRequest()
self.recognitionRequest?.shouldReportPartialResults = true
guard let request = self.recognitionRequest else { return }
let input = self.audioEngine.inputNode
let format = input.outputFormat(forBus: 0)
input.removeTap(onBus: 0)
let tapBlock = Self.makeAudioTapAppendCallback(request: request)
input.installTap(onBus: 0, bufferSize: 2048, format: format, block: tapBlock)
self.audioEngine.prepare()
try self.audioEngine.start()
self.recognitionTask = recognizer.recognitionTask(with: request) { [weak self] result, error in
guard let self else { return }
if let error {
if !self.isSpeaking {
self.statusText = "Speech error: \(error.localizedDescription)"
}
self.logger.debug("speech recognition error: \(error.localizedDescription, privacy: .public)")
}
guard let result else { return }
let transcript = result.bestTranscription.formattedString
Task { @MainActor in
await self.handleTranscript(transcript: transcript, isFinal: result.isFinal)
}
}
}
private func stopRecognition() {
self.recognitionTask?.cancel()
self.recognitionTask = nil
self.recognitionRequest?.endAudio()
self.recognitionRequest = nil
self.audioEngine.inputNode.removeTap(onBus: 0)
self.audioEngine.stop()
self.speechRecognizer = nil
}
private nonisolated static func makeAudioTapAppendCallback(request: SpeechRequest) -> AVAudioNodeTapBlock {
{ buffer, _ in
request.append(buffer)
}
}
private func handleTranscript(transcript: String, isFinal: Bool) async {
let trimmed = transcript.trimmingCharacters(in: .whitespacesAndNewlines)
if self.isSpeaking, self.interruptOnSpeech {
if self.shouldInterrupt(with: trimmed) {
self.stopSpeaking()
}
return
}
guard self.isListening else { return }
if !trimmed.isEmpty {
self.lastTranscript = trimmed
self.lastHeard = Date()
}
if isFinal {
self.lastTranscript = trimmed
}
}
private func startSilenceMonitor() {
self.silenceTask?.cancel()
self.silenceTask = Task { [weak self] in
guard let self else { return }
while self.isEnabled {
try? await Task.sleep(nanoseconds: 200_000_000)
await self.checkSilence()
}
}
}
private func checkSilence() async {
guard self.isListening, !self.isSpeaking else { return }
let transcript = self.lastTranscript.trimmingCharacters(in: .whitespacesAndNewlines)
guard !transcript.isEmpty else { return }
guard let lastHeard else { return }
if Date().timeIntervalSince(lastHeard) < self.silenceWindow { return }
await self.finalizeTranscript(transcript)
}
private func finalizeTranscript(_ transcript: String) async {
self.isListening = false
self.statusText = "Thinking…"
self.lastTranscript = ""
self.lastHeard = nil
self.stopRecognition()
await self.reloadConfig()
let prompt = self.buildPrompt(transcript: transcript)
guard let bridge else {
self.statusText = "Bridge not connected"
self.logger.warning("finalize: bridge not connected")
await self.start()
return
}
do {
let startedAt = Date().timeIntervalSince1970
let sessionKey = self.mainSessionKey
await self.subscribeChatIfNeeded(sessionKey: sessionKey)
self.logger.info(
"chat.send start sessionKey=\(sessionKey, privacy: .public) chars=\(prompt.count, privacy: .public)")
let runId = try await self.sendChat(prompt, bridge: bridge)
self.logger.info("chat.send ok runId=\(runId, privacy: .public)")
let completion = await self.waitForChatCompletion(runId: runId, bridge: bridge, timeoutSeconds: 120)
if completion == .timeout {
self.logger.warning(
"chat completion timeout runId=\(runId, privacy: .public); attempting history fallback")
} else if completion == .aborted {
self.statusText = "Aborted"
self.logger.warning("chat completion aborted runId=\(runId, privacy: .public)")
await self.start()
return
} else if completion == .error {
self.statusText = "Chat error"
self.logger.warning("chat completion error runId=\(runId, privacy: .public)")
await self.start()
return
}
guard let assistantText = try await self.waitForAssistantText(
bridge: bridge,
since: startedAt,
timeoutSeconds: completion == .final ? 12 : 25)
else {
self.statusText = "No reply"
self.logger.warning("assistant text timeout runId=\(runId, privacy: .public)")
await self.start()
return
}
self.logger.info("assistant text ok chars=\(assistantText.count, privacy: .public)")
await self.playAssistant(text: assistantText)
} catch {
self.statusText = "Talk failed: \(error.localizedDescription)"
self.logger.error("finalize failed: \(error.localizedDescription, privacy: .public)")
}
await self.start()
}
private func subscribeChatIfNeeded(sessionKey: String) async {
let key = sessionKey.trimmingCharacters(in: .whitespacesAndNewlines)
guard !key.isEmpty else { return }
guard let bridge else { return }
guard !self.chatSubscribedSessionKeys.contains(key) else { return }
do {
let payload = "{\"sessionKey\":\"\(key)\"}"
try await bridge.sendEvent(event: "chat.subscribe", payloadJSON: payload)
self.chatSubscribedSessionKeys.insert(key)
self.logger.info("chat.subscribe ok sessionKey=\(key, privacy: .public)")
} catch {
self.logger
.warning(
"chat.subscribe failed sessionKey=\(key, privacy: .public) err=\(error.localizedDescription, privacy: .public)")
}
}
private func unsubscribeAllChats() async {
guard let bridge else { return }
let keys = self.chatSubscribedSessionKeys
self.chatSubscribedSessionKeys.removeAll()
for key in keys {
do {
let payload = "{\"sessionKey\":\"\(key)\"}"
try await bridge.sendEvent(event: "chat.unsubscribe", payloadJSON: payload)
} catch {
// ignore
}
}
}
private func buildPrompt(transcript: String) -> String {
let interrupted = self.lastInterruptedAtSeconds
self.lastInterruptedAtSeconds = nil
return TalkPromptBuilder.build(transcript: transcript, interruptedAtSeconds: interrupted)
}
private enum ChatCompletionState: CustomStringConvertible {
case final
case aborted
case error
case timeout
var description: String {
switch self {
case .final: "final"
case .aborted: "aborted"
case .error: "error"
case .timeout: "timeout"
}
}
}
private func sendChat(_ message: String, bridge: BridgeSession) async throws -> String {
struct SendResponse: Decodable { let runId: String }
let payload: [String: Any] = [
"sessionKey": self.mainSessionKey,
"message": message,
"thinking": "low",
"timeoutMs": 30000,
"idempotencyKey": UUID().uuidString,
]
let data = try JSONSerialization.data(withJSONObject: payload)
let json = String(decoding: data, as: UTF8.self)
let res = try await bridge.request(method: "chat.send", paramsJSON: json, timeoutSeconds: 30)
let decoded = try JSONDecoder().decode(SendResponse.self, from: res)
return decoded.runId
}
private func waitForChatCompletion(
runId: String,
bridge: BridgeSession,
timeoutSeconds: Int = 120) async -> ChatCompletionState
{
let stream = await bridge.subscribeServerEvents(bufferingNewest: 200)
return await withTaskGroup(of: ChatCompletionState.self) { group in
group.addTask { [runId] in
for await evt in stream {
if Task.isCancelled { return .timeout }
guard evt.event == "chat", let payload = evt.payloadJSON else { continue }
guard let data = payload.data(using: .utf8) else { continue }
guard let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else { continue }
if (json["runId"] as? String) != runId { continue }
if let state = json["state"] as? String {
switch state {
case "final": return .final
case "aborted": return .aborted
case "error": return .error
default: break
}
}
}
return .timeout
}
group.addTask {
try? await Task.sleep(nanoseconds: UInt64(timeoutSeconds) * 1_000_000_000)
return .timeout
}
let result = await group.next() ?? .timeout
group.cancelAll()
return result
}
}
private func waitForAssistantText(
bridge: BridgeSession,
since: Double,
timeoutSeconds: Int) async throws -> String?
{
let deadline = Date().addingTimeInterval(TimeInterval(timeoutSeconds))
while Date() < deadline {
if let text = try await self.fetchLatestAssistantText(bridge: bridge, since: since) {
return text
}
try? await Task.sleep(nanoseconds: 300_000_000)
}
return nil
}
private func fetchLatestAssistantText(bridge: BridgeSession, since: Double? = nil) async throws -> String? {
let res = try await bridge.request(
method: "chat.history",
paramsJSON: "{\"sessionKey\":\"\(self.mainSessionKey)\"}",
timeoutSeconds: 15)
guard let json = try JSONSerialization.jsonObject(with: res) as? [String: Any] else { return nil }
guard let messages = json["messages"] as? [[String: Any]] else { return nil }
for msg in messages.reversed() {
guard (msg["role"] as? String) == "assistant" else { continue }
if let since, let timestamp = msg["timestamp"] as? Double,
TalkHistoryTimestamp.isAfter(timestamp, sinceSeconds: since) == false
{
continue
}
guard let content = msg["content"] as? [[String: Any]] else { continue }
let text = content.compactMap { $0["text"] as? String }.joined(separator: "\n")
let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines)
if !trimmed.isEmpty { return trimmed }
}
return nil
}
private func playAssistant(text: String) async {
let parsed = TalkDirectiveParser.parse(text)
let directive = parsed.directive
let cleaned = parsed.stripped.trimmingCharacters(in: .whitespacesAndNewlines)
guard !cleaned.isEmpty else { return }
let requestedVoice = directive?.voiceId?.trimmingCharacters(in: .whitespacesAndNewlines)
let resolvedVoice = self.resolveVoiceAlias(requestedVoice)
if requestedVoice?.isEmpty == false, resolvedVoice == nil {
self.logger.warning("unknown voice alias \(requestedVoice ?? "?", privacy: .public)")
}
if let voice = resolvedVoice {
if directive?.once != true {
self.currentVoiceId = voice
self.voiceOverrideActive = true
}
}
if let model = directive?.modelId {
if directive?.once != true {
self.currentModelId = model
self.modelOverrideActive = true
}
}
self.statusText = "Generating voice…"
self.isSpeaking = true
self.lastSpokenText = cleaned
do {
let started = Date()
let language = ElevenLabsTTSClient.validatedLanguage(directive?.language)
let resolvedKey =
(self.apiKey?.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty == false ? self.apiKey : nil) ??
ProcessInfo.processInfo.environment["ELEVENLABS_API_KEY"]
let apiKey = resolvedKey?.trimmingCharacters(in: .whitespacesAndNewlines)
let preferredVoice = resolvedVoice ?? self.currentVoiceId ?? self.defaultVoiceId
let voiceId: String? = if let apiKey, !apiKey.isEmpty {
await self.resolveVoiceId(preferred: preferredVoice, apiKey: apiKey)
} else {
nil
}
let canUseElevenLabs = (voiceId?.isEmpty == false) && (apiKey?.isEmpty == false)
if canUseElevenLabs, let voiceId, let apiKey {
let desiredOutputFormat = (directive?.outputFormat ?? self.defaultOutputFormat)?
.trimmingCharacters(in: .whitespacesAndNewlines)
let requestedOutputFormat = (desiredOutputFormat?.isEmpty == false) ? desiredOutputFormat : nil
let outputFormat = ElevenLabsTTSClient.validatedOutputFormat(requestedOutputFormat ?? "pcm_44100")
if outputFormat == nil, let requestedOutputFormat {
self.logger.warning(
"talk output_format unsupported for local playback: \(requestedOutputFormat, privacy: .public)")
}
let modelId = directive?.modelId ?? self.currentModelId ?? self.defaultModelId
func makeRequest(outputFormat: String?) -> ElevenLabsTTSRequest {
ElevenLabsTTSRequest(
text: cleaned,
modelId: modelId,
outputFormat: outputFormat,
speed: TalkTTSValidation.resolveSpeed(speed: directive?.speed, rateWPM: directive?.rateWPM),
stability: TalkTTSValidation.validatedStability(directive?.stability, modelId: modelId),
similarity: TalkTTSValidation.validatedUnit(directive?.similarity),
style: TalkTTSValidation.validatedUnit(directive?.style),
speakerBoost: directive?.speakerBoost,
seed: TalkTTSValidation.validatedSeed(directive?.seed),
normalize: ElevenLabsTTSClient.validatedNormalize(directive?.normalize),
language: language,
latencyTier: TalkTTSValidation.validatedLatencyTier(directive?.latencyTier))
}
let request = makeRequest(outputFormat: outputFormat)
let client = ElevenLabsTTSClient(apiKey: apiKey)
let stream = client.streamSynthesize(voiceId: voiceId, request: request)
if self.interruptOnSpeech {
do {
try self.startRecognition()
} catch {
self.logger.warning(
"startRecognition during speak failed: \(error.localizedDescription, privacy: .public)")
}
}
self.statusText = "Speaking…"
let sampleRate = TalkTTSValidation.pcmSampleRate(from: outputFormat)
let result: StreamingPlaybackResult
if let sampleRate {
self.lastPlaybackWasPCM = true
var playback = await self.pcmPlayer.play(stream: stream, sampleRate: sampleRate)
if !playback.finished, playback.interruptedAt == nil {
let mp3Format = ElevenLabsTTSClient.validatedOutputFormat("mp3_44100")
self.logger.warning("pcm playback failed; retrying mp3")
self.lastPlaybackWasPCM = false
let mp3Stream = client.streamSynthesize(
voiceId: voiceId,
request: makeRequest(outputFormat: mp3Format))
playback = await self.mp3Player.play(stream: mp3Stream)
}
result = playback
} else {
self.lastPlaybackWasPCM = false
result = await self.mp3Player.play(stream: stream)
}
self.logger
.info(
"elevenlabs stream finished=\(result.finished, privacy: .public) dur=\(Date().timeIntervalSince(started), privacy: .public)s")
if !result.finished, let interruptedAt = result.interruptedAt {
self.lastInterruptedAtSeconds = interruptedAt
}
} else {
self.logger.warning("tts unavailable; falling back to system voice (missing key or voiceId)")
if self.interruptOnSpeech {
do {
try self.startRecognition()
} catch {
self.logger.warning(
"startRecognition during speak failed: \(error.localizedDescription, privacy: .public)")
}
}
self.statusText = "Speaking (System)…"
try await TalkSystemSpeechSynthesizer.shared.speak(text: cleaned, language: language)
}
} catch {
self.logger.error(
"tts failed: \(error.localizedDescription, privacy: .public); falling back to system voice")
do {
if self.interruptOnSpeech {
do {
try self.startRecognition()
} catch {
self.logger.warning(
"startRecognition during speak failed: \(error.localizedDescription, privacy: .public)")
}
}
self.statusText = "Speaking (System)…"
let language = ElevenLabsTTSClient.validatedLanguage(directive?.language)
try await TalkSystemSpeechSynthesizer.shared.speak(text: cleaned, language: language)
} catch {
self.statusText = "Speak failed: \(error.localizedDescription)"
self.logger.error("system voice failed: \(error.localizedDescription, privacy: .public)")
}
}
self.stopRecognition()
self.isSpeaking = false
}
private func stopSpeaking(storeInterruption: Bool = true) {
guard self.isSpeaking else { return }
let interruptedAt = self.lastPlaybackWasPCM
? self.pcmPlayer.stop()
: self.mp3Player.stop()
if storeInterruption {
self.lastInterruptedAtSeconds = interruptedAt
}
_ = self.lastPlaybackWasPCM
? self.mp3Player.stop()
: self.pcmPlayer.stop()
TalkSystemSpeechSynthesizer.shared.stop()
self.isSpeaking = false
}
private func shouldInterrupt(with transcript: String) -> Bool {
let trimmed = transcript.trimmingCharacters(in: .whitespacesAndNewlines)
guard trimmed.count >= 3 else { return false }
if let spoken = self.lastSpokenText?.lowercased(), spoken.contains(trimmed.lowercased()) {
return false
}
return true
}
private func resolveVoiceAlias(_ value: String?) -> String? {
let trimmed = (value ?? "").trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty else { return nil }
let normalized = trimmed.lowercased()
if let mapped = self.voiceAliases[normalized] { return mapped }
if self.voiceAliases.values.contains(where: { $0.caseInsensitiveCompare(trimmed) == .orderedSame }) {
return trimmed
}
return Self.isLikelyVoiceId(trimmed) ? trimmed : nil
}
private func resolveVoiceId(preferred: String?, apiKey: String) async -> String? {
let trimmed = preferred?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
if !trimmed.isEmpty {
if let resolved = self.resolveVoiceAlias(trimmed) { return resolved }
self.logger.warning("unknown voice alias \(trimmed, privacy: .public)")
}
if let fallbackVoiceId { return fallbackVoiceId }
do {
let voices = try await ElevenLabsTTSClient(apiKey: apiKey).listVoices()
guard let first = voices.first else {
self.logger.warning("elevenlabs voices list empty")
return nil
}
self.fallbackVoiceId = first.voiceId
if self.defaultVoiceId == nil {
self.defaultVoiceId = first.voiceId
}
if !self.voiceOverrideActive {
self.currentVoiceId = first.voiceId
}
let name = first.name ?? "unknown"
self.logger
.info("default voice selected \(name, privacy: .public) (\(first.voiceId, privacy: .public))")
return first.voiceId
} catch {
self.logger.error("elevenlabs list voices failed: \(error.localizedDescription, privacy: .public)")
return nil
}
}
private static func isLikelyVoiceId(_ value: String) -> Bool {
guard value.count >= 10 else { return false }
return value.allSatisfy { $0.isLetter || $0.isNumber || $0 == "-" || $0 == "_" }
}
private func reloadConfig() async {
guard let bridge else { return }
do {
let res = try await bridge.request(method: "config.get", paramsJSON: "{}", timeoutSeconds: 8)
guard let json = try JSONSerialization.jsonObject(with: res) as? [String: Any] else { return }
guard let config = json["config"] as? [String: Any] else { return }
let talk = config["talk"] as? [String: Any]
let session = config["session"] as? [String: Any]
let rawMainKey = (session?["mainKey"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
self.mainSessionKey = rawMainKey.isEmpty ? "main" : rawMainKey
self.defaultVoiceId = (talk?["voiceId"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines)
if let aliases = talk?["voiceAliases"] as? [String: Any] {
var resolved: [String: String] = [:]
for (key, value) in aliases {
guard let id = value as? String else { continue }
let normalizedKey = key.trimmingCharacters(in: .whitespacesAndNewlines).lowercased()
let trimmedId = id.trimmingCharacters(in: .whitespacesAndNewlines)
guard !normalizedKey.isEmpty, !trimmedId.isEmpty else { continue }
resolved[normalizedKey] = trimmedId
}
self.voiceAliases = resolved
} else {
self.voiceAliases = [:]
}
if !self.voiceOverrideActive {
self.currentVoiceId = self.defaultVoiceId
}
let model = (talk?["modelId"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines)
self.defaultModelId = (model?.isEmpty == false) ? model : Self.defaultModelIdFallback
if !self.modelOverrideActive {
self.currentModelId = self.defaultModelId
}
self.defaultOutputFormat = (talk?["outputFormat"] as? String)?
.trimmingCharacters(in: .whitespacesAndNewlines)
self.apiKey = (talk?["apiKey"] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines)
if let interrupt = talk?["interruptOnSpeech"] as? Bool {
self.interruptOnSpeech = interrupt
}
} catch {
self.defaultModelId = Self.defaultModelIdFallback
if !self.modelOverrideActive {
self.currentModelId = self.defaultModelId
}
}
}
private static func configureAudioSession() throws {
let session = AVAudioSession.sharedInstance()
try session.setCategory(.playAndRecord, mode: .voiceChat, options: [
.duckOthers,
.mixWithOthers,
.allowBluetoothHFP,
.defaultToSpeaker,
])
try session.setActive(true, options: [])
}
private nonisolated static func requestMicrophonePermission() async -> Bool {
await withCheckedContinuation(isolation: nil) { cont in
AVAudioApplication.requestRecordPermission { ok in
cont.resume(returning: ok)
}
}
}
private nonisolated static func requestSpeechPermission() async -> Bool {
await withCheckedContinuation(isolation: nil) { cont in
SFSpeechRecognizer.requestAuthorization { status in
cont.resume(returning: status == .authorized)
}
}
}
}

View File

@@ -0,0 +1,70 @@
import SwiftUI
struct TalkOrbOverlay: View {
@Environment(NodeAppModel.self) private var appModel
@State private var pulse: Bool = false
var body: some View {
let seam = self.appModel.seamColor
let status = self.appModel.talkMode.statusText.trimmingCharacters(in: .whitespacesAndNewlines)
VStack(spacing: 14) {
ZStack {
Circle()
.stroke(seam.opacity(0.26), lineWidth: 2)
.frame(width: 320, height: 320)
.scaleEffect(self.pulse ? 1.15 : 0.96)
.opacity(self.pulse ? 0.0 : 1.0)
.animation(.easeOut(duration: 1.3).repeatForever(autoreverses: false), value: self.pulse)
Circle()
.stroke(seam.opacity(0.18), lineWidth: 2)
.frame(width: 320, height: 320)
.scaleEffect(self.pulse ? 1.45 : 1.02)
.opacity(self.pulse ? 0.0 : 0.9)
.animation(.easeOut(duration: 1.9).repeatForever(autoreverses: false).delay(0.2), value: self.pulse)
Circle()
.fill(
RadialGradient(
colors: [
seam.opacity(0.95),
seam.opacity(0.40),
Color.black.opacity(0.55),
],
center: .center,
startRadius: 1,
endRadius: 112))
.frame(width: 190, height: 190)
.overlay(
Circle()
.stroke(seam.opacity(0.35), lineWidth: 1))
.shadow(color: seam.opacity(0.32), radius: 26, x: 0, y: 0)
.shadow(color: Color.black.opacity(0.50), radius: 22, x: 0, y: 10)
}
.contentShape(Circle())
.onTapGesture {
self.appModel.talkMode.userTappedOrb()
}
if !status.isEmpty, status != "Off" {
Text(status)
.font(.system(.footnote, design: .rounded).weight(.semibold))
.foregroundStyle(Color.white.opacity(0.92))
.padding(.horizontal, 12)
.padding(.vertical, 8)
.background(
Capsule()
.fill(Color.black.opacity(0.40))
.overlay(
Capsule().stroke(seam.opacity(0.22), lineWidth: 1)))
}
}
.padding(28)
.onAppear {
self.pulse = true
}
.accessibilityElement(children: .combine)
.accessibilityLabel("Talk Mode \(status)")
}
}

Some files were not shown because too many files have changed in this diff Show More