shmup/colabbd
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add a new plan

Browse source
This commit is contained in:
Jared Miller 2026-01-27 20:33:35 -05:00
parent 853fcba641
commit cd4b1def5b
Signed by: shmup
GPG key ID: 22B5C6D66A38B06C
1 changed files with 982 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

982
docs/plans/2026-01-27-cursor-persistence-adapters.md Normal file
View file

@ -0,0 +1,982 @@
# Cursor Sync, Persistence & Adapter Guide
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** Add cursor/selection awareness, persistent rooms via bun:sqlite, and document how to write editor adapters.
**Architecture:** Yjs awareness protocol for cursors (ephemeral state, separate from document CRDT). SQLite stores yjs update log per room - replay on daemon restart. Adapter guide documents the vim pattern for other editors.
**Tech Stack:** yjs awareness, bun:sqlite, vim9script
---
## Task 1: Cursor Sync - Protocol & Session
**Files:**
- Modify: `src/protocol.ts:7,12` (awareness types exist, need payload shape)
- Modify: `src/session.ts` (add awareness state tracking)
- Create: `src/session.test.ts` (add awareness tests)
**Step 1: Define awareness payload type in protocol**
In `src/protocol.ts`, the awareness message exists but payload is just `number[]`. Add a typed structure:
```typescript
// After line 14, add:
export type AwarenessState = {
clientId: number;
cursor?: { line: number; col: number };
selection?: { startLine: number; startCol: number; endLine: number; endCol: number };
name?: string;
};
```
**Step 2: Write failing test for awareness broadcast**
Add to `src/session.test.ts`:
```typescript
import { describe, test, expect, mock } from "bun:test";
describe("awareness", () => {
test("broadcasts awareness to other clients", () => {
const session = new Session();
const sent1: unknown[] = [];
const sent2: unknown[] = [];
const client1 = { ws: { send: (m: string) => sent1.push(JSON.parse(m)) } } as Client;
const client2 = { ws: { send: (m: string) => sent2.push(JSON.parse(m)) } } as Client;
session.join(client1);
session.join(client2);
const awareness = { clientId: 1, cursor: { line: 5, col: 10 } };
session.broadcastAwareness(client1, awareness);
// client1 should NOT receive their own awareness
expect(sent1.filter(m => m.type === "awareness")).toHaveLength(0);
// client2 should receive it
expect(sent2.filter(m => m.type === "awareness")).toHaveLength(1);
expect(sent2.find(m => m.type === "awareness")?.data).toEqual(awareness);
});
});
```
**Step 3: Run test to verify it fails**
```bash
bun test src/session.test.ts
```
Expected: FAIL - `broadcastAwareness` is not a function
**Step 4: Implement broadcastAwareness in session.ts**
Add after `broadcastPeerCount()`:
```typescript
broadcastAwareness(sender: Client, state: AwarenessState): void {
const message = encode({ type: "awareness", data: state });
for (const client of this.clients) {
if (client !== sender) {
try {
client.ws.send(message);
} catch {
// client disconnected, ignore
}
}
}
}
```
Update imports at top:
```typescript
import { encode, decode, type AwarenessState } from "./protocol";
```
**Step 5: Run test to verify it passes**
```bash
bun test src/session.test.ts
```
Expected: PASS
**Step 6: Commit**
```bash
git add src/protocol.ts src/session.ts src/session.test.ts
git commit -m "Add awareness broadcast to session"
```
---
## Task 2: Cursor Sync - Daemon Routing
**Files:**
- Modify: `src/index.ts:68-73` (add awareness case)
- Create: `src/index.test.ts` (integration test for awareness routing)
**Step 1: Write failing integration test**
Create `src/awareness.test.ts`:
```typescript
import { describe, test, expect, beforeAll, afterAll } from "bun:test";
import type { Server } from "bun";
describe("awareness routing", () => {
let server: Server;
const PORT = 4042;
beforeAll(async () => {
// Import and start server on test port
process.env.PORT = String(PORT);
const mod = await import("./index");
server = mod.server;
});
afterAll(() => {
server?.stop();
});
test("awareness message routes to other peers in same room", async () => {
const ws1 = new WebSocket(`ws://localhost:${PORT}/ws`);
const ws2 = new WebSocket(`ws://localhost:${PORT}/ws`);
const received: unknown[] = [];
await Promise.all([
new Promise(r => ws1.onopen = r),
new Promise(r => ws2.onopen = r),
]);
ws2.onmessage = (e) => {
received.push(JSON.parse(e.data));
};
// Both join same room
ws1.send(JSON.stringify({ type: "join", room: "test" }));
ws2.send(JSON.stringify({ type: "join", room: "test" }));
await Bun.sleep(50);
// ws1 sends awareness
ws1.send(JSON.stringify({
type: "awareness",
data: { clientId: 1, cursor: { line: 10, col: 5 } }
}));
await Bun.sleep(50);
const awareness = received.find(m => m.type === "awareness");
expect(awareness).toBeDefined();
expect(awareness.data.cursor).toEqual({ line: 10, col: 5 });
ws1.close();
ws2.close();
});
});
```
**Step 2: Run test to verify it fails**
```bash
bun test src/awareness.test.ts
```
Expected: FAIL - awareness not received (no routing in index.ts yet)
**Step 3: Add awareness routing to index.ts**
In `src/index.ts`, the message handler switch (around line 43), add after the `update` case:
```typescript
case "awareness":
if (client.room) {
const session = getSession(client.room);
if (session && "data" in msg) {
session.broadcastAwareness(client, msg.data);
}
}
break;
```
Also update `isClientMessage` in protocol.ts to validate awareness data shape:
```typescript
// In isClientMessage(), add validation for awareness:
if (msg.type === "awareness") {
return typeof msg.data === "object" && msg.data !== null;
}
```
**Step 4: Run test to verify it passes**
```bash
bun test src/awareness.test.ts
```
Expected: PASS
**Step 5: Commit**
```bash
git add src/index.ts src/protocol.ts src/awareness.test.ts
git commit -m "Route awareness messages between peers"
```
---
## Task 3: Cursor Sync - Bridge
**Files:**
- Modify: `adapters/vim/bridge.ts` (send/receive awareness)
- Modify: `adapters/vim/bridge.test.ts` (test awareness flow)
**Step 1: Write failing test for awareness in bridge**
Add to `adapters/vim/bridge.test.ts`:
```typescript
describe("awareness", () => {
test("forwards awareness from daemon to vim", async () => {
// Start daemon
const server = Bun.serve({
port: 4043,
fetch(req, server) {
if (new URL(req.url).pathname === "/ws") {
server.upgrade(req);
return;
}
return new Response("not found", { status: 404 });
},
websocket: {
open(ws) {
// Simulate sending awareness after join
setTimeout(() => {
ws.send(JSON.stringify({
type: "awareness",
data: { clientId: 99, cursor: { line: 3, col: 7 }, name: "peer" }
}));
}, 100);
},
message() {},
close() {},
},
});
const output: string[] = [];
const bridge = Bun.spawn(["bun", "run", "adapters/vim/bridge.ts"], {
env: { ...process.env, DAEMON_URL: "ws://localhost:4043/ws" },
stdin: "pipe",
stdout: "pipe",
});
const reader = bridge.stdout.getReader();
const decoder = new TextDecoder();
// Collect output
(async () => {
while (true) {
const { done, value } = await reader.read();
if (done) break;
output.push(decoder.decode(value));
}
})();
await Bun.sleep(50);
bridge.stdin.write(JSON.stringify({ type: "connect", room: "test" }) + "\n");
await Bun.sleep(200);
const awarenessMsg = output.join("").split("\n")
.filter(Boolean)
.map(l => JSON.parse(l))
.find(m => m.type === "cursor");
expect(awarenessMsg).toBeDefined();
expect(awarenessMsg.data.line).toBe(3);
expect(awarenessMsg.data.col).toBe(7);
expect(awarenessMsg.data.name).toBe("peer");
bridge.kill();
server.stop();
});
});
```
**Step 2: Run test to verify it fails**
```bash
bun test adapters/vim/bridge.test.ts
```
Expected: FAIL - no cursor message received
**Step 3: Handle awareness in bridge.ts**
In `adapters/vim/bridge.ts`, in the `ws.onmessage` handler, add a case for awareness:
```typescript
case "awareness":
// Forward cursor info to vim
if (parsed.data?.cursor) {
send({
type: "cursor",
data: {
clientId: parsed.data.clientId,
line: parsed.data.cursor.line,
col: parsed.data.cursor.col,
name: parsed.data.name || `peer-${parsed.data.clientId}`,
},
});
}
break;
```
**Step 4: Run test to verify it passes**
```bash
bun test adapters/vim/bridge.test.ts
```
Expected: PASS
**Step 5: Commit**
```bash
git add adapters/vim/bridge.ts adapters/vim/bridge.test.ts
git commit -m "Forward awareness/cursor to vim from bridge"
```
---
## Task 4: Cursor Sync - Vim Plugin
**Files:**
- Modify: `adapters/vim/collab.vim` (receive cursor, display highlight)
**Step 1: Add cursor handling to vim plugin**
In `adapters/vim/collab.vim`, in `OnOutput()` function, add a case for cursor messages:
```vim
elseif msg.type ==# 'cursor'
call s:ShowPeerCursor(msg.data)
```
Add the display function:
```vim
var peer_match_ids: dict<number> = {}
def ShowPeerCursor(data: dict<any>)
const client_id = string(data.clientId)
# Clear previous highlight for this peer
if has_key(peer_match_ids, client_id)
silent! matchdelete(peer_match_ids[client_id])
endif
# Highlight the cursor position (line, col are 0-indexed from bridge)
const line_nr = data.line + 1
const col_nr = data.col + 1
# Create a 1-char highlight at cursor position
const pattern = '\%' .. line_nr .. 'l\%' .. col_nr .. 'c.'
peer_match_ids[client_id] = matchadd('PeerCursor', pattern, 10)
enddef
def ClearPeerCursors()
for id in values(peer_match_ids)
silent! matchdelete(id)
endfor
peer_match_ids = {}
enddef
```
Add highlight group setup at top of file:
```vim
highlight PeerCursor ctermbg=yellow guibg=yellow ctermfg=black guifg=black
```
Call `ClearPeerCursors()` in `Disconnect()`.
**Step 2: Manual test**
No automated test for vim highlight rendering. Test manually:
1. Open two vim instances
2. `:CollabJoin testroom` in both
3. Move cursor in one - should see yellow highlight in other
**Step 3: Commit**
```bash
git add adapters/vim/collab.vim
git commit -m "Display peer cursors in vim with yellow highlight"
```
---
## Task 5: Cursor Sync - Send Local Cursor
**Files:**
- Modify: `adapters/vim/collab.vim` (send cursor on CursorMoved)
- Modify: `adapters/vim/bridge.ts` (forward cursor to daemon)
**Step 1: Send cursor position from vim**
In `adapters/vim/collab.vim`, add autocmd in `Connect()`:
```vim
autocmd CursorMoved,CursorMovedI <buffer> call s:SendCursor()
```
Add the send function:
```vim
def SendCursor()
if !connected
return
endif
const pos = getpos('.')
# pos is [bufnum, line, col, off] - line/col are 1-indexed
Send({type: 'cursor', line: pos[1] - 1, col: pos[2] - 1})
enddef
```
**Step 2: Forward cursor from bridge to daemon**
In `adapters/vim/bridge.ts`, in the stdin message handler, add:
```typescript
case "cursor":
if (ws && msg.line !== undefined && msg.col !== undefined) {
ws.send(JSON.stringify({
type: "awareness",
data: {
clientId: Math.floor(Math.random() * 1000000), // TODO: stable client id
cursor: { line: msg.line, col: msg.col },
},
}));
}
break;
```
**Step 3: Manual test**
1. Two vim instances, same room
2. Move cursor in one - yellow highlight appears in other
3. Move cursor in other - highlight updates
**Step 4: Commit**
```bash
git add adapters/vim/collab.vim adapters/vim/bridge.ts
git commit -m "Send local cursor position to peers"
```
---
## Task 6: Persistence - Schema & Setup
**Files:**
- Create: `src/db.ts` (database module)
- Create: `src/db.test.ts` (database tests)
**Step 1: Write failing test for db module**
Create `src/db.test.ts`:
```typescript
import { describe, test, expect, beforeEach, afterEach } from "bun:test";
import { unlinkSync } from "fs";
import { initDb, saveUpdate, getUpdates, close } from "./db";
describe("persistence", () => {
const TEST_DB = ":memory:";
beforeEach(() => {
initDb(TEST_DB);
});
afterEach(() => {
close();
});
test("saves and retrieves updates for a room", () => {
const room = "testroom";
const update1 = new Uint8Array([1, 2, 3]);
const update2 = new Uint8Array([4, 5, 6]);
saveUpdate(room, update1);
saveUpdate(room, update2);
const updates = getUpdates(room);
expect(updates).toHaveLength(2);
expect(updates[0]).toEqual(update1);
expect(updates[1]).toEqual(update2);
});
test("returns empty array for unknown room", () => {
const updates = getUpdates("nonexistent");
expect(updates).toEqual([]);
});
test("rooms are isolated", () => {
saveUpdate("room1", new Uint8Array([1]));
saveUpdate("room2", new Uint8Array([2]));
expect(getUpdates("room1")).toHaveLength(1);
expect(getUpdates("room2")).toHaveLength(1);
expect(getUpdates("room1")[0]).toEqual(new Uint8Array([1]));
});
});
```
**Step 2: Run test to verify it fails**
```bash
bun test src/db.test.ts
```
Expected: FAIL - module not found
**Step 3: Implement db.ts with bun:sqlite**
Create `src/db.ts`:
```typescript
import { Database } from "bun:sqlite";
let db: Database | null = null;
export function initDb(path: string = "collabd.db"): void {
db = new Database(path);
db.run(`
CREATE TABLE IF NOT EXISTS updates (
id INTEGER PRIMARY KEY AUTOINCREMENT,
room TEXT NOT NULL,
data BLOB NOT NULL,
created_at INTEGER DEFAULT (unixepoch())
)
`);
db.run(`CREATE INDEX IF NOT EXISTS idx_room ON updates(room)`);
}
export function saveUpdate(room: string, data: Uint8Array): void {
if (!db) throw new Error("Database not initialized");
db.run("INSERT INTO updates (room, data) VALUES (?, ?)", [room, data]);
}
export function getUpdates(room: string): Uint8Array[] {
if (!db) throw new Error("Database not initialized");
const rows = db.query("SELECT data FROM updates WHERE room = ? ORDER BY id").all(room) as { data: Uint8Array }[];
return rows.map(r => new Uint8Array(r.data));
}
export function close(): void {
db?.close();
db = null;
}
```
**Step 4: Run test to verify it passes**
```bash
bun test src/db.test.ts
```
Expected: PASS
**Step 5: Commit**
```bash
git add src/db.ts src/db.test.ts
git commit -m "Add bun:sqlite persistence layer"
```
---
## Task 7: Persistence - Hook Into Session
**Files:**
- Modify: `src/session.ts` (save updates, load on create)
- Modify: `src/session.test.ts` (test persistence integration)
**Step 1: Write failing test for session persistence**
Add to `src/session.test.ts`:
```typescript
import { initDb, getUpdates, close as closeDb } from "./db";
describe("session persistence", () => {
beforeEach(() => {
initDb(":memory:");
});
afterEach(() => {
closeDb();
});
test("saves updates to database", () => {
const session = getOrCreateSession("persist-test");
const client = mockClient();
session.join(client);
// Create a yjs update
const doc = new Y.Doc();
const text = doc.getText("content");
text.insert(0, "hello");
const update = Y.encodeStateAsUpdate(doc);
session.applyUpdate(client, Array.from(update));
// Check database has the update
const saved = getUpdates("persist-test");
expect(saved.length).toBeGreaterThan(0);
});
test("loads existing updates on session create", () => {
// First session creates content
const session1 = getOrCreateSession("reload-test");
const client1 = mockClient();
session1.join(client1);
const doc = new Y.Doc();
const text = doc.getText("content");
text.insert(0, "persisted");
const update = Y.encodeStateAsUpdate(doc);
session1.applyUpdate(client1, Array.from(update));
// Clear in-memory sessions
removeSession("reload-test");
// New session should load from db
const session2 = getOrCreateSession("reload-test");
const content = session2.doc.getText("content").toString();
expect(content).toBe("persisted");
});
});
```
**Step 2: Run test to verify it fails**
```bash
bun test src/session.test.ts
```
Expected: FAIL - updates not persisted
**Step 3: Modify session.ts to persist updates**
In `src/session.ts`:
```typescript
import { saveUpdate, getUpdates } from "./db";
// In getOrCreateSession():
export function getOrCreateSession(room: string): Session {
let session = sessions.get(room);
if (!session) {
session = new Session();
// Load persisted updates
const updates = getUpdates(room);
for (const update of updates) {
Y.applyUpdate(session.doc, update);
}
sessions.set(room, session);
}
return session;
}
// In Session.applyUpdate(), after Y.applyUpdate():
applyUpdate(sender: Client, data: number[]): void {
const update = new Uint8Array(data);
Y.applyUpdate(this.doc, update);
// Persist the update (need room name - add to session)
if (this.room) {
saveUpdate(this.room, update);
}
// ... rest of broadcast logic
}
```
Add room tracking to Session:
```typescript
class Session {
doc: Y.Doc;
clients: Set<Client>;
room?: string; // Add this
// Update getOrCreateSession to set it:
session.room = room;
}
```
**Step 4: Run test to verify it passes**
```bash
bun test src/session.test.ts
```
Expected: PASS
**Step 5: Commit**
```bash
git add src/session.ts src/session.test.ts
git commit -m "Persist yjs updates to sqlite, reload on session create"
```
---
## Task 8: Persistence - Init on Daemon Start
**Files:**
- Modify: `src/index.ts` (init db on startup)
**Step 1: Add db initialization**
In `src/index.ts`, at the top after imports:
```typescript
import { initDb } from "./db";
// Initialize database
const DB_PATH = process.env.COLLABD_DB || "collabd.db";
initDb(DB_PATH);
```
**Step 2: Manual test**
1. Start daemon: `just dev`
2. Join room, make edits
3. Kill daemon (Ctrl+C)
4. Restart daemon
5. Join same room - content should be there
**Step 3: Commit**
```bash
git add src/index.ts
git commit -m "Initialize sqlite database on daemon startup"
```
---
## Task 9: Write Adapter Guide
**Files:**
- Create: `docs/ADAPTERS.md`
**Step 1: Write the adapter guide**
Create `docs/ADAPTERS.md`:
```markdown
# Writing Editor Adapters for collabd
This guide explains how to write an adapter that lets any editor participate
in collaborative editing sessions.
## Architecture
```
┌─────────────┐ json/channel ┌─────────────┐ websocket ┌─────────────┐
│ Editor │ ◄──────────────────► │ Bridge │ ◄────────────────► │ Daemon │
│ (plugin) │ │ (bun) │ │ :4040 │
└─────────────┘ └─────────────┘ └─────────────┘
```
The **bridge** is a Bun process that:
- Manages the Yjs document (CRDT state)
- Translates editor buffer changes to Yjs operations
- Translates remote Yjs updates to buffer content
- Speaks a simple JSON protocol with the editor plugin
The **plugin** is editor-specific code that:
- Hooks buffer change events
- Sends buffer content to bridge
- Applies remote content from bridge
- Optionally displays peer cursors
## Why a Bridge?
Most editors can't embed Yjs directly:
- Vim: No npm/node, limited async
- Helix: Rust-only plugins
- Kakoune: Shell-based scripting
The bridge isolates CRDT complexity. Plugins stay simple.
## Protocol: Plugin ↔ Bridge
Messages are newline-delimited JSON.
### Plugin → Bridge
```json
{"type": "connect", "room": "myroom"}
{"type": "disconnect"}
{"type": "content", "text": "full buffer contents"}
{"type": "cursor", "line": 5, "col": 10}
```
### Bridge → Plugin
```json
{"type": "ready"}
{"type": "connected", "room": "myroom"}
{"type": "disconnected"}
{"type": "content", "text": "full buffer contents"}
{"type": "peers", "count": 2}
{"type": "cursor", "data": {"clientId": 123, "line": 5, "col": 10, "name": "peer-123"}}
{"type": "error", "message": "connection failed"}
```
## Implementing a Plugin
### 1. Spawn the Bridge
```
bun run adapters/vim/bridge.ts
```
Or use your own bridge (see next section).
Communication via stdin/stdout with JSON lines.
### 2. Wait for Ready
Bridge sends `{"type": "ready"}` when it starts. Then send connect:
```json
{"type": "connect", "room": "myroom"}
```
### 3. Handle Content Updates
When you receive `{"type": "content", "text": "..."}`:
1. Save cursor position
2. Replace buffer contents with received text
3. Restore cursor position (clamped to valid range)
**Critical:** Do NOT trigger your change handler when applying remote content,
or you'll create an infinite loop.
### 4. Send Local Changes
When user edits the buffer:
1. Get full buffer text
2. Send `{"type": "content", "text": "full text"}`
The bridge handles diffing - just send the whole buffer.
### 5. Cursor Sync (Optional)
On cursor move:
```json
{"type": "cursor", "line": 5, "col": 10}
```
On receiving cursor:
```json
{"type": "cursor", "data": {"line": 5, "col": 10, "name": "peer"}}
```
Render a highlight/virtual text at that position.
## Implementing a Custom Bridge
If you need a bridge in a different language (e.g., Rust for Helix),
implement the same protocol. Key responsibilities:
1. **Connect to daemon** at `ws://localhost:4040/ws`
2. **Manage Y.Doc** - create on connect, apply updates
3. **Diff buffer changes** - when receiving content from plugin:
```
old_text = current Y.Text content
new_text = received buffer content
diff = compute_diff(old, new)
apply diff as Y.Text operations
```
4. **Forward updates** - when Y.Doc changes from remote:
```
send full text to plugin as {"type": "content", ...}
```
5. **Handle awareness** - forward cursor positions both ways
### Yjs Libraries by Language
- **JavaScript/TypeScript:** `yjs` (npm)
- **Rust:** `yrs` (crates.io) - Yjs port
- **Python:** `pycrdt` (pypi)
- **Go:** `yjs-go` (experimental)
## Reference: Vim Adapter
See `adapters/vim/` for a complete example:
- `collab.vim` - 135 lines of Vim9script
- `bridge.ts` - 160 lines of TypeScript
The vim plugin:
1. Spawns bridge as job
2. Communicates via channels (vim's async IPC)
3. Sets TextChanged autocmd to detect edits
4. Applies remote content with `:delete` + `setline()`
## Testing Your Adapter
1. Start daemon: `just dev`
2. Open editor A, join room "test"
3. Open editor B (vim works), join room "test"
4. Type in one - should appear in other within 100ms
5. Type in both simultaneously - should converge
## Common Pitfalls
**Feedback loops:** Applying remote content triggers your change handler,
which sends it back. Use a flag to suppress during remote apply.
**Cursor jitter:** Cursor moves during remote apply. Save/restore position.
**Large files:** Sending full buffer on every keystroke is fine for <1MB.
For larger files, consider debouncing or incremental updates.
**Encoding:** Always UTF-8. Line endings should be `\n`.
```
**Step 2: Commit**
```bash
git add docs/ADAPTERS.md
git commit -m "Add adapter implementation guide"
```
---
## Summary
After completing all tasks:
- **Cursor sync:** Peers see each other's cursor positions with yellow highlights
- **Persistence:** Room state survives daemon restarts via SQLite
- **Adapter guide:** Clear documentation for adding new editors
The vim adapter remains simple (~150 lines). The bridge handles CRDT complexity.
New adapters follow the same pattern: spawn bridge, send JSON, apply content.