diff --git a/docs/cola.txt b/docs/cola.txt new file mode 100644 index 0000000..f878fe3 --- /dev/null +++ b/docs/cola.txt @@ -0,0 +1,57 @@ +Cola - Text CRDT for Real-Time Collaborative Editing +===================================================== + +https://github.com/nomad/cola + +What is it? + A Rust library implementing a Conflict-free Replicated Data Type (CRDT) + specifically designed for collaborative text editing. Allows multiple peers + to edit the same document concurrently without a central server. + + +Why it's interesting +-------------------- + - Peer-to-peer: no server needed, peers sync directly + - Convergence guaranteed: all replicas eventually reach same state + - Designed for text: not a generic CRDT, optimized for editing operations + - Rust: fast, safe, could compile to WASM for browser or FFI for other langs + + +How CRDTs work (simplified) +--------------------------- + Instead of "insert char at position 5", operations are like "insert char + after unique-id-xyz". Each character gets a unique ID based on who inserted + it and when. This means concurrent edits never conflict - they just get + ordered deterministically. + + +Potential uses +-------------- + - Build an editor-agnostic collab layer + - Terminal multiplexer with shared buffers + - Plugin backend for vim/emacs/helix + - Pair with a simple transport (WebRTC, TCP, WebSocket) + + +To explore +---------- + 1. Clone the repo, run the examples + 2. Look at the Replica and Insertion types + 3. See how edits are encoded and merged + 4. Think about what transport layer you'd use + 5. Consider: could this power a "collab daemon" that editors connect to? + + +Related projects +---------------- + - Automerge: more general CRDT, bigger community + - Yjs: JavaScript CRDT, powers many web editors + - diamond-types: another Rust text CRDT, by the Automerge folks + + +Links +----- + Repo: https://github.com/nomad/cola + CRDTs: https://crdt.tech + Automerge: https://automerge.org + Yjs: https://yjs.dev diff --git a/docs/notes.txt b/docs/notes.txt new file mode 100644 index 0000000..749517f --- /dev/null +++ b/docs/notes.txt @@ -0,0 +1,125 @@ +CLI Collaborative Editing Research +=================================== + +The problem: Zed/VSCode have great collab features. What about terminal folks +who want to use vim/emacs/whatever but still pair/mob program in real-time? + + +HOW THE BIG PLAYERS DO IT +------------------------- + +See detailed breakdowns: + - vscode-liveshare.txt (host-guest model, SSH relay, no CRDT) + - zed-collab.txt (true CRDT, anchors, tombstones, SumTree) + +Quick comparison: + + VSCode Live Share: + - Host-guest model (not true P2P) + - All content stays on host machine + - SSH tunnel (P2P or via Microsoft relay) + - No conflict resolution needed - only one source of truth + - Simpler but dependent on host connection + + Zed: + - True CRDT - every replica is equal + - Anchors instead of offsets (insertion_id + offset) + - Tombstone deletions with version vectors + - Lamport timestamps for ordering concurrent edits + - Per-user undo via undo map + - SumTree (copy-on-write B+ tree) everywhere + +Key insight: + VSCode = "remote desktop for code" + Zed = "Google Docs for code" + +For CLI collab, the Zed approach is more interesting because it's +truly decentralized and doesn't require a persistent host. + + +NEOVIM-SPECIFIC +--------------- + +instant.nvim + https://github.com/jbyuki/instant.nvim + - Pure Lua, no dependencies, CRDT-based + - Run a server, others connect, real-time sync + - Virtual cursors show where others are editing + - Can share single buffer or entire session + - Built-in localhost server, default port 8080 + - Commands: :InstantStartSingle, :InstantJoinSingle, :InstantStartSession + - Separate undo/redo per user + - This is probably the closest to Zed collab for terminal users + +live-share.nvim + https://github.com/azratul/live-share.nvim + https://dev.to/azratul/live-sharenvim-real-time-collaboration-for-neovim-1kn2 + - Builds on instant.nvim with nicer UX + - Still actively developed + + +TERMINAL SHARING (any editor) +----------------------------- + +Upterm + https://github.com/owenthereal/upterm + https://upterm.dev + - Modern tmate alternative, written in Go + - NOT a tmux fork so you keep your tmux config + - GitHub/GitLab/SourceHut/Codeberg auth + - Community server: uptermd.upterm.dev + - Supports scp/sftp file transfer + - WebSocket fallback when SSH blocked + - Can integrate with GitHub Actions for SSH debugging + +tmate + https://tmate.io + - Fork of tmux 2.x, shares terminal sessions + - Simple but stuck on old tmux, config conflicts + +bottlerocketlabs/pair + https://github.com/bottlerocketlabs/pair + - Wrapper around tmux for quick pairing + - Good for vim/emacs users + + +CRDT LIBRARIES (build your own) +------------------------------- + +Cola (Rust) + https://github.com/nomad/cola + - Text CRDT for real-time collaborative editing + - Peer-to-peer, no central server required + - Could theoretically power an editor-agnostic collab layer + - See: cola.txt in this dir for deep dive + +Automerge + https://automerge.org + - More general CRDT library + - Has bindings for many languages + + +THE MISSING PIECE +----------------- + +Nobody has built the "any editor" dream yet. Would need: + 1. Shared CRDT document layer (cola/automerge) + 2. LSP forwarding to share language intelligence + 3. Thin clients for each editor connecting to shared state + +This could be a fun project to explore. + + +QUICK START +----------- + +To try instant.nvim: + 1. Install the plugin + 2. One person runs :InstantStartServer 0.0.0.0 8080 + 3. Same person runs :InstantStartSession [ip] 8080 + 4. Others run :InstantJoinSession [ip] 8080 + +To try Upterm: + 1. brew install owenthereal/upterm/upterm (or build from source) + 2. upterm host -- tmux new -s shared + 3. Share the SSH connection string with your pair diff --git a/docs/synthesis.txt b/docs/synthesis.txt new file mode 100644 index 0000000..356dfb2 --- /dev/null +++ b/docs/synthesis.txt @@ -0,0 +1,217 @@ +Research Synthesis - Editor-Agnostic CLI Collaboration + +THE CORE PROBLEM: + +Zed and VSCode have beautiful real-time collaboration. But they lock you into their editors. If you're a vim/helix/kakoune user and want to pair program with a friend, you shouldn't have to make them switch editors. The goal: divorce collaborative editing from any specific editor. + +EXISTING APPROACHES ANALYZED: + +1. Terminal Multiplexing (upterm, tmate, tmux sharing) + + How it works: Share a PTY over the network. Everyone sees the same terminal output, keystrokes forwarded to the shell. + + Upterm specifically: Reverse SSH tunnel to a central server, clients connect through it. MultiWriter pattern broadcasts output to all connected clients. + + Pros: Works TODAY with any CLI editor. Zero editor integration needed. Good for "let me show you something" pair programming. + + Cons: No concurrent editing (everyone's typing goes to same shell). No offline. No semantic awareness. Last keystroke wins. Not true collaborative editing. + + Verdict: Great for terminal screenshare, not for document collaboration. + +2. File-Level Sync (VSCode LiveShare style) + + How it works: Host owns the workspace. Guests get proxied file access. SSH protocol with relay fallback. + + Not actually CRDT-based - more like remote desktop for code. + + Sessions expire after 24 hours. P2P when possible, Microsoft relay otherwise. + + Verdict: Doesn't solve editor-agnostic problem. Guests are still locked to host's environment. + +3. CRDT-Based Document Sync (Zed, instant.nvim) + + How it works: Each character gets a unique ID. Operations are "insert after ID xyz" not "insert at position 5". Concurrent edits automatically merge correctly. + + Zed's architecture: Anchors (logical positions), tombstone deletions, Lamport timestamps, version vectors, per-user undo maps. Server for auth/discovery, CRDT for document state. + + instant.nvim: Pure Lua implementation for Neovim. WebSocket server routes messages. Position IDs (tombstone vector clocks) for conflict-free ordering. + + Key insight from instant.nvim: 70% of the code is editor-agnostic (transport + CRDT algorithm). Only 30% is neovim-specific (buffer events, manipulation, cursor display). + +THE PROPOSED ARCHITECTURE: + +CRDT Daemon + Thin Editor Adapters + +The daemon handles all the hard parts: +- CRDT text buffer (using cola or diamond-types) +- Network sync (WebSocket for remote, Unix socket for local) +- Session management +- Peer discovery/auth + +Each editor gets a minimal adapter that: +1. Hooks into buffer change events +2. Serializes changes as (offset, length, text) +3. Sends to daemon +4. Receives remote operations from daemon +5. Applies changes to local buffer +6. Optionally: displays peer cursors + +Why this split works: +- Solving CRDT correctly is hard. Do it once in the daemon. +- Each editor's adapter is simple. Just event hooks and buffer manipulation. +- Adding new editors is cheap. Write a small plugin, done. +- Multiple different editors can collaborate simultaneously. + +THE EDITOR ADAPTER REQUIREMENTS: + +For any CLI editor to participate, the adapter needs: + +1. Change event hook - Know when user edits the buffer + - Neovim: nvim_buf_attach with on_lines callback + - Helix: LSP-based or custom events + - Kakoune: FIFO-based extension system + - Vim: +clientserver or plugin + +2. Buffer manipulation - Apply remote changes + - Neovim: nvim_buf_set_lines + - Others: Similar APIs exist + +3. Cursor visualization (optional but nice) - Show where peers are editing + - Neovim: nvim_buf_set_extmark with virtual text + - Others: Editor-specific + +THE LSP ANGLE: + +Many CLI editors already speak LSP (Language Server Protocol). This is interesting because: + +- textDocument/didChange already notifies of edits +- textDocument/didOpen and didClose handle lifecycle +- workspace/executeCommand can carry custom operations + +A "collaboration language server" could: +1. Receive didChange notifications +2. Run them through CRDT +3. Push remote changes back via workspace edits + +This would reduce per-editor work to almost zero - editors already have LSP clients. Worth exploring. + +CRDT LIBRARY CHOICE: + +Cola (https://github.com/nomad/cola): +- Operation-based CRDT for text +- Buffer-agnostic: doesn't store text, just manages coordinates +- Clean API: Replica, Insertion, Deletion +- Real-time P2P focus +- Serialization via serde or custom encode +- Handles out-of-order delivery via backlog +- Benchmarks show 1.4-2x faster than diamond-types in some cases + +Diamond-types (https://github.com/josephg/diamond-types): +- "World's fastest CRDT" +- 5000x-80000x speedup through aggressive RLE +- Stores full history (temporal DAG + spatial state) +- More complex (OpLog, Branch, CausalGraph concepts) +- Great for: large documents, offline-first, audit trails +- WASM support for browser + +For our use case: Cola wins. +- Simpler API, easier to integrate +- Real-time focus matches our needs +- We don't need full history storage +- Less cognitive overhead to work with + +Diamond-types is overkill for initial prototyping. Could revisit for optimization later. + +COMMUNICATION PROTOCOL OPTIONS: + +1. Unix socket - Simple, local only. Good for same-machine testing. + +2. WebSocket - Works remote. Browser-friendly if we ever want web UI. Good default. + +3. stdio pipe - Simplest for CLI tools. Editor spawns daemon, communicates via stdin/stdout. + +4. LSP protocol - Leverage existing infrastructure. Interesting but might be awkward fit. + +Recommendation: WebSocket as primary (works local and remote), Unix socket as fast local alternative. + +REFERENCE IMPLEMENTATIONS: + +repos/cola/ +- src/replica.rs: Main API, 1200+ lines of docs +- src/insertion.rs, deletion.rs: Operation types +- examples/basic.rs: Simple Document wrapper pattern +- Key pattern: editor maintains buffer + Replica, calls inserted/deleted for local ops, integrate_* for remote ops + +repos/instant.nvim/ +- lua/instant.lua: Main logic, mixed nvim + algorithm +- lua/instant/websocket_*.lua: Transport layer (portable) +- Position ID generation (genPID): Tombstone vector clocks +- Shows exactly what adapters need to do + +repos/upterm/ +- host/host.go: Session lifecycle +- io/writer.go: MultiWriter for output broadcast +- Different paradigm but useful for understanding terminal collaboration UX + +repos/diamond-types/ +- Complex internals, good for understanding CRDT optimization +- INTERNALS.md, BINARY.md explain the RLE approach + +NEXT STEPS TO PROTOTYPE: + +Phase 1: Minimal daemon +- Rust binary using cola +- Single document support +- WebSocket server +- Two clients can connect, edits sync + +Phase 2: Neovim adapter +- Lua plugin +- Connects to daemon via WebSocket +- Hooks nvim_buf_attach for changes +- Applies remote changes via nvim_buf_set_lines +- Test: two neovim instances editing same file + +Phase 3: Multi-document +- Session management +- File path mapping +- Join/leave notifications + +Phase 4: Second editor +- Helix adapter (or kakoune, or vim) +- Prove the architecture works across editors + +Phase 5: Polish +- Peer cursors +- User presence indicators +- Better auth (SSH keys, GitHub) +- Discovery service + +OPEN QUESTIONS: + +1. Where does the daemon run? + - Local daemon per machine? Central server? Hybrid? + - For local-first: daemon on each machine, P2P sync + - For easy setup: central server handles routing + +2. How to handle file paths? + - Relative to project root? Absolute? UUID-based? + - Need consistent naming across different machines + +3. Undo/redo coordination? + - Per-user undo (like Zed) or global? + - Cola doesn't handle this - need to build on top + +4. Cursor/selection sync? + - Nice to have, not essential for MVP + - Adds complexity (need to track peer positions) + +5. Permissions? + - Can anyone edit anything? Read-only viewers? + - Future concern, not MVP + +THE DREAM: + +You're in helix. Friend is in neovim. Another friend is in kakoune. You all open the same project, connect to a session, and just... edit together. Changes flow seamlessly. Each person uses their preferred editor with their preferred config. No one had to install anything they don't normally use. + +That's the goal. diff --git a/docs/vscode-liveshare.txt b/docs/vscode-liveshare.txt new file mode 100644 index 0000000..f4f6be7 --- /dev/null +++ b/docs/vscode-liveshare.txt @@ -0,0 +1,106 @@ +VSCode Live Share - Technical Architecture +========================================== + +https://learn.microsoft.com/en-us/visualstudio/liveshare/ + + +ARCHITECTURE MODEL +------------------ + +Host-Guest model, NOT peer-to-peer CRDT: + - One host owns the workspace + - Guests connect to host's machine + - All content stays on host, never synced to cloud or guest machines + - Sessions expire after 24 hours + +Connection flow: + 1. Host starts session, gets unique URL + 2. Guests join via URL + 3. Live Share attempts P2P connection first + 4. Falls back to Microsoft cloud relay if P2P fails (firewalls/NATs) + 5. Some guests can be P2P while others relay in same session + + +SYNCHRONIZATION +--------------- + +NOT using CRDTs - this is a remote workspace model: + - File system level sync, not document-level CRDT + - Host's LSP, terminals, debuggers are shared + - Guests get proxied access to host's environment + - More like "remote desktop for code" than true collaborative editing + +Why this matters: + - Simpler to implement (no conflict resolution needed) + - But requires constant connection to host + - If host disconnects, session ends + - Latency depends on connection to host + + +PROTOCOL & SECURITY +------------------- + +Transport: + - SSH protocol for all data + - P2P: direct SSH connection (ports 5990-5999) + - Relay: SSH over TLS-encrypted WebSockets + +Encryption: + - Diffie-Hellman key exchange for shared secret + - AES symmetric encryption derived from shared secret + - Keys rotated periodically during session + - Keys only in memory, never persisted + +Authentication: + - JWT tokens signed by Live Share service + - Claims include user identity (MSA/AAD/GitHub) + - Session-specific RSA keypair generated by host + - Private key never leaves host memory + + +RELAY SERVICE +------------- + +Microsoft's cloud relay: + - Only used when P2P fails + - Does NOT store or inspect content + - Just routes encrypted SSH packets + - End-to-end encryption means relay can't read traffic + +Enterprise option: + - Private relay servers possible + - Requires additional infrastructure + + +WHAT GETS SHARED +---------------- + + - File system (read/write based on permissions) + - Language services (IntelliSense, go-to-definition) + - Debugging sessions + - Terminal instances (optional, read-only or read-write) + - Localhost servers (port forwarding) + - Cursor positions and selections + + +IMPLICATIONS FOR CLI COLLAB +--------------------------- + +Live Share's approach could work for terminal editors: + 1. One person hosts their tmux/vim session + 2. Others connect via relay or P2P + 3. All editing happens on host machine + 4. No conflict resolution needed + +But: + - Not truly decentralized + - Dependent on host's connection + - Less elegant than CRDT approach + + +LINKS +----- + +Docs: https://learn.microsoft.com/en-us/visualstudio/liveshare/ +Security: https://learn.microsoft.com/en-us/visualstudio/liveshare/reference/security +Connect: https://learn.microsoft.com/en-us/visualstudio/liveshare/reference/connectivity diff --git a/docs/zed-collab.txt b/docs/zed-collab.txt new file mode 100644 index 0000000..9e81935 --- /dev/null +++ b/docs/zed-collab.txt @@ -0,0 +1,175 @@ +Zed Editor - Collaboration Architecture +======================================= + +https://zed.dev/docs/collaboration/overview +https://zed.dev/blog/crdts + + +PHILOSOPHY +---------- + +Collaboration is "part of Zed's DNA" - not bolted on. +Built from ground up with multiplayer in mind. +Uses CRDTs instead of Operational Transformation. + + +WHY CRDT OVER OT +---------------- + +Operational Transformation (OT): + - Transform concurrent operations to apply in different orders + - Requires central server to sequence operations + - Complex correctness proofs + - What Google Docs uses + +CRDT approach: + - Structure data so operations are inherently commutative + - No transformation needed - apply directly on any replica + - Express edits in terms of logical locations, not absolute offsets + - Decentralized by nature + + +CRDT IMPLEMENTATION DETAILS +--------------------------- + +Anchors (logical positions): + - Each position is (insertion_id, offset) pair + - insertion_id = replica_id + sequence_number + - Replicas get unique IDs from server, then generate IDs locally + - No collision risk for concurrent operations + +Fragments: + - Text organized into fragments + - Each fragment knows its insertion ID and offset + - Remote operations can find insertion points regardless of local changes + +Immutable insertions: + - Every piece of inserted text is immutable forever + - Edits mark deletions, they don't remove content + - This is key to conflict-free merging + + +DELETION HANDLING +----------------- + +Tombstones: + - Deleted text gets tombstone marker + - Text is hidden, not removed + - Allows insertions within deleted ranges to resolve correctly + - "the deleted text is merely hidden rather than actually thrown away" + +Version vectors: + - Each deletion has version vector + - Encodes "latest observed sequence number for each replica" + - Prevents concurrent insertions from being incorrectly tombstoned + + +CONFLICT RESOLUTION +------------------- + +Concurrent insertions at same location: + - Sorted by Lamport timestamps (descending) + - Preserves user intent + - Guarantees same ordering on all replicas + +Lamport clocks: + - Logical timestamps for causal ordering + - Each operation increments local clock + - Receiving operation updates clock to max(local, received) + 1 + + +UNDO/REDO +--------- + +Undo map: + - Associates operation IDs with counts + - Odd count = undone + - Even count = redone (or never undone) + - Enables arbitrary-order undo in collaborative context + - Your undo doesn't affect others' operations + + +DATA STRUCTURES +--------------- + +SumTree (the "soul of Zed"): + - Thread-safe, snapshot-friendly, copy-on-write B+ tree + - Leaf nodes contain items + summaries + - Internal nodes contain summary of subtree + - Used EVERYWHERE in Zed (20+ uses) + +Rope: + - B-tree of 128-byte string chunks (fixed size) + - Summaries enable fast offset-to-line/column conversion + - Concurrent access safe via copy-on-write snapshots + +Where SumTree is used: + - Text buffers (via Rope) + - File lists in project + - Git blame info + - Chat messages + - Diagnostics + - Syntax trees + + +SERVER ARCHITECTURE +------------------- + +Components: + - Collaboration server (Rust) + - PostgreSQL database + - LiveKit for voice/screenshare (optional) + +Protocol: + - Protocol buffers (proto/zed.proto) + - RPC over WebSocket + - Server routes messages, manages rooms, auth + +Key crates: + - crates/proto/ - message definitions + - crates/rpc/ - generic RPC framework + - crates/collab/ - collaboration server + + +CHANNELS +-------- + +IRC-like but for code: + - Each channel = ongoing project or work-stream + - Join channel = enter shared room + - See what everyone is working on (ambient awareness) + - Easy to jump into someone's context + +Features: + - Shared cursors with zero latency + - Following (your view follows their navigation) + - Voice chat built-in + - Text chat in editor + - Screen sharing + + +WHAT A CLI VERSION WOULD NEED +----------------------------- + +From Zed's approach: + 1. CRDT text buffer (like Cola) + 2. Anchor-based positions instead of offsets + 3. Tombstone deletions with version vectors + 4. Lamport timestamps for ordering + 5. Undo map for per-user undo + 6. Some transport (WebSocket, WebRTC, etc) + 7. Optional: server for discovery/auth, or pure P2P + +The SumTree/Rope is optional but helps with: + - Large file performance + - Efficient snapshot creation for background tasks + + +LINKS +----- + +CRDT blog: https://zed.dev/blog/crdts +Rope/SumTree: https://zed.dev/blog/zed-decoded-rope-sumtree +Channels: https://zed.dev/blog/channels +Collab docs: https://zed.dev/docs/collaboration/overview +Local dev: https://zed.dev/docs/development/local-collaboration