mud/docs/how/if-journey.rst

if journey — from arcade terminal to moldable worlds
=====================================================

This doc tracks the IF (interactive fiction) integration effort for the MUD engine. It's a living document — updated as research progresses and decisions are made. For detailed technical analysis, see the two interpreter audits (``viola-embedding-audit.rst`` and ``zvm-embedding-audit.rst``) and the original integration notes (``if-integration.txt``).

the vision — five levels of integration
----------------------------------------

Five levels emerged from design discussion. They represent a spectrum of how deeply IF worlds integrate with the MUD world, from simplest to most ambitious.

Level 1 — terminal mode
~~~~~~~~~~~~~~~~~~~~~~~~

Subprocess (dfrotz), text in/out, spectators see text scrolling on a screen. Player sits at an arcade terminal, plays Zork. Others in the room see the output. The IF world is opaque — a black box.

Implemented. See ``docs/how/if-terminal.txt`` for how the system works.

Level 2 — inspectable world
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Embedded interpreter. MUD can READ z-machine state. Know what room the player is in, describe it to spectators, track progress. Spectators don't just see text — they know "jared is in the Trophy Room." Read-only bridge between MUD and IF world.

Level 3 — moldable world
~~~~~~~~~~~~~~~~~~~~~~~~~

Embedded interpreter. MUD can READ AND WRITE z-machine state. Inject items into IF game objects. Put a note in the Zork mailbox. The z-machine object tree (parent/child/sibling with attributes and properties) becomes accessible from MUD code. Two-way bridge. Game world is modifiable from outside.

Level 4 — shared world
~~~~~~~~~~~~~~~~~~~~~~~

Multiple MUD players mapped into the same z-machine instance. Each has their own player object. Independent inventory and position. MojoZork's MultiZork proved this works for V3 games. The IF world is a zone in the MUD that multiple players inhabit simultaneously.

Level 5 — transcendent world
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Z-machine object tree and MUD entity model are unified. An item in Zork IS a MUD item. Pick it up in the IF world, carry it back to the MUD. The mailbox exists at coordinates in both worlds simultaneously. Full bidirectional entity bridge.

Note: Level 1 uses subprocess. Levels 2-5 require being inside the interpreter. Once you're at level 2, the jump to 3 is small (reading memory vs writing it). Level 4 is the MojoZork leap. Level 5 is the dream.

what we know — audit findings
------------------------------

Two Python z-machine interpreters were audited in detail. Don't repeat everything here — see the audit docs for details. Focus on decision-relevant facts.

viola (DFillmore/viola)
~~~~~~~~~~~~~~~~~~~~~~~

Can run games today. All V1-V5 opcodes implemented, partial V6-V8. But global state is deeply tangled (13/18 modules with mutable globals). Multiple instances in one process: not feasible without major refactor.

pygame dependency is cleanly separable (adapter pattern, unidirectional). ``sys.exit()`` in error paths (8+ locations) — needs patching for server use. Memory leaks (5+ unbounded growth patterns) — fixable with cleanup hooks.

Object tree accessors exist and are wired through all opcodes. Full quetzal save/restore working.

See: ``docs/how/viola-embedding-audit.rst``

zvm (sussman/zvm)
~~~~~~~~~~~~~~~~~

Cannot run any game. ~46/108 opcodes implemented, including ZERO input opcodes. But instance-based state (mostly clean, two minor global leaks fixable in ~20 lines). Multiple instances in one process: structurally possible.

IO abstraction is excellent — purpose-built for embedding (abstract ZUI with stubs). Proper exception hierarchy, zero ``sys.exit()`` calls. No memory leaks, clean bounded state.

Object tree parser has complete read API, mostly complete write API. BUT: many object opcodes not wired up at CPU level. Save parser works, save writer is stubbed.

See: ``docs/how/zvm-embedding-audit.rst``

The verdict from the audits: "zvm has the architecture you'd want. viola has the implementation you'd need."

MojoZork (C, Ryan C. Gordon)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

V3 only but has working multiplayer telnet server (MultiZork). Proves multiplayer z-machine works (up to 4 players, separate inventories). ``runInstruction()`` for single-step execution — the async pattern we want. Excellent reference architecture, not directly usable (C only).

the hybrid path — option D
---------------------------

This emerged from comparing the audits side by side. Use zvm's architecture (clean IO abstraction, instance-based state, exception model) as the skeleton. Port viola's working opcode implementations into it.

Why this is attractive:

- zvm's opcodes take ``self`` (ZCpu) and access state through instance attributes
- viola's opcodes use module-level globals (``zcode.game.PC``, ``zcode.memory.data``, etc)
- porting means translating global lookups to instance attribute lookups
- that's mechanical, not creative — could port 5-10 opcodes per hour once the pattern is established
- gets the clean design with the working code

Why this could be harder than it sounds:

- viola and zvm may represent z-machine internals differently
- memory layout assumptions, stack frame format, string encoding details
- porting opcodes may require porting the data structures they operate on
- need to verify each ported opcode against the z-machine spec, not just translate

Estimated effort: medium. Less than finishing zvm from scratch, less than refactoring viola's globals. But not trivial.

the object tree — key to moldable worlds
-----------------------------------------

This is what makes levels 3-5 possible. The z-machine has an object tree — every game entity is a node with parent/child/sibling pointers, attributes (boolean flags), and properties (variable-length data).

What the object tree gives us:

- Read what room the player is in (player object's parent)
- Read container contents (children of the container object)
- Inject items (create objects, parent them to containers)
- Modify game state (set/clear attributes, change properties)
- Query the dictionary (what words the parser recognizes)

Both interpreters have object tree parsers:

- viola: complete read + write, all opcodes wired, working
- zvm: complete read, mostly complete write (missing ``set_attr``/``clear_attr``), many opcodes unwired at CPU level, bug in ``insert_object``

The dictionary problem (level 3+)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Injecting an object into the mailbox works for "look in mailbox" — the game iterates children and prints short names. But "take [new item]" fails unless the word exists in the game's dictionary. The dictionary is baked into the story file.

Options:

- Add words to dictionary at runtime (memory surgery — relocating/expanding the dictionary)
- Intercept input before the parser and handle custom items at the MUD layer
- Use existing dictionary words for injected items ("note", "scroll", "key" are common)
- Hybrid: intercept unrecognized words, check if they match MUD-injected items, handle outside z-machine

This is a level 3-5 problem. Not a blocker for levels 1-2.

games we care about
-------------------

The games that motivated this work:

The Wizard Sniffer (Buster Hudson, 2017)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You play a pig who sniffs out wizards. IFComp winner, XYZZY winner. Screwball comedy. The pig/wizard game that started this. Glulx format (.gblorb) — out of scope for embedded interpreter unless we subprocess to a Glulx interpreter.

Lost Pig (Admiral Jota, 2007)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Grunk the orc chases an escaped pig. IFComp winner, 4 XYZZY awards. Famous for its responsive parser and comedy writing. Z-machine V8 format (not V5 as originally assumed). Now playable in the hybrid interpreter.

Zork I, II, III
~~~~~~~~~~~~~~~

The classics. Everyone should play them. V3.

Hitchhiker's Guide to the Galaxy
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Funny, frustrating, great spectator game. V3.

Also: Anchorhead, Photopia, Spider and Web, Shade, Colossal Cave.

V3 covers the Infocom catalog. V5/V8 covers most modern IF. Lost Pig is V8 (same opcodes as V5, different packed address scaling). Wizard Sniffer is Glulx (out of scope for embedded). V8 support is now implemented.

Note: no Python Glulx interpreter exists. Games that target Glulx (some modern Inform 7) are out of scope unless we subprocess to a C interpreter.

architecture fit
----------------

How this fits the existing MUD architecture. The codebase is ready:

Mode stack
~~~~~~~~~~

Push "if" mode. All input routes to IF handler, bypassing command dispatch. Same pattern as editor mode. Already proven.

Input routing
~~~~~~~~~~~~~

``server.py`` shell loop checks ``player.mode``. Add elif for "if" mode, route to ``if_game.handle_input()``. Same as ``editor.handle_input()``.

Room-local broadcasting
~~~~~~~~~~~~~~~~~~~~~~~~

Implemented for IF. ``broadcast_to_spectators()`` in ``if_session.py`` sends to all players at the same (x,y) location. Pattern will be reused for ambient messages, weather, room events.

State storage
~~~~~~~~~~~~~

Quetzal saves stored as files in ``data/if_saves/{player}/{game}.qzl``. Filesystem approach (simpler than SQLite blobs for dfrotz, which already writes to disk).

Terminal game object
~~~~~~~~~~~~~~~~~~~~

A room object (or coordinate-anchored object) that hosts IF sessions. Players "use" it to enter IF mode. Pattern extends to other interactive objects.

open questions
--------------

Things we haven't figured out yet. Update this as questions get answered.

1. V3 opcode footprint
~~~~~~~~~~~~~~~~~~~~~~

How many of the ~62 missing zvm opcodes are actually exercised by V3 games? V3 uses a smaller subset. If we target V3 first, the hybrid might need 30 ported, not 62.

UPDATE: Opcode tracing (via ``scripts/trace_zmachine.py``) found Zork 1 uses 69 opcodes. zvm had 36 implemented. 33 were ported from viola. All 69 are now implemented in the hybrid interpreter (``src/mudlib/zmachine/``).

All V3 gaps have been resolved. sread tokenization works correctly. save/restore is not yet functional (see question 7).

2. zvm/viola memory layout compatibility
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Do they represent z-machine memory the same way? Both use bytearrays, but header parsing, object table offsets, string encoding — are these compatible enough that porting opcodes is translation, or is it a deeper rewrite?

UPDATE: Resolved. The hybrid interpreter now works end-to-end for both V3 and V8. V8 uses the same object model as V5 (14-byte frames, 48 attributes, 65535 objects) and the same opcodes. The only difference is packed address scaling (×8 vs ×4). Memory layout handled correctly for all supported versions.

3. Async model
~~~~~~~~~~~~~~

Both interpreters have blocking run loops. Options:

- ``run_in_executor`` (thread pool) — standard pattern, adds latency
- extract ``step()`` and call from async loop — zvm audit says this is ~5 lines
- run in separate thread with queue-based IO — more complex but natural

Which is best for the MUD's tick-based game loop?

4. Multiplayer z-machine
~~~~~~~~~~~~~~~~~~~~~~~~

MojoZork does this for V3. What would it take for V5? The V5 object model is larger (65535 objects vs 255). Do V5 games assume single-player in ways that break multiplayer?

5. Game file licensing
~~~~~~~~~~~~~~~~~~~~~~

Infocom games are abandonware but not legally free. Modern IF games (Lost Pig, Wizard Sniffer) are freely distributable. Need to figure out what we can bundle vs what players bring.

6. Dictionary injection feasibility
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

How hard is it to add words to a z-machine dictionary at runtime? The dictionary is in static memory. Adding words means expanding it, which means relocating it if there's no space. Is this practical?

7. Save/restore in the hybrid interpreter
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

RESOLVED: save/restore is now fully implemented and working. Key pieces:

- ``QuetzalWriter`` chunk generators implemented (``IFhd`` for header, ``CMem`` for XOR-compressed dynamic memory, ``Stks`` for stack frame serialization)
- ``op_save`` and ``op_restore`` wired to filesystem layer via ``TrivialFilesystem``
- round-trip tested: save game state, restore it, continue playing
- fixed Quetzal ``Stks`` field mapping: ``return_pc`` belongs on the caller frame's ``program_counter``, not the current frame. ``varnum`` is the store variable on the current frame. round-trip tests masked this because writer and parser had the same bug symmetrically
- fixed V3 save branch processing on restore: in-game saves store PC pointing at branch data after the save opcode (0xB5). ``_try_restore()`` detects this and calls ``_branch(True)`` to advance past it. without this, branch bytes were decoded as instructions
- fixed restored local var padding: save files store only declared locals, runtime expects 15 slots. now zero-pads on restore

Quetzal format is now fully supported for both reading and writing saves. Diagnostic tooling: ``scripts/zmachine_inspect.py`` for offline state inspection, instruction trace deque (last 20) auto-dumps on crash.

what to do next
---------------

Concrete next steps, roughly ordered. Update as items get done.

- [x] trace V3 opcode usage: run zork through viola with opcode logging, get the actual set of opcodes a real game uses. this tells us how much porting work the hybrid path actually requires. (done — found 69 opcodes, see ``scripts/trace_zmachine.py``)

- [ ] compare memory layouts: look at how viola and zvm represent z-machine memory, object tables, string tables. determine if opcode porting is mechanical translation or deeper adaptation.

- [x] prototype the hybrid: pick 5-10 common opcodes, port them from viola to zvm's architecture. see how the pattern feels. if it's smooth, the hybrid is viable. if every opcode is a battle, reconsider. (done — all 69 Zork 1 opcodes ported, hybrid interpreter lives in ``src/mudlib/zmachine/``)

- [x] build level 1 prototype: regardless of interpreter choice, implement the terminal object, IF mode, and subprocess dfrotz path. this proves the MUD-side architecture (mode stack, spectators, save/restore) independently of the interpreter question. (done — see ``docs/how/if-terminal.txt``)

- [x] implement save/restore: finished ``QuetzalWriter`` chunk generators (IFhd, CMem, Stks) and wired ``op_save``/``op_restore`` to filesystem. quetzal round-trip now works — can save during gameplay, restore, and continue. also fixed parser off-by-one bug in return_pc.

- [x] wire embedded interpreter to MUD: connected the hybrid interpreter to the MUD's mode stack via ``EmbeddedIFSession``. .z3 files use the embedded interpreter; other formats fall back to dfrotz. save/restore works via QuetzalWriter/QuetzalParser. state inspection (room name, objects) enables level 2. found and fixed a quetzal parser bug (bit slice for local vars was 3 bits, needed 4). (done — see ``src/mudlib/embedded_if_session.py``, ``src/mudlib/zmachine/mud_ui.py``)

- [ ] study MojoZork's multiplayer model: read the MultiZork source for how it handles multiple players in one z-machine. document the pattern for our eventual level 4.

- [x] find the game files: locate freely distributable z-machine story files for the games we care about. Wizard Sniffer, Lost Pig, Zork (if legally available). (zork1.z3 bundled in content/stories/)

- [x] V8 support and Lost Pig: relaxed version gates (V8 = V5 with ×8 packed addresses), implemented extended opcode decoder, ported V5+ opcodes (aread, call_vn/vn2/vs2, save_undo stub, log/art shift, scan_table, tokenize, copy_table, print_unicode). found and fixed: insert_object wrong removal order, double-byte operand decoder reading type bytes interleaved with operands instead of all types first, opcode detection using 7-bit mask instead of 5-bit. Lost Pig now runs to completion (101K steps, 61 opcodes). (done — LostPig.z8 bundled in content/stories/, see ``scripts/trace_lostpig.py``)

- [x] wire V8 games to MUD: ``play.py`` routes .z3/.z5/.z8 to ``EmbeddedIFSession``. Lost Pig playable via ``play lostpig``. fixed upper window leak: V5+ games write room names to window 1 (status line) via ``select_window``. ``MudScreen`` now tracks the active window and suppresses writes to the upper window, preventing status line text from appearing in game output.

- [ ] implement real save_undo: currently stubs returning -1 ("not available"). a proper implementation needs in-memory state snapshots (dynamic memory + call stack). Lost Pig works without undo but players expect it.

milestone — Zork 1 playable in hybrid interpreter
--------------------------------------------------

The hybrid interpreter (zvm architecture + ported viola opcodes) can now run Zork 1. This is the first working implementation targeting levels 2-5 — inspectable, moldable, and shared worlds. Level 1 (terminal mode) uses subprocess dfrotz; this is the embedded path.

What works:

- 69 V3 opcodes ported, all Zork 1-required opcodes implemented
- key implementations: ``op_test`` (conditional logic), ``op_verify`` (story file checksums), ``sread`` with ZLexer tokenization (parsing player input)
- ``step()`` method for async MUD integration — single instruction at a time, no blocking loop
- instruction trace deque (last 20 instructions) for debugging state errors
- smoke test: ``scripts/run_zork1.py`` runs the game headless, exercises core opcode paths
- parser and lexer: all Zork 1 commands work (look, open mailbox, read leaflet, inventory, take, drop, navigation)
- save/restore: full quetzal format support for persisting and restoring game state
- the interpreter is fully playable for Zork 1

What this enables:

- Read z-machine state (object tree, variables) from MUD code (level 2)
- Write z-machine state, inject items, modify world (level 3)
- Multiplayer instances (level 4, following MojoZork patterns)
- Entity bridge (level 5, further out)

The step-based execution model means IF sessions can run in the async MUD game loop without blocking. Each player command advances their z-machine instance by N instructions (until output or a stopping condition). The trace deque captures the last 20 instructions for debugging unexpected state.

milestone — Level 2: embedded interpreter wired to MUD
-------------------------------------------------------

The embedded z-machine interpreter is now connected to the MUD engine. Players can ``play zork1`` and the game runs inside the MUD process — no dfrotz subprocess needed for .z3 files.

What works:

- ``EmbeddedIFSession`` wraps the hybrid interpreter with the same interface as the dfrotz-based ``IFSession``
- MUD ZUI components: ``MudScreen`` (buffered output), ``MudInputStream`` (thread-safe input with events), ``MudFilesystem`` (quetzal saves to disk), ``NullAudio``
- interpreter runs in a daemon thread; ``MudInputStream`` uses ``threading.Event`` for async bridge — interpreter blocks on ``read_line()``, async side feeds input and waits for next prompt
- save/restore via ``::save`` and ``::quit`` escape commands (QuetzalWriter), auto-restore on session start (QuetzalParser)
- state inspection: ``get_location_name()`` reads global variable 0 (player location object), ``get_room_objects()`` walks the object tree
- .z3 files use embedded interpreter, other formats fall back to dfrotz
- fixed quetzal parser bug: ``_parse_stks`` bit slice was ``[0:3]`` (3 bits, max 7 locals), should be ``[0:4]`` (4 bits, max 15 locals) — Zork uses 15
- 558 tests pass including unit tests for MUD UI components and integration tests with real zork1.z3

What this enables:

- spectators can see what room the IF player is in (``get_location_name()``)
- MUD code can read the object tree, variables, and attributes
- foundation for level 3 (moldable world — write z-machine state from MUD)
- no external dependency on dfrotz for V3 games

milestone — V8 support: Lost Pig playable
------------------------------------------

The hybrid interpreter now supports V8 (and by extension V5) story files. Lost Pig — the 4-time XYZZY award winner about an orc chasing a pig — runs to completion in the interpreter.

What was needed:

- version gates relaxed: V8 uses the same object model and opcodes as V5, just different packed address scaling (×8 vs ×4) and larger max file size (512KB)
- extended opcode decoder: V5+ uses 0xBE prefix for extended opcodes. the decoder was stubbed. implemented reading opcode number + type byte after prefix
- double-byte operand fix: ``call_vs2`` and ``call_vn2`` have two operand type bytes. the old decoder read them interleaved with operands (type1, operands, type2, operands) instead of both types first. refactored into ``_read_type_byte()`` + ``_parse_operand_list()``
- double-byte detection fix: the check used ``opcode[0:7]`` (7-bit mask) instead of ``opcode_num`` (5-bit). never matched, so double-byte opcodes were always mis-parsed
- ``insert_object`` fix: was inserting into new parent before removing from old. corrupted sibling chains when moving within same parent. now removes first per spec
- 15 new opcodes: ``aread``, ``call_vn``/``vn2``/``vs2``, ``catch``, ``check_arg_count``, ``save_undo`` (stub), ``restore_undo`` (stub), ``log_shift``, ``art_shift``, ``scan_table``, ``tokenize``, ``copy_table``, ``set_font``, ``print_unicode``, ``check_unicode``

Lost Pig trace: 101K instructions, 61 unique opcodes, full gameplay loop working (room descriptions, parser, object manipulation). ``scripts/trace_lostpig.py`` for tracing.

Performance profiling and optimization work is documented in ``docs/how/zmachine-performance.rst``. Baseline: ~30K opcodes/command at 60 ops/ms (500ms steady-state). Optimization brought cold-start from 4720ms to 786ms (6x improvement).

What this enables:

- modern IF games compiled with Inform 6/7 to Z-machine V5/V8 format are now playable
- same level 2 inspection works (object tree, location, state reading)
- level 3 write APIs already handle V4-8 object format (14-byte frames, 48 attributes, 65535 objects)
- Wizard Sniffer is Glulx (out of scope), but Lost Pig is the better game anyway

related documents
-----------------

``docs/how/if-terminal.txt`` — how the level 1 IF system works (implementation reference)

``docs/how/if-integration.txt`` — original research and integration plan (predates audits)

``docs/how/viola-embedding-audit.rst`` — detailed viola architecture audit

``docs/how/zvm-embedding-audit.rst`` — detailed zvm architecture audit with comparison

``docs/how/architecture-plan.txt`` — MUD engine architecture plan

``DREAMBOOK.md`` — project vision and philosophy