diff --git a/notes/charset-vs-mtts.txt b/notes/charset-vs-mtts.txt new file mode 100644 index 0000000..59ee38f --- /dev/null +++ b/notes/charset-vs-mtts.txt @@ -0,0 +1,161 @@ +charset negotiation vs mtts: why tintin++ says WILL then REJECTED +================================================================= + +the problem +----------- + +when connecting to a telnetlib3 server, there's a 4-second delay before the +shell starts. the server is waiting for encoding to resolve. it never does. + +what the logs show +------------------ + +the full option negotiation between telnetlib3 and tintin++ takes about 500ms. +everything resolves quickly - TTYPE, SGA, NAWS, NEW_ENVIRON. but CHARSET +negotiation goes like this: + + 26.220 server sends DO CHARSET + 26.332 tintin responds WILL CHARSET ("i can do charset negotiation") + 26.332 server sends SB CHARSET REQUEST UTF-8 UTF-16 LATIN1 ... US-ASCII + 26.442 tintin responds SB CHARSET REJECTED + +tintin++ said "i support charset negotiation" and then rejected a list of 16 +charsets including UTF-8 and US-ASCII. every charset the server knows about. + +the server marks encoding as unresolved, waits for connect_maxwait (4.0s), +gives up, falls back to US-ASCII, and finally starts the shell. + + 30.122 server "encoding failed after 4.00s: US-ASCII" + 30.122 server "negotiation complete after 4.00s." + +so the actual work is done in 500ms. the remaining 3.5 seconds is dead air. + +but tintin++ already told us the answer +---------------------------------------- + +in the same connection, tintin++ sent this through NEW_ENVIRON: + + CHARSET=ASCII + CLIENT_NAME=TinTin++ + CLIENT_VERSION=2.02.61 + MTTS=2825 + TERMINAL_TYPE=tmux-256color + +and through TTYPE cycling (3 rounds): + + ttype1: TINTIN++ + ttype2: tmux-256color + ttype3: MTTS 2825 + +MTTS is the MUD Terminal Type Standard, a bitmask the MUD community created +for advertising client capabilities. 2825 in binary: + + 2825 = 2048 + 512 + 256 + 8 + 1 + + bit 0 (1) = ANSI color + bit 3 (8) = UTF-8 + bit 8 (256) = 256 colors + bit 9 (512) = OSC color palette + bit 11 (2048) = true color + +bit 3 is set. tintin++ explicitly advertises UTF-8 support through MTTS. + +two standards, one winner +------------------------- + +RFC 2066 CHARSET (1997) is a proper telnet standard. it defines a full +negotiation dance: + + 1. one side sends DO CHARSET, other responds WILL CHARSET + 2. requester sends SB CHARSET REQUEST ... + 3. responder either ACCEPTED or REJECTED + +this is thorough but cumbersome. the MUD community needed something simpler. + +MTTS piggybacks on the TTYPE mechanism that already existed. instead of adding +a new option negotiation, it uses the third round of TTYPE cycling to send a +bitfield encoding all client capabilities in a single integer. no extra +round-trips, no subnegotiation, works with any server that already does TTYPE. + + round 1: client name "TINTIN++" + round 2: terminal type "tmux-256color" + round 3: MTTS bitfield "MTTS 2825" + +the MUD world settled on MTTS. clients implement it properly. RFC 2066 CHARSET +gets a polite WILL ("sure i know what that option is") followed by REJECTED +("but i don't actually want to use it"). tintin++ is not alone in this - most +MUD clients behave the same way. + +what telnetlib3 could do better +------------------------------- + +right now telnetlib3 waits for CHARSET to resolve encoding. when it gets +REJECTED, it has no fallback path except running out the clock. but by the time +CHARSET is rejected (26.442), the server already has: + + - MTTS=2825 via NEW_ENVIRON (26.595) or will have it momentarily + - MTTS 2825 via TTYPE round 3 (26.595) + +if the server checked MTTS bit 3 after CHARSET rejection, it could resolve +encoding to UTF-8 immediately and start the shell in under 600ms instead of +waiting the full 4 seconds. + +this would be a good upstream contribution to telnetlib3: when CHARSET is +rejected, check MTTS flags for encoding hints before falling back to the +timeout. + +the full annotated session +-------------------------- + +timestamps are seconds within the connection. total negotiation takes ~477ms. +the 4-second mark is the connect_maxwait timeout. + + +0.000 server Connection from + + -- phase 1: TTYPE discovery -- + + +0.000 server send IAC DO TTYPE + +0.101 tintin recv IAC WILL TTYPE + +0.101 server send IAC SB TTYPE SEND IAC SE (ask for ttype1) + + -- phase 2: advanced negotiation begins after WILL TTYPE -- + + +0.102 server send IAC WILL SGA + +0.102 server send IAC WILL BINARY + +0.102 server send IAC DO NAWS + +0.102 server send IAC DO CHARSET + + +0.212 tintin recv IAC SB TTYPE IS 'TINTIN++' (ttype1: client name) + +0.212 server (recognizes MUD client, skips WILL ECHO) + +0.213 server send IAC DO NEW_ENVIRON + +0.213 server send IAC SB TTYPE SEND IAC SE (ask for ttype2) + + +0.213 tintin recv IAC DO SGA -> local_option[SGA] = True + +0.214 tintin recv IAC DONT BINARY -> local_option[BINARY] = False + +0.214 tintin recv IAC WILL NAWS + +0.214 tintin recv IAC SB NAWS (cols=89, rows=56) (terminal size!) + +0.214 tintin recv IAC WILL CHARSET + +0.214 server send IAC WILL CHARSET (reciprocating) + +0.214 server send IAC SB CHARSET REQUEST UTF-8 ... US-ASCII IAC SE + + -- charset rejected, but negotiation continues -- + + +0.323 tintin recv IAC WILL NEW_ENVIRON + +0.323 server send IAC SB NEW_ENVIRON SEND ... + +0.323 tintin recv IAC SB TTYPE IS 'tmux-256color' (ttype2: terminal) + +0.324 server send IAC SB TTYPE SEND IAC SE (ask for ttype3) + +0.324 tintin recv IAC SB CHARSET REJECTED *** rejected *** + + +0.434 tintin recv NEW_ENVIRON IS: CHARSET=ASCII + +0.477 tintin recv NEW_ENVIRON IS: CLIENT_NAME=TinTin++ + +0.477 tintin recv NEW_ENVIRON IS: CLIENT_VERSION=2.02.61 + +0.477 tintin recv NEW_ENVIRON IS: MTTS=2825 *** utf-8 is here *** + +0.477 tintin recv NEW_ENVIRON IS: TERMINAL_TYPE=tmux-256color + +0.477 tintin recv IAC SB TTYPE IS 'MTTS 2825' (ttype3: capability bits) + + -- all options resolved, but encoding still "pending" -- + + +4.004 server "encoding failed after 4.00s: US-ASCII" + +4.004 server "negotiation complete after 4.00s." + + -- shell finally starts, 3.5 seconds of dead air later --