charset negotiation vs mtts: why tintin++ says WILL then REJECTED ================================================================= the problem ----------- when connecting to a telnetlib3 server, there's a 4-second delay before the shell starts. the server is waiting for encoding to resolve. it never does. what the logs show ------------------ the full option negotiation between telnetlib3 and tintin++ takes about 500ms. everything resolves quickly - TTYPE, SGA, NAWS, NEW_ENVIRON. but CHARSET negotiation goes like this: 26.220 server sends DO CHARSET 26.332 tintin responds WILL CHARSET ("i can do charset negotiation") 26.332 server sends SB CHARSET REQUEST UTF-8 UTF-16 LATIN1 ... US-ASCII 26.442 tintin responds SB CHARSET REJECTED tintin++ said "i support charset negotiation" and then rejected a list of 16 charsets including UTF-8 and US-ASCII. every charset the server knows about. the server marks encoding as unresolved, waits for connect_maxwait (4.0s), gives up, falls back to US-ASCII, and finally starts the shell. 30.122 server "encoding failed after 4.00s: US-ASCII" 30.122 server "negotiation complete after 4.00s." so the actual work is done in 500ms. the remaining 3.5 seconds is dead air. but tintin++ already told us the answer ---------------------------------------- in the same connection, tintin++ sent this through NEW_ENVIRON: CHARSET=ASCII CLIENT_NAME=TinTin++ CLIENT_VERSION=2.02.61 MTTS=2825 TERMINAL_TYPE=tmux-256color and through TTYPE cycling (3 rounds): ttype1: TINTIN++ ttype2: tmux-256color ttype3: MTTS 2825 MTTS is the MUD Terminal Type Standard, a bitmask the MUD community created for advertising client capabilities. 2825 in binary: 2825 = 2048 + 512 + 256 + 8 + 1 bit 0 (1) = ANSI color bit 3 (8) = UTF-8 bit 8 (256) = 256 colors bit 9 (512) = OSC color palette bit 11 (2048) = true color bit 3 is set. tintin++ explicitly advertises UTF-8 support through MTTS. two standards, one winner ------------------------- RFC 2066 CHARSET (1997) is a proper telnet standard. it defines a full negotiation dance: 1. one side sends DO CHARSET, other responds WILL CHARSET 2. requester sends SB CHARSET REQUEST ... 3. responder either ACCEPTED or REJECTED this is thorough but cumbersome. the MUD community needed something simpler. MTTS piggybacks on the TTYPE mechanism that already existed. instead of adding a new option negotiation, it uses the third round of TTYPE cycling to send a bitfield encoding all client capabilities in a single integer. no extra round-trips, no subnegotiation, works with any server that already does TTYPE. round 1: client name "TINTIN++" round 2: terminal type "tmux-256color" round 3: MTTS bitfield "MTTS 2825" the MUD world settled on MTTS. clients implement it properly. RFC 2066 CHARSET gets a polite WILL ("sure i know what that option is") followed by REJECTED ("but i don't actually want to use it"). tintin++ is not alone in this - most MUD clients behave the same way. what telnetlib3 could do better ------------------------------- right now telnetlib3 waits for CHARSET to resolve encoding. when it gets REJECTED, it has no fallback path except running out the clock. but by the time CHARSET is rejected (26.442), the server already has: - MTTS=2825 via NEW_ENVIRON (26.595) or will have it momentarily - MTTS 2825 via TTYPE round 3 (26.595) if the server checked MTTS bit 3 after CHARSET rejection, it could resolve encoding to UTF-8 immediately and start the shell in under 600ms instead of waiting the full 4 seconds. this would be a good upstream contribution to telnetlib3: when CHARSET is rejected, check MTTS flags for encoding hints before falling back to the timeout. the full annotated session -------------------------- timestamps are seconds within the connection. total negotiation takes ~477ms. the 4-second mark is the connect_maxwait timeout. +0.000 server Connection from -- phase 1: TTYPE discovery -- +0.000 server send IAC DO TTYPE +0.101 tintin recv IAC WILL TTYPE +0.101 server send IAC SB TTYPE SEND IAC SE (ask for ttype1) -- phase 2: advanced negotiation begins after WILL TTYPE -- +0.102 server send IAC WILL SGA +0.102 server send IAC WILL BINARY +0.102 server send IAC DO NAWS +0.102 server send IAC DO CHARSET +0.212 tintin recv IAC SB TTYPE IS 'TINTIN++' (ttype1: client name) +0.212 server (recognizes MUD client, skips WILL ECHO) +0.213 server send IAC DO NEW_ENVIRON +0.213 server send IAC SB TTYPE SEND IAC SE (ask for ttype2) +0.213 tintin recv IAC DO SGA -> local_option[SGA] = True +0.214 tintin recv IAC DONT BINARY -> local_option[BINARY] = False +0.214 tintin recv IAC WILL NAWS +0.214 tintin recv IAC SB NAWS (cols=89, rows=56) (terminal size!) +0.214 tintin recv IAC WILL CHARSET +0.214 server send IAC WILL CHARSET (reciprocating) +0.214 server send IAC SB CHARSET REQUEST UTF-8 ... US-ASCII IAC SE -- charset rejected, but negotiation continues -- +0.323 tintin recv IAC WILL NEW_ENVIRON +0.323 server send IAC SB NEW_ENVIRON SEND ... +0.323 tintin recv IAC SB TTYPE IS 'tmux-256color' (ttype2: terminal) +0.324 server send IAC SB TTYPE SEND IAC SE (ask for ttype3) +0.324 tintin recv IAC SB CHARSET REJECTED *** rejected *** +0.434 tintin recv NEW_ENVIRON IS: CHARSET=ASCII +0.477 tintin recv NEW_ENVIRON IS: CLIENT_NAME=TinTin++ +0.477 tintin recv NEW_ENVIRON IS: CLIENT_VERSION=2.02.61 +0.477 tintin recv NEW_ENVIRON IS: MTTS=2825 *** utf-8 is here *** +0.477 tintin recv NEW_ENVIRON IS: TERMINAL_TYPE=tmux-256color +0.477 tintin recv IAC SB TTYPE IS 'MTTS 2825' (ttype3: capability bits) -- all options resolved, but encoding still "pending" -- +4.004 server "encoding failed after 4.00s: US-ASCII" +4.004 server "negotiation complete after 4.00s." -- shell finally starts, 3.5 seconds of dead air later --