ssh-workbench/docs/TERMINAL_PARSER.md

Terminal Parser — Architecture & Implementation Guide

Last updated: 2026-04-01 Module: lib-terminal-view Package: com.roundingmobile.terminalview.engine

---

Overview

Our terminal emulator implements a complete VT100/VT220/xterm parser, ported from the TellNext C++ ANSICtrl engine. It also supports VT52 compatibility mode. No external terminal library is used — this is fully owned code.

The parser processes raw byte streams from SSH/Telnet/PTY connections and updates a ScreenBuffer model, which is then rendered by the TerminalRenderer.

---

Parser Hierarchy

BaseTermParser          ← C0/C1, CSI dispatch, SGR, SM/RM, DSR (all VT100 features)
  └─ Vt220Parser        ← DECSCA, DHDW, G2/G3 charsets, locking/single shifts
       └─ XtermParser    ← OSC (title/color), 256-color/RGB SGR, mouse, bracketed paste

Vt52Parser              ← Standalone delegate (NOT in inheritance chain)

There is no intermediate Vt100Parser class — all VT100 features are implemented directly in BaseTermParser. Vt220Parser extends BaseTermParser directly.

Vt52Parser is a separate class used as a delegate by BaseTermParser when VT52 compatibility mode is active. It does not extend BaseTermParser — it has its own independent state machine.

---

State Machine

ANSI Mode (normal operation)

BaseTermParser uses an enum-based state machine:

enum class State {
    GROUND,       // Normal text / C0 controls
    ESCAPE,       // Got ESC (0x1B), waiting for next byte
    CSI_PARAMS,   // Inside CSI (ESC[), collecting parameters
    CSI_GT,       // Inside CSI > (ESC[>), DA2 / kitty keyboard
    CSI_LT,       // Inside CSI < (ESC[<), kitty keyboard pop
    OSC_STRING,   // Inside OSC (ESC]), collecting string until BEL/ST
    DCS_STRING,   // Inside DCS (ESCP), collecting until ST
    CHARSET       // ESC followed by (, ), *, + — waiting for designator
}

Byte flow: process(data) → loop over bytes → dispatch by state.

In GROUND:

• 0x1B (ESC) → enter ESCAPE state
• 0x00-0x1F → processC0() (BEL, BS, HT, LF, CR, etc.)
• 0x7F (DEL) → ignore
• 0x80-0x9F → processC1() (override in Vt220Parser)
• 0x20+ → processPrintable() (batch UTF-8 decode + screen.putChar())

In ESCAPE:

• [ → enter CSI_PARAMS
• ] → enter OSC_STRING
• P → enter DCS_STRING
• (, ), *, + → enter CHARSET
• Other → dispatch via processEscChar()

In CSI_PARAMS:

• Digits → accumulate in paramDigitBuf (max MAX_PARAM_LEN=16 chars)
• ; / , → flush param, increment paramCount
• ?, $, !, etc. → set csiFlags
• Final byte (0x40-0x7E) → dispatchCsi(ch)

VT52 Mode (compatibility)

Activated by CSI ?2l (DECANM reset). When screen.vt52Mode == true, the main process() loop delegates every byte to Vt52Parser.processByte() instead of the ANSI state machine.

VT52 has its own simpler state machine:

enum class State {
    GROUND,       // Normal text / C0 controls
    ESCAPE,       // Got ESC, waiting for VT52 command byte
    CURSOR_ROW,   // ESC Y received, waiting for row byte
    CURSOR_COL    // Got row, waiting for col byte
}

VT52 mode is exited by ESC < (sets screen.vt52Mode = false).

---

CSI Parameter Parsing

CSI sequences: ESC [ <params> <final-byte>

params: IntArray(20)           // up to 20 numeric parameters
paramMissed: BooleanArray(20)  // true if param was not explicitly given
paramCount: Int                // number of params collected
paramDigitBuf: StringBuilder   // current digit accumulator (max 16 chars)
csiFlags: Int                  // bitmask: FLAG_QMARK, FLAG_DOLLAR, FLAG_EXCLAM, FLAG_EQUAL, FLAG_QUOTE

MAX_PARAM_LEN = 16

This controls the maximum digits accumulated per CSI parameter. It was increased from 10 to 16 to handle vttest's leading-zero tests, which send parameters like ESC[00000000004;000000001H (11 digits). With 10, the significant digit was silently dropped after the 10 leading zeros.

Quirk: If a CSI parameter exceeds 16 digits, additional digits are silently dropped. The parameter value will be whatever was accumulated from the first 16 digits. In practice, no real terminal application sends more than 5-6 digits.

CSI Flags

Modifier characters before the final byte set flags:

Flag	Character	Meaning
FLAG_QMARK	?	DEC private mode (CSI ? n h/l)
FLAG_DOLLAR	$	DECRQM, selective erase
FLAG_EXCLAM	!	DECSTR (soft reset)
FLAG_EQUAL	=	Various extended
FLAG_QUOTE	"	DECSCA, DECSASD

Characters +, *, &, / are recognized as intermediate bytes but do not set flags (no dispatch uses them). < and > switch to dedicated CSI_LT/CSI_GT states instead of setting flags.

---

Mode Handling (SM/RM)

DEC Private Modes (CSI ? n h/l)

Mode	Name	Description
1	DECCKM	Application cursor keys (ESC O A vs ESC [ A)
2	DECANM	ANSI mode. Reset = enter VT52 mode, set = exit VT52
3	DECCOLM	80/132 column mode switching
5	DECSCNM	Reverse screen (swap fg/bg for entire screen)
6	DECOM	Origin mode (cursor moves relative to scroll region)
7	DECAWM	Auto-wrap mode
25	DECTCEM	Text cursor enable (show/hide cursor)
47	—	Alternate screen buffer (no save/restore cursor)
67	DECBKM	Backspace mode
1047	—	Alternate screen buffer (no save/restore cursor)
1049	—	Alternate screen buffer (with save/restore cursor)
9	—	X10 mouse tracking
1000	—	Normal mouse tracking
1002	—	Button-event mouse tracking
1003	—	Any-event mouse tracking
1006	—	SGR mouse encoding
1015	—	URXVT mouse encoding
2004	—	Bracketed paste mode

ANSI Modes (CSI n h/l)

Mode	Name	Description
2	KAM	Keyboard lock (no-op, handled externally)
4	IRM	Insert/replace mode
12	SRM	Local echo (no-op, handled externally)
20	LNM	Line feed / new line mode (no-op)

---

DECCOLM (Mode 3) — 80/132 Column Switching

DECCOLM is one of the trickiest escape sequences because it triggers a full terminal resize mid-parse.

How it works

1. Parser encounters CSI ?3h (132 cols) or CSI ?3l (80 cols)
2. setDecPrivateMode(3, on) clears the screen, resets scroll region, homes cursor
3. Calls listener.onColumnModeChange(cols) which reaches TerminalService
4. TerminalService.replaceEngine() creates a new ScreenBuffer and XtermParser

The mid-chunk problem (fixed)

Problem: parser.process(bytes) is iterating a byte array. When DECCOLM fires at byte N, replaceEngine() replaces the entry's screenBuffer and parser. But the OLD parser instance continues executing process() for bytes N+1..end, writing to its screen field — which still points to the old (discarded) buffer. All output after the DECCOLM in the same TCP chunk is lost.

Fix: In replaceEngine(), the line entry.parser.screen = newBuffer redirects the still-executing old parser to write to the new buffer. This works because:

• BaseTermParser.screen is var (mutable)
• Kotlin's synchronized is reentrant (same thread), so the nested lock acquisition in replaceEngine() → called from parser.process() → under parserLock works

fun replaceEngine(sessionId: Long, newBuffer: ScreenBuffer, newParser: XtermParser) {
    val entry = sessions[sessionId] ?: return
    synchronized(entry.parserLock) {
        newBuffer.listener = entry.screenBuffer.listener
        entry.parser.screen = newBuffer  // ← critical: redirect old parser mid-process
        entry.screenBuffer = newBuffer
        entry.parser = newParser
    }
}

Testing DECCOLM

vttest test 1 exercises DECCOLM with these screens:

• Screen 3: 132-column border test (auto-wrap at col 132)
• Screen 4: 80-column border test (auto-wrap at col 80)
• Screen 5: 132-column auto-wrap test
• Screen 6: 80-column auto-wrap test

All require the mid-chunk fix to display the menu after the mode switch.

---

VT52 Compatibility Mode

Entry and exit

Direction	Mechanism	Code path
Enter VT52	CSI ?2l (DECANM reset)	setDecPrivateMode(2, false) → screen.vt52Mode = true
Exit VT52	ESC < (from VT52)	Vt52Parser.processEscape('<') → screen.vt52Mode = false
Exit VT52	CSI ?2h (from ANSI — not reachable from VT52)	setDecPrivateMode(2, true) → screen.vt52Mode = false

Important: CSI ?2h cannot exit VT52 mode because CSI sequences (ESC [) don't exist in VT52. The [ after ESC is treated as an unrecognized VT52 command. The ONLY way to exit VT52 from within VT52 is ESC <.

VT52 Escape Sequences

Sequence	Action	ScreenBuffer call
ESC A	Cursor up	cursorUp(1)
ESC B	Cursor down	cursorDown(1)
ESC C	Cursor right	cursorRight(1)
ESC D	Cursor left	cursorLeft(1)
ESC F	Enter graphics mode	Sets internal graphicsMode flag
ESC G	Exit graphics mode	Clears internal graphicsMode flag
ESC H	Cursor home	locate(0, 0)
ESC I	Reverse linefeed	reverseIndex()
ESC J	Erase to end of screen	clearToEndOfScreen()
ESC K	Erase to end of line	clearToEndOfLine()
ESC Y r c	Direct cursor address	locate(r - 32, c - 32)
ESC Z	Identify (DECID)	Responds with ESC / Z
ESC <	Exit VT52 mode	screen.vt52Mode = false
ESC =	Keypad application mode	(handled by key encoder)
ESC >	Keypad numeric mode	(handled by key encoder)

ESC Y — Direct Cursor Address

This is a 3-byte sequence: ESC Y <row> <col>, where row and col are the target position + 32 (ASCII space = position 0). This requires two extra parser states (CURSOR_ROW, CURSOR_COL).

The row and col bytes can arrive in separate TCP chunks — the state machine correctly handles this case.

DECID Response

In ANSI mode, ESC Z (DECID) sends the primary DA response: ESC [ ? 62 ; 1 ; 2 ; 6 ; 7 ; 8 ; 9 c.

In VT52 mode, ESC Z sends the VT52 identify response: ESC / Z.

C0 Controls in VT52

C0 controls (0x00-0x1F) work identically in both ANSI and VT52 modes:

Byte	Name	Action
0x07	BEL	listener.onBell()
0x08	BS	screen.backspace()
0x09	HT	screen.tab()
0x0A	LF	screen.lineFeed()
0x0B	VT	screen.lineFeed()
0x0C	FF	screen.lineFeed()
0x0D	CR	screen.carriageReturn()
0x18	CAN	Cancel — return to GROUND
0x1A	SUB	Cancel — return to GROUND

---

VT52 Graphics Character Set

VT52 has a graphics mode (ESC F enters, ESC G exits) that maps ASCII bytes 0x5F-0x7E to DEC Special Graphics glyphs (line-drawing characters, math symbols). This is the same character set used by VT100's charset 0 (selected via ESC ( 0).

Mapping Table

Byte	ASCII	Glyph	Unicode	Description
0x5F	_		U+0020	Blank
0x60	`	◆	U+25C6	Diamond
0x61	a	▒	U+2592	Checkerboard
0x62	b	␉	U+2409	HT symbol
0x63	c	␌	U+240C	FF symbol
0x64	d	␍	U+240D	CR symbol
0x65	e	␊	U+240A	LF symbol
0x66	f	°	U+00B0	Degree
0x67	g	±	U+00B1	Plus-minus
0x68	h	␤	U+2424	NL symbol
0x69	i	␋	U+240B	VT symbol
0x6A	j	┘	U+2518	Lower-right corner
0x6B	k	┐	U+2510	Upper-right corner
0x6C	l	┌	U+250C	Upper-left corner
0x6D	m	└	U+2514	Lower-left corner
0x6E	n	┼	U+253C	Crossing
0x6F	o	⎺	U+23BA	Scan line 1
0x70	p	⎻	U+23BB	Scan line 3
0x71	q	─	U+2500	Horizontal line
0x72	r	⎼	U+23BC	Scan line 7
0x73	s	⎽	U+23BD	Scan line 9
0x74	t	├	U+251C	Left tee
0x75	u	┤	U+2524	Right tee
0x76	v	┴	U+2534	Bottom tee
0x77	w	┬	U+252C	Top tee
0x78	x	│	U+2502	Vertical line
0x79	y	≤	U+2264	Less-than-or-equal
0x7A	z	≥	U+2265	Greater-than-or-equal
0x7B	{	π	U+03C0	Pi
0x7C	|	≠	U+2260	Not-equal
0x7D	}	£	U+00A3	Pound sign
0x7E	~	·	U+00B7	Centered dot

Characters below 0x5F are NOT mapped — they pass through as normal ASCII.

Implementation

The mapping table lives in Vt52Parser.GRAPHICS_MAP (companion object) as a CharArray. The static function Vt52Parser.mapGraphicsChar(ch) performs the lookup.

When graphicsMode is true, processPrintable() calls mapGraphicsChar() on each ASCII byte before passing it to screen.putChar(). UTF-8 multi-byte characters are not affected by graphics mode.

State reset

Graphics mode is cleared when:

• ESC G is received (explicit exit)
• ESC < exits VT52 mode (graphics mode is part of VT52 state, not screen state)
• Vt52Parser.reset() is called (on any DECANM mode change)
• ScreenBuffer.reset() clears vt52Mode, and the next DECANM entry calls vt52.reset()

---

Delegation Architecture

Why VT52 is a separate class

VT52 is not part of the VT100→VT220→Xterm evolution. It's a completely different, simpler protocol from 1975 that DEC terminals supported as a compatibility mode. The escape sequences are incompatible:

Feature	ANSI (VT100+)	VT52
Cursor up	ESC [ A	ESC A
Cursor address	ESC [ row ; col H	ESC Y row col
Clear screen	ESC [ 2 J	ESC J (erase to end only)
Parameters	CSI with numeric params	No parameters
Colors/SGR	ESC [ 1 ; 31 m	Not supported

Putting VT52 in the inheritance chain would add complexity to every parser level. As a delegate:

• BaseTermParser holds private val vt52 = Vt52Parser { screen }
• The process() main loop checks screen.vt52Mode on every byte
• When true, delegates to vt52.processByte() — the ANSI state machine is bypassed entirely
• The lambda { screen } ensures Vt52Parser always uses the current screen reference (important for replaceEngine())

Screen reference synchronization

Vt52Parser accesses the screen via a () -> ScreenBuffer lambda, not a stored reference. This handles the replaceEngine() scenario where BaseTermParser.screen is swapped mid-parse — the lambda always reads the current value.

---

Device Attribute Responses

Primary DA (CSI c / ESC Z in ANSI mode)

Response: ESC [ ? 62 ; 1 ; 2 ; 6 ; 7 ; 8 ; 9 c

Meaning: VT220 (62), with capabilities:

• 1: 132 columns
• 2: Printer port
• 6: Selective erase
• 7: DRCS (soft character sets)
• 8: UDK (user-defined keys)
• 9: National replacement character sets

Secondary DA (CSI > c)

Response: ESC [ > 41 ; 0 ; 0 c

Meaning: Model 41 (VT420 class), firmware 0, no hardware options.

VT52 DECID (ESC Z in VT52 mode)

Response: ESC / Z

Meaning: "I am a VT52."

---

Known Quirks and Gotchas

1. Mid-chunk DECCOLM data loss

Already described above in the DECCOLM section. The fix (entry.parser.screen = newBuffer) is critical — without it, any bytes after a DECCOLM switch in the same TCP chunk are silently lost to the discarded buffer.

2. UTF-8 chunk boundary buffering (FIXED 2026-04-01)

Multi-byte UTF-8 sequences can be split across TCP chunks. For example, a 3-byte UTF-8 character (e.g., € = 0xE2 0x82 0xAC) might arrive as [..., 0xE2] in chunk N and [0x82, 0xAC, ...] in chunk N+1. The parser now buffers incomplete lead bytes in utf8Buffer/utf8BufferLen between process() calls. At the start of the next call, completeUtf8() pulls continuation bytes from the new data and emits the completed character before resuming normal parsing.

3. DCS String Terminator (FIXED 2026-04-01)

The DCS parser now correctly handles ESC \ (String Terminator) as a 2-byte sequence, matching the OSC parser's approach. Previously only 1 byte was consumed, leaking the backslash \ to the screen as a visible character.

4. CSI leading zeros

vttest sends parameters like ESC[00000000004;000000001H (11 digits of leading zeros). With MAX_PARAM_LEN < 11, the significant digit is silently dropped and the parameter defaults to 0→1. Fixed by setting MAX_PARAM_LEN = 16.

3. VT52 ESC Y byte values

ESC Y row/col bytes are raw byte values (position + 32), NOT printable digit characters. Row 5 = byte value 37 (not '5'). This is unlike CSI cursor addressing where parameters are decimal digit strings.

4. VT52 mode is NOT screen state (mostly)

screen.vt52Mode lives on ScreenBuffer because it needs to survive across parser state resets and be checked in the process() loop. But graphicsMode lives on Vt52Parser because it's parser state that should reset when entering/exiting VT52 mode.

5. ANSI state machine is paused during VT52

When VT52 mode is active, the ANSI state field (BaseTermParser.state) is frozen at whatever it was when VT52 was entered. When VT52 exits, ANSI parsing resumes from that state. This is fine because:

• DECANM is a CSI command, so state is GROUND after dispatching it
• The VT52→ANSI transition (ESC <) always leaves the parser clean

6. Chunk size and log visibility

TerminalService only hex-dumps chunks smaller than 128 bytes. Larger chunks (like the vttest menu after a DECCOLM switch) are processed silently. When debugging, use session recordings (/sdcard/Download/SshWorkbench/recordings/) to capture all bytes.

7. Reentrant synchronized

parser.process() runs under parserLock. When DECCOLM fires, onColumnModeChange() calls replaceEngine() which takes the same parserLock. This works because Kotlin/JVM synchronized is reentrant for the same thread. But code in callbacks must NEVER spawn a new thread that also takes parserLock — it would deadlock.

---

Testing

Unit tests

File	Count	Coverage
VtParserTest.kt	30+	SGR, CSI, OSC, modes, cursor, colors, UTF-8 chunk splitting
Vt52ParserTest.kt	37	Mode entry/exit, cursor, erase, ESC Y, graphics charset, C0, DECID, split chunks
VttestParityTest.kt	5+	DECCOLM, DECALN, vttest cursor movements (verified against Termux)
ScreenBufferTest.kt	100+	Scrollback, reflow, resize, wrap, insert/delete

vttest verification

All main vttest tests (1-11) pass:

Test	Name	Status
1	Cursor movements	Pass (all 6 screens, 80+132 cols)
2	Screen features	Pass (SGR, scroll, wrap, tabs, save/restore)
3	Character sets	Pass (all 13 sub-tests)
4	Double-sized characters	Pass (DHDW rendering)
5	Keyboard	Skipped (requires physical interaction)
6	Terminal reports	Pass (DA, DSR, CPR, DECREQTPARM)
7	VT52 mode	Pass (cursor, erase, graphics charset, DECID)
8	VT102 insert/delete	Pass (all screens)
9	Known bugs	Pass (all sub-tests)
10	Reset and self-test	Pass (RIS)
11	Non-VT100 (VT220/XTERM)	Pass (colors, BCE, VT220 features)

Running vttest

# Install and launch with auto-connect
adb -s 22160523026079 install -r app/build/outputs/apk/pro/debug/ssh-workbench.v1.0.0-pro-debug.apk
adb -s 22160523026079 shell am start -n com.roundingmobile.sshworkbench.pro/com.roundingmobile.sshworkbench.ui.MainActivity --es profile "SSHTest"

# Send input via broadcast
adb -s 22160523026079 shell "am broadcast -a com.roundingmobile.sshworkbench.INPUT --es text 'vttest'"
adb -s 22160523026079 shell "am broadcast -a com.roundingmobile.sshworkbench.INPUT --es text $'\r'"

# Take screenshot for verification
adb -s 22160523026079 shell screencap -p /sdcard/Download/screen.png
adb pull /sdcard/Download/screen.png /tmp/screen.png

---

File Reference

File	Lines	Purpose
BaseTermParser.kt	~855	Main loop, C0, ESC dispatch, CSI params, SGR, SM/RM, DSR, DA
Vt220Parser.kt	~180	C1 controls, DECSCA, DHDW, G2/G3, charset shifts
XtermParser.kt	~7	OSC title/color (inherits everything else)
Vt52Parser.kt	~130	VT52 delegate: escape dispatch, ESC Y, graphics charset
ScreenBuffer.kt	~900	2D grid, scrollback, modes, cursor, erase, scroll, reflow
ScreenRow.kt	~60	Single row: chars + attrs arrays, wrap flag
TextAttr.kt	~200	64-bit packed cell attributes
MouseMode.kt	~20	Mouse tracking mode/encoding enums
TerminalListener.kt	~15	Callback interface: bell, title, sendToHost, columnModeChange