[FEATURE] Count UTF-8 terminal scalars as native cells #4

Closed
opened 2026-10-24 02:17:43 +02:00 by erikinkinen · 1 comment
Owner

Problem and motivation

The ordinary terminal currently advances once per UTF-8 byte. A multibyte character therefore occupies several fallback cells, making cursor geometry and narrow manual-page layout misleading.

Proposed behavior and scope

Frame well-formed UTF-8 scalars across fragmented ordinary terminal writes. Each non-ASCII scalar uses one existing fallback glyph and one native cell. Keep the current ASCII font lookup; full Unicode glyph selection, shaping and wide-cell behavior remain separate work. Apply identical rules to framebuffer rendering and geometry-only tracking.

Authority, security and reliability

The decoder stores only bounded framing state. It creates no framebuffer or endpoint authority. Decode only ordinary ground text; non-ASCII bytes inside CSI remain invalid. Malformed prefixes cannot swallow ASCII controls or synthesize controls from encoded bytes. Writer/trusted boundaries discard incomplete framing. Literal log rendering keeps its existing behavior.

Acceptance criteria

  • Keep scalar cell counts stable across every write split and native row wrapping.
  • Reject overlong, surrogate and out-of-range framing without changing CSI refusal or ASCII recovery.
  • Preserve framebuffer/geometry parity and discard partial state at writer boundaries.
  • Document fallback behavior and coordinate the physical manual pager.

Alternatives and tradeoffs

Counting bytes preserves the old fallback output but prevents useful scalar-aligned pagination. This bounded framing change improves the existing fixed-cell renderer without introducing a font engine or allocation.

Tracking and rollout

Coordinate LibConsole PR2, manual geometry and Phase 6. The original implementation follows RFC 3629.

## Problem and motivation The ordinary terminal currently advances once per UTF-8 byte. A multibyte character therefore occupies several fallback cells, making cursor geometry and narrow manual-page layout misleading. ## Proposed behavior and scope Frame well-formed UTF-8 scalars across fragmented ordinary terminal writes. Each non-ASCII scalar uses one existing fallback glyph and one native cell. Keep the current ASCII font lookup; full Unicode glyph selection, shaping and wide-cell behavior remain separate work. Apply identical rules to framebuffer rendering and geometry-only tracking. ## Authority, security and reliability The decoder stores only bounded framing state. It creates no framebuffer or endpoint authority. Decode only ordinary ground text; non-ASCII bytes inside CSI remain invalid. Malformed prefixes cannot swallow ASCII controls or synthesize controls from encoded bytes. Writer/trusted boundaries discard incomplete framing. Literal log rendering keeps its existing behavior. ## Acceptance criteria - [x] Keep scalar cell counts stable across every write split and native row wrapping. - [x] Reject overlong, surrogate and out-of-range framing without changing CSI refusal or ASCII recovery. - [x] Preserve framebuffer/geometry parity and discard partial state at writer boundaries. - [x] Document fallback behavior and coordinate the physical manual pager. ## Alternatives and tradeoffs Counting bytes preserves the old fallback output but prevents useful scalar-aligned pagination. This bounded framing change improves the existing fixed-cell renderer without introducing a font engine or allocation. ## Tracking and rollout Coordinate [LibConsole PR2](https://git.erikinkinen.fi/erix/lib-console/pulls/2), [manual geometry](https://git.erikinkinen.fi/erix/coreutils/issues/7) and [Phase 6](https://git.erikinkinen.fi/erix/integration/issues/65). The original implementation follows [RFC 3629](https://www.rfc-editor.org/rfc/rfc3629.html#section-4).
Author
Owner

e8faed60 frames ordinary UTF-8 text across fragmented writes. Each complete non-ASCII scalar uses one fallback cell; rendering and cursor tracking share the same actions. Malformed prefixes preserve ASCII recovery, CSI refusal stays intact, and writer suspension discards partial text. The physical manual pager now has this dependency; broader font and shaping work remains separate.

[e8faed60](https://git.erikinkinen.fi/erix/lib-console/commit/e8faed60fa4da7c1c17e34da77fb452e1cb5621d) frames ordinary UTF-8 text across fragmented writes. Each complete non-ASCII scalar uses one fallback cell; rendering and cursor tracking share the same actions. Malformed prefixes preserve ASCII recovery, CSI refusal stays intact, and writer suspension discards partial text. The physical manual pager now has this dependency; broader font and shaping work remains separate.
erikinkinen added this to the Alpha 1 milestone 2026-10-30 09:34:38 +01:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
erix/lib-console#4
No description provided.