[FEATURE] Keep lexical continuations in caller-provided storage #25

Open
opened 2026-10-18 21:27:16 +02:00 by erikinkinen · 3 comments
Owner

Problem and motivation

Mixed quotes and expansions pass through mutually calling lexical scanners. Move their continuation state into caller-provided storage so nesting does not grow the Rust call chain.

Proposed behavior and scope

Use one iterative scanner with explicit continuation frames. Migrate lexer, parser, here-document and runtime callers together, and retire the recursive storage-free entry points. Keep token ranges, quoting, line joins, errors and grammar-produced substitution boundaries unchanged.

Authority, security and reliability

Use separate scratch from live grammar-discovery frames. Add no allocation, capability or fixed nesting limit. Report the exact required storage separately from lexical errors; clear scratch and restore the original lexer state on shortage, so a retry with more storage has the same result. Work frames cannot authorize a substitution boundary.

Acceptance criteria

  • Define and document the caller-owned frame and progress APIs, including overflow, erasure and retry behavior.
  • Replace the complete scanner call cycle and migrate every production caller; remove obsolete recursive APIs.
  • Preserve literal tokens, flags, error precedence, source offsets and checked grammar-boundary conversion across every scanner context.
  • Cover deep mixed nesting, exact and insufficient capacities, retries, malformed input and complete scratch cleanup with independent controls.
  • Update native storage layouts, capacity growth, component documentation and the technical manual together.
  • Complete affected Rust and native checks without warnings. Retain the original frame budgets, 60-second gate and remaining source, route and artifact requirements.

Alternatives and tradeoffs

A fixed local stack adds a nesting policy; hidden allocation breaks caller ownership. A constant-space replay algorithm would need a separate correctness and performance argument. Explicit caller storage fits the existing parser model and makes exhaustion visible.

Tracking and rollout

Continue the frame work in #4 and PR #3. Coordinate the API migration and catalog adoption before removing old entry points. Phase completion remains in Integration #65.

## Problem and motivation Mixed quotes and expansions pass through mutually calling lexical scanners. Move their continuation state into caller-provided storage so nesting does not grow the Rust call chain. ## Proposed behavior and scope Use one iterative scanner with explicit continuation frames. Migrate lexer, parser, here-document and runtime callers together, and retire the recursive storage-free entry points. Keep token ranges, quoting, line joins, errors and grammar-produced substitution boundaries unchanged. ## Authority, security and reliability Use separate scratch from live grammar-discovery frames. Add no allocation, capability or fixed nesting limit. Report the exact required storage separately from lexical errors; clear scratch and restore the original lexer state on shortage, so a retry with more storage has the same result. Work frames cannot authorize a substitution boundary. ## Acceptance criteria - [x] Define and document the caller-owned frame and progress APIs, including overflow, erasure and retry behavior. - [x] Replace the complete scanner call cycle and migrate every production caller; remove obsolete recursive APIs. - [x] Preserve literal tokens, flags, error precedence, source offsets and checked grammar-boundary conversion across every scanner context. - [x] Cover deep mixed nesting, exact and insufficient capacities, retries, malformed input and complete scratch cleanup with independent controls. - [x] Update native storage layouts, capacity growth, component documentation and the technical manual together. - [ ] Complete affected Rust and native checks without warnings. Retain the original frame budgets, 60-second gate and remaining source, route and artifact requirements. ## Alternatives and tradeoffs A fixed local stack adds a nesting policy; hidden allocation breaks caller ownership. A constant-space replay algorithm would need a separate correctness and performance argument. Explicit caller storage fits the existing parser model and makes exhaustion visible. ## Tracking and rollout Continue the frame work in [#4](https://git.erikinkinen.fi/erix/exsh/issues/4) and [PR #3](https://git.erikinkinen.fi/erix/exsh/pulls/3). Coordinate the API migration and catalog adoption before removing old entry points. Phase completion remains in [Integration #65](https://git.erikinkinen.fi/erix/integration/issues/65).
Author
Owner

The iterative lexer and production callers are published in 411998e6. Integration a28bf98a adopts the version-23 workspace producers, including explicit lexical capacity. The first four criteria are checked. The generated API/manual and remaining native source, route and frame work stay open, with the original 60-second gate unchanged.

The iterative lexer and production callers are published in [411998e6](https://git.erikinkinen.fi/erix/exsh/commit/411998e6c67d3f77d003888349b4ffcfa762f8a1). [Integration a28bf98a](https://git.erikinkinen.fi/erix/integration/commit/a28bf98ac5a9d9d941078816e2256cfba2d8633b) adopts the version-23 workspace producers, including explicit lexical capacity. The first four criteria are checked. The generated API/manual and remaining native source, route and frame work stay open, with the original 60-second gate unchanged.
Author
Owner

The public Rustdoc JSON is now generated from the published bc058d90 source, using its native library metadata and exact pinned dependencies. This supplies the input for the maintained manual converter. Manual regeneration and the remaining native frame and deployment requirements are still open.

The public Rustdoc JSON is now generated from the published [bc058d90 source](https://git.erikinkinen.fi/erix/exsh/commit/bc058d90cdb2fcf5928cc190bfa482259851f8d8), using its native library metadata and exact pinned dependencies. This supplies the input for the maintained manual converter. Manual regeneration and the remaining native frame and deployment requirements are still open.
Author
Owner

The storage-layout and manual criterion is complete: Docs f3f144b publishes the updated chapter and generated reference, and Integration 61b0d41 selects it. The remaining criterion includes the frame checker at its existing deadline; #4 tracks that work.

The storage-layout and manual criterion is complete: [Docs f3f144b](https://git.erikinkinen.fi/erix/docs/commit/f3f144b0f4aeccbaf724ccfa91199a1fcb0cc0d4) publishes the updated chapter and generated reference, and [Integration 61b0d41](https://git.erikinkinen.fi/erix/integration/commit/61b0d414f12469f06249841ac9261cadae67af0d) selects it. The remaining criterion includes the frame checker at its existing deadline; [#4](https://git.erikinkinen.fi/erix/exsh/issues/4) tracks that work.
erikinkinen added this to the Alpha 1 milestone 2026-10-30 09:33:50 +01:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
erix/exsh#25
No description provided.