[FEATURE] Tokenize strings with caller-owned parsing state #11

Open
opened 2026-10-26 20:12:05 +01:00 by erikinkinen · 1 comment
Owner

Problem and motivation

C programs need to split mutable byte strings while keeping independent parsing state. LibCstd does not yet provide the POSIX reentrant tokenizer.

Proposed behavior and scope

Add strtok_r with caller-owned saved state, namespaced Rust declarations and partial C/C++ headers. Skip separator spans, return nonempty tokens and replace the terminating separator with NUL. Permit a different separator set on each call and preserve all bytes outside the specified mutation.

Follow the POSIX tokenization contract. Supply the implementation through the original memory/compatibility companion, including the conventional native alias.

Authority, security and reliability

Use only the caller's mutable input, read-only separators and saved pointer. Retain no global parser state, allocate nothing and preserve errno. Document lifetime, exclusive access and C restrict requirements. Independent parses must not interfere.

Acceptance criteria

  • Implement caller-owned tokenization without a private input or separator limit.
  • Provide documented Rust declarations, C/C++ headers and explicit native symbols.
  • Cover empty and repeated separators, changing delimiters, high bytes, interleaved parses, guard boundaries and exact mutation.
  • Integrate the original source roles and exercise tokenization in the ordinary C consumer.
  • Document the interface and demonstrate warning-free component and native workload acceptance.

Alternatives and tradeoffs

Reuse the existing instruction companion. The caller-owned cursor avoids hidden shared state and does not need allocation or a thread runtime.

Tracking and rollout

Part of the C runtime, with consumer adoption through ordinary C startup. Track overall work in Phase 6.

## Problem and motivation C programs need to split mutable byte strings while keeping independent parsing state. LibCstd does not yet provide the POSIX reentrant tokenizer. ## Proposed behavior and scope Add `strtok_r` with caller-owned saved state, namespaced Rust declarations and partial C/C++ headers. Skip separator spans, return nonempty tokens and replace the terminating separator with NUL. Permit a different separator set on each call and preserve all bytes outside the specified mutation. Follow the [POSIX tokenization contract](https://pubs.opengroup.org/onlinepubs/007904975/functions/strtok.html). Supply the implementation through the original memory/compatibility companion, including the conventional native alias. ## Authority, security and reliability Use only the caller's mutable input, read-only separators and saved pointer. Retain no global parser state, allocate nothing and preserve errno. Document lifetime, exclusive access and C restrict requirements. Independent parses must not interfere. ## Acceptance criteria - [x] Implement caller-owned tokenization without a private input or separator limit. - [x] Provide documented Rust declarations, C/C++ headers and explicit native symbols. - [ ] Cover empty and repeated separators, changing delimiters, high bytes, interleaved parses, guard boundaries and exact mutation. - [ ] Integrate the original source roles and exercise tokenization in the ordinary C consumer. - [ ] Document the interface and demonstrate warning-free component and native workload acceptance. ## Alternatives and tradeoffs Reuse the existing instruction companion. The caller-owned cursor avoids hidden shared state and does not need allocation or a thread runtime. ## Tracking and rollout Part of [the C runtime](https://git.erikinkinen.fi/erix/lib-cstd/issues/1), with consumer adoption through [ordinary C startup](https://git.erikinkinen.fi/erix/lib-cstd/issues/9). Track overall work in [Phase 6](https://git.erikinkinen.fi/erix/integration/issues/65).
Author
Owner

The tokenizer is implemented locally, with caller-owned continuation state, partial C/C++ headers and the native instruction alias. It handles changing separator sets and unsigned byte values without shared parser state. The C startup fixture now checks interleaved parses and exact buffer mutation; packaging adoption is in progress.

The tokenizer is implemented locally, with caller-owned continuation state, partial C/C++ headers and the native instruction alias. It handles changing separator sets and unsigned byte values without shared parser state. The C startup fixture now checks interleaved parses and exact buffer mutation; packaging adoption is in progress.
erikinkinen added this to the Alpha 1 milestone 2026-10-30 09:34:40 +01:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
erix/lib-cstd#11
No description provided.