[FEATURE] Tokenize strings with caller-owned parsing state #11
Open
opened 2026-10-26 20:12:05 +01:00 by erikinkinen
·
1 comment
No Branch/Tag specified
main
feature/offline-workspace-source-catalog
feature/remaining-memory-string-family
feature/initialized-thread-main-c418114
feature/initialized-thread-main-f90e6ad
feature/initialized-thread-runtime-68fd72b
feature/locale-byte-comparisons-9a95378
feature/cargo-cleanup-evidence-5905f80
feature/closed-runtime-custody-39c62c7
feature/closed-runtime-custody-b001946
feature/runtime-dependency-routing-c8778a1
feature/bounded-strings-current-main-8df9df7
feature/bounded-string-span-reconciled-26bd172
feature/initialized-thread-images
feature/bounded-string-copy-concatenation
feature/string-set-queries-reconciled-8cace44
feature/bounded-writers-reconciled-2000b5c
feature/thread-image-rustdoc-213525a
feature/qualify-105-9842dc5
feature/terminal-symbol-coverage-merge-ready-a3d1e0dd
feature/native-c-owner-cohort-merge-ready-762b46b0
feature/thematic-runtime-command-owner-merge-ready-3db6ea3d
feature/physical-terminal-modem-reception-merge-ready-b8355361
feature/physical-terminal-line-configuration-merge-ready-a455e079
feature/terminal-timed-break-merge-ready-c8651d49
feature/terminal-software-flow-merge-ready-dba77fad
feature/terminal-drain-deferred-settings-merge-ready-4ba8a472
feature/terminal-queue-flush-merge-ready-566a529c
feature/namespaced-terminal-attributes-merge-ready-79e125e8
feature/native-producer-cleanup-receipts
feature/caller-owned-termios-speeds-merge-ready-408f60a7
feature/terminal-identification-window-controls-merge-ready-5007e5e4
feature/explicit-c-clock-sleep-merge-ready-defebcf2
feature/checked-c-time-arithmetic-merge-ready-8bec6563
feature/caller-owned-multibyte-conversion-merge-ready-9c424681
feature/mediated-c-pipe-creation-merge-ready-09240feb
feature/mediated-poll-select-merge-ready-f4908182
feature/native-command-lifetime-closure-merge-ready-0de5d405
feature/mediated-directory-iteration-merge-ready-5df7b52c
feature/mediated-process-directory-reports-merge-ready-46c5a51e
feature/freestanding-string-call-controls-merge-ready-751731dd
feature/cxx-end-copy-refusal-diagnostics-merge-ready-5422af14
feature/counted-c-file-timestamps-merge-ready-8fd5fbde
feature/mediated-c-file-interfaces-merge-ready-88a608ef
feature/generated-cargo-evidence-retirement-merge-ready-c63ca132
feature/generated-cargo-evidence-retirement-merge-ready-079bc00b
feature/checked-runtime-helper-closure-merge-ready-1b16f95d
feature/checked-runtime-helper-closure-merge-ready-0caf969f
feature/process-root-descriptor-adapters-merge-ready-15e72138
feature/mutable-process-environment-merge-ready-bf3fd43c
feature/caller-owned-string-tokenization-merge-ready-26200b2a
feature/caller-owned-string-tokenization-merge-ready-15850f6b
feature/heap-owned-string-duplication-merge-ready-50d7996c
feature/native-process-startup-exit-merge-ready-95249cec
feature/heap-owned-string-duplication-merge-ready-1239c6a6
feature/native-process-startup-exit-merge-ready-06cd4f4d
feature/native-process-startup-exit-merge-ready-2055caa2
feature/shared-process-heap-merge-ready-3f6b2353
feature/shared-process-heap-merge-ready-66e7ead8
feature/shared-process-heap-merge-ready-d9c56e70
feature/cxx-memory-string-headers-merge-ready-e25a9efa
feature/destination-end-byte-copy-merge-ready-a535f50a
feature/destination-end-string-copies-merge-ready-72ecd27a
feature/destination-end-string-copies-merge-ready-8b7dc2a1
feature/fixed-locale-character-operations-merge-ready-06dd1a8b
feature/fixed-locale-integer-conversions-merge-ready-a939433f
feature/fixed-c-error-catalog-merge-ready-8519ed2d
feature/constant-storage-substring-search-merge-ready-e5e99498
feature/terminal-symbol-coverage
feature/native-c-pure-owner-cohort
feature/canonical-native-owner-cohort-merge-ready-132c7116
feature/native-runtime-source-profile-merge-ready-46258a04
feature/canonical-native-owner-cohort
feature/native-runtime-source-profile
feature/generic-caller-native-owner-merge-ready-6cdd34c8
feature/native-memory-intrinsics
feature/current-locale-multibyte
feature/native-c-owner-cohort
feature/freestanding-string-call-controls
feature/fixed-locale-character-operations
feature/fixed-c-error-catalog
feature/dependency-review-policy-merge-ready-bdfcfbd3
feature/runtime-preparation-diagnostics-merge-ready-3c35f2d9
fix/runtime-preparation-diagnostics-merge-ready-3c35f2d9
feature/source-closure-entry-diagnostics-merge-ready-3f309b84
fix/source-closure-entry-diagnostics-merge-ready-3f309b84
feature/generic-caller-native-owner-merge-ready-e90c0113
perf/generic-caller-native-owner-merge-ready-e90c0113
feature/standard-multibyte-headers-merge-ready-d4324a13
feature/native-process-owner-cohort-merge-ready-31cd92b2
feature/terminal-speed-alias-controls-merge-ready-8c4601e0
feature/cxx-header-callback-controls-merge-ready-e7cfcc9e
feature/explicit-c-sort-search-merge-ready-b7495bce
feature/native-runtime-scope-contract-merge-ready-0b8dcc3b
feature/native-c-owner-compiler-cohort
feature/owned-directory-streams
feature/bounded-string-span-queries
feature/counted-c-file-timestamps
feature/mediated-c-file-interfaces
feature/generated-cargo-evidence-retirement
feature/explicit-c-clock-sleep
feature/terminal-identification-window-controls
feature/mediated-c-pipe-creation
feature/mediated-poll-select
feature/mediated-process-directory-reports
feature/compatibility-runtime-packaging
feature/thread-image-alignment
fix/ci-owned-helper-exclusion
feature/locale-byte-string-comparisons
feature/runtime-custody-hold-diagnostics
fix/workspace-companion-link-ownership
perf/generic-caller-native-owner
feature/raw-live-scope-freshness
fix/runtime-preparation-diagnostics
fix/source-closure-entry-diagnostics
fix/retained-rust-probe-dependencies
fix/deliberate-crash-core-policy
fix/startup-source-contract
fix/wide-string-link-rejection
feature/native-c-owner-input-cohort
feature/standard-multibyte-headers
fix/runtime-toolchain-compatibility
fix/retained-runtime-probe-dependencies
fix/ci-workspace-source-exclusions
build/native-c-owner-compiler-cohort
feature/process-thread-locales
feature/owned-thread-runtime-state
feature/cxx-memory-string-headers
feature/dependency-review-policy
feature/destination-end-byte-copy
feature/destination-end-string-copies
feature/explicit-c-sort-search
feature/constant-storage-substring-search
feature/fixed-locale-integer-conversions
feature/native-runtime-scope-contract
feature/fixture-rustdoc-boundaries
feature/freestanding-length-probe
feature/string-span-probe-framing
feature/native-producer-receipt-contract
feature/freestanding-c-memory
feature/readonly-byte-string-queries
feature/native-process-owner-cohort
feature/terminal-speed-alias-controls
feature/physical-terminal-modem-reception
feature/posix-runtime-formatting
feature/thematic-runtime-command-owner
feature/physical-terminal-line-configuration
feature/terminal-timed-break
feature/terminal-drain-deferred-settings
feature/terminal-queue-flush
feature/terminal-software-flow
feature/caller-owned-termios-speeds
feature/namespaced-terminal-attributes
feature/checked-c-time-arithmetic
feature/caller-owned-multibyte-conversion
feature/native-command-lifetime-closure
feature/cxx-header-callback-controls
feature/mediated-directory-iteration
feature/cxx-end-copy-refusal-diagnostics
feature/checked-runtime-helper-closure
feature/process-root-descriptor-adapters
feature/caller-owned-string-tokenization
feature/heap-owned-string-duplication
feature/mutable-process-environment
feature/native-process-startup-exit
feature/shared-process-heap
feature/ordinary-startup-owner-cohort
feature/owned-c-runtime-ci-graph
feature/ci-source-acquisition
feature/posix-compat
No results found.
Labels
Clear labels
alpha-1
EriX Alpha 1 release acceptance
bug
Something is not working
ci
docs
duplicate
This issue or pull request already exists
enhancement
New feature
help wanted
Need some help
invalid
Something is wrong
performance
Measured performance profiling, bottleneck analysis, and regression budgets.
phase-6
Phase 6: POSIX compatibility, full EriX self-hosting, and its required audits and validation.
question
More information is needed
refactor
Code cleanup, removal of obsolete paths, and structural improvements preserving behavior.
security
wontfix
This won't be fixed
No labels
alpha-1
bug
ci
docs
duplicate
enhancement
help wanted
invalid
performance
phase-6
question
refactor
security
wontfix
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
erix/lib-cstd#11
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem and motivation
C programs need to split mutable byte strings while keeping independent parsing state. LibCstd does not yet provide the POSIX reentrant tokenizer.
Proposed behavior and scope
Add
strtok_rwith caller-owned saved state, namespaced Rust declarations and partial C/C++ headers. Skip separator spans, return nonempty tokens and replace the terminating separator with NUL. Permit a different separator set on each call and preserve all bytes outside the specified mutation.Follow the POSIX tokenization contract. Supply the implementation through the original memory/compatibility companion, including the conventional native alias.
Authority, security and reliability
Use only the caller's mutable input, read-only separators and saved pointer. Retain no global parser state, allocate nothing and preserve errno. Document lifetime, exclusive access and C restrict requirements. Independent parses must not interfere.
Acceptance criteria
Alternatives and tradeoffs
Reuse the existing instruction companion. The caller-owned cursor avoids hidden shared state and does not need allocation or a thread runtime.
Tracking and rollout
Part of the C runtime, with consumer adoption through ordinary C startup. Track overall work in Phase 6.
The tokenizer is implemented locally, with caller-owned continuation state, partial C/C++ headers and the native instruction alias. It handles changing separator sets and unsigned byte values without shared parser state. The C startup fixture now checks interleaved parses and exact buffer mutation; packaging adoption is in progress.