[FEATURE] Reuse actual retained census traversal ancestry #614

Open
opened 2026-11-08 17:47:30 +01:00 by erikinkinen · 0 comments
Owner

Problem and motivation

Repeated retained-space censuses consume a growing part of the original native producer budget. A read-only output-data diagnostic traversed 66,127 entries using 19,686 fresh directory views. One profiled pass performed about 1.9 million stat/fstat calls; full ancestor validation dominated the traversal. Twelve repeated passes remained roughly stable despite retaining the actual closed capture history. This is diagnostic evidence, not complete image acceptance.

The current traversal validates a parent chain before capturing each child and immediately validates the full child chain again before enumerating it. The design should consolidate genuinely overlapping observations within that compound operation while preserving the original checks and finite limits.

Proposed behavior and scope

Integration should own one explicit, synchronous retained-census traversal context, registered with the actual defining scope before its first descriptor acquisition. It borrows the same actual output-root and parent receivers. A pathname, inode tuple, saved inventory or previous successful census cannot create that relationship.

Combine child-directory capture with its first guarded enumeration. The context may reuse the actual registered parent frame during this compound step, but complete root-to-child path validation must finish before directory names or other observations are returned or consumed. Every directory retains its complete before/after identity and namespace checks. All original file metadata and source-body checks remain on their existing boundaries.

An implementation must document the exact observation it consolidates and show that it is redundant within this actual retained traversal. It must retain a check if equivalence cannot be established. Structural relationships may be retained; successful byte, metadata or namespace verdicts may not authorize later steps or another census.

The root view must be a newly captured original parent/name opening. An exhausted original directory cursor, duplicated OFD or independently opened same-inode descriptor cannot replace it. The context accepts no external FD, owner lookup by class/path, provider callback or serialized completion.

This changes Integration's internal census implementation. It does not change guest ABI, source selection, compiled artifacts, allocation/capacity policy or scenario acceptance.

Authority, security and reliability

Security, then reliability, then performance remains the priority. Each directory and file operation must derive from the original registered retained root. Every existing path edge, literal name, body and metadata check must remain enforced, including complete before/after mutation detection. The census observes namespace and extent; it cannot authenticate source bytes or provide compiler ownership.

Keep the original capture-before-open and once-only close machinery. Preserve actual capture identity across FD-number reuse. Positively closed captures may be absent from a live traversal index, but their failure/diagnostic history must remain separate and no unknown open/close outcome may disappear. Partial acquisition, failed traversal or failed close stays registered with the original owner and remains sticky after restoration.

The context is withdrawn on completion or failure and cannot cross a compiler, tool or native-child effect. Its defined synchronous operations are metadata descriptor acquisition, checks, enumeration and close. It cannot be copied, replayed, reconstructed from ordinary data or used after original retirement. Original total, quiet, graph and retirement limits, material charges, output bounds and aggregate acceptance remain unchanged. There is no timer heartbeat, stat-only approval cache, skipped census or broader filesystem grant.

Acceptance criteria

  • Publish the exact traversal/observation and retirement contract before implementation.
  • Exercise genuine held-FD trees and prove complete entry, extent and metadata results match the existing census, including sparse files and complete fresh views after an exhausted original cursor.
  • Deny foreign owners and independently opened or duplicated same-inode/OFD substitutes; refuse retired, copied or replayed traversal contexts.
  • Use deterministic actual mutations at parent/child boundaries: replacement, rename and restoration, symlink substitution, changed metadata and altered namespace. Preserve the first failure after restoration; do not accept a partial or smaller inventory.
  • Preserve capture-before-open duties, partial-open and unknown-close state, and test actual FD-number reuse without closing or accepting the replacement.
  • Preserve every source-body check and original budget/end; prove no native effect can use an active traversal borrow.
  • Compare actual same-boundary call counts and timings on equivalent fresh FD owners. Demonstrate a substantive reduction in redundant ancestry work, without attributing profiler overhead to production.
  • Pass focused FD/lifecycle controls, complete source-helper declarations, original native-owner byte checks, syntax checks and documentation.
  • Run a fresh uninstrumented original-budget producer/graph control after the implementation is frozen. Keep full image and authenticated guest acceptance separately reported.

Alternatives and tradeoffs

Dropping source checks, caching successful inventories and widening deadlines are excluded. Closed-history-only changes are not yet supported as the dominant traversal fix by the repeat-FD profile. Removing genuinely unused compiler scratch may complement this work only after proving that no actual product, source or metadata receiver still needs it.

The bounded experiment must decide whether a combined capture/enumeration step can preserve every original observation with fewer repeated ancestor walks. If it cannot, retain the current checks and evaluate another explicit design rather than weakening admission.

Tracking and rollout

Related performance bug: #611. Current completed-work accounting prerequisite: #612. Implement on a separate feature branch based on that coherent source, refresh the whole actual helper cohort and retain the existing original owner bodies. Preserve previous failures and distinguish diagnostic, source, native graph, image and guest evidence. No complete native/image/guest acceptance is claimed by this design.

## Problem and motivation Repeated retained-space censuses consume a growing part of the original native producer budget. A read-only output-data diagnostic traversed 66,127 entries using 19,686 fresh directory views. One profiled pass performed about 1.9 million stat/fstat calls; full ancestor validation dominated the traversal. Twelve repeated passes remained roughly stable despite retaining the actual closed capture history. This is diagnostic evidence, not complete image acceptance. The current traversal validates a parent chain before capturing each child and immediately validates the full child chain again before enumerating it. The design should consolidate genuinely overlapping observations within that compound operation while preserving the original checks and finite limits. ## Proposed behavior and scope Integration should own one explicit, synchronous retained-census traversal context, registered with the actual defining scope before its first descriptor acquisition. It borrows the same actual output-root and parent receivers. A pathname, inode tuple, saved inventory or previous successful census cannot create that relationship. Combine child-directory capture with its first guarded enumeration. The context may reuse the actual registered parent frame during this compound step, but complete root-to-child path validation must finish before directory names or other observations are returned or consumed. Every directory retains its complete before/after identity and namespace checks. All original file metadata and source-body checks remain on their existing boundaries. An implementation must document the exact observation it consolidates and show that it is redundant within this actual retained traversal. It must retain a check if equivalence cannot be established. Structural relationships may be retained; successful byte, metadata or namespace verdicts may not authorize later steps or another census. The root view must be a newly captured original parent/name opening. An exhausted original directory cursor, duplicated OFD or independently opened same-inode descriptor cannot replace it. The context accepts no external FD, owner lookup by class/path, provider callback or serialized completion. This changes Integration's internal census implementation. It does not change guest ABI, source selection, compiled artifacts, allocation/capacity policy or scenario acceptance. ## Authority, security and reliability Security, then reliability, then performance remains the priority. Each directory and file operation must derive from the original registered retained root. Every existing path edge, literal name, body and metadata check must remain enforced, including complete before/after mutation detection. The census observes namespace and extent; it cannot authenticate source bytes or provide compiler ownership. Keep the original capture-before-open and once-only close machinery. Preserve actual capture identity across FD-number reuse. Positively closed captures may be absent from a live traversal index, but their failure/diagnostic history must remain separate and no unknown open/close outcome may disappear. Partial acquisition, failed traversal or failed close stays registered with the original owner and remains sticky after restoration. The context is withdrawn on completion or failure and cannot cross a compiler, tool or native-child effect. Its defined synchronous operations are metadata descriptor acquisition, checks, enumeration and close. It cannot be copied, replayed, reconstructed from ordinary data or used after original retirement. Original total, quiet, graph and retirement limits, material charges, output bounds and aggregate acceptance remain unchanged. There is no timer heartbeat, stat-only approval cache, skipped census or broader filesystem grant. ## Acceptance criteria - [ ] Publish the exact traversal/observation and retirement contract before implementation. - [ ] Exercise genuine held-FD trees and prove complete entry, extent and metadata results match the existing census, including sparse files and complete fresh views after an exhausted original cursor. - [ ] Deny foreign owners and independently opened or duplicated same-inode/OFD substitutes; refuse retired, copied or replayed traversal contexts. - [ ] Use deterministic actual mutations at parent/child boundaries: replacement, rename and restoration, symlink substitution, changed metadata and altered namespace. Preserve the first failure after restoration; do not accept a partial or smaller inventory. - [ ] Preserve capture-before-open duties, partial-open and unknown-close state, and test actual FD-number reuse without closing or accepting the replacement. - [ ] Preserve every source-body check and original budget/end; prove no native effect can use an active traversal borrow. - [ ] Compare actual same-boundary call counts and timings on equivalent fresh FD owners. Demonstrate a substantive reduction in redundant ancestry work, without attributing profiler overhead to production. - [ ] Pass focused FD/lifecycle controls, complete source-helper declarations, original native-owner byte checks, syntax checks and documentation. - [ ] Run a fresh uninstrumented original-budget producer/graph control after the implementation is frozen. Keep full image and authenticated guest acceptance separately reported. ## Alternatives and tradeoffs Dropping source checks, caching successful inventories and widening deadlines are excluded. Closed-history-only changes are not yet supported as the dominant traversal fix by the repeat-FD profile. Removing genuinely unused compiler scratch may complement this work only after proving that no actual product, source or metadata receiver still needs it. The bounded experiment must decide whether a combined capture/enumeration step can preserve every original observation with fewer repeated ancestor walks. If it cannot, retain the current checks and evaluate another explicit design rather than weakening admission. ## Tracking and rollout Related performance bug: #611. Current completed-work accounting prerequisite: #612. Implement on a separate feature branch based on that coherent source, refresh the whole actual helper cohort and retain the existing original owner bodies. Preserve previous failures and distinguish diagnostic, source, native graph, image and guest evidence. No complete native/image/guest acceptance is claimed by this design.
erikinkinen added this to the Alpha 1 milestone 2026-11-08 17:47:30 +01:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
erix/integration#614
No description provided.