[FEATURE] Persist original POSIX metadata with atomic FAT-family updates #8

Closed
opened 2026-09-30 22:29:03 +02:00 by erikinkinen · 0 comments
Owner

Problem and motivation

FAT/exFAT native observations intentionally lack original POSIX identity, owner,
mode, link and change-time evidence. Entry coordinates and clusters cannot replace
persistent object serials, and mount/status clocks cannot reconstruct historical
time. Complete status and permission mediation needs an explicit persistent source
joined atomically to the original filesystem mutations.

Proposed behavior and scope

Implement an administrator-selected FAT-family metadata extension inside the
original partition and outside its declared file-data volume. Selection is explicit
through the existing mount-control route. Existing imported media keeps its honest
unknown fields; ordinary file bytes never opt a mount into a new metadata policy.

A versioned checksummed immutable footer at the final original partition block
binds the original block width, FAT/exFAT volume length, metadata extent, journal
extent and nonzero filesystem serial. All extents must fit the same original
partition. The metadata extent begins at the end of the declared volume, remains
inside the journal target prefix, and ends before the disjoint journal. The footer
is outside both writable target and journal extents. Require exact geometry and
canonical reserved bytes before any recovery or filesystem repair.

The metadata catalog retains a nonwrapping object-serial allocator and original
root/object records. Records bind separately allocated serials to exact original
entry geometry and checked original entry evidence; geometry is a lookup key,
never the serial. Retain complete mode, original UID/GID, actual links and
independent 24-byte signed source/precision-qualified event observations. No
recipient identity, path, cluster, handle, mount time or newly queried status clock
can fill a missing original fact. Preserve imported local FAT time as unknown UTC
unless original qualified evidence exists. Birth is not a prerequisite for POSIX
status. The extension's initial policy is explicit mode/ownership with known ACL
absence; later ACL support must be explicit and cannot silently reinterpret media.

Use the implemented original-device undo adapter to join normal FAT/exFAT writes
and metadata changes. Reserve original mount state and reply capacity first, then
admit explicit extension geometry and recover before normal mount validation can
repair mirrors. Complete validation and metadata publication follow recovery.
Mount-level read-only selection independently prohibits recovery writes, even on a
physically writable device. Metadata and journal sectors are excluded from the
ordinary filesystem view. No automatic formatting or older-format fallback.

One provider operation owns one transaction across nested timestamped, exact-entry
and ordinary helpers. Preserve all pre-effect admission. Failed block operations
cannot be swallowed, and storage uncertainty fails stopped without an ordinary
reply. On known rollback, restore affected volatile object state without reviving
consumed navigation evidence, cleanup keys or monotonic identities. Reserve any
rollback storage through the existing deployment arena; do not impose a fixed
record ceiling. Original mount/handle retirement must clear new policy state.

Implement shared ordinary/acquired status and permission observations, actual
creation attribution, event updates, rename/replacement/deletion/reuse and alias
lifetime under this transaction policy. New format selection is not full feature
acceptance until these consumers and native workloads are integrated.

Authority, security and reliability

Source authority remains the existing administrator mount route and the original
retained block device. New fields are descriptive policy and geometry, never a
capability grant or an alternate endpoint selector. An on-media record cannot
address another device or an extent outside those already admitted. Scope,
read-only ceilings, one-use permission evidence and original cleanup remain in
force. Checksums establish corruption detection, not authentication against a
malicious storage owner. The explicit mount owner selects this persistent source.

Require complete prefix/recovery admission, reject corrupt or missing selected
metadata, and never substitute plausible defaults. A reopened owner must recover
uncertain original media before ordinary use. Copies of volatile state cannot
restore spent authority-bearing keys or evidence. Security, reliability and then
performance determine implementation choices. Derive capacity from actual media
and deployment storage, share codecs and avoid repeated whole-directory discovery.
No measured improvement is claimed without execution.

Acceptance criteria

Checklist
  • Define exact explicit format-selection and original extent admission with canonical media codecs, hostile-geometry controls and complete mount/producer/consumer compatibility.
  • Integrate journal recovery before mount repair, preserve read-only policy and original-key retirement, and author corruption/interrupted-recovery/native controls.
  • Persist original object allocation, exact media binding, root metadata, ownership/mode/links and independent event observations without invented history.
  • Join all supported filesystem and metadata mutations in one durable transaction; restore volatile state without reviving keys or evidence and fail stopped on uncertainty.
  • Integrate shared ordinary/acquired status, fresh permissions, authenticated creation, aliases and rename/replacement/deletion/reuse semantics with independent controls.
  • Author native mounted format/status/mutation/recovery workloads and retain existing authority and lifecycle gates; execution is waived by user, not represented as a pass.
  • Complete authority/rustdoc/size audits, component and manual documentation, coherent signed revisions, linked reviews and phase tracking before full acceptance.

Alternatives and tradeoffs

Plain FAT/exFAT cannot supply the missing original fields. Import defaults and
clock queries at mount/status time would invent evidence. An ordinary sidecar file
would expose policy through file access and separate publication lifetimes. The
explicit original-partition extent avoids both. A generic journal alone cannot
restore provider caches or establish metadata provenance; its consumer integration
is part of this feature. Full record layout and indexed lookup are refined with
provider invariants before their source changes; no alternate authority or fixed
catalog ceiling is introduced for convenience.

Tracking and rollout

Implemented and closed under source acceptance; test and CI execution remain waived.

Complete original metadata and status/directory source acceptance — 1 October 2026

Source acceptance covers explicit format admission/recovery, persistent original identities and full metadata, shared ordinary/acquired observations, fresh permissions, atomic filesystem/catalog updates, alias and cleanup lifetime, compatibility stat families and shared seekable directory descriptions. Imported unavailable facts remain explicit refusals.

Runtime and independent fixture producers emit the mandatory stream allocation flag, which admission and image verification require.

This checkpoint authors 14 Rust and four Python controls plus two selected exFAT VM scenarios; two existing FAT scenarios now also require six-child directory growth, post-growth observations and complete cleanup.

Both complete EriX guest generations and external Rust/LLVM guest rebuilding remain required and unperformed.

Related records and contracts
Repository Links
docs PR 4
fatd issue 10, issue 8, issue 9, PR 3, exfat-directory-mutation.md, original-catalog.md
integration issue 65, PR 12
posixd issue 25, PR 5
rootd PR 2
## Problem and motivation FAT/exFAT native observations intentionally lack original POSIX identity, owner, mode, link and change-time evidence. Entry coordinates and clusters cannot replace persistent object serials, and mount/status clocks cannot reconstruct historical time. Complete status and permission mediation needs an explicit persistent source joined atomically to the original filesystem mutations. ## Proposed behavior and scope Implement an administrator-selected FAT-family metadata extension inside the original partition and outside its declared file-data volume. Selection is explicit through the existing mount-control route. Existing imported media keeps its honest unknown fields; ordinary file bytes never opt a mount into a new metadata policy. A versioned checksummed immutable footer at the final original partition block binds the original block width, FAT/exFAT volume length, metadata extent, journal extent and nonzero filesystem serial. All extents must fit the same original partition. The metadata extent begins at the end of the declared volume, remains inside the journal target prefix, and ends before the disjoint journal. The footer is outside both writable target and journal extents. Require exact geometry and canonical reserved bytes before any recovery or filesystem repair. The metadata catalog retains a nonwrapping object-serial allocator and original root/object records. Records bind separately allocated serials to exact original entry geometry and checked original entry evidence; geometry is a lookup key, never the serial. Retain complete mode, original UID/GID, actual links and independent 24-byte signed source/precision-qualified event observations. No recipient identity, path, cluster, handle, mount time or newly queried status clock can fill a missing original fact. Preserve imported local FAT time as unknown UTC unless original qualified evidence exists. Birth is not a prerequisite for POSIX status. The extension's initial policy is explicit mode/ownership with known ACL absence; later ACL support must be explicit and cannot silently reinterpret media. Use the implemented original-device undo adapter to join normal FAT/exFAT writes and metadata changes. Reserve original mount state and reply capacity first, then admit explicit extension geometry and recover before normal mount validation can repair mirrors. Complete validation and metadata publication follow recovery. Mount-level read-only selection independently prohibits recovery writes, even on a physically writable device. Metadata and journal sectors are excluded from the ordinary filesystem view. No automatic formatting or older-format fallback. One provider operation owns one transaction across nested timestamped, exact-entry and ordinary helpers. Preserve all pre-effect admission. Failed block operations cannot be swallowed, and storage uncertainty fails stopped without an ordinary reply. On known rollback, restore affected volatile object state without reviving consumed navigation evidence, cleanup keys or monotonic identities. Reserve any rollback storage through the existing deployment arena; do not impose a fixed record ceiling. Original mount/handle retirement must clear new policy state. Implement shared ordinary/acquired status and permission observations, actual creation attribution, event updates, rename/replacement/deletion/reuse and alias lifetime under this transaction policy. New format selection is not full feature acceptance until these consumers and native workloads are integrated. ## Authority, security and reliability Source authority remains the existing administrator mount route and the original retained block device. New fields are descriptive policy and geometry, never a capability grant or an alternate endpoint selector. An on-media record cannot address another device or an extent outside those already admitted. Scope, read-only ceilings, one-use permission evidence and original cleanup remain in force. Checksums establish corruption detection, not authentication against a malicious storage owner. The explicit mount owner selects this persistent source. Require complete prefix/recovery admission, reject corrupt or missing selected metadata, and never substitute plausible defaults. A reopened owner must recover uncertain original media before ordinary use. Copies of volatile state cannot restore spent authority-bearing keys or evidence. Security, reliability and then performance determine implementation choices. Derive capacity from actual media and deployment storage, share codecs and avoid repeated whole-directory discovery. No measured improvement is claimed without execution. ## Acceptance criteria <details> <summary>Checklist</summary> - [x] Define exact explicit format-selection and original extent admission with canonical media codecs, hostile-geometry controls and complete mount/producer/consumer compatibility. - [x] Integrate journal recovery before mount repair, preserve read-only policy and original-key retirement, and author corruption/interrupted-recovery/native controls. - [x] Persist original object allocation, exact media binding, root metadata, ownership/mode/links and independent event observations without invented history. - [x] Join all supported filesystem and metadata mutations in one durable transaction; restore volatile state without reviving keys or evidence and fail stopped on uncertainty. - [x] Integrate shared ordinary/acquired status, fresh permissions, authenticated creation, aliases and rename/replacement/deletion/reuse semantics with independent controls. - [x] Author native mounted format/status/mutation/recovery workloads and retain existing authority and lifecycle gates; execution is waived by user, not represented as a pass. - [x] Complete authority/rustdoc/size audits, component and manual documentation, coherent signed revisions, linked reviews and phase tracking before full acceptance. </details> ## Alternatives and tradeoffs Plain FAT/exFAT cannot supply the missing original fields. Import defaults and clock queries at mount/status time would invent evidence. An ordinary sidecar file would expose policy through file access and separate publication lifetimes. The explicit original-partition extent avoids both. A generic journal alone cannot restore provider caches or establish metadata provenance; its consumer integration is part of this feature. Full record layout and indexed lookup are refined with provider invariants before their source changes; no alternate authority or fixed catalog ceiling is introduced for convenience. ## Tracking and rollout Implemented and closed under source acceptance; test and CI execution remain waived. ### Complete original metadata and status/directory source acceptance — 1 October 2026 Source acceptance covers explicit format admission/recovery, persistent original identities and full metadata, shared ordinary/acquired observations, fresh permissions, atomic filesystem/catalog updates, alias and cleanup lifetime, compatibility stat families and shared seekable directory descriptions. Imported unavailable facts remain explicit refusals. Runtime and independent fixture producers emit the mandatory stream allocation flag, which admission and image verification require. This checkpoint authors 14 Rust and four Python controls plus two selected exFAT VM scenarios; two existing FAT scenarios now also require six-child directory growth, post-growth observations and complete cleanup. Both complete EriX guest generations and external Rust/LLVM guest rebuilding remain required and unperformed. <details> <summary>Related records and contracts</summary> | Repository | Links | | --- | --- | | docs | [PR 4](https://git.erikinkinen.fi/erix/docs/pulls/4) | | fatd | [issue 10](https://git.erikinkinen.fi/erix/fatd/issues/10), [issue 8](https://git.erikinkinen.fi/erix/fatd/issues/8), [issue 9](https://git.erikinkinen.fi/erix/fatd/issues/9), [PR 3](https://git.erikinkinen.fi/erix/fatd/pulls/3), [exfat-directory-mutation.md](https://git.erikinkinen.fi/erix/fatd/src/branch/feature/posix-compat/docs/exfat-directory-mutation.md), [original-catalog.md](https://git.erikinkinen.fi/erix/fatd/src/branch/feature/posix-compat/docs/original-catalog.md) | | integration | [issue 65](https://git.erikinkinen.fi/erix/integration/issues/65), [PR 12](https://git.erikinkinen.fi/erix/integration/pulls/12) | | posixd | [issue 25](https://git.erikinkinen.fi/erix/posixd/issues/25), [PR 5](https://git.erikinkinen.fi/erix/posixd/pulls/5) | | rootd | [PR 2](https://git.erikinkinen.fi/erix/rootd/pulls/2) | </details>
erikinkinen added this to the Alpha 1 milestone 2026-10-30 09:33:59 +01:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
erix/fatd#8
No description provided.