[BUG] RTC preparation fails at staged commit during VM startup #122

Open
opened 2026-10-14 16:02:51 +02:00 by erikinkinen · 4 comments
Owner

Summary

VM scenarios stop during the shared RTC preparation path before reaching their
intended assertions. Deviced receives an internal error for staged materialization
commit, and startup aborts.

Reproduction

The original CI 2615
at 9d76db96c4ec17452352377f401060287417d0e2 reproduces this in the complete
VM catalog. Its workflow selects scenarios with scripts/list-ci-scenarios.py
and runs scripts/run-ci-vm-scenarios.sh out/ci-scenarios.txt after the configured
build and image checks. At 13:58 UTC on 14 October, the first 13 of 525 scenarios
had failed; the workflow was still running.

Expected and actual behavior

RTC preparation should finish through the existing staged-process protocol so
startup and each scenario can continue. Instead, the serial log records:

ERIX_DEVICED:RTC_LOADER_RESULT=0000000000000005
ERIX_DEVICED:RTC_LOADER_FAILURE=0000000000000014
ERIX_DEVICED:START_FAIL:DRV_RTC_CMOS
ERIX_KERNEL:ROOTD_WARN:FAIL_LAST_STAGE=0000000000000151

Result 5 is IPC_RESULT_INTERNAL; failure 20 is
IPC_LOADER_FAILURE_MATERIALIZATION_STAGED_COMMIT. QEMU exits 41 where the
scenario expects 33, or the test cannot observe the later terminal-ready marker.
Two defects caused the failure. Procd supplied the child's ordinary IPC endpoint
where initial TLS requires its held Process control endpoint. Executable links
also placed generated metadata on pages shared with code or writable data,
giving those pages conflicting permissions. Native TLS initialization correctly
rejected the mappings. The route repair
and metadata layout repair
preserve the existing grant, generation and native permission checks.

Environment and identity

The original x86-64 Docker job runs on Linux 6.14.0-1-pve. A retained development
image has profile identity
579fa1d15c7e1181466a8e5d7638d590c51ebc2c396d15dddf8eb9af4fbb5a4a.
The commit's component catalog and Cargo manifests select the complete source
graph. No guest authority or resource limits were changed for this observation.
A last working revision for this graph has not been established.

Evidence and prior checks

The linked workflow passes its host, source-catalog and build gates before the
VM failures. This is separate from the repaired catalog mismatch.
The original serial output and incomplete workflow remain retained; no successful
rerun is being substituted for the failure.

Tracking

Procd PR #2,
manual clarification,
Integration PR #12,
phase completion.

  • Identify and repair the staged-commit failure without weakening receipt, grant or cleanup checks.
  • Add a regression at the responsible boundary and reproduce successful RTC startup in a VM.
  • Complete the existing VM catalog and subsequent image/interactive gates.
## Summary VM scenarios stop during the shared RTC preparation path before reaching their intended assertions. Deviced receives an internal error for staged materialization commit, and startup aborts. ## Reproduction The original [CI 2615](https://git.erikinkinen.fi/erix/integration/actions/runs/2615) at `9d76db96c4ec17452352377f401060287417d0e2` reproduces this in the complete VM catalog. Its workflow selects scenarios with `scripts/list-ci-scenarios.py` and runs `scripts/run-ci-vm-scenarios.sh out/ci-scenarios.txt` after the configured build and image checks. At 13:58 UTC on 14 October, the first 13 of 525 scenarios had failed; the workflow was still running. ## Expected and actual behavior RTC preparation should finish through the existing staged-process protocol so startup and each scenario can continue. Instead, the serial log records: ```text ERIX_DEVICED:RTC_LOADER_RESULT=0000000000000005 ERIX_DEVICED:RTC_LOADER_FAILURE=0000000000000014 ERIX_DEVICED:START_FAIL:DRV_RTC_CMOS ERIX_KERNEL:ROOTD_WARN:FAIL_LAST_STAGE=0000000000000151 ``` Result 5 is `IPC_RESULT_INTERNAL`; failure 20 is `IPC_LOADER_FAILURE_MATERIALIZATION_STAGED_COMMIT`. QEMU exits 41 where the scenario expects 33, or the test cannot observe the later terminal-ready marker. Two defects caused the failure. Procd supplied the child's ordinary IPC endpoint where initial TLS requires its held Process control endpoint. Executable links also placed generated metadata on pages shared with code or writable data, giving those pages conflicting permissions. Native TLS initialization correctly rejected the mappings. The [route repair](https://git.erikinkinen.fi/erix/procd/commit/fcf6bf042b07e9cbd27115710dd2f6d1684394f5) and [metadata layout repair](https://git.erikinkinen.fi/erix/integration/commit/10d91c79e61f9bfeb243a3e54b93d1e60ea1073c) preserve the existing grant, generation and native permission checks. ## Environment and identity The original x86-64 Docker job runs on Linux 6.14.0-1-pve. A retained development image has profile identity `579fa1d15c7e1181466a8e5d7638d590c51ebc2c396d15dddf8eb9af4fbb5a4a`. The commit's component catalog and Cargo manifests select the complete source graph. No guest authority or resource limits were changed for this observation. A last working revision for this graph has not been established. ## Evidence and prior checks The linked workflow passes its host, source-catalog and build gates before the VM failures. This is separate from the repaired [catalog mismatch](https://git.erikinkinen.fi/erix/integration/issues/121). The original serial output and incomplete workflow remain retained; no successful rerun is being substituted for the failure. ## Tracking [Procd PR #2](https://git.erikinkinen.fi/erix/procd/pulls/2), [manual clarification](https://git.erikinkinen.fi/erix/docs/commit/f02ed28ccbd4b1280a335d5493f97a3f87eea048), [Integration PR #12](https://git.erikinkinen.fi/erix/integration/pulls/12), [phase completion](https://git.erikinkinen.fi/erix/integration/issues/65). - [x] Identify and repair the staged-commit failure without weakening receipt, grant or cleanup checks. - [x] Add a regression at the responsible boundary and reproduce successful RTC startup in a VM. - [ ] Complete the existing VM catalog and subsequent image/interactive gates.
Author
Owner

Procd fcf6bf04 fixes the initial TLS control route. The regression fails on the old route and passes on the repair; the component's CI 489 also passes.

The same VM scenario with the repaired Procd still returns RTC result 5/failure 20. Its other source selections and limits are unchanged. The issue remains open while the failing native boundary is traced. Integration b31ac269 selects the repair in both source catalogs.

[Procd fcf6bf04](https://git.erikinkinen.fi/erix/procd/commit/fcf6bf042b07e9cbd27115710dd2f6d1684394f5) fixes the initial TLS control route. The regression fails on the old route and passes on the repair; the component's [CI 489](https://git.erikinkinen.fi/erix/procd/actions/runs/489) also passes. The same VM scenario with the repaired Procd still returns RTC result 5/failure 20. Its other source selections and limits are unchanged. The issue remains open while the failing native boundary is traced. [Integration b31ac269](https://git.erikinkinen.fi/erix/integration/commit/b31ac269ec7cd7171b50c8b8c89a7e8b228b753f) selects the repair in both source catalogs.
Author
Owner

Cancelled the superseded CI 2615 and CI 2617 after retaining their available logs. Both selected the old Procd route; leaving them queued would delay the current revision. Their failures remain recorded, and cancellation counts as no acceptance. The original workflow for b31ac269 is preserved.

Cancelled the superseded [CI 2615](https://git.erikinkinen.fi/erix/integration/actions/runs/2615) and [CI 2617](https://git.erikinkinen.fi/erix/integration/actions/runs/2617) after retaining their available logs. Both selected the old Procd route; leaving them queued would delay the current revision. Their failures remain recorded, and cancellation counts as no acceptance. The original workflow for b31ac269 is preserved.
Author
Owner

10d91c79 separates linker-generated metadata into pages with consistent permissions and rejects conflicting layouts before packaging. The native linker regression reproduces both old conflicts; cache tests cover layout changes in both key paths.

With the published Process-route fix, the original lookup-manifest VM now passes its existing markers and exit check under the same 120-second, single-CPU guest limit. All 65 final executable and shared-object layouts have compatible page permissions. The original failures remain retained. The complete VM catalog and later image/interactive checks are still pending.

[10d91c79](https://git.erikinkinen.fi/erix/integration/commit/10d91c79e61f9bfeb243a3e54b93d1e60ea1073c) separates linker-generated metadata into pages with consistent permissions and rejects conflicting layouts before packaging. The native linker regression reproduces both old conflicts; cache tests cover layout changes in both key paths. With the published Process-route fix, the original lookup-manifest VM now passes its existing markers and exit check under the same 120-second, single-CPU guest limit. All 65 final executable and shared-object layouts have compatible page permissions. The original failures remain retained. The complete VM catalog and later image/interactive checks are still pending.
Author
Owner

b0cea358 corrects the syntax-order test expectation for the added page-permission test. Both the complete command-order check and the first-failure check pass. The original CI failures are retained; the current workflow is running with Rust and Markdown already passing.

[b0cea358](https://git.erikinkinen.fi/erix/integration/commit/b0cea358d53451a9991154e094b702b13afd5273) corrects the syntax-order test expectation for the added page-permission test. Both the complete command-order check and the first-failure check pass. The original CI failures are retained; the current workflow is running with Rust and Markdown already passing.
erikinkinen added this to the Alpha 1 milestone 2026-10-30 09:34:16 +01:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
erix/integration#122
No description provided.