[BUG] Descendant-cleanup host fixture errors when the observed process disappears #29

New issue

Open

opened 2026-09-12 18:14:02 +02:00 by erikinkinen · 0 comments

erikinkinen commented

2026-09-12 18:14:02 +02:00

Owner

Summary

The descendant-cleanup host fixture errors when the observed process disappears
while its /proc/<pid>/stat file is being read. This stops Integration CI before
the VM catalog even though an absent process is an intended successful outcome
of this assertion.

Reproduction

Observed once in push CI 1575
at Integration 96f1c21848d534683fc077779dd92d332e3e4779. The maintained workflow
runs:

python3 tests/immutable_source_resolution_test.py

The failing case is
MetadataProcessTests.test_returning_parent_cannot_leave_a_running_descendant.
It starts an owned host Python process whose forked child ignores TERM after a
readiness handshake; the immediate parent prints that child's PID and exits.
After run_owned returns, the test observes whether the descendant is absent or
a zombie. No isolated rerun or deterministic reproduction has been performed,
and no failure-frequency estimate is established.

Expected and actual behavior

Expected: the observation accepts an absent or zombie descendant, continues to
reject a running descendant, and propagates unrelated observation errors.

Actual: test line 705 opens /proc/<pid>/stat, then its read raises
ProcessLookupError: [Errno 3] No such process. The handler catches only
FileNotFoundError, so the suite reports Ran 36 tests in 22.673s and
FAILED (errors=1). The workflow step and main job fail. The exception comes
from the post-cleanup observation, not a raised failure from run_owned.

Linux documents that descriptors for departed processes may fail with ESRCH and
do not refer to a later process that reuses the numeric PID. This is a supported
process-disappearance outcome missing from this assertion's handling; it does
not prove a live descendant escaped cleanup or identify the complete timing of
the CI process. See the kernel proc filesystem documentation.

Environment and identity

Integration: 96f1c21848d534683fc077779dd92d332e3e4779; push run 1575,
API run 9059, integration job 19484, attempt 1; terminal failure.
The exact test and scripts/cargo_source_metadata.py bytes are unchanged from
Integration 381bccd87c5632e9d1af8aa9d0aca295cbc00e1f and
e2b1cf5995bcf6334839cda9a013e90fa97787d5. This does not identify a newly
introduced causal commit.
Linux container host fixture; traceback identifies Python 3.13 library paths.
No exact host kernel or interpreter executable identity is established by the
log. No guest image, firmware or VM was involved in this failure.
Production helper ownership remains the selected process group, with its
existing TERM/KILL and bounded immediate-child waits. The observer retains its
existing 100 iterations and 0.01-second sleeps; it sends no new signals.

Evidence and prior checks

The complete terminal job log and run/job metadata are retained. The exception
is reported at 2026-09-12 16:04:48.5103022 UTC; the workflow step returns 1 at
16:04:48.5319669 UTC. The 21-case console-watchdog group passed earlier in this
same job, so this is a distinct subsequent failure. Both Rust and Markdown jobs
pass; Rust records 320 default and 321 all-feature tests without warning
candidates in the retained logs.

PR CI 1576, at the
same exact head, passed this host group and was still running later component
builds at the observation. It is not terminal replacement acceptance. The older
push run 1573 success and PR run 1574 watchdog failure remain separate evidence.
No rerun, timeout change, cleanup-policy change or source edit was used to
classify this failure.

The test-only correction is signed and pushed as Integration 194f01c61f61de9b9fcd41589c14d556cb6d59e3 in PR #12. The observer catches only missing-process and process-lookup errors around the actual proc read, retaining Z-only parsed acceptance, the 100-iteration/0.01-second poll contract and propagation of unrelated errors. Seven deterministic controls cover open/read disappearance, stream closure, live-to-absent transition, live-state exhaustion, malformed text and EACCES/EIO; the real fork/readiness test and production cleanup remain unchanged.

The exact original observation loop fails a deterministic after-open ESRCH control. The corrected complete file passes all 43 tests in 19.803 seconds with warnings denied, owned exit zero and no cleanup failure. Its first broader local run failed because the chosen outer environment disabled Git replacement objects during an unchanged negative fixture setup; that failure is retained. A new environment removes only that outer setting, while production original-object checks remain intact. New current-head CI is pending. These tests preserve the original numeric-PID observation limits and do not establish complete process-generation cleanup.

Tracking

Related original-source preparation,
validation stream,
component audit, and
WIP PR 12.
The console-progress fixture failure
is distinct and its corrected group passes here.

Add a narrow test-only observation correction with deterministic before-open
absence and after-open ESRCH controls. Retain zombie acceptance, running-process
failure, unrelated-error propagation and the real descendant cleanup fixture.
Do not broaden the production retirement rules or treat every I/O error as
successful cleanup.

## Summary The descendant-cleanup host fixture errors when the observed process disappears while its `/proc/<pid>/stat` file is being read. This stops Integration CI before the VM catalog even though an absent process is an intended successful outcome of this assertion. ## Reproduction Observed once in [push CI 1575](https://git.erikinkinen.fi/erix/integration/actions/runs/1575) at Integration `96f1c21848d534683fc077779dd92d332e3e4779`. The maintained workflow runs: ```sh python3 tests/immutable_source_resolution_test.py ``` The failing case is `MetadataProcessTests.test_returning_parent_cannot_leave_a_running_descendant`. It starts an owned host Python process whose forked child ignores TERM after a readiness handshake; the immediate parent prints that child's PID and exits. After `run_owned` returns, the test observes whether the descendant is absent or a zombie. No isolated rerun or deterministic reproduction has been performed, and no failure-frequency estimate is established. ## Expected and actual behavior Expected: the observation accepts an absent or zombie descendant, continues to reject a running descendant, and propagates unrelated observation errors. Actual: test line 705 opens `/proc/<pid>/stat`, then its read raises `ProcessLookupError: [Errno 3] No such process`. The handler catches only `FileNotFoundError`, so the suite reports `Ran 36 tests in 22.673s` and `FAILED (errors=1)`. The workflow step and main job fail. The exception comes from the post-cleanup observation, not a raised failure from `run_owned`. Linux documents that descriptors for departed processes may fail with ESRCH and do not refer to a later process that reuses the numeric PID. This is a supported process-disappearance outcome missing from this assertion's handling; it does not prove a live descendant escaped cleanup or identify the complete timing of the CI process. See the [kernel proc filesystem documentation](https://docs.kernel.org/filesystems/proc.html). ## Environment and identity - Integration: `96f1c21848d534683fc077779dd92d332e3e4779`; push run 1575, API run 9059, integration job 19484, attempt 1; terminal failure. - The exact test and `scripts/cargo_source_metadata.py` bytes are unchanged from Integration `381bccd87c5632e9d1af8aa9d0aca295cbc00e1f` and `e2b1cf5995bcf6334839cda9a013e90fa97787d5`. This does not identify a newly introduced causal commit. - Linux container host fixture; traceback identifies Python 3.13 library paths. No exact host kernel or interpreter executable identity is established by the log. No guest image, firmware or VM was involved in this failure. - Production helper ownership remains the selected process group, with its existing TERM/KILL and bounded immediate-child waits. The observer retains its existing 100 iterations and 0.01-second sleeps; it sends no new signals. ## Evidence and prior checks The complete terminal job log and run/job metadata are retained. The exception is reported at 2026-09-12 16:04:48.5103022 UTC; the workflow step returns 1 at 16:04:48.5319669 UTC. The 21-case console-watchdog group passed earlier in this same job, so this is a distinct subsequent failure. Both Rust and Markdown jobs pass; Rust records 320 default and 321 all-feature tests without warning candidates in the retained logs. [PR CI 1576](https://git.erikinkinen.fi/erix/integration/actions/runs/1576), at the same exact head, passed this host group and was still running later component builds at the observation. It is not terminal replacement acceptance. The older push run 1573 success and PR run 1574 watchdog failure remain separate evidence. No rerun, timeout change, cleanup-policy change or source edit was used to classify this failure. The test-only correction is signed and pushed as Integration `194f01c61f61de9b9fcd41589c14d556cb6d59e3` in [PR #12](https://git.erikinkinen.fi/erix/integration/pulls/12). The observer catches only missing-process and process-lookup errors around the actual proc read, retaining Z-only parsed acceptance, the 100-iteration/0.01-second poll contract and propagation of unrelated errors. Seven deterministic controls cover open/read disappearance, stream closure, live-to-absent transition, live-state exhaustion, malformed text and EACCES/EIO; the real fork/readiness test and production cleanup remain unchanged. The exact original observation loop fails a deterministic after-open ESRCH control. The corrected complete file passes all 43 tests in 19.803 seconds with warnings denied, owned exit zero and no cleanup failure. Its first broader local run failed because the chosen outer environment disabled Git replacement objects during an unchanged negative fixture setup; that failure is retained. A new environment removes only that outer setting, while production original-object checks remain intact. New current-head CI is pending. These tests preserve the original numeric-PID observation limits and do not establish complete process-generation cleanup. ## Tracking Related [original-source preparation](https://git.erikinkinen.fi/erix/integration/issues/17), [validation stream](https://git.erikinkinen.fi/erix/integration/issues/5), [component audit](https://git.erikinkinen.fi/erix/integration/issues/11), and [WIP PR 12](https://git.erikinkinen.fi/erix/integration/pulls/12). The [console-progress fixture failure](https://git.erikinkinen.fi/erix/integration/issues/25) is distinct and its corrected group passes here. Add a narrow test-only observation correction with deterministic before-open absence and after-open ESRCH controls. Retain zombie acceptance, running-process failure, unrelated-error propagation and the real descendant cleanup fixture. Do not broaden the production retirement rules or treat every I/O error as successful cleanup.

erikinkinen added the

bug

phase-6

labels

2026-09-12 18:14:02 +02:00

~~erikinkinen referenced this issue 2026-09-12 18:32:06 +02:00~~

WIP: Coordinate native authority, shell workspace and source transport #12