[BUG] Initial shell extop scenario stalls after CPU rows #141

Open
opened 2026-10-16 15:07:41 +02:00 by erikinkinen · 2 comments
Owner

Summary

The initial Exsh extop scenario stops making serial progress after displaying its CPU rows. The existing 45-second watchdog terminates QEMU. The cause is unknown.

Reproduction

CI 2732, VM job 47544 attempt 1, worker 7:

bash scripts/run-scenario.sh tests/scenarios/appliance-disk-image-initial-exsh-extop-positive.toml components.toml

The log records the stall at 12:38:20 UTC on 16 October 2026. The same scenario also fails in the earlier CI 2720 matrix.

Expected and actual behavior

Extop should display CPU, memory and process information, then the scripted shell should exit with the required success markers.

Instead, the serial tail shows the extop header and both CPU rows as --.--% online, followed by initial-shell failure telemetry. After 45 seconds without serial changes, the runner terminates QEMU and reports image-profile preflight failure. Nearby delegated-fault diagnostics do not establish the cause.

Environment and identity

Integration 0e93075a477efa06a39007c206ebf5bc8fd023be, push CI 2732/API 22574, VM job 47544 attempt 1, task 34524. Release image, development runtime policy, two virtual CPUs and the original 120-second QEMU ceiling. No introducing revision is known.

Evidence and prior checks

The retained log prefix includes the scenario header, serial tail and watchdog. The job was still running when it was read; its final matrix outcome and inner VM artifacts remain pending. All 1,670 preceding Python tests pass.

The first failure-context call records endpoint 166, operation 1678, received length 120 and r=0 f=1 n=80; the last records endpoint 2636, operation 1668, received length 40 and r=0 f=0 n=0. These are observations, not a cause diagnosis.

Tracking

Implementation PR, regression acceptance, phase completion.

  • Retain the complete original job outcome and recover the inner VM evidence.
  • Fix the responsible behavior with focused regression coverage.
  • Pass the original extop scenario with its authority and deadline checks.
## Summary The initial Exsh extop scenario stops making serial progress after displaying its CPU rows. The existing 45-second watchdog terminates QEMU. The cause is unknown. ## Reproduction [CI 2732](https://git.erikinkinen.fi/erix/integration/actions/runs/2732), VM job 47544 attempt 1, worker 7: ```sh bash scripts/run-scenario.sh tests/scenarios/appliance-disk-image-initial-exsh-extop-positive.toml components.toml ``` The log records the stall at 12:38:20 UTC on 16 October 2026. The same scenario also fails in the earlier CI 2720 matrix. ## Expected and actual behavior Extop should display CPU, memory and process information, then the scripted shell should exit with the required success markers. Instead, the serial tail shows the extop header and both CPU rows as `--.--% online`, followed by initial-shell failure telemetry. After 45 seconds without serial changes, the runner terminates QEMU and reports image-profile preflight failure. Nearby delegated-fault diagnostics do not establish the cause. ## Environment and identity Integration `0e93075a477efa06a39007c206ebf5bc8fd023be`, push CI 2732/API 22574, VM job 47544 attempt 1, task 34524. Release image, development runtime policy, two virtual CPUs and the original 120-second QEMU ceiling. No introducing revision is known. ## Evidence and prior checks The retained log prefix includes the scenario header, serial tail and watchdog. The job was still running when it was read; its final matrix outcome and inner VM artifacts remain pending. All 1,670 preceding Python tests pass. The first failure-context call records endpoint 166, operation 1678, received length 120 and `r=0 f=1 n=80`; the last records endpoint 2636, operation 1668, received length 40 and `r=0 f=0 n=0`. These are observations, not a cause diagnosis. ## Tracking [Implementation PR](https://git.erikinkinen.fi/erix/integration/pulls/12), [regression acceptance](https://git.erikinkinen.fi/erix/integration/issues/6), [phase completion](https://git.erikinkinen.fi/erix/integration/issues/65). - [ ] Retain the complete original job outcome and recover the inner VM evidence. - [ ] Fix the responsible behavior with focused regression coverage. - [ ] Pass the original extop scenario with its authority and deadline checks.
Author
Owner

CI run 2822, Integration f06bf722, reproduces the stall after the extop heading and both CPU rows. The 45-second watchdog fires at 15:21:47 UTC on 17 October.

The retained guest output also records FAULT_NO_PAGER and FAULT_KILLED for TCB 0x23: RIP 0x68005c02, RSP 0x68225d10, fault address 0x68225d08, error 0x6. These registers give us a concrete fault to map against the exact guest binaries.

[CI run 2822](https://git.erikinkinen.fi/erix/integration/actions/runs/2822), Integration `f06bf722`, reproduces the stall after the extop heading and both CPU rows. The 45-second watchdog fires at 15:21:47 UTC on 17 October. The retained guest output also records `FAULT_NO_PAGER` and `FAULT_KILLED` for TCB `0x23`: RIP `0x68005c02`, RSP `0x68225d10`, fault address `0x68225d08`, error `0x6`. These registers give us a concrete fault to map against the exact guest binaries.
Author
Owner

CI 2846 also reaches the 45-second serial stall in the initial Exsh extop scenario, at 01:48:57 UTC on October 18. The retained serial output ends with the initial-shell failure-context record.

The 525-case workflow is still running. This confirms the symptom recurs; the cause and a complete passing run remain open.

[CI 2846](https://git.erikinkinen.fi/erix/integration/actions/runs/2846/jobs/1) also reaches the 45-second serial stall in the initial Exsh extop scenario, at 01:48:57 UTC on October 18. The retained serial output ends with the initial-shell failure-context record. The 525-case workflow is still running. This confirms the symptom recurs; the cause and a complete passing run remain open.
erikinkinen added this to the Alpha 1 milestone 2026-10-30 09:34:18 +01:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
erix/integration#141
No description provided.