Skip to content

Don't check collected streams for kernel exceptions - #741

Merged
maleadt merged 1 commit into
mainfrom
tb/collected-stream-error
Oct 24, 2026
Merged

maleadt merged 1 commit into
mainfrom
tb/collected-stream-error

Conversation

@maleadt

@maleadt maleadt commented Oct 24, 2026

Copy link
Copy Markdown
Member

#739 stopped waits from synchronizing destroyed command lists, but take_error! still synchronizes the stream's immediate list directly to check for kernel exceptions. When memory whose owning stream was collected gets reused, drain gets past the guarded waits and then segfaults there:

signal 11 (1): Segmentation fault
unchecked_zeCommandListHostSynchronize at lib/level-zero/libze.jl:2515
take_error! at src/context.jl:492
drain at src/memory.jl:89
synchronize_owner! at src/memory.jl:111
take_ownership! at src/memory.jl:123

This is what still fails the array tests on Aurora on main (pipelines 71762 and 71764). #739 was tested on a base without #731, which introduced that call, so its CI didn't see it. #739's regression test reproduces it on an Arc A770 every time (3/3 runs on main).

A collected stream has nothing left to report, and its report slot was released along with it, possibly to another stream, so take_error! now returns early for it. With that, the array tests pass 3/3 on the A770 and so does the full suite.

This also unblocks #740, whose ALCF failure is the same crash (from before #739).

Memory freed together with the stream that last used it still names that
stream as its owner, so reusing the memory waits for the collected stream.
#739 made the waits treat a destroyed command list as done, but
take_error! still synchronized the list directly to check for kernel
exceptions, which segfaults in the driver on the destroyed handle. #739 was
tested on a base without #731, which introduced that call.

A collected stream has nothing left to report, and its report slot was
released along with it, possibly to another stream, so return early.
@codecov

codecov Bot commented Oct 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.37%. Comparing base (c1dea4f) to head (e3ed929).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #741      +/-   ##
==========================================
+ Coverage   86.03%   86.37%   +0.34%     
==========================================
  Files          55       55              
  Lines        5077     4984      -93     
==========================================
- Hits         4368     4305      -63     
+ Misses        709      679      -30     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@maleadt
maleadt merged commit d90fdd8 into main Oct 24, 2026
5 checks passed
@maleadt
maleadt deleted the tb/collected-stream-error branch October 24, 2026 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant