Repository navigation
Don't check collected streams for kernel exceptions - #741
Merged
Merged
Conversation
Memory freed together with the stream that last used it still names that stream as its owner, so reusing the memory waits for the collected stream. #739 made the waits treat a destroyed command list as done, but take_error! still synchronized the list directly to check for kernel exceptions, which segfaults in the driver on the destroyed handle. #739 was tested on a base without #731, which introduced that call. A collected stream has nothing left to report, and its report slot was released along with it, possibly to another stream, so return early.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #741 +/- ##
==========================================
+ Coverage 86.03% 86.37% +0.34%
==========================================
Files 55 55
Lines 5077 4984 -93
==========================================
- Hits 4368 4305 -63
+ Misses 709 679 -30 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#739 stopped waits from synchronizing destroyed command lists, but
take_error!still synchronizes the stream's immediate list directly to check for kernel exceptions. When memory whose owning stream was collected gets reused,draingets past the guarded waits and then segfaults there:This is what still fails the
arraytests on Aurora on main (pipelines 71762 and 71764). #739 was tested on a base without #731, which introduced that call, so its CI didn't see it. #739's regression test reproduces it on an Arc A770 every time (3/3 runs on main).A collected stream has nothing left to report, and its report slot was released along with it, possibly to another stream, so
take_error!now returns early for it. With that, thearraytests pass 3/3 on the A770 and so does the full suite.This also unblocks #740, whose ALCF failure is the same crash (from before #739).