Skip to content

tests, task: fix load-dependent test failures - #4674

Open
grandixximo wants to merge 13 commits into
LinuxCNC:masterfrom
grandixximo:flaky-tests-4669
Open

grandixximo wants to merge 13 commits into
LinuxCNC:masterfrom
grandixximo:flaky-tests-4669

Conversation

@grandixximo

@grandixximo grandixximo commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Part of #4669. @hdiethelm, this covers the failures you listed. I reproduced them by pinning the test and a few busy loops to one CPU (taskset -c 7).

  • touchy, touchy-quit: the sim loaded the gladevcp test panel without -W, so touchy_postgui.hal could run before touchy_test existed and touchy exited. Now loadusr -Wn touchy_test.
  • touchy-fit: xwininfo -tree also lists unmapped windows, so window-fit measured touchy before it was mapped. It now polls for a visible window of stable size; an oversized window still fails.
  • motion-logger tests: the canon commands queued at startup reached motion only when the UI sent its first command late. On your question about the wait: INTERP_IDLE is already true while interp_list still holds them, so the race stays. The test UIs now wait for s.state == linuxcnc.RCS_DONE, and the expected logs include those commands. startup-gcode-abort waits for EXEC_WAITING_FOR_DELAY instead, since its startup code sits in a dwell.
  • hal-show: halcmd's hal_init() right after fork races the loaded module for the next uspace component id. A 200 ms delay there reproduces your 13 => 11 diff. The owner column is masked together with its padding (thanks for catching the 9 vs 10 width).
  • gmoccapy: when homing took over 15 s, the retry asked for STATE_ESTOP_RESET with the machine already on, and task never reports that state while traj is enabled. Forcing the first homing wait to 50 ms reproduces your log exactly.
  • multiclick, mux: a stream of depth N held only N-1 samples, so the 100 line input never fit before the thread started, and it was preloaded behind a fixed 0.5 s sleep. A late halstreamer left the streamer empty and shifted the end of the output. The tests now use depth 128 and wait for halstreamer with loadusr -w; the stream capacity itself is fixed in hal: a stream of depth N holds N samples #4678.
  • halmodule/stream: the reader printed "pass" after ready(), and halrun ends the test at ready, so a late reader was interrupted before printing. It prints first now.
  • gmoccapy homing: when homing outlasted the 15 s wait, the retry homed again. X homes past the switch it shares with Z, so the restarted Z began on an active switch and aborted. The test now waits while any joint is still homing.
  • gmoccapy sim: X and Z had one shared home switch, so a second home all always failed; each now has its own.
  • remap-io: the test polled status after setting the input pin, so under load the "pre" line already showed the new value. It polls first now.
  • stop-button-crazy-move: the waits allow 60 s instead of 10 s; each ends as soon as its condition holds.
  • linuxcncrsh: this one is in task, not the test. Motion reports a G38.2/G38.4 move that ended without contact one servo cycle after the planner goes idle, but task already counted motion DONE in that cycle. The MDI then finished without the error, and the error was returned for the next command. Task now waits for emcmotStatus.probing to clear before reporting motion DONE, which costs at most one servo cycle. With a program, the same window could let task issue the next line before the probe error. A 50 cycle delay before the verdict reproduces your diff, and it passes with the fix. @BsAtHome, since this is what your linuxcncrsh probe test was hitting, I'd like your view on fixing it in task.

Under load the affected tests went from failing to passing in every run (motion-logger 16/16, touchy tests 3/3 each). The full suite passes apart from blendmath and tool-frame, which don't build with GCC 14 because of EXPORT_SYMBOL (#4673).

touchy_postgui.hal links touchy_test pins, but the panel was loaded
without -W. Under load the pins did not exist yet and touchy exited.

Part of LinuxCNC#4669.
xwininfo -tree also lists unmapped windows, so under load window-fit
measured touchy at its creation size. Poll for a viewable window of
stable size.

Part of LinuxCNC#4669.
Whether the canon commands queued by interpreter init and the startup
code reach motion depended on how fast the UI sent its first command.
Wait for RCS_DONE (the dwell in startup-gcode-abort) and expect them.

Part of LinuxCNC#4669.
halcmd's hal_init() after fork races the loaded module for the next
uspace id, so the owner column changed under load. Mask it as the
mb2hal tests do.

Part of LinuxCNC#4669.
…ine on

ESTOP_RESET leaves traj enabled, so task keeps reporting ON and the
retry after a slow first homing attempt could never succeed.

Part of LinuxCNC#4669.
@hdiethelm

Copy link
Copy Markdown
Contributor

Thanks for taking this on! And a good idea to use taskset, so I don't have to reboot my VM between compile and testruns.

Would it be worth creating a scripts/runtests_flaky that basically runs taskset -c 0 scripts/runtests -p -j 32 for a few 100 rounds to check for check flaky tests? Probably not for CI but just to run once in a while and might be before a release.

I was not shure if INTERP_IDLE is the right state... I don't know these parts of linuxcnc well (yet).

I pulled your commits in and will run paralell tests for a few hours. Until now, it looks good! Exept one test that went under in the noise, see: #4669 (comment)

@hdiethelm

Copy link
Copy Markdown
Contributor

Caught a new one:

--- expected	2026-10-12 19:59:49.609852828 +0200
+++ result	2026-10-12 21:17:22.596376249 +0200
@@ -1,6 +1,6 @@
 Component Pins:
 Owner   Type  Dir                 Value  Name
-    (ignored comp id)  bool  IN                  FALSE  conv-bool-uint.0.in <== net-conv-bool-uint.0.in
+     (ignored comp id)  bool  IN                  FALSE  conv-bool-uint.0.in <== net-conv-bool-uint.0.in
     (ignored comp id)  real  IN                      0  conv-real-sint.0.in <== net-conv-real-sint.0.in
     (ignored comp id)  sint  IN                      0  conv-sint-real.0.in <== net-conv-sint-real.0.in
     (ignored comp id)  sint  IN                      0  conv-sint-uint.0.in <== net-conv-sint-uint.0.in
@@ -36,7 +36,7 @@
 
 Component Pins:
 Owner   Type  Dir                 Value  Name
-    (ignored comp id)  bool  IN                   TRUE  conv-bool-uint.0.in <== net-conv-bool-uint.0.in
+     (ignored comp id)  bool  IN                   TRUE  conv-bool-uint.0.in <== net-conv-bool-uint.0.in
     (ignored comp id)  real  IN           2.147484e+09  conv-real-sint.0.in <== net-conv-real-sint.0.in
     (ignored comp id)  sint  IN            -2147483648  conv-sint-real.0.in <== net-conv-sint-real.0.in
     (ignored comp id)  sint  IN   -9223372036854775808  conv-sint-uint.0.in <== net-conv-sint-uint.0.in
*** /home/hannes/linuxcnc-src/tests/hal-show: FAIL: checkresult exited with 1

Probably an when the ID changes from 10 to 9.
echo " 10 bool IN FALSE conv-bool-uint.0.in <== net-conv-bool-uint.0.in" | sed --regexp-extended --expression='s/^( *)[0-9]+( +[a-z0-9]+ +(IN|OUT|I\/O) .*)$/\1(ignored comp id)\2/'
vs
echo " 9 bool IN FALSE conv-bool-uint.0.in <== net-conv-bool-uint.0.in" | sed --regexp-extended --expression='s/^( *)[0-9]+( +[a-z0-9]+ +(IN|OUT|I\/O) .*)$/\1(ignored comp id)\2/'

sed --regexp-extended --expression='s/^( *)[0-9]+( +[a-z0-9]+ +(IN|OUT|I\/O) .*)$/(ignored comp id)\2/ not adding the spaces before the (ignored comp id) and updating the expected should fix it. I have to admit, chatgpt was of help, regexp are not compatible with my brain.

The owner column is right-aligned, so an id below 10 gets one more
leading space and still failed the diff. Reported by Hannes Diethelm.

Part of LinuxCNC#4669.
Motion flags a G38.2/G38.4 move that ended without contact one servo
cycle after the planner goes idle. Task could read that cycle as DONE,
finish the MDI command and miss the error, which linuxcncrsh then
returned for the next command. Wait for emcmotStatus.probing to clear.

Part of LinuxCNC#4669.
A 100 deep fifo holds 99 samples, and the 100 line input was preloaded
with a fixed 0.5s sleep, so a late halstreamer left the streamer empty
and shifted the end of the output. Use deeper fifos and loadusr -w.

Part of LinuxCNC#4669.
@grandixximo grandixximo changed the title tests: fix five load-dependent test failures tests, task: fix load-dependent test failures Oct 13, 2026
@grandixximo

Copy link
Copy Markdown
Contributor Author

Pushed three more commits: the hal-show padding fix, multiclick/mux, and a task fix for linuxcncrsh (see description).

Comment thread tests/multiclick/test.hal
Comment on lines +5 to +6
loadrt sampler depth=128 cfg=bbbbbbbbbsbbbbbbbbbs
loadrt streamer depth=128 cfg=bb

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is essentially a bug in the streamer/sampler interface in HAL.

When a buffer is requested of size N, then you should be able to put N elements in it. The bug causes only N-1 elements to be available.
The hal_stream* code should be fixed to allocate a buffer of sufficient size to encapsulate the requested size. But the reported max size should also just be N, regardless whether the allocation is one larger. Therefore, this is not the most trivial of fixes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I fixed it in hal_lib: a stream of depth N now allocates N+1 slots, holds N samples and still reports N as maxdepth. histobinstream and latencybinstream had a +1 to work around it, which is gone now. The multiclick and mux streamers are back to depth 100, and with loadusr -w halstreamer the test would hang if a 100 deep stream still held only 99. tests/halmodule/stream asserted the old capacity, so it now fills 10 of 10.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seen the fix. That should do it.

This is such a specific fix that I think it needs to be in its own PR. It isn't a "test-under-load fix", but a bug in HAL stream code.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Split out into #4678. This PR is back to streamer depth 128 for multiclick and mux, so it doesn't depend on that fix.

@BsAtHome

Copy link
Copy Markdown
Contributor

The linuxcncrsh issue, well, it was a probe error that I never was able to find and bothered only when I hit it ;-)

Now that, under load, it has surfaced to be in task, that would be the right place to fix it.

halrun ends the test once the reader is ready, so a reader descheduled
between ready() and print() was interrupted before it printed "pass".

Part of LinuxCNC#4669.
…gain

When homing outlasted the 15s attempt, home_all() issued home(-1) again.
In the gmoccapy sim X and Z share one home switch, and X homes to 10,
past its switch, so Z then started on an active switch, backed off
without clearing it and aborted ("j2 end of move in home state 5"),
on every further attempt. Keep waiting while any joint is homing.

Part of LinuxCNC#4669.
BsAtHome pushed a commit that referenced this pull request Oct 14, 2026
The ring keeps one slot free to tell full from empty, so a stream
created with depth N took only N-1 samples. Allocate N+1 slots and keep
reporting N as maxdepth. Drop the +1 that histobinstream and
latencybinstream added for it, and fill the halmodule stream test to
10. Found by Bertho Stultiens in #4674.
The gmoccapy sim joined the X and Z switches through or2. X homes to 10,
past its switch, so after a home all the shared input stayed active and
a second home all always failed: Z started on the switch, backed off
without clearing it and aborted ("j2 end of move in home state 5").
Under load any interrupted first home led the ui-smoke test into this.
Wire each switch to its own joint, as lib/hallib/simulated_home.hal does.

Part of LinuxCNC#4669.
The test set the input pin and then polled status for the "pre" line,
expecting the old value. Motion copies the pin within a servo cycle, so
under load the poll saw the new one. Poll first.

Part of LinuxCNC#4669.
Each wait ends as soon as its condition holds, so the limit only matters
on a slow machine. Under heavy parallel load the queue had reached 948
of the 1000 segments after 10s.

Part of LinuxCNC#4669.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants