Repository navigation
[SC26 SCC] NTU Team Supernova's GPU optimizations to MFC - #2093
Draft
aliencaocao wants to merge 2 commits into
Draft
aliencaocao wants to merge 2 commits into
aliencaocao wants to merge 2 commits into
Conversation
|
Claude Code Review Head SHA: 0221014 Files changed:
Findings:
Checked and found consistent:
Coverage note: about 35k diff lines were not reviewed hunk-by-hunk. Only |
GPU performance optimizations for the simulation solver, mostly for the NVIDIA OpenACC path, together with fixes to existing numerical bugs. Most optimizations are opt-in via compiler arguments; the remainder are always-on (guarded by the relevant compiler/GPU where necessary) because guarding them would duplicate too much code. The work is directed by three findings: with case optimization MFC is largely not memory-capacity constrained; on modern GPUs the solver is memory-bandwidth bound through excessive device round-trips; and communication is a bottleneck. The changes therefore trade extra memory for on-device data residency and fewer H2D/D2H trips, maximize on-device cache/register/shared-memory reuse, and improve communication on both GPU-direct MPI and a new NVSHMEM transport.
Contributor
Author
|
The Intel compiler failures is due to a regression in ifx 2026.2. I tested locally with 2026.1 and it compiles fine. Seems to be happening on unmodified ifx segfaults on m_variables_conversion.mod. Flag-independent (segfault at -O0, no -ipo/-march). I will fix all the lint/format tests |
…-CMake renders MFC_FUSION_STAGING_DEFAULT is injected into the generated case.fpp by the toolchain only (toolchain/mfc/run/input.py). Documentation and other plain-CMake renders resolve case.fpp from src/common/include/defaults, which does not define it, so fypp aborted with a NameError and the documentation build failed. Keep the toolchain define first (same value) and fall back to the literal, mirroring the sibling constants below and toolchain/mfc/tuning.py. Co-Authored-By: Claude Code <noreply@anthropic.com>
Lines of Code
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[SC26 SCC] NTU Team Supernova's GPU optimizations to MFC
Note: I am still running the full CI on my own hardware. This notice will be edited away once they all pass. If any fixes are needed arising from my own CI run, I will push to this PR. This PR is not ready for review yet but will be ready once my own CI runs pass. I will not be marking it ready for review myself to not flood the runner.
For the full version of the PR description, please refer the full version below, as they are too long for Github to accept.
PR.md
This PR adds GPU performance optimizations for the simulation solver and communication optimizations, mostly for NVIDIA OpenACC path. It also fixes some existing numerical bugs. Most of the optimizations are guarded via opt-in compiler argument, except for some that will cause too much duplicate code when guarded. For those, they are always-on and guarded by the relevant compiler/GPU where necessary.
If I should be adding the options as case.py arguments, please let me know too, I know that is the current pattern, but I think case.py should focous on defining case-specific options and not general performance opt-ins. This is my own decision, and I am happy to change the approach if the maintainers prefer otherwise.
This is a large PR and if the maintainers prefer to split them into smaller PRs, I can do that too, but the goal was to not flood the CI runners so I merged them into 1 single PR.
If the maintainers are satisfied with some of the lower-risk optimizations and would like to make them always-on as default, I can also remove their respective opt-in toggles by replacing the existing implementation. This reduces code duplication. A significant part of the changed LOCs are from preserving existing code paths.
This PR assumes existing bug-fix PRs by me will be merged, and I will handle any conflicts arising from other team's contributions where necessary.
In general, the direction that I am working towards to is based on the following findings:
Thus, you will find many optimizations that trades some extra memory usage for data residency/reduced H2D/D2H trips, and maximizing on-device cache/register/shared memory reuse.
Unfortunately I do not have access to any larger cluster like Frontier to test scalability of my changes, so I want to focus more on improving compute-bound parts, and I do not want to claim anything in scalability for communication changes beyond what I have tested on.
Benchmark environment
All performance figures in this description were measured on 1-8 NVIDIA H200 NVL (141 GB) with 4-way NVLink bridge at 600W TDP · NVHPC 26.9 · CUDA 13.4 · NVSHMEM 3.8.0 · OpenACC · FP64 unless otherwise specified.
For fair comparison, all runs enabled GPU-direct MPI exchange
(
rdma_mpi) to achieve optimal performance on NVLinked NVIDIA GPUs. This is by default off in the current code and this PR does not change that default.Primary Results
Measured on https://github.com/MFlowCode/MFC/blob/master/examples/3D_performance_test/case.py 200³ performance test, 300 steps, one rank per GPU.
The tables give the median grind time of the runs at each point as reported by MFC. As reported in #2088, current
mastercode has a significiant regression in this test case (and many others). For our hardware we expected pre-regression timing to be around 0.34ns. The regression has been identified and fixed in this PR, but the numbers below is unaffected by it because one of the optimizations prevented the regression by construction.This PR includes an improved version of #1628. To ensure fairness and isolate geniue contributions, the number and scaling study below EXCLUDES this optimization. It also varies greatly with each case so it is unfair to base it all on the 3D performance test case. In Section 12, I separately report how the optimization further improves the performance.
8-GPU runs were 2 groups of 4xH200 NVL within a single node running dual AMD EPYC 9755 CPU.
Step time in ms/step
The two transports agree on the grind within 2% at 2, 4 and 8 GPUs, however due to removal of MPI host-stall outside of the grind time region, NVSHMEM runs 8%, 7% and 25% faster at 2, 4 and 8 GPUs (§6), judging from step time. This is meaningful to total runtime which the current grind time fail to account for. More info on NVSHMEM below.
Strong scaling
The original code scales 1.89×, 3.44× and 5.85× at 2, 4 and 8 GPUs. This PR
scales 1.79×, 3.06× and 4.40× on NVSHMEM, and 1.80×, 3.11× and 4.38× on
GPU-direct MPI. The speedup over the original code falls from 8.0× at one GPU
to 6.0× at 8 GPUs.
In host step time the original scales 1.89×, 3.45× and 5.84×. This PR scales 1.52×, 2.60× and 3.41× on NVSHMEM, and 1.40×, 2.44× and 2.74× on GPU-direct MPI. The step time contains the host side of the exchange, which the device-timed grind excludes, so it scales less than the grind.
Weak scaling
The work per GPU stays at 100³ cells. The grids are 100³, 200×100×100, 200×200×100 and
200³ on 1, 2, 4 and 8 GPUs.
The original code holds perfect weak efficiency at our scale. This PR runs every point 6.0× to 6.9× faster in grind. Its weak efficiency is 1.12, 1.10 and 0.98 at 2, 4 and 8 GPUs on NVSHMEM. On GPU-direct MPI it is 1.13, 1.13 and 0.97.
The 8-GPU runs place the ranks in a 1×2×4 grid which is most optimized for our specific topology. The 2-GPU and
4-GPU runs and the original code use the built-in most-cubic Cartesian cuts.
The performance-test case applies characteristic boundary conditions (CBC) on all six faces. At 1 GPU the single rank need to do the CBC staging on all 6 physical faces. On multi rank, the domain is split and the cut faces become rank interfaces, which exchange halos but run no CBC staging. The slowest rank's physical-face count falls 6 → 5 (2 GPUs) → 4 (4 GPUs) → 3–4 (8, where x is unsplit). With less boundary work per rank, the weak-scaling efficiency become higher than 100% at 2 and 4.
masterstays flat at 1.00 because its per-rank time is ~6× larger, so the same boundary saving is ~1–2% and invisible.Weak scaling at 8 GPU drops sharply partly due to the additional cross-NUMA traffic instead of NVLink, since we are not using a full-mesh node, so communication become much more bottlenecked. On H200 SXM I do not expect such a sharp drop.
Step time in ms/step
In host step time the original stays flat at 14.7 ms. This PR goes from 1.89ms at one GPU to 2.53 ms at eight on NVSHMEM, and to 3.15 ms on GPU-direct MPI. The grind-based weak efficiency at 8 GPUs is 0.98 on NVSHMEM, so the rise in step time is host-side work outside the device-timed region.
New Build options
Most of the optimizations in this PR are default-off and can be enabled by these new build options. Each option is a
./mfc.sh buildflag and a CMake option of the same name. For example,--fused-rhssetsMFC_FUSED_RHS=ON, and--no-fused-rhssets it toOFF. Every option is offby default. The toolchain (
toolchain/mfc/args.py) and CMake check the requirements below andstop with a message when any incompatible options are used, since some depends on others. Some options automatically enable the other dependent ones, unless the user explicitly passed --no-*, in which case they conflict and it will error with a custom error message.
--fused-rhs--case-optimization, OpenACC GPU build with NVHPC--hllc-register-tuning--case-optimization, OpenACC GPU build with NVHPC--cbc-no-host-sync--cbc-parallel-scatter--cbc-no-host-sync--async-queue--no-unified,--accurate-grind-time--accurate-grind-time--cuda-graphs--no-unified,--async-queueand therefore--accurate-grind-time--halo-single-round--mpi--halo-index-cache--halo-single-round--nvshmem--mpi, OpenACC GPU build with NVHPC,--no-unified, the NVSHMEM library--weno-direct-loads--case-optimization, OpenACC GPU build with NVHPC,--fused-rhsor--hllc-register-tuning--muscl-direct-loads--direct-riemann-flux--case-optimization, OpenACC GPU build with NVHPC--persistent-bc--halo-single-round.--fused-bubble-source--qbmm-device-inversion--case-optimization--igr-tuned-sweeps--case-optimization, otherwise it will be partially enabled.--igr-shallow-halo--lagrangian-device-cell-list--active-boxMFC_ACTIVE_BOXturns it on at run time.--fused-rhs, checked whenMFC_ACTIVE_BOXis setNew Environment variables
Run time
The binaries read these variables at start-up. An unset variable gives the behavior of the
original code.
MFC_ACTIVE_BOX<steps>[,<growth>[,<boxes>]]--active-box --fused-rhsMFC_ACTIVE_BOX_FLOOR--active-boxMFC_ACTIVE_BOX_INNERk≥ 1ksteps. It needs the single-box window.--active-boxMFC_HALO_OVERLAP1--fused-rhs --halo-single-round --async-queueMFC_HALO_EPOCH_DEVICE1--nvshmem --halo-single-roundMFC_HALO_26_TRIM1--nvshmem --halo-single-round --active-boxMFC_IGR_PINGPONG1MFC_SFC_WEIGHTSMFC_DECOMP_PROCESS_GRID<px>,<py>,<pz>parallel_io = T.MFC_DECOMP_AXIS_ORDER1,2,3parallel_io = T.1,2,3MFC_DECOMP_X_OFFSETS,MFC_DECOMP_Y_OFFSETS,MFC_DECOMP_Z_OFFSETS0,<cut>,...,<size>CUDA_VISIBLE_DEVICES,ROCR_VISIBLE_DEVICES,HIP_VISIBLE_DEVICES--accurate-grind-timeNVSHMEM_REMOTE_TRANSPORT,NVSHMEM_IB_ENABLE_IBGDA1--nvshmemBuild time
The toolchain reads these variables when it generates and builds a case.
MFC_TUNING_FILE<path>valueskey.--fused-rhsor--fused-bubble-sourceMFC_FUSION_*--fused-rhsMFC_BUBBLE_VECTOR_LENGTH--fused-bubble-sourceMFC_STATE_ALIGNMENT_BYTES--fused-rhsMFC_REGISTER_CAPS<module>=<cap>[,<module>=<cap>...]src/simulation. The cap is an integer from 1 to 255.--gpu accwith NVHPCNVSHMEM_ROOT<path>--nvshmemThe source gives the default of every constant. A value that is not set keeps the default.
Tests
All tests pass on NVHPC + OpenACC, AMD + OpenMP on all golden cases (assuming that the numerical bug PRs are merged.) This PR adds 1 golden
AE748A63to catch #2048.Build verification on the default configuration (every optional optimization off):
B44AF4A1(axisymmetric LF),AFBCBDFA(IGR),8EAC3DA7(viscous), and63CD6ADF(grcbc) pass.Design notes
files, and post-processing pipelines stay compatible. The optimization flags
compose through compile-time configuration.
layer and module conventions of the source. The fused kernel uses fypp templates over
direction and physics, like the existing preprocessor usage.
Fixes #2088 #2048
Addresses the hang observed in #1927
Related:
#2090 where I believe my implementation is better for the cases where it covers, and #2090 is better for other cases.
#2092 came up with similar idea independently but chose to go the more extreme route of optimizing away the loads using fused kernels. also optimize away "
mp_wenoandint_comp > 0" mentioned in the PR.#1628 where I improved upon by supporting multi-rank natively.
Contribution Policy
All contributions are expected to demonstrate:
If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.
Acknowledgement
This PR is prepared by Billy Cao from Nanyang Technological University SCC team: Supernova for SC26 Student Cluster Competition. It unintentionally overlapped with ideas that existing PRs from other teams have raised, and I have fully acknowledged them in these cases. It also took inspiration from existing PRs by the MFC team, which I have also fully credited where due. For each of such cases, I have explained the differences/improvements on top of existing PRs.