Repository navigation
Conversation
PARMETIS_ANISOTROPY_WEIGHT (default YES) weights the ParMETIS graph edges by cell aspect ratio, gives the edges that chain points into wall-normal lines the largest weight, and moves each line to the rank holding most of its points after partitioning. Lines are then not split between ranks, which keeps the block-Jacobi ILU and line-based agglomeration from seeing fragments of lines. On stretched boundary-layer meshes this turned stalled or diverging parallel runs into converging ones (2D shear layer, 2D bump, 3D bump, flat plate with multigrid) and made iteration counts match the serial run more closely, with no case that converged without it failing with it. The load imbalance stays within 1.01-1.05 in 2D and up to about 1.3 on a large 3D mesh. The option has no effect on isotropic meshes or without ParMETIS; a warning is printed when it is set explicitly in a build without ParMETIS. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
I bumped black up a version and precommit removed a couple of empty lines in some python files. |
…eights In AD builds MPI_DOUBLE maps to AMPI_ADOUBLE, which expects active su2double buffers. The passivedouble buffers were overrun, corrupting the heap and crashing the AD regression cases that partition with ParMETIS. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ode/su2 into feature_parmetis_anisotropy
Every rank holds a copy of the pressure drop, so the adjoint extracted on one rank was only its local contribution and Adjoint_SolExtra depended on the partition. The extracted adjoint is now summed over the ranks, and the seed is divided by the number of ranks because the reverse Allreduce of the massflow sums the seeds of all copies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TransposeInPlace turns the blocks of halo rows into owned rows of the transposed matrix. Boundary conditions modify only owned rows (symmetry planes project out the normal momentum, walls delete velocity, energy and turbulence rows), so wherever a partition boundary met such a boundary the adjoint linear system had spurious couplings. The primal never reads halo rows and was unaffected. On da_sp_pinArray_cht_2d_mf at CFL 100 this made the adjoint diverge on some partitions (2 ranks with anisotropic ParMETIS weights, 3 and 4 ranks without), with the adjoint linear solver reaching only 1e-1.4 instead of 1e-3. All partitions from 1 to 4 ranks now give the same results. The reference values of that case are updated, they include the pressure drop adjoint fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TransposeInPlace now takes the geometry and copies the owner rows into the halo rows itself, so the two callers cannot forget it. Comments shortened and the new code run through clang-format. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
there was also an mpi error in AD that popped up with this PR, it is fixed with 50639a4 |
rois1995
left a comment
There was a problem hiding this comment.
Please address the two code issues described in the inline comments.
|
The assessed head ( Please also validate the changed pressure-drop derivative against finite differences at 1, 2, and 4 MPI ranks. The refreshed two-rank reference passes, but an independent comparison is still needed. Local validation: the MPI normal build passes, and 201-iteration RAE2822 SST/MG runs complete with finite histories in serial and on two ranks with the option enabled and disabled. These short runs do not establish convergence. |
Send global column indices in an unsigned long buffer, apart from the scalar blocks, so single-precision builds keep indices above 2^24 exact. Reject whole-line moves in KeepLinesTogether when the resulting vertex weight imbalance exceeds max(1.5, the ParMETIS partition's imbalance). Add unit tests for the halo block buffer and the imbalance measure. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ode/su2 into feature_parmetis_anisotropy
…ode/su2 into feature_parmetis_anisotropy
|
@rois1995 I fixed the code issues, and now going through the regressions to see if there is anything out of order. |
This PR splits off the parmetis improvements from #2931. It turns out that this parmetis improvement is 'almost' always better, and in the worst case performs equally well. It improves MPI performance for cases with inflation layers.
PARMETIS_ANISOTROPY_WEIGHT (default YES) weights the ParMETIS graph edges by cell aspect ratio, gives the edges that chain points into wall-normal lines the largest weight, and moves each line to the rank holding most of its points after partitioning. Lines are then not split between ranks, which keeps the block-Jacobi ILU and line-based agglomeration from seeing fragments of lines.
On stretched boundary-layer meshes this turned stalled or diverging parallel runs into converging ones (2D shear layer, 2D bump, 3D bump, flat plate with multigrid) and made iteration counts match the serial run more closely, with no case that converged without it failing with it. The load imbalance stays within 1.01-1.05 in 2D and up to about 1.3 on a large 3D mesh. The option has no effect on isotropic meshes or without ParMETIS; a warning is printed when it is set explicitly in a build without ParMETIS.
Proposed Changes
Give a brief overview of your contribution here in a few sentences.
Related Work
Resolve any issues (bug fix or feature request), note any related PRs, or mention interactions with the work of others, if any.
PR Checklist
Put an X by all that apply. You can fill this out after submitting the PR. If you have any questions, don't hesitate to ask! We want to help. These are a guide for you to know what the reviewers will be looking for in your contribution.
pre-commit run --allto format old commits.