move unpack out of the flux, krea2 and ltx decoders into the core denoise groups by akshan-main · Pull Request #14737 · huggingface/diffusers

akshan-main · 2026-09-08T07:59:07Z

What does this PR do?

Adds a test to the modular tester mixin that checks the shape of the latents each pipeline leaves in the state after denoising, and every tester now declares expected_latents_shape. Running it across all families catches the three from the issue, plus ltx2 (being moved in #14612, so it's declared at its current packed shape until then) and wan_animate_2, which decodes inside its segment loop and never leaves latents behind, so it opts out with latents_output_name = None.

For flux, krea2 and ltx the unpack now happens at the end of the core denoise group, same shape as Flux2UnpackLatentsStep. The decoders just denormalize and decode. They still accept packed latents with a deprecation warning, which is why height/width (and num_frames for ltx) are still on them, and there's a test per family for that path.

Outputs are identical to main on the test repos and on the real checkpoints (images in akshan-main/latents-form-verification). Loading LTX-Video with ModularPipeline.from_pretrained needs #14736. Also wrote the principle down in .ai/references/modular.md in a separate commit; feel free to drop it or change it.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Requested by @yiyixuxu in [modular] move unpack out of the decoder blocks for flux, krea2 and ltx #14730
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

Who can review?

@yiyixuxu

…eaves in the state

…oise groups

…ts-form test

…rately Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

HuggingFaceDocBuilderDev · 2026-10-17T22:04:50Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

yiyixuxu

thanks a lot! really nice work
I left some feedbacks

yiyixuxu · 2026-10-17T22:08:04Z

@@ -28,6 +28,64 @@


 # auto_docstring


ohh I think this #auto_docstring maker is in wrong place

yiyixuxu · 2026-10-17T22:09:55Z

can you update the description here?

yiyixuxu · 2026-10-17T22:11:17Z

+            "the VAE takes (still normalized). Closes the core denoise group, so the blocks that follow take the same "
+            "form the VAE produces and need no geometry inputs."


Suggested change

"the VAE takes (still normalized). Closes the core denoise group, so the blocks that follow take the same "

"form the VAE produces and need no geometry inputs."

"the VAE takes (still normalized)."

yiyixuxu · 2026-10-17T22:13:32Z

+            "VAE takes (still normalized). Closes the core denoise group, so the blocks that follow take the same "
+            "form the VAE encoder produces and need no geometry inputs."


Suggested change

"VAE takes (still normalized). Closes the core denoise group, so the blocks that follow take the same "

"form the VAE encoder produces and need no geometry inputs."

"VAE takes (still normalized)."

yiyixuxu · 2026-10-17T22:25:26Z

+                type_hint=torch.Tensor,
+                description="The denoised latents from the denoising step, packed, of shape `[B, S, C]`.",
+            ),
+            InputParam("height", default=1024),


height/width should be required with no default

yiyixuxu · 2026-10-17T22:32:45Z

+            "the VAE takes (still normalized). Closes the core denoise group, so the blocks that follow take the same "
+            "form the VAE encoder produces and need no geometry inputs."


Suggested change

"the VAE takes (still normalized). Closes the core denoise group, so the blocks that follow take the same "

"form the VAE encoder produces and need no geometry inputs."

"the VAE takes (still normalized)."

yiyixuxu · 2026-10-17T22:36:39Z

+    # Still the packed `[B, S, C]` form: #14612 moves ltx2 to the VAE form and updates this.
+    expected_latents_shape = (1, 768, 4)


Suggested change

# Still the packed `[B, S, C]` form: #14612 moves ltx2 to the VAE form and updates this.

expected_latents_shape = (1, 768, 4)

let's not give the wrong "expected_latents_shape" just so that the test pass - it's misleading

we can skip the test explicitly with something like this

@pytest.mark.skip(reason="ltx2 still leaves packed `[B, S, C]` latents in the state; #14612 moves it to the VAE form.") def test_latents_output_in_canonical_form(self): pass

yiyixuxu · 2026-10-17T22:53:15Z

+        if self.latents_output_name is None:
+            pytest.skip("This family decodes inside its denoise loop and leaves no latents in the state.")


Suggested change

if self.latents_output_name is None:

pytest.skip("This family decodes inside its denoise loop and leaves no latents in the state.")

the subclass can skip it with `@pytest.mark.skip(...)

sayakpaul

I like the PR. I think it aims at a standardization that we have been discussing in other threads. Thank you!

sayakpaul · 2026-10-19T02:54:56Z

+
+Each should be applied and undone in mirrored pairs, and the core denoise group hands latents back in the same form it received them. The `latents` output of the denoise group should always be in one form: unpacked and still normalized, the same form its VAE encoder blocks produce. Whatever runs next (a decoder, a latent upsampler, a second denoise group) consumes that form, and no block outside the group needs `height`/`width`/`num_frames` just to interpret tokens.
+
+The pattern: `encode -> norm -> pack -> denoise -> unpack -> denorm -> decode`


Should provide a reference from the codebase here.

sayakpaul · 2026-10-19T02:56:17Z

+                    "packed latents",
+                    "1.0.0",


I think we can deprecate before. Like give it ~5 releases? I would also add a test to ensure the deprecation is actually raised (if not already added).

sayakpaul · 2026-10-19T02:57:16Z

+            InputParam.template("height", default=1024),
+            InputParam.template("width", default=1024),


No defaults here following https://github.com/huggingface/diffusers/pull/14737/changes#r4304769059?

…ath directly

yiyixuxu · 2026-10-20T02:59:08Z

    pipeline_blocks_class = LTX25AutoBlocks
    pretrained_model_name_or_path = LTX25_REPO_ID
+    # Still the packed `[B, S, C]` form: #14612 moves ltx2 to the VAE form and updates this.
+    expected_latents_shape = (1, 768, 4)


can we explicitly skip this one too?

yiyixuxu · 2026-10-20T03:08:09Z

/diffusers-bot pytest tests/modular_pipelines/flux tests/modular_pipelines/krea2 tests/modular_pipelines/ltx -k test_decode_accepts_packed_latents_with_deprecation

github-actions · 2026-10-20T03:08:27Z

❌ pytest tests/modular_pipelines/flux tests/modular_pipelines/krea2 tests/modular_pipelines/ltx -k test_decode_accepts_packed_latents_with_deprecation failed on GPU — view logs.

yiyixuxu · 2026-10-20T03:14:22Z

    output_name = "images"
+    # State name of the latents the denoise group hands to the decoder. Chunked families that hand over a list of
+    # per-chunk latents (e.g. `latent_chunks`) are checked chunk by chunk. Families that decode inside their
+    # denoise loop and leave no latents in the state set this to `None`.


can you update the comment here? better to skip explicitly with a reason

Added this
State name of the latents the denoise group hands to the decoder. Chunked families that hand over a list of per-chunk latents (e.g. latent_chunks) are checked chunk by chunk. A family that leaves no latents in the state, or not yet in its canonical form, skips test_latents_output_in_canonical_form with a reason.

yiyixuxu · 2026-10-20T03:17:52Z

+        decoder.to(torch_device)
+
+        # The 9-frame 32x32 dummy video is a `[1, 8, 5, 16, 16]` latent, packed to 1280 tokens of 8 channels.
+        packed_latents = torch.randn(1, 1280, 8, generator=self.get_generator())


Suggested change

packed_latents = torch.randn(1, 1280, 8, generator=self.get_generator())

packed_latents = torch.randn(1, 1280, 8, generator=self.get_generator()).to(torch_device)

can you update the other 2 tests too? see https://github.com/huggingface/diffusers/actions/runs/39362689598/job/118432333376#step:8:58

yiyixuxu

thanks a lot for working on this!

akshan-main added 5 commits September 7, 2026 21:14

add a modular test that pins the canonical latents form each family l…

1314e35

…eaves in the state

move unpack out of the flux, krea2 and ltx decoders into the core den…

9005916

…oise groups

pop the unpack step from a copy of the blocks in the deprecation tests

713dbbe

let families that decode inside the denoise loop opt out of the laten…

bd54dd8

…ts-form test

document the canonical latents form in the modular reference

7abc2c0

github-actions Bot added tests modular-pipelines size/L PR with diff > 200 LOC fixes-issue labels Sep 8, 2026

yiyixuxu and others added 2 commits October 16, 2026 23:07

Merge branch 'main' into modular-latents-form

2ef9288

[docs] canonical latents form: name normalize, patchify and pack sepa…

76984ea

…rately Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

yiyixuxu mentioned this pull request Oct 17, 2026

Follow-up to #14737: drop output_type="latent" from modular decode blocks #15071

Open

yiyixuxu reviewed Oct 17, 2026

View reviewed changes

yiyixuxu requested a review from sayakpaul October 17, 2026 23:04

yiyixuxu added this to the Release 0.42.0 milestone Oct 17, 2026

sayakpaul reviewed Oct 19, 2026

View reviewed changes

require geometry on the unpack steps and test the deprecated decode p…

2c2c0cb

…ath directly

akshan-main requested review from sayakpaul and yiyixuxu October 19, 2026 04:15

dhvani7014 mentioned this pull request Oct 19, 2026

Remove output_type="latent" from modular decoders #15078

Open

7 of 10 tasks

Merge branch 'main' into modular-latents-form

1b50989

yiyixuxu reviewed Oct 20, 2026

View reviewed changes

akshan-main added 2 commits October 19, 2026 20:09

skip the canonical latents test explicitly for ltx 2.5 too

80c734b

remove the packed latents path in 0.47.0, five releases after 0.42

060e96d

yiyixuxu reviewed Oct 20, 2026

View reviewed changes

update the latents_output_name comment for the explicit skips

09903d8

yiyixuxu reviewed Oct 20, 2026

View reviewed changes

move the packed test latents to the test device

436058a

yiyixuxu approved these changes Oct 20, 2026

View reviewed changes

yiyixuxu merged commit 6184688 into huggingface:main Oct 20, 2026
15 of 16 checks passed

		"the VAE takes (still normalized). Closes the core denoise group, so the blocks that follow take the same "
		"form the VAE produces and need no geometry inputs."

	"the VAE takes (still normalized). Closes the core denoise group, so the blocks that follow take the same "
	"form the VAE produces and need no geometry inputs."
	"the VAE takes (still normalized)."

		"VAE takes (still normalized). Closes the core denoise group, so the blocks that follow take the same "
		"form the VAE encoder produces and need no geometry inputs."

	"VAE takes (still normalized). Closes the core denoise group, so the blocks that follow take the same "
	"form the VAE encoder produces and need no geometry inputs."
	"VAE takes (still normalized)."

		# Still the packed `[B, S, C]` form: #14612 moves ltx2 to the VAE form and updates this.
		expected_latents_shape = (1, 768, 4)

		if self.latents_output_name is None:
		pytest.skip("This family decodes inside its denoise loop and leaves no latents in the state.")


		Each should be applied and undone in mirrored pairs, and the core denoise group hands latents back in the same form it received them. The `latents` output of the denoise group should always be in one form: unpacked and still normalized, the same form its VAE encoder blocks produce. Whatever runs next (a decoder, a latent upsampler, a second denoise group) consumes that form, and no block outside the group needs `height`/`width`/`num_frames` just to interpret tokens.

		The pattern: `encode -> norm -> pack -> denoise -> unpack -> denorm -> decode`

		InputParam.template("height", default=1024),
		InputParam.template("width", default=1024),

	packed_latents = torch.randn(1, 1280, 8, generator=self.get_generator())
	packed_latents = torch.randn(1, 1280, 8, generator=self.get_generator()).to(torch_device)

Conversation

akshan-main commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Who can review?

Uh oh!

HuggingFaceDocBuilderDev commented Oct 17, 2026

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yiyixuxu commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

akshan-main commented Sep 8, 2026 •

edited

Loading

github-actions Bot commented Oct 20, 2026 •

edited

Loading