fix(flux2): remove inert guidance UI, add Klein 4B Base variant, fix metadata recall#8995
Conversation
Pfannkuchensack
requested review from
JPPhoto,
blessedcoolant,
dunkeroni and
lstein
as code owners
github-actions
Bot
added
python
JPPhoto
left a comment
JPPhoto
left a comment
There was a problem hiding this comment.
Some changes and recommendations:
-
Klein 9B Base guidance still is not written to image metadata, so the main recall/regeneration fix is incomplete. In buildFLUXGraph.ts, the FLUX.2 metadata object only includes
model,steps, andscheduler; it never addsguidance, even though the denoise node now receives it. The parser in parsing.tsx looks formetadata.guidance, so FLUX.2 images still will not display or recall guidance. -
The metadata viewer will show duplicate VAE entries for FLUX.2 images. ImageMetadataActions.tsx already renders the generic
VAEModelhandler, and this change addsKleinVAEModeltoo. I think both handlers parse the samemetadata.vaekey in parsing.tsx, and both accept FLUX.2, so the UI will end up with two recall controls for the same VAE. -
More automated coverage is needed for this feature. I did not find new tests covering the added FLUX.2 guidance path, scheduler metadata path, or the new metadata recall entries. Consider targeted unit/integration coverage: negative cases for guidance being omitted for distilled Klein variants and outside
klein_9b_base, plus positive cases for guidance and scheduler being persisted into graph metadata and for the new metadata fields to parse and recall correctly. This PR changes graph construction and metadata handling, and tests at those seams would help lock the behavior down.
… Base, and fix metadata recall Klein 4B and 9B (distilled) have guidance_embeds=False, while Klein 9B Base (undistilled) has guidance_embeds=True. This commit: - Sets guidance_embed=False for Klein 4B/9B and adds Klein9BBase with True - Adds guidance parameter to Flux2DenoiseInvocation (used by Klein 9B Base) - Passes real guidance value instead of hardcoded 1.0 in flux2/denoise.py - Hides guidance slider for distilled Klein models, shows it for Klein 9B Base - Shows Flux scheduler dropdown for all Flux2 Klein models - Passes scheduler to Flux2 denoise node and saves it in metadata - Adds KleinVAEModel and KleinQwen3EncoderModel to recall parameters panel
…all dedupe Add a mock-based harness for buildFLUXGraph that locks in the FLUX.2 orchestration: guidance is written to metadata and the flux2_denoise node only for klein_9b_base, distilled variants (klein_9b, klein_4b) omit it, the FLUX scheduler is persisted into both metadata and the denoise node, and separately selected Klein VAE / Qwen3 encoder land in metadata. Add parsing tests for the metadata recall handlers: KleinVAEModel and KleinQwen3EncoderModel only fire when the current main model is FLUX.2, and the generic VAEModel handler now bails out for flux2 / z-image so the metadata viewer no longer renders duplicate VAE rows next to the dedicated Klein / Z-Image handlers.
Pfannkuchensack
force-pushed
the
fix/flux2-klein-guidance-and-metadata
branch
from
de68d29 to
edbc705
Compare
…/flux2-klein-guidance-and-metadata
JPPhoto
left a comment
JPPhoto
left a comment
There was a problem hiding this comment.
This looks good to me. I think we should ask other people to specifically test the undistilled model since I can't really run that easily on my end.
…adata' into fix/flux2-klein-guidance-and-metadata # Conflicts: # invokeai/frontend/web/src/features/settingsAccordions/components/GenerationSettingsAccordion/GenerationSettingsAccordion.tsx
…dance-and-metadata
|
I hope I haven't misunderstood the intention of the PR, I tested it but I couldn't get it to work as I expected. What I tested and what failed:I started by testing the I tried manually editing the model's What I tried next and what I found:Since the Guidance parameter wasn't affecting the output, I wondered if the model might respond to traditional CFG instead. I bypassed the linear UI and used the Node Editor to feed different This worked! The model responded to the CFG changes and produced completely different, properly guided images. I also went back and tested the Testing with Kontext Conditioning:I also tested using reference images via Kontext Conditioning. I initially ran into an issue where the output would be a pure black image when using a CFG scale > 1.0. However, I realised this was due to an incompatible VAE. When I ensured the specific What I think should be done:Based on these tests, it looks like the Klein Base models rely on standard CFG rather than guidance embeddings. Since the backend
|
Recognize FLUX.2-klein-base-4B on import via filename heuristic. The variant shares Klein4B's architecture (Qwen3-4B encoder, context_in_dim=7680) and reports guidance_embeds=False in its HF config, consistent with Klein 9B Base. UI behavior stays identical to distilled Klein4B until CFG support is wired up in a follow-up.
…dance-and-metadata
|
I will address this in another PR. |
…/flux2-klein-guidance-and-metadata
…b.com/Pfannkuchensack/InvokeAI into fix/flux2-klein-guidance-and-metadata
|
@Pfannkuchensack Request a re-review from me when you and @skunkworxdark are satisfied with the state of things, please. |
|
My comments remain the same at the moment. With a slight difference, I don't think we need to have the negative prompt visible now. The current guidance setting doesn't create a different image output at all. I still believe that it should be cfg_scale that is used instead, but this requires a negative prompt to be passed. This neg prompt in the diffusers pipeline is internally defaulted to "", but in the Invoke implementation, it needs to be passed manually. However, using a cfg_scale > 1 will double the generation time. In my testing, I don't believe that exposing the negative prompt to the user would provide any benefit over it being automatically set to "". As for the rest of the PR I have done some research into guidance_embeds=true for the base model and I can't find any reference that states this. Infact the BFL base model for both 4B and 9B base have the transformer/config.json containing guidance_embeds=false. For me, this should probably be the source of truth on this matter. Unless there is a paper or repo that states otherwise. |
All current FLUX.2 Klein variants (4B, 4B Base, 9B, 9B Base) report guidance_embeds=false in their HF transformer config (or have zeroed projection weights), so the guidance scalar has no effect on output. The linear UI previously exposed a guidance slider for klein_9b_base and wrote the value into metadata, which misled users into thinking it was steering generation.
Pfannkuchensack
changed the title
|
OK Updated the PR. |
|
@Pfannkuchensack Did you mean to check something in? I don't see any recent commits. |
…b.com/Pfannkuchensack/InvokeAI into fix/flux2-klein-guidance-and-metadata
|
|
The generic Guidance metadata handler unconditionally parsed `metadata.guidance` and dispatched `setGuidance(value)` into the shared params slice. For images generated before the Klein guidance cleanup, this still fired — silently writing a stale guidance value into the global state, which then leaked back into FLUX.1 on model switch. Gate the handler on `metadata.model.base`: reject parsing when the image was generated with a FLUX.2 model. The handler is then skipped for both display and recall on legacy FLUX.2 metadata, matching the "silently ignored" contract stated in the PR. - parsing.tsx: check metadata.model.base in Guidance.parse() - parsing.test.tsx: three new cases covering FLUX.2 gating, FLUX.1 pass-through, and back-compat for metadata without a model field
5 participants
Summary
Several independent fixes around FLUX.2 Klein model support plus a SDXL DoRA + partial-loading regression fix:
FLUX.2 Klein guidance & metadata cleanup
guidance_embeds=falsein their HF transformer config, or ship with absent/zeroed guidance projection weights. The scalar therefore has no effect on output for any Klein model currently on HuggingFace.util.py: Setguidance_embed=Falsefor all four Klein variants (Klein9BBase was previouslyTrue, but the loader already zeroed the weights — cosmetic drift that misled readers).klein_9b_basewhere it appeared to do something but didn't.guidanceinto theflux2_denoisenode and into Flux2 image metadata.flux2_denoiseinvocation field description to reflect the inert behaviour.guidance,cfg_scale, andnegative_text_conditioningfields onflux2_denoisefor experimentation — only the linear UI gating changes.FLUX.2 Klein 4B Base variant recognition
Flux2VariantType.Klein4BBaseenum entry alongside the existingKlein9BBase."base" in name.lower()) extended to Klein 4B, mirroring the existing 9B Base logic.FLUX.2-klein-base-4B(https://huggingface.co/black-forest-labs/FLUX.2-klein-base-4B) is now detected as its own variant on import.qwen3_source_modelfallback, and default settings extended to includeKlein4BBase(Qwen3_4B encoder, 28-step default).QA Instructions
FLUX.2-klein-4bdiffusers or safetensors model. Linear UI advanced panel shows steps + scheduler; no guidance slider. Generate — output identical to prior behaviour.FLUX.2-klein-base-4B(diffusers or a safetensors file whose name contains "base"). Verify it is identified asklein_4b_basein the model manager (FLUX.2 Klein 4B Base). Selecting it loads the Qwen3-4B encoder (not Qwen3-8B). Generate — single-pass image, no guidance slider, no CFG slider.guidance.guidancemetadata: Recalling an older Flux2 image that was generated with the previous build should still work —guidanceis silently ignored, no errors.guidance.flux2_denoisestill exposesguidance,cfg_scale, andnegative_text_conditioningas editable fields. Wiring a negative text encoder + cfg_scale > 1.0 still produces the two-pass CFG output that worked before.Merge Plan
No special ordering required. Backend enum change is additive (
Klein4BBaseis a new value, not a rename), so older clients reading the schema will simply not see the new variant.Checklist
What's Newcopy (if doing a release after this PR)