NicoLab28/ClipProj-MiniMax-H3

ClipProj — MiniMax H3 conditioning from a Qwen3-VL-4B or 8B

Projection matrices that let a small Qwen3-VL replace the Qwen3-VL-32B text encoder of MiniMax H3.

15.7 GB → 4.9 GB of VRAM, with no change to the diffusion model, the VAEs or the sampler. The 32B is nvfp4, the small encoders int8; the projection itself costs 52 MB to 604 MB depending on the file.

Five renders of one scene, in this order: 4B matrix-only, 8B matrix-only, the 32B reference, 8B residual, 4B residual. The reference sits in the middle, so each half is read against it.

conditioning	encoder	projection on card	total
Qwen3-VL-32B nvfp4	15.69 GB	—	15.7 GB
Qwen3-VL-8B int8 + v3-mlp	10.01 GB	604 MB	10.6 GB
Qwen3-VL-8B int8 + v3	10.01 GB	84 MB	10.1 GB
Qwen3-VL-4B int8 + v3-mlp	4.83 GB	503 MB	5.3 GB
Qwen3-VL-4B int8 + v3	4.83 GB	52 MB	4.9 GB

Two things this table makes explicit, because both would otherwise flatter the result. The quantisations differ: the 32B is nvfp4, the students are int8 — part of the size gap is format, not parameter count. And the residual is not free: the node loads the matrix in float32 and keeps the residual in the dtype it was saved in, so a -mlp file costs roughly half a gigabyte on the card where the plain matrix costs a rounding error.

diffusion model	minimax_h3_fl2va_pruned_int8_convrot
turbo LoRA	minimax_h3_fl2v_turbo_8step_v1.0_comfyui_bf16, strength 1.0
sampler / scheduler	res_multistep / simple
steps	8
seed	42
resolution	16:9 at 0.8 MP, upscaled 2× by RTX Video Super Resolution ULTRA
frames	192 at 24 fps — 8.00 s
video VAE	minimax_h3_video_vae_int8_convrot
audio VAE	minimax_h3_audio_vae_fp32

Only the projection changes between the five. Everything else is identical, and on one machine the pipeline is deterministic — running the same configuration twice gives byte-identical decoded video and audio, verified by MD5 — so every difference you see comes from the projection and nothing else.

You will not reproduce these files, and that is expected

Run the demo prompt with seed 42 on your own machine and you will get the same scene, not the same file. The result depends on the model of GPU the encoder runs on.

This came out of an unrelated test — checking that three loading modes gave the same output — and the cards happened to be at hand. Four of them is not a study, and none of this was the point of the exercise; it is written down because it would otherwise look like something is broken. Same prompt, same seed, same everything else:

card	decoded video MD5
RTX 4070	1daf9be3…
RTX 3090	0a415964…
RTX 3060	dcb2f965…
RTX 4070 Ti SUPER	b3b185e7…

Four cards, four results. Two different RTX 3090s gave byte-identical output, so it is the model that decides, not the individual card — and not the architecture either, since the 3060 and the 3090 are both Ampere and disagree.

The cause is small and the consequence is not. Encoding the same prompt on two cards gives conditioning that agrees to a relative error of 7 × 10⁻⁷ — cosine 1.00000000, largest single-component difference 0.002. Different numbers of compute units mean different reduction orders, so floating-point additions do not happen in the same sequence. Eight denoising steps turn that into a different piece of furniture, or a wristwatch that is there on one card and absent on another. That watch is nowhere in the prompt, which is exactly why it is free to move.

So: on one machine, with one card, everything here is reproducible to the bit — that is what makes the five-way comparison above meaningful. Across machines, expect the same scene with different details. This is a property of the diffusion model and its sampler, not of the projections: the reference 32B behaves identically.

The full prompt is in demo/chess-prompt.txt, the settings above in machine-readable form in demo/generation-settings.json, and the five renders are in demo/ one by one if you want to step through them.

What to look at. Everything the prompt states is there, on all five: the seated pose, the red dress, the white pieces on her side, the two captured black pawns, the cat, the straw hat, the laundry — and her knee, asked for three times and ending on a sentence of its own, "Her knee never stops bouncing." A continuous involuntary motion with no narrative purpose is the clearest single sign that a projection carried what was written, and it carries on the plain matrices too.

One thing none of the five gets right, the 32B included: she lifts a knight and does not set it back on the same square, and on the 8B residual there is no knight on the board at all. Object permanence behind an occluding hand, on a grid of sixty-four identical squares, is a limit of the video model rather than of the conditioning.

And the terrace is furnished differently from one render to the next — that is not infidelity. The prompt asks for a densely lived-in terrace without anchoring most of it: the cat is "stretched out asleep in the sun", and nothing says where. What is left open, the model invents, and it invents differently depending on the projection, the seed, and the model of GPU — all three act on that same free space and none of them touches what was written. Two renders side by side give the impression of a different seed; that impression is what an unconstrained description looks like.

⚠️ Proof of concept — working, but a proof of concept. It runs and produces good video, and every number below was measured on real hardware. Built and tested on a single setup (Windows 11, NVIDIA, ComfyUI 0.31.0) with deliberately limited exploration.

Arriving from a tutorial or an article? Anything published before this release names older files. Nothing has been deleted — the previous sets are still here — but take the current one: mmh3-4b-ClipProj-v3-mlp for a Qwen3-VL-4B, mmh3-8b-ClipProj-v3-mlp for an 8B. They need node 0.1.13 or later, and an earlier node raises KeyError: 'W' on them rather than falling back quietly.

These files are useless on their own. They require the custom node: github.com/nicolab28/ComfyUI-ClipProj

Where this came from

I am not an ML researcher. I work in imaging, and programming is a tool and a hobby rather than my trade. This started as something to tinker with: I wanted to understand how a diffusion model actually uses its text encoder, and the only way I know how to understand something is to take it apart and see whether it still runs afterwards.

So the question was never "how do I save VRAM". It was "is this even possible at all". I expected it to fail. A linear map between two models that were never trained together, fitted in a single pass with no gradients and no learning rate, has no business producing usable video.

It did, and the first results were good enough that keeping them on my own disk seemed silly. That is the whole story, and it is why this is labelled a proof of concept rather than a tool: it was never designed as one.

It is also why there are so many measurements on the model card. Before showing this to anyone I had to convince myself I was not fooling myself, and most of what I tried along the way turned out to be wrong. Those attempts are written down as well, in MEASUREMENTS.md and CALIBRATION.md.

What changed in v3

Calibrated against the stock encoder. The previous matrices were fitted against a modified 32B. While testing them we found that naming one part of a body could rewrite the whole of it — build, height and face shifting together, none of it asked for. v3 targets qwen3vl_32b_minimax_h3_nvfp4_awq, the encoder a plain Load CLIP gives you, and we no longer observe the problem.

The 8B now sees image tokens. Until v3 the image corpus had only ever been encoded with a 4B student, so both 8B matrices projected vision tokens without having seen a single one — while the node accepts a reference image. Measured on 100 held-out images, on the raw conditioning the diffusion model actually receives:

	vision tokens	text in the same sequences
8B residual, before	0.7692	0.9085
8B residual, after	0.8578	0.9605
8B matrix, before	0.7845	0.8926
8B matrix, after	0.8457	0.9361

That costs 0.0027 of pure-text cosine on the residual and 0.0013 on the matrix. Both students now share the same corpus, so the 4B and the 8B are comparable to each other for the first time.

Node 0.1.13 is required. The -v3-mlp files carry no linear matrix at all — the non-linear part does the whole job — and an earlier node raises KeyError: 'W' when it opens one.

And ClipProjApply now works with an int8 encoder. Loading a quantised Qwen3-VL through ComfyUI's own Load CLIP and handing it to ClipProjApply used to fail inside the vision tower as soon as a reference image was present: dequantize_int8_embedding was called on a tensor the cast context had already dequantised to bf16, and the error named nothing useful. The position embedding now falls back to a plain lookup when that happens. A text-only prompt never triggered it, which is why it stayed hidden — it only appeared with an image, on the pageable path. The five demo renders above use that path.

One number, measured the same way for everything. The cos_test written inside each file is its training-time figure against its own campaign's target, and it does not compare across versions. The comparable number is stored separately as cos_prompt_reference: every projection encoded against the same stock 32B, on the same prompt.

projection	vs 32B
mmh3-8b-ClipProj-v3-mlp	0.9449
v2 mmh3-8b-ClipProj-celeb-mlp	0.9393
mmh3-4b-ClipProj-v3-mlp	0.9381
v2 mmh3-4b-ClipProj-celeb-mlp	0.9293
mmh3-8b-ClipProj-v3	0.9289
mmh3-4b-ClipProj-v3	0.9193

Prefer the -mlp files, on the measurement — not on this scene. They sit closer to the 32B, 0.9449 against 0.9289 on the 8B. But watch the video before assuming that shows: on this prompt the five renders are faithful, plain matrices included. The pose, the red dress, the white pieces, the cat, the straw hat, the laundry, the bouncing knee — all of it holds on all five.

The only thing nobody gets right is the knight. The prompt asks her to lift one of her own pieces and set it back down without committing to the move; on every render it lands somewhere else, and on one of them — the 8B residual, the best-measuring file of the set — there is no knight on the board at all. That is object permanence behind an occluding hand on a grid of sixty-four identical squares, a limit of the video model rather than of the conditioning, and the 32B reference fails it too.

So the honest reading is: a tightly written prompt survives even the linear baseline, and the difference between the files shows up in the numbers well before it shows up on screen.

What this is

MiniMax H3 conditions on a Qwen3-VL-32B truncated to 50 layers — 15.7 GB in NVFP4 — solely to turn a prompt into a [seq, 5120] tensor. This repository provides a learned map that lets a much smaller Qwen3-VL produce the same conditioning:

cond = ((h - mean_in) / std_in) @ W * std_out + mean_out

and, in the -mlp files, plus the output of a small residual network fed the same standardised input.

It works because every Qwen3-VL shares the same tokenizer (151936 tokens): a prompt yields the same tokens at the same positions in both models, so a position-by-position mapping between their hidden states can be learned. The matrix is fitted by plain ridge regression — no gradients, no epochs, no learning rate. The residual network is the only part that is trained.

Files

Put them in ComfyUI/models/clip_projections/.

Start with mmh3-8b-ClipProj-v3-mlp if you have the VRAM, mmh3-4b-ClipProj-v3-mlp otherwise. Both need node 0.1.13 or later.

File	Encoder	Structure	Size	vs 32B
mmh3-4b-ClipProj-v3-mlp	any Qwen3-VL-4B	non-linear	503 MB	0.9381
mmh3-8b-ClipProj-v3-mlp	any Qwen3-VL-8B	non-linear	604 MB	0.9449
mmh3-4b-ClipProj-v3	any Qwen3-VL-4B	matrix only	26 MB	0.9193
mmh3-8b-ClipProj-v3	any Qwen3-VL-8B	matrix only	42 MB	0.9289
mmh3-ClipProj-control-zero	—	control, run it once	52 MB	—
mmh3-ClipProj-control-identity	—	control, run it once	52 MB	—

The v2 files — mmh3-4b-ClipProj-celeb-mlp and the seven beside it — are kept and still work. They were calibrated against a modified 32B and against a corpus containing no image tokens, so prefer v3.

The last column is the one number measured identically for every row: same stock 32B, same prompt, cosine averaged token by token. It is also written inside each file as cos_prompt_reference.

The -v3-mlp files are larger than the v2 residuals — 503 and 604 MB against 304 and 386 — because the hidden width went from 16 384 to 32 768. That is the whole reason for the extra download.

Every matrix works on any variant of its own size: the measured cosine gap between a bf16-calibrated matrix applied to an abliterated fp8 encoder is 0.0023. You do not need the exact checkpoint a matrix was calibrated on. The 8B matrices need an 8B encoder though — 4096 input dimensions instead of 2560 — and the node checks the width and refuses a mismatch.

Named people

This is what changed in 0.1.3, and it was a corpus problem.

The calibration corpus named a person on about 70 lines out of 8632, roughly 0.02 % of the training tokens. The directions of the hidden space that carry an identity were therefore constrained by nothing at all, and the fit put whatever minimised the error on landscape descriptions there. Named people came out as somebody else.

The -celeb matrices add 500 people, ranked by popularity, with five short prompts and two long ones each. What it buys and what it costs:

	name tokens	rest of the sentence	general test set
without	0.8265	0.9358	0.7944
with	0.8844	0.9516	0.7930

Seven thousandths of cosine on the general corpus, for six points on the tokens that carry an identity. The rest of the sentence improves too, because the celebrity prompts are short and the general corpus had nothing under fifteen words.

Two findings that decide how far this is worth pushing.

Two contexts per person are enough. Measured on contexts held out for people the matrix had seen: 0.9875 at two, 0.9945 at five, 0.9986 at twenty. Forty is a waste.

Five hundred names generalise to names never seen. A held-out band at popularity ranks 501 to 540, absent from every calibration, reconstructs at 0.8795 against 0.8844 for the covered ones. Covering 500 people does not teach 500 names; it teaches the map how to handle that region of the space. Going to several thousand would buy very little.

What still fails is not the corpus. Characters whose identity is a mask rather than a face come out as a stranger wearing the right costume. People whose fame predates the era when everything was photographed come out wrong or generic. And some names fail on the plain 32B too, so run the reference before blaming the projection — that check has overturned three of my own conclusions.

Where the calibration data comes from

The general corpus is GokuScraper/seedance-2-prompts-datasets, filtered to prompts of fifteen words or more and deduplicated: 8632 lines, median 128 words. The 500 named people come from a TMDB export published on Kaggle, ranked by popularity, with transliterated names dropped beyond rank 1000.

Around each name, five short prompts are generated from templates, and two longer ones in MiniMax H3's section format are written by Mistral Small and Gemini Flash Lite, half each. Everything needed to rebuild the corpus is in the node's calibration/ folder, including the system prompt the long prompts were written from.

The residual network

The -mlp files carry a d_in → 16384 → 5120 network with a GELU, added to the matrix rather than replacing it. Its last layer is initialised to zero, so at the first step the model reproduces the matrix exactly and can only improve on it. It is worth 0.05 to 0.08 of cosine, four times what multiplying the corpus by eleven buys the linear map.

Which of the two renders better is not settled. The cosine does not predict it — that is the single most repeated lesson of this project. Try both on your own prompts.

Two things measured while building it. Width beats depth: at equal parameter count, two hidden layers of 8192 reach 0.7691 against 0.7944 for one layer of 16384. And a residual extrapolates worse than a matrix does — outside the corpus it saw, a linear map degrades gracefully while the network collapses.

Measured results

Two scales of cosine appear on this page and they do not compare. The table just below is the v2 training-time figure, measured on held-out prompts against that campaign's target. The 0.93–0.94 figures higher up are cos_prompt_reference: one prompt, every projection encoded against the same stock 32B. A number is only ever comparable to another measured the same way — mixing them is how a set of matrices can look like it improved when nothing was established.

The v2 campaign, on its own held-out set:

	4B	8B
matrix, no names	0.7169	0.7528
matrix + residual	0.7944	0.7970
matrix, names covered	0.7095	0.7466
matrix + residual, names covered	0.7930	0.8037

A cosine of 0.79 sounds poor and is not — the DiT tolerates far more than the metric suggests. What holds up in actual generation: simple prompts, structured multi-shot prompts with several distinct cuts and no bleed between them, fl2va with first and last frame, ref2va with a reference image, and since 0.1.3 ref2va with a reference video.

Fidelity does not collapse on short prompts: measured per-token cosine goes from 0.937 at 80 words to 0.908 at 2 words, once the attention sink is handled.

Speech

The first release lost non-English speech: a French line came out half Spanish, and the 8B put everything in English. That was the clearest regression and I could not explain it then.

With mmh3-8b-ClipProj-celeb-mlp, a three-shot clip carrying English, French and Spanish comes out like the 32B does, and the audio level gap measured against the reference has gone from 7.6 dB to 3.5.

Part of what was blamed on the projection was not the projection. A line that fills more than about two thirds of its shot comes out slurred whatever encoder produced the conditioning — the fix is a longer shot, not a better matrix. And MiniMax H3 expects speech wrapped in <d>[Language] ...</d> with a stable speaker id declared beforehand; without that, one voice with one accent is used for the whole clip. Neither of those is documented here because neither is ours, but both cost me a day.

Run the controls first

The two control matrices exist to prove the learned matrix is doing the work rather than the diffusion model. Same prompt, same seed, only the matrix changes:

Matrix	Output for "a red ball on a wood table"
mmh3-ClipProj-control-zero	a countryside landscape — the prompt is entirely ignored
mmh3-ClipProj-control-identity	a golden object in flames — unusable
a learned matrix	the red ball on a wood table

‖W_identity‖ = 50.6 against ‖W_learned‖ = 52.4 — near-identical energy, so the difference is structural, not a matter of scale.

If the identity control ever looks fine, the learned matrix adds nothing — and you want to know that before trusting it.

What is in obsolete/

The previous matrices, kept because a comparison posted on r/StableDiffusion ran on them and the links have to keep working. They have no name coverage and are calibrated on a corpus thirty times smaller. There is no reason to prefer them.

Among them, the CONDPROJ pair, and the story is worth telling because the mistake was instructive.

The DiT does not consume the conditioning as it arrives: it first passes it through condition_proj, a Linear(5120 → 5376) feeding the token refiner. That layer's spectrum is very uneven — a factor of 45 between the top and bottom deciles of its singular values, 52 % of the energy in 10 % of the directions. Plain ridge regression ignores this and spends as much effort on a direction the DiT will multiply by 0.10 as on one it will multiply by 37. Calibrating against the output of that layer instead, then mapping back through the pseudo-inverse, should therefore minimise the error the DiT actually sees. The cosine went from 0.697 to 0.845 on the 4B and 0.731 to 0.860 on the 8B.

Then I compared what the two matrices actually output:

4B   CONDPROJ against unweighted, same corpus     cosine 0.999998
8B   CONDPROJ against unweighted, same corpus     cosine 0.999999

They are the same function. Unregularised least squares is invariant to an invertible linear transform of the targets, so fitting in one space and mapping back recovers the same map; only the ridge penalty breaks that invariance, and with 37 851 training tokens against λ = 1000 it barely binds. The entire gain was an artefact of measuring in a different space.

The idea came from u/stddealer on r/StableDiffusion, and it was a good one. The measurement is on me: I published the cosine before checking whether the matrix had changed at all.

Known limitations

Quantisation costs facts. Comparing int8_convrot against bf16 on factual recall shows errors appearing under quantisation. Fine for general use, worth knowing if your prompts lean on proper nouns.

Masks defeat identity. A character recognised by a costume rather than a face comes out as an unknown person in the right suit. No corpus fixes that, because the identity is not in the name's representation to begin with.

Counting is unreliable, and not because of the projection. Ask for three of something and you get four, on the 32B too. Enumerating works better than announcing a number.

Required models

Role	Model
Diffusion model + VAEs	Comfy-Org/MiniMax-H3
Text encoder, 4B	Comfy-Org/Krea-2 → text_encoders/qwen3vl_4b_fp8_scaled.safetensors
Text encoder, 8B	any ComfyUI-format Qwen3-VL-8B (the 8B matrices expect 4096 input dims)

The 32B text encoder is no longer needed — that is the entire point.

Licence and responsibility

These matrices are released under MIT, like the node.

They are derived from the activations of both models, and their legal status is unclear. They are provided as-is, for research, with no claim of ownership over anything derived from the underlying models.

• Qwen3-VL is published by Alibaba under Apache 2.0. Read and comply with its terms and acceptable-use policy.
• MiniMax H3 ships under a custom licence. Read it before any use, particularly commercial.

This project is not affiliated with, endorsed by, or connected to Alibaba / Qwen, MiniMax, or Comfy Org.

You remain responsible for what you generate and for complying with the licences of every model you load.

Credits

Vibe-coded with Anthropic Claude Code (Opus 5). Every number quoted was measured on real hardware, not estimated: where a prediction turned out wrong, the measurement won and the text was corrected. Three claims in the previous version of this file were wrong and are corrected here.