Dean Byrne PRO

Hey

Two landed yesterday while I was building a GPU buffer pool, and I got to bin them live.

Bug one: pooled readbacks came back bucket-sized instead of logical-sized and a Float32Array.set exploded on the first step. Shape. Loud, cheap, dead in five minutes. Your prior holds: the math-adjacent ones announce themselves.

Bug two is the interesting one. Pooled buffers are not zero initialized, and the rowmax kernel accumulates with atomicMax assuming zeros. Here is the trap: every buffer is FRESH on its first acquisition, and the init gates run first. So the gates pass on virgin zeroed memory, then step two reuses a dirty buffer and the scales go quietly wrong. A gate that provably cannot see the bug class, not because it is weak but because it runs at the wrong time. That is not shape, scale, checker, or liveness. It is state: wrong only after reuse. Fifth bin, or liveness's little sibling, since both live on the time axis and both are invisible to anything that checks a single call once.

I caught it in review, and the continuous audit would have caught it live, because it recomputes random cells of the real GEMMs mid-training, so it samples the dirty steps and not just the virgin one. The instrument for the state axis already existed. I just had not named the axis it was covering.

So, before the fix: if it is loud, bet shape or scale and relax. If it is quiet and the init gates are green, bet the time axis, state or liveness, and reach for whatever samples repeatedly instead of once. And your tower note carries: the audit that owns the state axis is itself mutation-tested, a planted wrong cell must trip it, so it joins the fixed point with the rest.

You called it, and funnily enough I got there a few hours before your message landed. Two anchors shipped: relu output can't go negative, and at unit scales the output IS the integer dot product. Same species as your matmul(A, I) = A. Your corpus went 0/4 to 2/4 to 4/4, and the last two took exactly what you said: stop writing relations, start pinning values. The c*out argument is a theorem and I stopped fighting it.

Then I ran your harder question, because it was the right one. My four July bugs, scored against everything I have:

properties 0/2. Both data bugs are the theorem again. The stripped binding is a per-column scalar, the round-once sum is a last-ulp schedule change, and my shiny new anchor sits at unit scales, which is precisely where both are invisible. The exact differential gate catches both.

But that's only half the corpus. The dead gate is a bug in a checker. No oracle sees it, only mutation-testing the gate does. The roster stall is a bug in the protocol. Every value on every peer was correct, so no data oracle CAN fire. Caught it with a liveness sim: drop one gradient asymmetrically, old protocol parks forever, repair protocol finishes with identical weights.

So, the honest answer to "what does your own corpus score" is: my oracles go 2/4, and the other two needed instruments that aren't oracles at all. The population defines the instrument. Your bugs taught me anchors. Mine taught me to test the tests and the protocol. It's test_selfcorpus.js in the repo now, red bar and all, so the next time I feel good about a scoreboard I can check whose bugs it was measured on.

Late reply. The work got done before the words did.

Corpus went in. Properties scored 0 out of 4.

Best failure: your multiple-of-128 bug zeroed the whole output and every property passed, because zero permutes fine, preserves zero rows, and decomposes across batches. My suite could be aced by doing nothing. Worse: both mutants I wrote myself were structural, so my mutation test had the same author problem as my kernel. You called that one from across the room.

Added non-triviality and sensitivity. Now 2/4, and the split is clean: 2/2 on loop bugs, 0/2 on math bugs. Differential gate 4/4. So properties are a loop-bug detector, full stop. There is no property that catches a uniform 2x without a reference. I stopped looking.

The oracle got built. BigInt, straight from the standard, no fround inside. Agrees with the mirror on 500k inputs, rejects my old round-once mirror on 34% of them. It caught a bug on its first run: mine, in its own harness. (a < 0) misses -0. Even the referee needed a referee.

Your closing question: no, the spec oracle does not fix indexing, and here's the tally you asked for. This month's bugs: a dead gate, a stripped binding, a sum that rounded once where it should round three times, and a gradient that reached the leader but not one follower and parked a live fleet for hours. That last one isn't math or indexing. It's delivery. Almost none of my bugs live where the oracles look. The oracles just make that claim checkable instead of comfortable.

One more, in your key: new GPU quantize kernel passed its exact gate, then I realized the old spec would have passed too, since random data never hits a rounding boundary. Searched 4000 inputs for one where the specs disagree, found it, GPU sided with the new spec. Mutation-score the gate. Allclose all the way down.

test_corpus.js has your bug names in the output. Partial is a number now. It was worse than the feeling, which is the point.

Last time I thought the weight hash covered kernel correctness. It doesn't, it covers replica agreement. Now you're pointing out the probe covers backend agreement, and I called that correctness too. Same mistake, one floor up. I keep looking at two things matching and reading it as two things being right.

The referee question is the good one, and the answer is: me. Which is not a great answer.

In my defense, for about one second, the rounding fix wasn't quite "shader disagreed so I changed the mirror." The WGSL spec says f32 rounds after every op, and my mirror was doing the whole chain in f64 and rounding once at the end. So, the spec is what told me the mirror was wrong. The GPU just happened to agree. But nothing in the repo enforced that. It was me, with a red test in front of me, wanting it green. The difference between "spec says the mirror is wrong" and "shader says the mirror is wrong" lived entirely in my head, and I know which one I'd reach for on a bad day.

One small pushback. The mul8 table isn't a reimplementation, it falls out of the Python side as a trained artifact, and the DP4A kernel never touches it. So, table vs hardware is a real differential test with a referee neither side wrote. That part I think survives. What doesn't survive is the loop around it. The indexing, the strides, the accumulation, all of that came out of the same head as the shader on the same afternoon. Two implementations, one brain, same blind spots. You're right and I've got nothing.

So, I went looking for something that doesn't need a second implementation at all, and it turns out you can just use properties. A zero row of A has to give a zero row out. Permute the rows of A and the output rows permute the same way. A batched call has to match running each one on its own. None of that comes from anyone's version of the thing, it comes from what the thing is. And when one fails there's no referee argument to have. It's just wrong.

They bite, too. The batch one catches a dropped stride with nothing to compare against. The permutation one catches swapped rows. And a bug sitting in shared WGSL, the exact thing the probe can't see, can't hide behind agreement, because every replica breaks the property at the same moment.

Catch is they're partial. My zero-row property walks straight past the row-swap bug that the permutation one catches. You only cover what you thought to write down, which is your original point again, one floor further down. This building has a lot of floors.

Next up is your actual tiebreaker: an IEEE-754 oracle done in exact integer math, straight from the standard, so the spec is something that runs instead of something I remembered right once.

Doesn't fix the author problem though. I wrote the kernel, the mirror, and the properties. That's what an outside corpus does that nothing in my repo can. Asking again in case it got lost in the noise the first time.

As always, I am appreciative of your comments. They always make me check my work.

Short answer: once, at startup. Longer answer: I went and re-read my own gates instead of answering from memory, and it's worse than I would have told you.

The one gate I had that was actually exact (a plain != check, no tolerance, since int8 x int8 into int32 has no rounding to hide in) was sitting on a kernel my transformer stopped calling when I switched to block-scaled quantization. It still ran at boot. It still passed. It still printed "HW verified vs units" in the UI. It was verifying a code path that does no work. That's worse than having no gate, because it looks like coverage.

Why my live kernels only had tolerance checks is the part I think you'll appreciate. Fusing the dequant epilogue into the kernel is what destroyed the exact oracle. The int32 accumulator gets multiplied by the block scales in f32 inside the kernel and never comes back to the host, so there was nothing exact left to compare against. I bought throughput and paid for it in observability, and I didn't notice the invoice until you asked.

The fix was to emit two variants of each shader from the same source string, where the only difference is the final write (one does the epilogue, one dumps the raw int32). Same indexing, same instruction, so gating the verify variant actually gates the real kernel. Then the surprising bit: once I made my JS mirror round the way WGSL rounds (f32 after the int cast, f32 after each multiply, instead of doing the whole chain in f64), the epilogue passed an exact check too, on real hardware. The tolerance was never covering hardware variance. It was covering a bug in my own mirror.

But the part you were really getting at is the one I had backwards. I'd been thinking of the weight hash as a partial backstop on kernel correctness. It isn't one at all. Weights only depend on the gradient bytes everyone receives, so if one device has a broken kernel it broadcasts a bad gradient, everybody averages the same bad bytes, and every replica stays bit identical and perfectly happy. There is no kernel disagreement the weight hash can detect. It's not a weak check on that axis, it just isn't a check on that axis.

So there's a canonical probe now. A fixed seeded GEMM run through each device's live kernel, hashed on the raw int32 accumulator (exact on every backend), sent with every gradient and re-run every 25 steps. Same input plus correct kernels means the same number whether you're on a phone GPU or a CPU fallback. My CPU mirror and my GPU DP4A path both come out to 2186402845, which is honestly the first evidence I've had that those two agree. There's also a random cell audit that re-checks live GEMMs against the units mid run at the real shapes, since toy shapes at boot were never going to catch thermal or shape dependent drift.

Then I did the thing your corpus implies and mutation tested my own gate. Five injected bugs (dropped batch stride, short K loop, missing column scale, dropped ReLU, wrong rounding) all rejected. The old allclose gate passes the rounding one that the exact gate catches, so your point reproduced itself in my own code. Live fault injection: 1.08e-7 drift, an order of magnitude below the old tolerance, caught mid run at 256x32x32.