The Aleph Moves Into a Pretrained Trunk: Relays, Registers, and the Two-Regime Dispatch Law

Autoregressive differentiation, part 2. Sequel to aleph-autoregressive-differentiation-ft1 and closer of the arc that began with the aleph-void keystone. Code, ledgers, checkpoints, and per-experiment sub-articles: AbstractPhil/geolip-aleph-qwen.

---

TL;DR

Ft1 ended inside a bespoke byte-level bed: the aleph address as the sole parameterization of a next-byte distribution, certified structural prior, sign-code parity, the consumption law. This installment asks the question that bed could not: does any of it survive contact with a real pretrained transformer? We froze Qwen2.5-0.5B(-Instruct) — every trunk parameter, permanently — and built the whole program as adapters riding it.

Nine experiments, six shipped packages in the new repo, two nights on one RTX 4090. Eight results:

1. The relay retrofit works at 0.5B. Aleph relay adapters after every frozen block: wikitext ppl 17.798 → 14.09, within 0.16 of a parameter-matched MLP adapter (13.93), at under 1% of trunk parameters, 2 seeds, replication spread <0.02. The ppl ordering inverted from the GPT-2 result (substrate-scoped), but the gate mechanism replicated: relay gates grow ~1.7× (0.047 → 0.081) while MLP gates shrink, on both substrates. And a stability asymmetry the means don't show: the wide-MLP control diverged to NaN in 1 of 2 seeds; across every relay variant in the campaign, divergences were zero.
2. The certified consumer. exp002's tournament settled the relay's internal structure: aleph addressing + a constellation patchwork consumer (relay_pw, ~230k params/block) — the first relay under 14 (13.997 mean, 2 seeds). The consumption law from ft1 held its shape in the new substrate: coverage beats concentration.
3. Registers differentiate, and it is measurable on the sign-code surface. Instruct-task training (grounded JSON extraction in three output registers) reduced grounding failures >10× (specialist 0.979/0.958 vs frozen 0.542, 2 seeds) — and exposed register capture: a single-register specialist answers every register in its own (perfect JSON, 0.0 task-validity elsewhere, both seeds). Multi-task training resolves it (all registers ≥0.875). The register probe — the gauge only the aleph relay has — showed instruct-trained stacks separating the registers ~2× a wikitext-trained stack at depth (inter−intra Hamming 0.22–0.30 vs 0.12–0.15 at L16/L23).
4. Termination is a property of sample semantics, not adapter capacity. Trained on packed 1024-token windows, the relay stack learned to write ~3,000-token grounded composite scenes flawlessly (0.0 malformed vs frozen 0.833) and never learned to stop (truncated 1.000; still 0.938 with the cap lifted to 3000 — true rows average 4,291 tokens, so packed windows almost never contain an ending). Row-aligned samples — each ends with the closing tag — fixed it completely: structural_ok 1.000 / 0.938, truncated 0.0, on 2 seeds. Packed streams train continuation; rows train the artifact.
5. The two-regime dispatch law. A dense signed aleph dispatch (no argmax, no top-k, no load-balancing loss — nothing selects) over frozen specialist stacks behaves in two regimes, decided by representational distance. At register distance the specialists are passengers: a control with random frozen stacks matched the real composite (0.979/0.979/1.0 vs 0.875/1.0/1.0) — the monolith-capacity trainable anchor carries everything. At domain distance (stories vs math vs extraction) the specialists are load-bearing and cleanly decoupled: story prompts put 0.82 of shadow-winner mass on the story anchor, and disabling it degrades stories while math stays perfect — in both directions. The register probe's separation numbers predict the regime in advance. (Each regime rests on 1 seed — a candidate law pending replication.)
6. The predictability principle. The same training recipe taught held-out function evaluation (+25pt on unseen f(x) instances, 0.688 → 0.938) and failed to teach a checkable keyword constraint (0.083, exactly frozen level, despite training on compliant stories). The LM loss teaches exactly what target-token predictability demands: math answers are unpredictable without using the question; story tokens are predictable without reading the instruction. (exp006/exp007, 1 seed each.)
7. A frozen text trunk can learn discrete image denoising. CIFAR-10 as 256 fixed-palette tokens (nothing fitted), uniform-replacement corruption, causal denoising through one relay stack: beats the copy-the-corrupted-token baseline at every noise level (1 seed) — 0.527 vs 0.252 at t=0.75, 0.378 vs 0.002 at pure noise. Two honest halves ride along: greedy iterative decoding mode-collapsed to the all-dark modal image (diversity 0.0 — a decoding property, not a model property; stochastic sampling restored diversity 0.92 and image-like layouts), and class-conditionality is not demonstrated (conditional ≈ shuffled histograms).
8. The family. Five frozen specialists (three extraction registers, story, math) plus one trainable anchor under one dense signed dispatch: eleven sub-competences at ~specialist level through a single routed model — including math code-register correctness at 1.0, above the specialist's own 0.938, and a 3× lift on the stubborn keyword register. One model that extracts, narrates, and computes, with surgical domain-level decoupling (1 seed; replication is the named next rung).

Every number above is asserted from a JSON ledger by a build_results.py shipped in its package; every package is standalone-reproducible from inside its folder.

---

1. Why leave the bespoke beds

The differentiation line (exp011–exp021, now on hold as a finished measurement program) produced laws but no artifact: enrichment lives in the aggregation channel; the address bottleneck is a prior, not a tax; memorization and generalization order inversely; durable memory needs a write-time-frozen key encoder. The honest assessment mid-program was that we were measuring a system we weren't building. The pivot: compose the certified components on a real pretrained substrate and see which laws are substrate properties and which were artifacts of tiny beds.

The substrate is deliberately modest — Qwen2.5-0.5B, fp32, one consumer GPU — because the question is architectural, not scale-flexing: what can a frozen trunk be taught through a geometric adapter surface alone?

2. The bed

Every experiment shares one skeleton. The trunk — Qwen2.5-0.5B(-Instruct) (Qwen Team, 2024) — loads frozen (requires_grad=False, asserted post-run: zero gradient ever reaches it). After every one of the 24 transformer blocks, an adapter reads the block's output and adds a gated residual. The lineage is the adapter/PEFT family (Houlsby et al., 2019; Hu et al., 2021), and the start-silent gating discipline has good company — Flamingo's tanh-gated cross-attention (Alayrac et al., 2022), ControlNet's zero convolutions (Zhang et al., 2023), LLaMA-Adapter's zero-init attention (Zhang et al., 2023b) — but the adapter body here is not a bottleneck MLP. The certified adapter is relay_pw: a projection into 16 four-dimensional slots, each slot addressed by the aleph closed form

M̂ = Σₖ sinh(uₖ)·Aₖ / Σₖ cosh(uₖ),   uₖ = cos(x, Aₖ)/τ

over 64 oriented half-axes — the addressing object whose code-verified core is the keystone article, inheriting the attractor geometry and binding constant of structural attractors — followed by a patchwork consumer (SquaredReLU expansion, LayerNorm, zero-init output; the constellation lineage from the diffusion-bottleneck article) entering through a gate initialized at −3.0 — adapters start nearly silent and must earn amplitude. ~230k parameters per block, ~5.5M per stack, ~1% of the trunk. Pure Adam, lr 1e-3, weight decay 0 (the optimizer discipline certified in geometric memory ft3: weight decay destroys the geometric harmonic — the geometry is the regularization). Perplexity work runs on wikitext-103 (Merity et al., 2016). No selector appears anywhere in any forward pass in this program; where multiple anchors must be mixed (exp005/exp009), the mixture is the dense signed dispatch described in §6, and the per-token winner is logged read-only.

The discrete surface comes free: each slot's winner half-axis and orientation is a sign code, cacheable per token per layer. The register probe (§4) and the two-regime prediction (§6) both read this surface; a parameter-matched MLP adapter has nothing comparable — which is the practical argument for the aleph design beyond raw ppl.

3. Retrofit and refinement (exp001, exp002)

exp001 asked the bare question: do relays retrofit? Five runs, two seeds: frozen 17.798 → relay 14.09; the parameter-matched MLP control 13.93. The GPT-2 ordering (relays beat MLPs) inverted here — worth stating plainly: which adapter wins on ppl is substrate-scoped. What replicated across substrates is the mechanism: relay gates grow ~1.7× while MLP gates shrink, and relay training never diverged where the wide MLP control lost 1 of 2 seeds to NaN.

exp002 ran the architecture tournament (12 runs): multi-tau strobing, depth, width, ordering, consumers. Verdicts: relay_pw certified at 13.997 (first sub-14 relay, 2 seeds); width and strobe saturated; coverage beats concentration (spreading the budget across all 24 blocks beat concentrating it on the deepest 12, −0.48 ppl) — the consumption law's signature, surviving translation from bespoke byte-beds to a pretrained trunk; ordering budget-stable at 6k steps.

4. Registers and the discrete gauge (exp003)

The instrument: 22k COCO captions (Lin et al., 2014) rendered through three structurally distinct output registers (grounded literal extraction / bracketed generics / positional placeholders) in native messages+tools format, caption-hash holdout, validators re-implemented from the dataset's own rules (json-coco-format).

Hallucination — ungrounded content in generation, the standing failure mode surveyed in Ji et al., 2023 — is measured here as a per-leaf grounding rate, and the reduction is quantified: the task_1 specialist grounds 0.979/0.958 of held-out extractions vs frozen 0.542 — grounding failures cut from 46% to 2–4%. Register capture, discovered: that same specialist answers every register in its own — perfect JSON shape, 0.0 task-validity — both seeds, with cross-register ppl damage. Multi-task training resolves capture (all registers ≥0.875).

The probe is the part that only exists because the adapter is an aleph: sign-codes collected at four depths for the same captions under four registers; separation = inter-register minus intra-register Hamming. Instruct-trained stacks: 0.22–0.30 at L16/L23. Wikitext-trained stack, same architecture: 0.12–0.15. Instruct training reorganizes the discrete surface into differentiated registers — and this gauge is what later predicts dispatch behavior (§6).

Instrument honesty, ledgered: the first validity pass used a non-greedy regex that truncated nested tool-call JSON — a perfect generation scored 0. Caught via a ppl-1.05-but-validity-0 contradiction; v2 validators re-judged every checkpoint post-hoc; both generations of rows ship in the ledger.

5. Termination is sample semantics (exp004)

Scaling the register from ~200-token outputs to multi-kilotoken composite scenes produced the cleanest process finding of the program. The targets are fused multi-entity JSON from a deterministic-first vision pipeline — Grounding DINO (Liu et al., 2023), SAM (Kirillov et al., 2023), Depth Anything V2 (Yang et al., 2024), SigLIP (Zhai et al., 2023) fused into one grounded scene per image (qwen-deepfashion-fused, qwen-synth-characters-fused); polygons excluded.

Act 1 (packed 1024-token windows): the relay learned the composite register flawlessly — json_error 0.0 vs frozen 0.833, held-out ppl 4.063 → 1.427 — and truncated every single generation at the 1400 cap. Lifting the cap to 3000 recovered almost nothing (0.938 still unclosed), and the one scene that closed was structurally near-perfect. The v2 cache then measured what the v1 estimate had missed: true rendered rows average 4,291 tokens (max 9,032). A packed 1024 window almost never contains an ending; stopping was never in the gradient. (Sequence packing is standard practice for throughput — Krell et al., 2021 — but its semantics for what a small adapter can learn about document boundaries is exactly what this gate measured.)

Act 2 (row-aligned SFT: every sample is one complete conversation ending with the closing tag, ≤4096 tokens, over-cap rows skipped and counted): structural_ok 1.000 (16/16) on seed 0, 0.938 on seed 1 (one malformed JSON; truncation 0.0 on both). The identical architecture that scored truncated 1.000 under packed windows scores ok 1.000 under rows. For long structured emission through frozen-trunk adapters, window semantics are part of the task definition.

The caveats ship asserted: entity over-emission (1.667/1.311× the reference — the model invents plausible extras) and weak caption-grounding. Construction is solved; faithfulness is the next line.

An ops law fell out too, the expensive way: at 4096-token rows with an adapter after every block, fp32 activations plus the seq×vocab logits exceed 24GB — and the Windows driver's sysmem-fallback silently spills to shared memory instead of OOMing (42.8GB observed; steps at PCIe speed; the tell is ~100W draw at "100% util"). Gradient checkpointing + chunked cross-entropy + a hard memory-fraction cap: 66GB spill → 8.8GB peak at 1.03s/step. The shipped beds print peak memory at step 50.

6. The dispatch, its confound, and the two-regime law (exp005, exp009)

Mixture-of-experts, in its standard form, is a selector: a gate picks top-k experts and a load-balancing loss fights the collapse that selection invites (Shazeer et al., 2017; Fedus et al., 2021). This program's standing failure class forbids exactly that — comparative selectors near the aleph collapse its paths — so the question is asked doctrine-legally, closest in spirit to fully dense soft mixtures (Puigcerver et al., 2023) but with signed weights and a geometric key: anchors are whole frozen relay stacks; per block, mixing weights are wₖ = sinh(uₖ)/Σⱼcosh(uⱼ) with u = cos(Px, C)/τ, key projection frozen at init (the write-time-frozen key law from the predecessor line's exp021, satisfied by construction), codebook trainable, weights dense and signed. No selection event exists, and no load-balancing loss — nothing pushes usage toward uniformity except the task mixture itself; the argmax|u| "shadow winner" is logged read-only.

exp005 (three frozen register specialists + one trainable anchor, 1 seed): the routed composite serves all registers (0.875/1.0/1.0), usage stays dense (perplexity 3.74–3.91 of 4, all paths alive — no collapse), the shadow winner leans to the domain anchor. The headline died in review before it shipped: the trainable anchor is itself monolith-capacity (5,524,008 params ≈ the multi-task stack's 5,517,864) trained on all registers interleaved — the exact recipe that already resolved capture — and the composite's numbers equal that monolith's seed-0 numbers exactly. So the package shipped with its own control: specialists replaced by random frozen stacks. The control matched (0.979/0.979/1.0). At register distance, the specialists were passengers; removal of one even ticked its neighbors up.

exp009 (1 seed) re-ran the architecture where the anchors are far apart: three extraction registers + a story anchor (four request registers) + a math anchor (four ask-registers), five frozen stacks + one trainable, one dispatch. Eleven sub-competences through one model: extraction 0.854/1.0/1.0; stories 1.0 on continuation/instruct/JSON; math perfect on all four registers including code at 1.0 — above its own specialist. Usage concentrated without a selector: story prompts 0.820 on the story anchor, math 0.508 on math, extraction registers blended. And the decouple tests are surgical in both directions: story anchor off → stories fall (instruct 1.0 → 0.667) while math stays perfect everywhere; math anchor off → math degrades (code 1.0 → 0.75) while stories hold untouched.

Together: whether a frozen anchor carries load or rides along is a function of representational separation — and the sign-code register probe measures that separation before you ever build the dispatch. Registers (sep 0.2–0.3) blend; domains (sep 0.35–0.5) specialize. The MoE design variable is not the gate; it is the distance between what the anchors know. Both regimes stand on 1 seed each — a candidate law, not yet replicated.

7. The predictability principle (exp006, exp007)

Two anchors (1 seed each) trained by the identical recipe, judged by checkable constraints:

• Story anchor (TinyStories, Eldan & Li, 2023; four request methods): continuation, instruct, and JSON registers all reach 1.0 validity (frozen: 0.667/0.833/0.833). The keyword register — "write a story using these three words," fully checkable — stays at 0.083, exactly frozen level, despite every training story containing its keywords.
• Math anchor (synthetic, four ask-methods, ground truth computable): format locks to 1.0 everywhere (frozen NL format: 0.312), correctness never degrades, and held-out f(x) evaluation improves 0.688 → 0.938 (audited: 5 of 16 judged code problems have zero recall channel — unseen coefficient combinations).

Same trunk, same adapter, same optimizer, same steps. The difference is in the targets: an arithmetic answer is unpredictable without using the question, so the loss routes computation through the adapter; a story's tokens are predictable without reading the instruction, so the constraint is ignorable and stays unlearned. Teacher forcing teaches what target-token predictability demands — nothing more. This is teacher forcing's known blind spot (Williams & Zipser, 1989; the train/inference mismatch that motivated scheduled sampling, Bengio et al., 2015) meeting shortcut learning (Geirhos et al., 2020) in its cleanest form: the shortcut is ignoring the instruction entirely, and a matched-everything pair of tasks makes it measurable. (The family's mixture pressure later tripled keyword compliance to 0.25 — real but far from solved; a keyword-first target format is the recorded follow-on.)

8. Images as tokens through a text trunk (exp008)

The most speculative directive — a CIFAR-10 diffusion model in tokens through the gated adapter, if possible — gets a split verdict (1 seed), all halves ledgered. The reference points: images as autoregressive token streams is iGPT (Chen et al., 2020); discrete diffusion over token corruption is D3PM (Austin et al., 2021) and its masked-parallel cousin MaskGIT (Chang et al., 2022); and frozen language models acquiring non-text competence through small trainable surfaces is the Frozen/universal-computation line (Tsimpoukelli et al., 2021; Lu et al., 2021). This bed sits at their intersection with one deliberate deviation: no fitted tokenizer — where iGPT used k-means color clusters and the VQ family learns its codebook (van den Oord et al., 2017), this program's failure class forbids fitted quantizers near the aleph, so the palette is fixed arithmetic: CIFAR-10 (Krizhevsky, 2009) at 16×16, 8 uniform levels per channel, palette id = 512 fixed vocabulary slots. Corruption replaces tokens uniformly at rate t; the training sample is [class, level, corrupted×256, SEP, clean×256] with loss on the clean half; inference walks the noise schedule down.

The denoiser is real: teacher-forced reconstruction beats the identity baseline (copy the corrupted token: (1−t)+t/512) at every level — 0.840 vs 0.751 (t=0.25), 0.689 vs 0.501, 0.527 vs 0.252, and 0.378 vs 0.002 at t=1.0, where the corrupted view carries nothing and reconstruction comes entirely from learned position/smoothness/class priors. The frozen trunk's own zero-shot 0.259 at t=1.0 is a finding in itself — an emergent copy-local-context strategy.

The two honest halves: greedy argmax in the iterative refinement loop mode-collapsed to a single all-dark image (diversity 0.0; conditional == shuffled to four decimals) — deterministic decoding in an iterative denoiser collapses to the global mode, the visual sibling of greedy text degeneration (Holtzman et al., 2019), and a decoding property: multinomial sampling from the same checkpoint restores diversity 0.92 — horizon lines, sky/ground splits, coherent palettes in the shipped grids. And class-conditionality is not demonstrated: conditional histogram distance ≈ shuffled. A frozen text trunk + 5.5M adapter learned image statistics, not yet image classes.

9. What ships, what's next

Six packages (exp004_composite through exp009_family), each with scrubbed standalone code, a self-asserting results builder run green before upload, the full JSON ledger including every failed act and instrument bug, checkpoints, and a sub-article. Two nights, one 4090, ~60 logged GPU runs.

The recorded next rungs: the family with the image-token anchor joined; sub-capacity trainable anchors (make specialists load-bearing at register distance); keyword-first targets for constraint binding; faithfulness training against entity over-emission; and the geofractal tower port — the family's anchors are homogeneous relay stacks, which is exactly what the tower machinery wants.

The aleph started this arc as a codebook differentiation device in toy reconstruction beds. It ends this installment as the addressing surface of a composable adapter family on a frozen pretrained transformer — with its discrete sign-code gauge doing real predictive work that a parameter-matched MLP cannot offer: telling you, before you build the mixture, whether your experts will carry or coast.

---

This program's corpus

The aleph line is self-published and cumulative; the installments this article stands on directly:

• aleph-autoregressive-differentiation-ft1 — the predecessor: bottleneck prior (7/7), the consumption law and the comparative-selector pathology this program's dispatch design obeys, sign-code parity, shell transit, projective objective-independence.
• geometric-vocabulary-patchwork-aleph-void — the keystone: the aleph closed form, code-verified.
• structural-attractors-in-latent-geometry — the binding constant and the attractor geometry the addressing objects inherit.
• constellation-diffusion-bottleneck — the 16s constellation system the patchwork consumer descends from.
• geometric-tri-band-ft2 — the projective codebook law, carried here unchanged.
• geometric-memory-ft3 — the optimizer discipline (pure Adam, wd 0): weight decay destroys the geometric harmonic; the geometry is the regularization.
• geolip-svae-transformer-ft1 — the two-year differentiation survivor: reconstruction-cultivated sign codes, the origin of "read the signs, not the probabilities."

Repositories: geolip-aleph-qwen (this line: code, ledgers, checkpoints, per-experiment sub-articles) and geolip-aleph-differentiation (the measurement program, exp011–exp021). Datasets: json-coco-format, qwen-deepfashion-fused, qwen-synth-characters-fused.

References

• Alayrac, J.-B., et al. (2022). Flamingo: a Visual Language Model for Few-Shot Learning. arXiv:2204.14198
• Austin, J., Johnson, D. D., Ho, J., Tarlow, D., & van den Berg, R. (2021). Structured Denoising Diffusion Models in Discrete State-Spaces. arXiv:2107.03006
• Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015). Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. arXiv:1506.03099
• Chang, H., Zhang, H., Jiang, L., Liu, C., & Freeman, W. T. (2022). MaskGIT: Masked Generative Image Transformer. arXiv:2202.04200
• Chen, M., et al. (2020). Generative Pretraining from Pixels. ICML 2020. PMLR 119
• Eldan, R., & Li, Y. (2023). TinyStories: How Small Can Language Models Be and Still Speak Coherent English? arXiv:2305.07759
• Fedus, W., Zoph, B., & Shazeer, N. (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961
• Geirhos, R., et al. (2020). Shortcut Learning in Deep Neural Networks. arXiv:2004.07780
• Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019). The Curious Case of Neural Text Degeneration. arXiv:1904.09751
• Houlsby, N., et al. (2019). Parameter-Efficient Transfer Learning for NLP. arXiv:1902.00751
• Hu, E. J., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685
• Ji, Z., et al. (2023). Survey of Hallucination in Natural Language Generation. arXiv:2202.03629
• Kirillov, A., et al. (2023). Segment Anything. arXiv:2304.02643
• Krell, M. M., Kosec, M., Perez, S. P., & Fitzgibbon, A. (2021). Efficient Sequence Packing without Cross-contamination: Accelerating Large Language Models without Impacting Performance. arXiv:2107.02027
• Krizhevsky, A. (2009). Learning Multiple Layers of Features from Tiny Images. Technical report
• Lin, T.-Y., et al. (2014). Microsoft COCO: Common Objects in Context. arXiv:1405.0312
• Liu, S., et al. (2023). Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. arXiv:2303.05499
• Lu, K., Grover, A., Abbeel, P., & Mordatch, I. (2021). Pretrained Transformers as Universal Computation Engines. arXiv:2103.05247
• Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016). Pointer Sentinel Mixture Models. arXiv:1609.07843
• Puigcerver, J., Riquelme, C., Mustafa, B., & Houlsby, N. (2023). From Sparse to Soft Mixtures of Experts. arXiv:2308.00951
• Qwen Team (2024). Qwen2.5 Technical Report. arXiv:2412.15115
• Shazeer, N., et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538
• Tsimpoukelli, M., et al. (2021). Multimodal Few-Shot Learning with Frozen Language Models. arXiv:2106.13884
• van den Oord, A., Vinyals, O., & Kavukcuoglu, K. (2017). Neural Discrete Representation Learning. arXiv:1711.00937
• Williams, R. J., & Zipser, D. (1989). A Learning Algorithm for Continually Running Fully Recurrent Neural Networks. Neural Computation, 1(2), 270–280. doi:10.1162/neco.1989.1.2.270
• Yang, L., et al. (2024). Depth Anything V2. arXiv:2406.09414
• Zhai, X., Mustafa, B., Kolesnikov, A., & Beyer, L. (2023). Sigmoid Loss for Language Image Pre-Training. arXiv:2303.15343
• Zhang, L., Rao, A., & Agrawala, M. (2023). Adding Conditional Control to Text-to-Image Diffusion Models. arXiv:2302.05543
• Zhang, R., et al. (2023b). LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention. arXiv:2303.16199

Code and ledgers: geolip-aleph-qwen. Predecessor line and laws: geolip-aleph-differentiation.