ceselder/nla-qwen36-27b-matryoshka

Matryoshka Natural Language Autoencoder — Qwen3.6-27B (layer 42)

A matryoshka NLA trained on Qwen3.6-27B residual-stream activations (layer 42 of 64, d=5120): the activation verbalizer (AV) is RL-trained with random-length truncation of its explanation before reconstruction (U[1,120] content tokens, shared per GRPO group), so the most important information is pushed to the front of every explanation.

Built with EasyNLA (branch qwen36-matryoshka), following the NLA recipe of Anthropic (2026) with the matryoshka truncation reward + position-tapered KL.

Results (400 GRPO steps, held-out)

prefix seen by reconstructor	FVE
first 10 tokens	42.9%
first 20 tokens	51.9%
first 40 tokens	58.0%
first 80 tokens	61.5%
full explanation	64.2%

The first 10 tokens carry 67% of the full-explanation FVE — explanations open with the single most predictive fact (usually the immediate next-token completion) and broaden from there.

Contents

path	what
rl_av_lora_iter400/	the matryoshka AV — LoRA (r128, attn + DeltaNet projections) on the text-only base, RL step 400
rl_critic_step400/	co-trained AR reconstructor (43-layer truncated backbone + value head) at step 400
warmstart_av_lora/	AV after SFT warm-start only (pre-RL baseline)
warmstart_ar_critic/	AR after SFT warm-start (48.2% held-out FVE on gold explanations)

Note: LoRAs apply to a text-only conversion of Qwen3.6-27B (Qwen3_5ForCausalLM: wrapper keys model.language_model.* remapped to model.*, vision tower + MTP head dropped — bit-identical text logits). The AR critic dirs are self-contained (truncated config + value_head.safetensors), loadable with EasyNLA's NLACriticModel.from_pretrained.

Injection: marker ㈜ (id 158983), Karvonen norm-matched addition at layer-1 output. Prompt format: bullets (untagged newline list), enable_thinking=False.

Training data: ceselder/nla-qwen36-27b-matryoshka-data.