MUGEN

A Unified Framework for Efficient Motion Understanding and Generation

MUGEN is a unified motion-language model. One model turns a description into human motion, and describes an observed motion in words. Both directions share a single motion representation, and that representation is continuous: no codebook, one draw.

• 📦 Code: github.com/JYe16/MUGEN
• 🏋️ This checkpoint: HumanML3D, K = 2 latent slots, 183.7M parameters

---

What makes it different

Unified motion-language systems have traditionally coupled generation and understanding through a shared discrete motion codebook, but quantization limits generation quality. The strongest generators buy that quality back at growing cost: stacked residual codebooks enlarge the representation, and masked decoding stages, long autoregressive rollouts and denoising chains of tens to hundreds of steps stretch inference. None of that decoding machinery serves understanding.

MUGEN pays neither cost. Generating a motion costs K language-model steps, one draw, and one decoder pass — with K = 2 here, that is two rollout steps.

Architecture

MugenForConditionalGeneration                                    183,690,603 params
├── motion_autoencoder (AdaptiveLengthAutoEncoder)                    48.9M
│   ├── encoder_backbone   dilated Conv1d ResNet, per-frame features
│   ├── latent_queries     K learnable queries -> the K slots
│   ├── encoder_blocks     4 x cross-attention (frames -> slots)
│   ├── decoder_blocks     4 x cross-attention (slots -> frames)
│   └── decoder_refiner    dilated Conv1d ResNet -> 263-d motion
├── language_model (GPT2LMHeadModel)                                  124.4M
│   └── 12 layers, 768 hidden, vocab 50261 (GPT-2 + <MOT> + 3 reserved)
├── layer router                                                        9.1M
│   ├── text_mem_proj      prompt tokens -> routing memory
│   ├── cross_attn_blocks  2 x cross-attention, per-slot routing queries
│   ├── static_router_logits   (K, 12) text-independent component
│   └── layer_router       MLP -> (K, 12) text-conditional component
├── calibrated latent head                                              0.9M
│   ├── projector          hidden -> (mu, logvar) per slot
│   ├── latent_factors     (K*512, 64) global covariance basis U
│   └── factor_scale_head  hidden -> per-factor log amplitudes
└── motion_in_projector    latent slots -> GPT-2 embeddings (captioning)  0.4M

Adaptive-length autoencoder. Cross-attention compresses a clip of any length into K = 2 continuous vectors of width 512, and a second cross-attention stack expands them back to any requested frame count. Decoder queries encode a frame's relative phase within the clip rather than an absolute frame index, which is why one decoder serves every length, and why you can decode the same latents at 60 frames and at 200 frames.

Depth-routed hidden states. A text-conditioned router gives each latent slot its own soft mixture over all twelve transformer layers, so a slot reads from the depth it needs instead of squeezing every piece of motion evidence through the final layer. Both halves of the routing logit are tanh-bounded, so no logit margin can saturate the routing softmax.

Calibrated latent head. The head predicts N(mu, U diag(a)^2 U^T + diag(sigma^2)) over the whole flattened K x 512 latent, with rank 64, trained by exact maximum likelihood. One draw therefore carries text-conditional variance that is correlated across slots, which is what a single-step sampler has to supply all at once.

Motion features	HumanML3D, 263-d per frame, 20 fps, 22 joints
Latent slots	K = 2, width 512
Language model	GPT-2 (124M), fine-tuned
Covariance rank	64
Training	Joint generation + understanding, 240 epochs (Stage 2)
Parameters	183,690,603
Precision	float32

---

Installation

pip install torch transformers safetensors numpy

trust_remote_code=True is required: the autoencoder, the router and the calibrated head are custom modules that ship with this repository.

Nothing else is needed. The model carries its own tokenizer and its own HumanML3D feature statistics, so it returns motion in real units without you downloading the dataset.

---

Quick start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "zy22b/MUGEN", trust_remote_code=True
).eval()
tokenizer = AutoTokenizer.from_pretrained("zy22b/MUGEN")

features = model.generate_motion(
    ["a person walks forward and then waves with the right hand."],
    lengths=[120],              # frames at 20 fps, so 6 seconds
    tokenizer=tokenizer,
)
print(features.shape)                       # torch.Size([1, 120, 263])

joints = model.features_to_joints(features)
print(joints.shape)                         # torch.Size([1, 120, 22, 3]) metres

print(model.generate_caption(features, tokenizer=tokenizer))
# ['a person walks forward, turns to the left, and walks back.']

That last line captions the motion the model just generated, so it changes from call to call: the motion is sampled, not a fixed function of the prompt.

---

Inference tutorial

1. Text to motion

prompts = [
    "a person walks forward and then waves with the right hand.",
    "the person jumps up and lands with both feet together.",
    "a man slowly sits down on a chair.",
    "a person runs in a circle to the left.",
]

features = model.generate_motion(prompts, lengths=120, tokenizer=tokenizer)

generate_motion returns (batch, frames, 263) denormalised HumanML3D features.

Argument	Meaning
texts	one description, or a list of them
lengths	frames per description at 20 fps; an int applies to the whole batch
temperature	multiplier on the sampled perturbation; defaults to the calibrated 1.0
generator	a torch.Generator for a reproducible draw
denormalize	True (default) returns raw HumanML3D units

Choosing a length. HumanML3D clips are multiples of 4 frames and at most 196 (9.8 s), so stay in [20, 196] and on the multiple-of-4 grid to stay in distribution. The decoder will happily produce other lengths, but you are then outside what it was trained on.

Per-row lengths. Because decoder queries are relative-phase, a row must be decoded at its own length rather than cropped from a longer one. Pass a list and the model handles it; the returned tensor is zero-padded to the longest row.

features = model.generate_motion(prompts[:2], lengths=[60, 196], tokenizer=tokenizer)
# (2, 196, 263); row 0 is valid up to frame 60

2. Temperature, and why the default is 1.0

The head predicts a distribution, and temperature scales the whole zero-mean perturbation around its mean.

# The calibrated conditional distribution. This is the protocol every reported
# number uses, and different calls give different motions for the same prompt.
features = model.generate_motion(prompts, lengths=120, tokenizer=tokenizer)

# The distribution mean. Deterministic, and a *regression* protocol: numbers
# obtained this way are not comparable to generative ones.
mean_motion = model.generate_motion(prompts, lengths=120, tokenizer=tokenizer,
                                    temperature=0.0)

# Reproducible sampling.
g = torch.Generator().manual_seed(1234)
features = model.generate_motion(prompts, lengths=120, tokenizer=tokenizer, generator=g)

Sampling at temperature 1.0 is what makes the same prompt yield genuinely different, plausible motions. Captioning five seeded draws of the same prompt shows how far apart they land:

prompt: "a person walks forward and then waves with the right hand."
  seed 0  a person walks forward and then turns around and walks back.
  seed 1  a person walks forward and reaches out with their right hand.
  seed 2  a person walks forward and then waves with their left hand.
  seed 3  a person walks forward and puts their right hand on their head.
  seed 4  a person walks forward and then puts something on a counter.

Lowering the temperature trades that diversity for proximity to the conditional mean.

3. Motion to text

import numpy as np

motion = np.load("000004.npy")                      # (T, 263) HumanML3D features
clip = torch.from_numpy(motion).float().unsqueeze(0)

print(model.generate_caption(clip, tokenizer=tokenizer))
# ['a person bends over to touch the ground with their hands, then stands up straight.']

Pass normalized=True if your features are already standardised by the dataset statistics. Captioning is greedy by default; max_new_tokens and num_beams override the config values.

4. The shared latent interface

The K slots are the only motion representation in the system: the language model generates them for text-to-motion and reads these same vectors back for captioning. You can work with them directly.

latents = model.encode_motion(clip)          # (1, 2, 512): a whole clip, two vectors
print(latents.shape)

# Decode at the original length, and at a different one.
same = model.decode_motion(latents, clip.shape[1])
longer = model.decode_motion(latents, 200)

# Caption straight from latents, skipping the encoder.
print(model.generate_caption(latents=latents, tokenizer=tokenizer))

# Text -> latents, without decoding.
z = model.text_to_latents(["a person waves."], tokenizer=tokenizer)

Interpolating between two clips is then a two-line operation:

a = model.encode_motion(clip_a)
b = model.encode_motion(clip_b)
blend = model.decode_motion(0.5 * a + 0.5 * b, 120)

5. Running on GPU, and in batches

model = model.to("cuda")
features = model.generate_motion(prompts, lengths=120, tokenizer=tokenizer)  # follows the model

Batched prompts are left-padded, which reproduces the training and evaluation behaviour exactly. Single-prompt calls involve no padding at all.

6. Saving and rendering

features_to_joints gives (batch, frames, 22, 3) positions in metres, on the HumanML3D skeleton, with the y axis vertical.

joints = model.features_to_joints(features)
np.save("sample.npy", joints[0].cpu().numpy())

For animation, feed those positions to any HumanML3D-compatible renderer. The code repository includes motGPT/utils/render_utils.py, which writes an mp4 directly from a [T, J, 3] array.

7. Sanity check

If you swap in different feature statistics, the failure is silent: shapes stay right and the numbers look plausible while the whole skeleton is mis-scaled. One assertion catches it.

joints = model.features_to_joints(features)
high, low = float(joints[..., 1].max()), float(joints[..., 1].min())
assert 1.0 < high < 2.5 and -0.5 < low < 0.5, "implausible body height"

---

Reproducibility

The inference path in modeling_mugen.py was verified against the original training code on the same checkpoint. Routed hidden states, mu, logvar, the factor log-amplitudes, the autoencoder encode and decode outputs, and a seeded draw all agree to max|diff| = 0.000e+00, and generated captions match token for token.

Two consequences worth knowing:

• A given seed reproduces the same draw here as it does in the research code, because the factor noise is drawn before the diagonal noise in both.
• router_eval_tau (1.5) is the converged end of the training temperature anneal. Changing it changes which transformer depth each slot reads from, so leave it alone unless you are deliberately studying the router.

Tested with transformers 4.47 and PyTorch 2.x. The rollout passes explicit position_ids, so behaviour does not drift with the transformers version.

Limitations

• Trained on HumanML3D only: everyday single-person motion, English descriptions, 20 fps, 22 joints. Multi-person interaction, object manipulation, and highly stylised motion are out of distribution.
• Length is an input, not something the model infers from the text. Ask for 120 frames and you get 120 frames whether or not the description warrants it.
• K = 2 slots is a deliberately tight budget, chosen for this dataset. It is not a universal setting: the SnapMoGen model in the code repository uses K = 4, and the best budget is dataset-dependent.
• Captions come from a fine-tuned GPT-2 (124M) and are short and generic relative to human references. They describe the dominant action reliably and fine detail unreliably. On HumanML3D clip 000004, whose reference reads "a man is pretending to be a chicken, constantly pecking at the ground and waving his arms like a chicken", the model produces "a person bends over to touch the ground with their hands, then stands up straight": the gross movement is right, the intent and the arm motion are gone.
• Not a safety-filtered model. It generates skeletal motion and short English captions, and it has no content filtering of any kind.

Acknowledgements

Built on HumanML3D, MotionGPT, MLD, and our earlier discrete-codebook system GeoMotionGPT.

License

MIT