philipp-zettl/modernbert-diffusion-openwebtext

./refinebert-openwebtext

Model Summary

A diffusion-style masked language model fine-tuned from philipp-zettl/modernbert-diffusion-universal on the Skylion007/openwebtext dataset.

Model Details

• Model ID: ./refinebert-openwebtext
• Base model: philipp-zettl/modernbert-diffusion-universal
• Training mode: Fine-tuning
• Task type: Masked token denoising / diffusion-style infilling

Intended Use

Intended for tasks related to Skylion007/openwebtext.

Example

from refinebert.diffusion_engine import MaskedDiffusionEngine

engine = MaskedDiffusionEngine("./refinebert-openwebtext")
prompt = "N/A (See generation logs)"
output = engine.generate(prompt, num_new_tokens=N/A, steps=N/A, guidance_scale=N/A)
print(output)

Training Data

Single-dataset fine-tuning.

Dataset Mix

| Skylion007/openwebtext | 100% | Fine-tuning Target |

Fine-tuned specifically on the Skylion007/openwebtext dataset.

Training Procedure

• Steps: 10000
• Batch size: 8
• Sequence length: 512
• Learning rate: 5e-05
• CFG dropout probability: N/A
• Samples loaded into RAM: N/A

Training Time & Hardware

• Duration: 3h 23m 12s
• Hardware: NVIDIA GeForce RTX 4070 Laptop GPU x1 (CUDA available)

Metrics (Training)

Metric	Value
Training Loss	4.4906
Epochs	1
Global Step	10000

Limitations & Considerations

• The model is trained with a masked-token diffusion objective and may not behave like an autoregressive LM.
• Data sources may have licensing or content constraints—review source dataset cards before deployment.
• Performance can vary substantially by mode (Fine-tuning) and prompt structure.