diegogs1451/qwen2.5-7B-Instruct-dUO-finetuned-20260706-3epochs

Qwen2.5-7B-Instruct LoRA — dUO Administrative Title Summarization

This model is a LoRA adapter fine-tuned from Qwen2.5-7B-Instruct for generating summarization titles from administrative and legal texts published in Spanish/European official journals.

Model Details

Model Description

The model takes Spanish administrative/legal texts from official journals and produces concise, formal titles ready for publication. It is specialized in the three official journals that make up the dUO acronym:

• DOUE — Diario Oficial de la Unión Europea (Official Journal of the European Union)

• BOE — Boletín Oficial del Estado (Spanish Official State Bulletin)

• BOPA — Boletín Oficial del Principado de Asturias (Official Bulletin of Asturias)

• Developed by: Diego Gonzalez Suarez

• Shared by: diegogs1451

• Funded by: University of Oviedo

• Model type: Qwen2ForCausalLM (decoder-only causal language model with LoRA adapters)

• Language(s) (NLP): Spanish

• License: MIT

• Finetuned from model: Qwen/Qwen2.5-7B-Instruct

Model Sources [optional]

• Repository: https://huggingface.co/diegogs1451/qwen2.5-7B-Instruct-dUO-finetuned-20260706-3epochs
• Contact: gonzalezsdiego@uniovi.es

Uses

Direct Use

The model is designed to generate summarization titles for administrative and legal texts from official journals (BOE, DOUE, BOPA). Given a Spanish administrative/legal text as input, it outputs a concise, clear, formal title preserving the essential administrative meaning while eliminating unnecessary details and redundancies.

Out-of-Scope Use

This model is specialized for Spanish administrative text summarization and may not perform well on general text summarization tasks, other languages, or informal content.

How to Get Started with the Model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_name = "Qwen/Qwen2.5-7B-Instruct"
adapter_path = "diegogs1451/qwen2.5-7B-Instruct-dUO-finetuned-20260706-3epochs"

tokenizer = AutoTokenizer.from_pretrained(base_model_name)
model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter_path)

system_prompt = (
    "Eres un asistente experto en resumir textos oficiales administrativos y jurídicos "
    "del DOUE (Diario Oficial de la Unión Europea), del BOE (Boletín Oficial del Estado) "
    "y del BOPA (Boletín Oficial del Principado de Asturias) en títulos breves, claros, "
    "formales y listos para publicación. Conserva el significado administrativo esencial "
    "eliminando detalles innecesarios y redundancias."
)

def generate_title(text):
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": text},
    ]
    inputs = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, return_tensors="pt"
    ).to(model.device)
    outputs = model.generate(inputs, max_new_tokens=128, temperature=0.1)
    return tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True)

Training Details

Training Data

Fine-tuned on a custom dataset of approximately 200 examples (~86k tokens) sourced from:

• DOUE (Diario Oficial de la Unión Europea)
• BOE (Boletín Oficial del Estado)
• BOPA (Boletín Oficial del Principado de Asturias)

The dataset field used for training is text, paired with a system prompt instructing the model to summarize administrative texts into formal titles.

Training Procedure

Training Hyperparameters

Parameter	Value
LoRA rank (r)	16
LoRA alpha	32
LoRA dropout	0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Per device batch size	2
Max steps	174
Num epochs	3
Learning rate	2e-5
LR scheduler	Linear
Optimizer	AdamW
Max sequence length	512 tokens
Precision	bfloat16 mixed precision
Training regime	bf16 mixed precision

Speeds, Sizes, Times [optional]

• Total FLOPs: 1.466e16

Evaluation

Testing Data, Factors & Metrics

Testing Data

A held-out portion of the DOUE/BOE/BOPA dataset was used for evaluation.

Metrics

• eval_loss — Cross-entropy loss on the evaluation set
• eval_mean_token_accuracy — Mean token-level accuracy

Results

Step	Epoch	Train Loss	Train Token Acc.	Eval Loss	Eval Token Acc.	Learning Rate
58	1	0.1538	96.4%	0.1182	96.4%	1.44e-5
116	2	0.1173	96.0%	0.1152	96.7%	7.47e-6
174	3	0.0812	97.8%	0.1108	96.8%	5.75e-7

Summary

The model achieves eval loss of 0.1108 and token accuracy of 96.8% on the held-out set after 3 epochs of fine-tuning, with consistent improvement in training metrics and stable evaluation performance across checkpoints.

Environmental Impact

• Hardware Type: Single GPU (T4, 16GB)
• Hours used: Not specified
• Cloud Provider: Not specified
• Compute Region: Not specified

Technical Specifications [optional]

Model Architecture and Objective

• Base architecture: Qwen2.5-7B-Instruct
• Hidden size: 3584
• Intermediate size: 18944
• Hidden layers: 28
• Attention heads: 28
• KV heads: 4 (Grouped-Query Attention)
• Max position embeddings: 32768
• Vocabulary size: 152064
• Activation function: SiLU
• Normalization: RMSNorm
• Objective: Causal language modeling (next-token prediction)

Framework versions

• PEFT 0.15.2
• Transformers (via TRL)
• TRL (SFTConfig / SFTTrainer)
• bitsandbytes (4-bit NF4 quantization)
• TensorBoard