plot-visualization-florence-2-lora-32

This is a LoRA fine-tuned version of microsoft/Florence-2-base-ft for detecting subplots in scientific figures.

Model Details

• Base Model: microsoft/Florence-2-base-ft
• LoRA Rank: 32
• Task: Object Detection (subplot detection)
• Labels: quantitative plot, qualitative plot

Usage

from pathlib import Path
from PIL import Image
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoProcessor
import torch

# Load model
base_model = "microsoft/Florence-2-base-ft"
processor = AutoProcessor.from_pretrained(base_model, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    trust_remote_code=True,
    torch_dtype=torch.float16,
)
model = PeftModel.from_pretrained(model, "amayuelas/plot-visualization-florence-2-lora-32")
model = model.merge_and_unload()
model = model.to("cuda")

# Run inference
image = Image.open("your_image.jpg").convert("RGB")
prompt = "<OD>"
inputs = processor(text=prompt, images=image, return_tensors="pt").to("cuda")
inputs["pixel_values"] = inputs["pixel_values"].to(torch.float16)

with torch.no_grad():
    generated_ids = model.generate(
        input_ids=inputs["input_ids"],
        pixel_values=inputs["pixel_values"],
        max_new_tokens=1024,
        num_beams=3,
        use_cache=False,
    )

output = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
print(output)

Training

This model was fine-tuned using PEFT/LoRA on a dataset of scientific figures with annotated subplots.