Skip to content
All projects
2025Live demo

Florence-2 Fine-Tuning for Visual Question Answering

I fine-tuned an about 230M-parameter vision-language model on 60,000 image-question pairs and shipped it live on HuggingFace Spaces.

Hypothesis

Can an about 230M-parameter vision-language model be fine-tuned on a single Colab GPU to a stable, deployable VQA endpoint without exceeding memory or destabilising gradients?

Problem
General vision-language models caption and detect objects well, but are not tuned to give short, direct answers to open questions. We wanted one that does, trained on one Colab GPU and served on free CPU hardware.
My role
Team of 3 (Data Mining and Machine Learning, Federico II) with Zahra Jafarinejad and Farshad Farahtaj. I built the pipeline: data preparation, the fine-tune, loss tracking and the deployed app.
Result
A full fine-tune of an about 230M-parameter model: validation loss fell from 0.240 after epoch 1 to 0.202 after epoch 3 on 30,000 held-out questions, live on Hugging Face.

Honest limits

  • No accuracy, BLEU or CIDEr was computed; only loss was tracked.
  • The loss counts padding tokens, so its absolute value is not comparable to other fine-tunes; the downward trend is real.
  • About 40% of training answers are yes or no, and the answer score in the demo is not a calibrated probability.
Try it live

Runs right here on this page. It loads from Hugging Face, which may set its own cookies, and a sleeping demo takes 30 to 60 seconds to wake up.

~230M params

Model size

60K train pairs

Training data

Full-parameter

Fine-tune

Colab GPU

Hardware

Dataset

VQA v2.0 Abstract Scenes, open-ended subset

We started from VQA v2.0's Abstract Scenes split: 50,000 fully coloured cartoon-style scenes, ~3 crowd-annotated questions per image, ~150K image-question pairs total (60K train, 30K val, 60K test) (dataset's published description). We used the open-ended question files.

We built a custom preprocessing pass that joined the raw questions and annotations JSONs on question_id, took the modal-vote answer per question, prefixed each prompt with the <VQA> task token, and emitted clean train / val / test JSONs. The images in the release are 224×224; the Florence processor resizes them again.

Data exploration

What the questions look like before fine-tuning

Before training, we profiled the open-ended questions and crowd-annotated answers to understand the linguistic surface area. The answer-frequency bar below is rebuilt from the notebook output as an interactive chart; the word cloud and question-length histogram stay as raw notebook artifacts.

Training-set EDA

Top 20 most common answers (training)

Each bar is one answer string; X is its frequency across 60,000 open-ended questions.

How to read it: yes and no (cyan) together carry roughly 24,000 of the 60,000 training questions, about 40 percent of the volume. A model that always guessed yes would already hit about 23 percent accuracy. The real benchmark is what the model does on the long tail past position 5, where colour and object words live.

Notebook output
Word cloud of the most common training question tokens
Training-set question vocabulary across 60,000 open-ended prompts.What it tells us: the questions cluster around color, counts, and object identification (color, many, dog, man, woman). The model needs strong colour recognition and basic counting more than abstract reasoning.
Notebook output
Histogram of validation question length in word count
Validation question length distribution. Bin width is one word.How to read it: the mode sits at 5 to 7 words and the tail extends past 15. So the model can rarely lean on template matching, even short prompts are linguistically varied.
Model selection

Florence-2 kept after two other models failed on Colab

We also tried PaliGemma (ran out of GPU memory) and BLIP (did not train stably); these runs were not logged. Florence-2-base-ft gave a stable full-parameter fine-tune, so we kept it.

We loaded microsoft/Florence-2-base-ft at revision refs/pr/6 with trust_remote_code=True. All parameters (about 230M) trainable, no frozen layers.

University team project (Data Mining and Machine Learning, Federico II) with Zahra Jafarinejad and Farshad Farahtaj. I built the pipeline: data preparation, the fine-tune, loss tracking and the deployed app.

Training and results

Training loss across 3 epochs

We optimised with AdamW (learning rate 1e-5, batch size 8), a linear LR schedule, no warmup, and 22,500 total steps over 3 epochs. Training ran on a single Google Colab GPU in roughly 9 hours wall clock, with a full validation pass at the end of every epoch.

EpochTrain lossVal loss
10.3070.240
20.1760.206
30.1110.202

Note: only cross-entropy loss is reported. No accuracy, BLEU or CIDEr was computed; only loss was tracked. The loss counts padding tokens, so its absolute value is not comparable to other fine-tunes; the downward trend is real.

Live demo

Try the fine-tuned model on HuggingFace Spaces

Configurable beam search and a per-answer confidence score. Upload any photo and ask a question about it.

Loads Florence-2 VQA from huggingface.co, which may set its own cookies. A sleeping demo takes 30 to 60 seconds to wake up.

Tech stack

Frameworks and infrastructure

PyTorchHuggingFace TransformersFlorence-2AdamWStreamlitHuggingFace SpacesGoogle Colab

Source code, technical report, and reproduction guide on GitHub.