Florence-2 Fine-Tuning for Visual Question Answering
I fine-tuned an about 230M-parameter vision-language model on 60,000 image-question pairs and shipped it live on HuggingFace Spaces.
Hypothesis
Can an about 230M-parameter vision-language model be fine-tuned on a single Colab GPU to a stable, deployable VQA endpoint without exceeding memory or destabilising gradients?
- Problem
- General vision-language models caption and detect objects well, but are not tuned to give short, direct answers to open questions. We wanted one that does, trained on one Colab GPU and served on free CPU hardware.
- My role
- Team of 3 (Data Mining and Machine Learning, Federico II) with Zahra Jafarinejad and Farshad Farahtaj. I built the pipeline: data preparation, the fine-tune, loss tracking and the deployed app.
- Result
- A full fine-tune of an about 230M-parameter model: validation loss fell from 0.240 after epoch 1 to 0.202 after epoch 3 on 30,000 held-out questions, live on Hugging Face.
Honest limits
- No accuracy, BLEU or CIDEr was computed; only loss was tracked.
- The loss counts padding tokens, so its absolute value is not comparable to other fine-tunes; the downward trend is real.
- About 40% of training answers are yes or no, and the answer score in the demo is not a calibrated probability.
Runs right here on this page. It loads from Hugging Face, which may set its own cookies, and a sleeping demo takes 30 to 60 seconds to wake up.
~230M params
Model size
60K train pairs
Training data
Full-parameter
Fine-tune
Colab GPU
Hardware
VQA v2.0 Abstract Scenes, open-ended subset
We started from VQA v2.0's Abstract Scenes split: 50,000 fully coloured cartoon-style scenes, ~3 crowd-annotated questions per image, ~150K image-question pairs total (60K train, 30K val, 60K test) (dataset's published description). We used the open-ended question files.
We built a custom preprocessing pass that joined the raw questions and annotations JSONs on question_id, took the modal-vote answer per question, prefixed each prompt with the <VQA> task token, and emitted clean train / val / test JSONs. The images in the release are 224×224; the Florence processor resizes them again.
What the questions look like before fine-tuning
Before training, we profiled the open-ended questions and crowd-annotated answers to understand the linguistic surface area. The answer-frequency bar below is rebuilt from the notebook output as an interactive chart; the word cloud and question-length histogram stay as raw notebook artifacts.
Training-set EDA
Top 20 most common answers (training)
Each bar is one answer string; X is its frequency across 60,000 open-ended questions.
How to read it: yes and no (cyan) together carry roughly 24,000 of the 60,000 training questions, about 40 percent of the volume. A model that always guessed yes would already hit about 23 percent accuracy. The real benchmark is what the model does on the long tail past position 5, where colour and object words live.


Florence-2 kept after two other models failed on Colab
We also tried PaliGemma (ran out of GPU memory) and BLIP (did not train stably); these runs were not logged. Florence-2-base-ft gave a stable full-parameter fine-tune, so we kept it.
We loaded microsoft/Florence-2-base-ft at revision refs/pr/6 with trust_remote_code=True. All parameters (about 230M) trainable, no frozen layers.
University team project (Data Mining and Machine Learning, Federico II) with Zahra Jafarinejad and Farshad Farahtaj. I built the pipeline: data preparation, the fine-tune, loss tracking and the deployed app.
Training loss across 3 epochs
We optimised with AdamW (learning rate 1e-5, batch size 8), a linear LR schedule, no warmup, and 22,500 total steps over 3 epochs. Training ran on a single Google Colab GPU in roughly 9 hours wall clock, with a full validation pass at the end of every epoch.
| Epoch | Train loss | Val loss |
|---|---|---|
| 1 | 0.307 | 0.240 |
| 2 | 0.176 | 0.206 |
| 3 | 0.111 | 0.202 |
Note: only cross-entropy loss is reported. No accuracy, BLEU or CIDEr was computed; only loss was tracked. The loss counts padding tokens, so its absolute value is not comparable to other fine-tunes; the downward trend is real.
Try the fine-tuned model on HuggingFace Spaces
Configurable beam search and a per-answer confidence score. Upload any photo and ask a question about it.
Loads Florence-2 VQA from huggingface.co, which may set its own cookies. A sleeping demo takes 30 to 60 seconds to wake up.
Frameworks and infrastructure
Source code, technical report, and reproduction guide on GitHub.