dmusingu/diffvqa-qwen3-0.6b-multiobjective-regsteps

Diff-VQA — Qwen3-0.6B visual-prefix decoder (multiobjective-regsteps encoder)

Online-trained Difference Visual Question Answering head for chest X-rays (MIMIC-CXR / Medical-Diff-VQA). A frozen ViT-L/14 vision encoder produces patch tokens for a current + reference image pair; a Qwen3-0.6B decoder, conditioned on those tokens as a visual prefix, generates the answer describing what changed.

This is a slimmed inference checkpoint: decoder weights + the frozen vision encoder weights + architecture metadata (optimizer/scheduler state stripped).

Contents (*_best.pt, torch.load(..., weights_only=False))

key	description
state_dict	DiffVQAQwen3Head weights (Qwen3-0.6B LLM + vis_proj + frame_emb)
encoder_state	frozen ViT-L/14 vision encoder (objective: multiobjective_caption+regioncontrastive)
decoder	Qwen/Qwen3-0.6B
tokenizer_name	Qwen/Qwen3-0.6B
vis_dim	1024
encoder_name	pretrain-multiobjective-regsteps
epoch / step / best_val	training position + best validation loss

Training

• Data: Medical-Diff-VQA difference questions over MIMIC-CXR image pairs (~100k train / ~16k val / ~16k test).
• Objective: next-token cross-entropy over the answer span (question tokens masked).
• Vision encoder is frozen; only the decoder (LLM + visual adapter) is trained.

Notes

The bundled encoder_state is the exact frozen encoder this decoder was trained against — load both together so the visual features match the space vis_proj expects (a mismatched encoder produces degenerate output).