Performance comparison of vision-language models in image classification

Vision-Language Models (VLMs) have introduced a new paradigm in image classification by integrating visual and textual modalities. Although these models achieve remarkable success in multimodal tasks, their effectiveness in purely visual classification has remained relatively underexplored. This thesis conducts a comprehensive, metric-driven comparative analysis of eight state-of-the-art VLMs—GPT-4o, GPT-4o-mini, Gemini-flash-1.5-8b, LLaMA-3.2-90B-vision-instruct, Grok-2-vision-1212, Qwen2.5-vl-7b-instruct, Claude-3.5-sonnet, and Pixtral-large-2411—across four datasets: CIFAR-10, ImageNet, COCO, and a domain-specific New Plant Diseases dataset. Model performance was systematically evaluated in zero-shot and few-shot settings using accuracy, precision, recall, F1-score, and robustness as primary metrics. The results demonstrate that GPT-4o consistently achieves the highest performance across standard benchmarks (e.g., accuracy: 0.91; F1-score: 0.91 on CIFAR-10), substantially outperforming lighter models such as Pixtral-large-2411 (accuracy: 0.13; F1-score: 0.13). While the near-optimal results observed on ImageNet and COCO are likely attributable to pre-training data overlap, the marked performance degradation on the New Plant Diseases dataset highlights persistent domain adaptation challenges. Overall, these findings underscore the necessity of robust, parameter-efficient, and domain-adaptive fine-tuning strategies to improve the applicability of VLMs in real-world image classification tasks.