Efficient visual question answering via matryoshka representation learning based adaptive granularity selection

Multimodal Large Language Models (MLLMs) achieve strong performance on vision-language tasks but suffer from computational bottlenecks when processing high-resolution images due to excessive visual token sequences. While adaptive granularity methods like AVG-LLaVA attempt to mitigate this, they rely on simple pooling operations that may lose critical information during coarse-grained extraction. This thesis presents MRL-LLaVA, a novel framework that integrates Matryoshka Representation Learning (MRL) to replace pooling-based extraction with hierarchical, information-enriched representations. Our approach involves two key components: (a) fine-tuning a pretrained CLIP visual encoder with an MRL layer to produce nested representations across embedding dimensions, and (b) integrating this MRL-CLIP encoder into AVG-LLaVA to create MRL-LLaVA. Evaluating across three VQA benchmarks (GQA, ChartQA, and DocVQA), MRL-LLaVA achieves substantial gains on document understanding (+7.60% on DocVQA) and scene reasoning (+3.97% on GQA) while reducing inference latency up to 43.6%. Routing analysis reveals that our approach may enable more adaptive granularity selection on certain VQA tasks, demonstrating diverse routing behavior where the baseline concentrates on fixed configurations. These results validate that Matryoshka-based nested representations provide richer visual granularities than pooling operations, offering improved efficiency-performance trade-offs for MLLMs.