The influence of context structurization on attention dynamics and representational spaces in bert for question answering

Understanding how Large Language Models (LLMs) like BERT process and represent information is important for improving their abilities in tasks such as question answering. This thesis investigates how context structurization, a technique informed by cognitive principles of information organization, influences attention dynamics and representational spaces within bert-base-cased models. To support this investigation, a new dataset, 'SQuAD with Structurized Contexts' (SQSC), was created by applying a structurization method to original SQuAD contexts. The research compares three BERT model states: a pre-trained vanilla model (BERT-V), a model fine-tuned on unstructured SQSC contexts (BERT-UN), and one fine-tuned on structurized SQSC contexts (BERT-ST). Using an out-of-distribution (OOD) data sample generated for this study, experiments analyze these models' internal mechanisms when processing both unstructured and structurized versions of this OOD input. The study utilizes a range of quantitative analyses to compare attention distributions and hidden state representations across the different models and input conditions, employing measures of distributional divergence, vector similarity, and comparative representational geometry. This research aims to provide detailed insights into how structural modifications to input context affect BERT's internal processing. The findings are expected to contribute a better understanding of the relationship between input data structure and model mechanisms, offer the SQSC dataset as a new public resource, and inform future work in developing more understandable and effective language models.