A faithfulness-aware pretraining strategy for abstractive text summarization

One of the main challenges in abstractive text summarizing is maintaining the faithfulness of the generated summaries compared to the source documents. In abstractive text summarizing, the term "faithfulness" refers to the degree to which a summary accurately and completely captures the essential information from the source text while maintaining the overall meaning and context. Recent works have made remarkable progress in addressing the issue of faithfulness in abstractive text summarization from several perspectives. For instance, some works suggested a post-process method to refine faithfulness. Others focused on the relationship between the decoding generation phase of the generative model and faithfulness. Furthermore, many studies put efforts into customizing the training phase in order to improve faithfulness. Nevertheless, these researches fail to adequately explore a central aspect, which is how pretraining strategies can impact and enhance the accuracy and reliability of faithfulness in abstractive text summarization. To address this problem, we have introduced an innovative pretraining strategy that stimulates the BART large language model to attend more to tokens and contexts correlated with faithfulness of the source text. To assess our approach, we conducted a thorough examination of its effects on both faithfulness and summarization. Our research revealed that the proposed technique improves the model's attention to the critical contexts that are strongly connected to the faithfulness of the original text. Furthermore, our experiments and analysis demonstrated that the introduced method outperforms the baseline model, which is pretrained using the traditional MLM techniques, in terms of different faithfulness metrics, such as QuestEval and BS-Fact metrics, in two downstream abstractive text summarization datasets. In addition, we investigated the possibility that the pretraining processes that were provided could improve the quality of the summaries that were created. This was determined by using summarization metrics such as ROUGE-N and BERT-Score.