Dataset cartography for compositional generalization

Neural networks have revolutionized language modelling and excelled in various downstream tasks. However, the extent to which these models achieve compositional generalization comparable to human cognitive abilities remains debatable. While existing approaches in the field have mainly focused on novel architectures and alternative learning paradigms, we introduce a pioneering method harnessing the power of dataset cartography [Swayamdipta et al., 2020]. By strategically identifying a subset of compositional generalization data using this approach, we achieve a remarkable improvement in model accuracy, yielding enhancements of up to 10% on CFQ and COGS datasets. Notably, our technique incorporates dataset cartography as a curriculum learning criterion, eliminating the need for hyperparameter tuning while consistently achieving superior performance. Moreover, as the data becomes the bottleneck in the current large language model (LLM) pipeline, covering every possible combination of known words or phrases becomes infeasible. Therefore, we focus on compositional generalization in LLMs to help LLMs process the combinations of unseen language parts faithfully. We expand the previously described setting above to LLMs and propose a new diversity-aware subset selection method named DiCart, a fusion of dataset cartography and determinantal point processes. DiCart results in better or on-par compositional generalization than baselines and even the full training set.