Knowledge distillation with foundation models for image segmentation

Recent advances in machine learning through model scaling achieves state-of-the-art results in various tasks. An example to this is foundation models, which are large models capable of generalizing well with zero-shot predictions. However, these models come with caveats of computational and memory costs. Knowledge distillation is a transfer learning technique where information from a bigger model can be distilled to a smaller architecture, saving memory and computational costs. In this thesis, we have investigated using foundation model as a teacher network in a knowledge distillation setting, compared to a large model fine-tuned on the same task.