A study about the effectiveness of ordering the samples by difficulty levels in the training of artificial neural networks

Curriculum Learning and Self-paced Learning are popular topics in machine learning which suggests to put the training samples in an order related with their difficulty levels. Studies about these topics show that starting with a small training set and continue to add new samples according to the difficulty levels improves the performance of the learner. It is seen that the learner has better performance with hard-to-easy ordering as well as easy-to-hard. Chapter 3 is about a method to automatically determine the difficulty levels of the samples to see the performance of the Curriculum and its reverse version on many application areas. Samples have been ordered from easy-to-hard and hard-to-easy with the proposed method and compared with classical training and each other. According to the results, methods which have ordered the training samples obtained statistically significant lower error rates. While it is expected to get better results by ordering the samples from easy-to-hard it is surprising to get also better results when ordering the samples from hard-to-easy. Because of this, the underlying reasons for the success of both the Curriculum Learning and its reverse version have been researched. It is suggested in Chapter 4 that the success of these methods with ordering are not due to the fact that they have ordering but the training set is growing from stage to stage. For this reason, it is thought that giving the samples in randomly growing groups rather than with a meaningful order will increase the learning performance. The theoretical perspective of the proposed method is explained and the proposed method is compared with previous methods in the experiments. As a result of the comparisons, it is seen that the random ordered growing sets method performed better than the classical method which all the samples were given in one step and closely with the successful Curriculum and Self-Paced Learning methods. As a result of the theoretical and experimental studies, it has been concluded that training with growing sets method, which is a common feature of Curriculum Learning and its reverse version, has a feature that allows to find a better local minimum during optimization and therefore can achieve lower error rates than training in a single stage.