Some applications in statistics entropy

Statistical entropy is a measure of uncertainty that a statistical experiment possesses. The more predictable the outcome of an experiment, the lower the entropy; and the less predictable the outcome, the higher the entropy . In addition there are some interesting application areas for statistical entropy. First of all, whenever the probability (or frequency) distribution is qualitative, entropy can be used as a measure of variation. In these cases, as known well, the statistics like variance and standard deviation can't be computed since arithmetic mean cannot be obtained as a measure of central tendency for qualitative distributions. Besides, whenever two variables are qualitative, classic correlation measures cannot be used. For this reason the application of measures of entropy and mutual information to the study of association between variables will be useful. Entropy and relative entropy are closely related to likelihood function. In general, relative entropy (Kullback-Leibler divergence) is closely associated with some goodness of fit statistics like chi-square statistics. In this thesis, after these issues are discussed in detail, by some applications, applicability of some of the measures based on entropy and relative entropy to basic problems of statistics mentioned previously is studied. Keywords: Shannon entropy, Rényi entropy, Tsallis entropy, Havrda-Charvat entropy, Kullback-Leibler divergence, Qualitative variation, Mutual information