The validity and reliability of clinicians' ratings and self-report scale scores in evaluating ICD-11 (international classification of diseases - 11TH edition) diagnostic criteria for general personality disorder

Purpose and hypothesis: The present study aimed to investigate the inter-rater reliability of the clinicians' diagnostic ratings concerning the ICD-11 criteria set for general personality disorder (PD), and to explore the validity of two self-report scales against clinicians' evaluations. We tested the research hypotheses that good levels of chance-corrected agreement (kappa or ICC ≥ 0.60) are to be found between criteria ratings by a pair of clinicians and between their ratings and the SASPD and WHODAS scores. Methods: The data were collected between March 2021 and November 2021. The sample consisted of 48 volunteering in- or out-patients ranging in age between 18-61 years. Each participant was administered the SASPD (Standardized Assessment of Severity of Personality Disorder) and 36-item WHODAS 2.0 (World Health Organization Disability Scale) and was interviewed and rated concerning the ICD-11 criteria for general PD by a clinician using a form designed for this study. Each participant was also rated by a second clinician listening to the audio-recorded interview. The agreement level between two clinicians in terms of dimensional ratings and categorical diagnoses were estimated by means of intraclass correlation coefficient (ICC) or kappa, respectively. Spearman rho and eta correlation coefficients were used to analyze the agreement between self-report scale scores and clinicians' severity ratings, and ROC analysis was employed to search for the optimal cut-off scores to be used to predict clinicians' categorical PD diagnoses. The internal consistency of the self-report scales was investigated through Cronbach's alpha coefficient and corrected item-total correlations. Since the Type-1 error rate was accepted as 0,05, the 95% confidence intervals of the obtained sample parameters were taken into account in estimating the population parameters. Findings: The clinicians agreed on the presence of a PD diagnosis in 37.5% and on the absence of a diagnosis in 58.3% of the sample, while disagreed on the 4.2%. The calculated kappa coefficient of 0.91 (95% CI = 0.77-1.00) indicates a high level of agreement beyond chance between the clinicians' diagnostic judgments. Likewise, the inter-rater reliability of the clinicians' dimensional ratings involving personality functioning, PD-specific symptoms, and social functionality was very good (i.e., ICC > 0.80). The correlations of the clinicians' individual and combined ratings of PD severity with SASPD total score and the WHODAS general disability score were around 0.60 and 0.70, respectively. Accordingly, the self-report scale scores of 18 patients with a consensus PD diagnosis were significantly (p ≤ 0.005) higher than those of the 28 patients without. Among the various scores obtained from WHODAS, the general disability score was associated with the largest difference between the two groups. Hence, we explored the performance of this score and the SASPD total score with optimal cut-offs. ROC analysis suggested the optimal cut-off point of 9 for SASPD scores maximizing the diagnostic accuracy to 76% with 78% sensitivity and 75% specificity. These values and the population validity estimates (kappa = 0.52; 95% CI = 0.26-0.76) suggested that the validity of the scale was within acceptable limits.The reliability of the SASPD, however, was unacceptably low, particularly for clinical use (alpha = 0.65; 95% CI = 0.46-0.79), mostly due to low item-total correlations (i.e., ≤ 0.30) observed for the final three items involving orderliness, caring for other people, and self-sufficiency. Conversely, the WHODAS-36 general disability score used with an optimal cut-off score of 1.30 as suggested by ROC analysis classified subjects with or without PD diagnosis with an accuracy of 85%, a sensitivity of 82%, and a specificity of 88%. Furthermore, the internal consistency of the general disability score was very high (Cronbach's alpha = 0.97; 95% CI= 0.96 - 0.99), and the entire corrected item-total correlations were over 0.30 level. These findings as well as the population validity estimates (kappa = 0.70; 95% CI = 0.46-0.90) suggested that the cut-off applied general disability score obtained from 36-item WHODAS is a valid and reliable tool to predict clinical PD diagnosis. As expected, the general disability score as a composite of 12 items of the WHODAS short form has similar psychometric properties when used with a cut-off score of 1.04. Briefly, the WHODAS-12 scores are associated with a Cronbach's alpha of 0.91 (95% CI = 0.86 - 0.95), kappa of 0.70 (95% CI = 0.46-0.90), sensitivity of 0.88, specificity of 0.83, and accuracy of 0.85. Conclusion: Our findings reveal high inter-rater reliability of diagnostic evaluations involving dimensional and categorical ratings of the ICD-11 criteria for general PD on a form designed for this study. The validity of the SASPD scores in predicting the PD diagnosis and severity was within acceptable limits, though the reliability was not. Therefore, use of this instrument to screen personality pathology can hardly be recommended. On the other hand, the general disability scores yielded either by short or long WHODAS self-report forms emerge as valid and highly reliable measures of ICD-11 defined PD. Therefore, data from the current psychometric study strongly support the use of WHODAS self-report forms to predict PD diagnosis for clinical and/or research purposes in Turkey.