When it matters to you: The impact of personal relevance on the evaluation of algorithmic versus human predictions

Although AI algorithms that surround our personal and professional lives are improving even further, the use of algorithmic predictions and recommendations in decision-making tasks is not solely guaranteed by their superior capabilities compared to humans. Previous research examining a range of algorithms, varying in complexity from simple decision rules to advanced AI systems, has long discussed the phenomena of both algorithm aversion and appreciation. Findings reveal that people's expectations and reactions to algorithms substantially vary under certain circumstances. Studies have primarily focused on the interaction between algorithm design and individual factors and lacked research on the role of task-related variables. Applying the Theory of Felt Involvement (Celsi & Olson, 1988), this study examines how the level of interest in a task affects the perceived accuracy of predictions that are presented as originating from an AI algorithm or a group of individuals. Participants (n = 78) were randomly assigned to one of three groups and asked to rate the accuracy of predictions for 30 trivia questions provided by one of the following sources: (a) the AI algorithm, (b) previous participants, and (c) the ambiguous source (control group). Analysis with a linear mixed-effects model showed that the accuracy ratings for the AI algorithm were higher than those for both the group of previous participants and the control condition, especially when predictions were made for subject categories participants indicated a higher felt interest. This finding suggests that the influence of the source on perceived prediction accuracy was more pronounced for questions with higher interest levels. We discuss the generalizability of our findings and provide possible future directions for human algorithm interaction research.