Experimental data on trial-by-trial AI assistance, feedback, human confidence and decisions during an AI-assisted decision-making chess puzzle task
Description
The data are collected from a human subjects experiment to examine how human confidence in AI and self-confidence change and impact their decision-making during an AI-assisted decision-making task. The experimental task includes three practice and 30 experimental chess puzzle problems, which 100 participants are asked to solve with AI assistance. For each problem, the goal is to make the best next chess move given a board state. The participants first select an independent move (bmove1), receive AI suggestion (aisugg), make a final move (bmove2) , receive feedback on the final move (feedback2), and report their self-confidence (selfconf) and confidence in AI (aiconf). All of these data indicated in the parentheses are recorded in the dataset, as well as some other information including opponent's last move (wmove), top seven moves with the highest evaluation scores (allgoodmoves), ranking of the AI suggestion (multiPV), goodness of the independent move (feedback1), chess board state before opponent's last move (fen before white move), and chess board state before the participants' move (fen before black move). There are two experimental conditions which differ in the order in which the AI performance (i.e., accuracy of AI suggestions) changes: 1) high-performing (80% accuracy) to low-performing AI (20% accuracy) and 2) low-performing (20%) to high-performing (80%) AI. Each CSV datafile ("data#_#") contains each participant's data, where the first # in the filename indicates the participant number and the second # indicates the condition number. For more detailed description of the dataset and the experiment, please refer to the Data In Brief article (publication in process) and the original research article (https://doi.org/10.1016/j.chb.2021.107018). This dataset can be utilized in various domains such as human-computer interaction, psychology, computer science, and team management in engineering/business that seek to understand human cognition and behavior in human-AI collaboration contexts.