Performance in Human-AI Teams: Experimental Evidence on Commitment Deficits Under Competition
Description
Overview: This research investigates why human-AI teams (HATs) often underperform despite AI's growing capabilities. While previous explanations have centered on coordination, communication failures, or miscalibrated trust in AI's abilities, this work proposes that teammate commitment—the sense of mutual moral and social obligation—is a key yet overlooked driver of HAT effectiveness, particularly in competitive settings. The research consists of two experimental studies that examine how different contextual factors influence human performance when working with AI versus human teammates: Study 1 Description Study 1 employed a between-subjects design with two conditions: Human-Human (HH) and Human-AI (HA). Participants were randomly assigned to one of these conditions. The study compared human performance across both individual and team competitive contexts while measuring trust through delegation behaviors. The experiment was an online study (N=1,988) conducted using Prolific for participant recruitment and oTree (Chen et al., 2016) for experiment programming. Participants completed three rounds of arithmetic reasoning tasks adapted from the American Armed Services Vocational Aptitude Battery (ASVAB). The three rounds consisted of: Individual Competition Round: Participants competed individually against either a human or AI opponent. Team Competition Round: Participants were paired with either a human or AI teammate and competed as a team against other teams. Delegation Choice Round: Participants were given the option to delegate both competitive rounds to their teammate. The data includes participant performance in the arithmetic reasoning tasks, as well as delegation choices and demographic information. Files Included: S1_Data_Otree&Prolific.csv: This CSV file contains the raw data from the Study 1 experiment. S1_Data_Analysis_JW.R: This R script is used for data structuring and analysis of the Study 1 data. Study 1_ Otree.zip: This zip file contains the oTree experiment code, including Python and HTML files, used to conduct Study 1. Study 2 Description: Study 2 employed a 2x2 within-subjects design, manipulating teammate type (Human vs. AI) and outcome structure (Independent vs. Interdependent). Participants completed a visual counting task across four rounds. The study examines whether the performance decrements observed in Study 1's competitive setting emerge when outcomes are interdependent. The experiment was a laboratory study (N=214) conducted at the Laboratory of Experimental Economics (LEE) at Warsaw University. Files Included: Study 2_allSessions.csv: This CSV file contains the raw data from the Study 2 experiment (all 20 lab sessions). S1_Data_Analysis_JW: Contains the R scripts used for data processing and analysis of the Study 2 data. Study 2.otreezip Study 2_ Otree.zip: This otreezip file contains the oTree experiment code, including Python and HTML files, used to conduct Study 2.
Files
Steps to reproduce
Study 1: Obtain the files: Download the files S1_Data_Otree&Prolific.csv, S1_Data_Analysis_JW.R, and Study 1_Otree.zip. Set up oTree (if needed): If you wish to run the experiment: Install oTree (refer to the oTree documentation for installation instructions). Extract the contents of Study 1_Otree.zip into your oTree projects directory. Use oTree commands to run the experiment. Data Structuring and Analysis: To analyze the data: Use R to execute the script S1_Data_Analysis_JW.R. This script will perform the necessary data structuring and statistical analyses. Ensure you have the required R packages installed. Data Exploration: The raw data is available in S1_Data_Otree&Prolific.csv. You can use any software that reads CSV files (e.g., R, Python, spreadsheet software) to explore the dataset. Study 2: Obtain the files: Download the files Study 2_allSessions.csv, S1_Data_Analysis_JW, and Study 2_Otree.zip. Set up oTree (if needed): If you wish to run the experiment: Install oTree (refer to the oTree documentation for installation instructions). Extract the contents of Study 2_Otree.zip into your oTree projects directory. Use oTree commands to run the experiment. Data Analysis: To analyze the data: Use R to execute the scripts provided in S1_Data_Analysis_JW. These scripts will perform the data processing and statistical analyses. Ensure you have the required R packages installed. Data Exploration: The raw data is available in Study 2_allSessions.csv. You can use software that reads CSV files to explore the dataset.
Institutions
- European University Institute