Pragmatic Task Battery: LLM Responses and Rubric Scores

Description

This dataset accompanies the manuscript "Pragmalinguistic competence without sociopragmatic calibration: an exploratory assessment of pragmatic performance in four large language models," submitted to the Journal of Pragmatics. It contains the complete Pragmatic Task Battery (PTB) — twelve Discourse Completion Task (DCT) scenarios — the verbatim responses produced by four large language models (ChatGPT / GPT-5.3 Instant, Gemini 3 Flash, Claude Sonnet 4.6, and Perplexity Sonar Pro) to each scenario, and the Python script used to compute the statistical analysis reported in the manuscript (Friedman test, Kendall's W, post-hoc Wilcoxon signed-rank comparisons).

Steps to reproduce

This dataset was generated for an exploratory study of pragmatic performance in four large language models (LLMs): ChatGPT / GPT-5.3 Instant, Gemini 3 Flash, Claude Sonnet 4.6, and Perplexity Sonar Pro. Data collection took place on 10 April 2026. The models were accessed through their default free-tier web interfaces. No custom system prompts, developer instructions, temperature settings, or other parameter modifications were used. The same Pragmatic Task Battery (PTB) was presented identically to each model. The PTB consists of 12 Discourse Completion Task (DCT) scenarios designed to elicit pragmatic responses in different communicative contexts. Each model produced one response for each scenario, resulting in 48 responses in total (4 models × 12 scenarios). The prompts and verbatim model responses are provided in PTB_Responses_All_Models.docx. Each response was assessed by the author using a five-criterion rubric addressing speech-act performance, sociopragmatic appropriateness, coherence, implicature, and naturalness. Each criterion was rated on a 1–5 scale, producing a total score ranging from 0 to 25 for each response. The scenario-level total scores used for the quantitative analysis are provided in PTB_Total_Scores.csv. Full rubric definitions and scoring anchors are reported in Section 3.4 of the associated manuscript. To reproduce the analysis, download all dataset files and place them in the same local folder. Install Python 3.10 or later and the required packages: pip install numpy pandas scipy matplotlib Then run: python analysis_script.py The script reads PTB_Total_Scores.csv and creates an outputs folder containing a line plot of total scores by scenario and model, descriptive statistics, the Friedman-test results, Kendall's W effect size, and pairwise Wilcoxon signed-rank tests with Bonferroni adjustment. This repository contains the scenario-level total scores used in the inferential analysis. Criterion-level scores are not included in this version of the dataset.

Institutions

• University of Alicante

  Valencia, Alicante

Categories

Licence