Pedagogical affordances of AI tools for reformulations in L2 writing: Profiling algorithmic personas
Description
These data include the meaning preservation and literal similarity for the four AI tools (ChatGPT, Wordvice, Writefull and QuillBot) as well as the lexical and syntactic complexity and readability for the original writing and the four AI tools.
Files
Steps to reproduce
3.2 L2 Corpus The corpus comprised 30 academic essays authored by second-year English majors at a university in mainland China. 3.3 AI tools and procedures Four distinct AI writing tools, namely, GPT-4o, Wordvice, Writefull, and QuillBot, were selected. To ensure consistency across applications, standardized protocols were established for each. Specifically, GPT-4o was provided with the prompt "Please paraphrase the following text in an academic way,” a directive designed to emulate a common student scenario aimed at academic enhancement. For QuillBot, Wordvice, and Writefull, the “Academic” mode was employed to maintain stylistic comparability with GPT-4o. 3.4 Linguistic metrics 3.4.1 Semantic fidelity We employed a semantic textual similarity (STS) approach using the Sentence-BERT (SBERT) framework (Reimers & Gurevych, 2019). Specifically, we utilized the pre-trained all-MiniLM-L6-v2 model to generate high-dimensional vector representations of the texts. 3.4.2 Literal similarity To rigorously assess the extent of verbatim retention, literal similarity was calculated using the SequenceMatcher algorithm from Python’s standard difflib library (Van Rossum & The Python Development Team, 2024). Unlike the probabilistic nature of neural networks, the SequenceMatcher provides a deterministic baseline. The calculation is mathematically exact and invariant. To ensure precision and replicability, all calculations were automated using standard string-matching libraries within the same Python-based environment. This eliminates human error in manual comparison and ensures a standardized evaluation protocol across the original text and all reformulated versions. 3.4.3 Linguistic complexity To capture the multi-dimensional nature of academic writing, we analyzed linguistic complexity across both lexical and syntactic domains using established computational frameworks. Lexical complexity was assessed using the Lexical Complexity Analyzer (LCA) (Lu, 2012), while syntactic complexity was quantified using the L2 Syntactic Complexity Analyzer (L2SCA) (Lu, 2010). 3.4.4 Readability assessment Readability was assessed using the Coleman-Liau Index (CLI), a character-based metric chosen for its reliability in evaluating technical and academic texts where syllable counting can be unreliable. We implemented the calculation programmatically within our Python data processing pipeline to ensure consistent tokenization with the previous metrics. The script extracted the density of letters (L) and sentences (S) per 100 words from each text sample (Coleman & Liau, 1975). These values were applied to the standard linear formula: CLI=0.0588L-0.296S-15.8