Step-level evidence integration in bayesian knowledge tracing: A controlled within-model ablation study
Description
This dataset and notebook support a study of whether intermediate step-level evidence provides additional predictive information in Bayesian Knowledge Tracing (BKT) beyond a single terminal correctness outcome. The central hypothesis is that collapsing a multi-step learning episode into one binary response discards meaningful information about the learner’s problem-solving process, and that integrating the observed scaffold-step responses can improve discrimination of subsequent performance. The analysis uses the corrected ASSISTments 2009–2010 Skill Builder dataset. A strict evaluation cohort is constructed from scaffolded episodes in which: (1) exactly one original problem interaction is observed; (2) that original response is incorrect; (3) all scaffold interactions occur after the original response; (4) the original problem is associated with exactly one non-null skill; (5) the learner completes the full scaffold sequence authored for the assistment; and (6) an immediate subsequent original opportunity on the same skill is observed. This procedure yields 8,995 evaluation instances from 1,494 learners across 33 skills. For each episode, step-level evidence is represented as the ordered binary correctness vector of the scaffold responses. The prediction target is correctness on the learner’s immediate next original opportunity on the same skill. Three matched BKT conditions are compared: Terminal-BKT, which represents the current episode only by the original failure; Sequential Step-BKT, which processes the same scaffold responses as sequential observations without learning transitions between scaffold steps; and Length-normalized Step-BKT, which integrates the same responses using the mean log-likelihood ratio and applies one learning transition after the episode. All conditions share the same fitted BKT parameters, pre-episode mastery state, learners, evaluation instances, and prediction targets. The primary findings show pooled out-of-fold AUCs of 0.7116, 0.7202, and 0.7382 for Terminal-BKT, Sequential Step-BKT, and Length-normalized Step-BKT, respectively. The length-normalized condition improves AUC over Terminal-BKT by 0.0266 and over the sequential control by 0.0180. Learner-cluster bootstrap confidence intervals are used for primary inference. RMSE and Brier-score results do not indicate a corresponding calibration improvement; therefore, the main finding should be interpreted as improved discrimination rather than improved calibration. The repository is intended to support reproducibility of this narrow empirical claim. It does not establish superiority over deep knowledge-tracing architectures, general multi-path DAG inference, or generalization beyond scaffold-completed failure episodes.
Files
Steps to reproduce
The analysis is implemented in the accompanying Jupyter notebook IEE_BKT.ipynb. It can be run in Google Colab or a compatible local Python environment. Place skill_builder_data_corrected.csv in a Data directory under the project root. If running outside Google Colab, edit PROJECT_ROOT in the notebook as needed. The analysis records the input file size and SHA-256 checksum before processing. The reference dataset used for the reported results has SHA-256 c425280d26a08f2f1b9b1829a7442c5da46b604100d57574d6b5741123e8fe85 and size 63,054,230 bytes. Run the notebook from the first cell without changing the predefined analysis settings. The main settings are random seed 42, 10 learner-level folds, 3,000 learner-cluster bootstrap replicates, a maximum of 200 EM iterations, convergence tolerance \(5\times10^{-7}\), four deterministic BKT initializations, and an upper bound of 0.49 for guess and slip parameters. The notebook loads the CSV using Latin-1 encoding and verifies the required fields. It then constructs the evaluation cohort by sequentially filtering scaffold episodes for one original event, an incorrect parent response, correctly ordered scaffold events, one parent skill, one original problem definition, completion of the full authored scaffold sequence, and availability of the immediately subsequent original same-skill opportunity. Learners are randomly assigned, using seed 42, to 10 mutually exclusive folds. For each held-out fold and skill, BKT parameters \(P(L_0)\), \(P(T)\), \(P(G)\), and \(P(S)\) are estimated by Baum-Welch/EM using only original interactions from training learners. Scaffold responses are not used for parameter estimation. For each evaluation episode, the learner’s pre-episode mastery state is reconstructed from earlier original interactions on the same skill. Terminal-BKT, Sequential Step-BKT, and Length-normalized Step-BKT predictions are then generated from this identical starting state. No learning transition is inserted between scaffold observations; one transition is applied after the episode. Run the remaining cells to reproduce pooled and fold-level AUC, RMSE, Brier score, 3,000-replicate learner-cluster bootstrap comparisons, Wilcoxon and leave-one-fold-out checks, chain-length and training-support sensitivity analyses, parameter-boundary and shrinkage analyses, calibration diagnostics, ROC figures, and EM-convergence diagnostics. Generated tables, figures, parameters, predictions, and metadata are written to the Output directory.
Institutions
- Hanoi National University of EducationHanoi, Hanoi