Learner experience incidents, coding records and expert validation in blended learning
Description
This seven-file package supports the reuse, audit and recalculation of the Traceable Critical-Incident Coding and Prioritisation (TCICP) method. It comprises one overall README and six linked Excel files covering anonymised source records, incident-level coding data, the codebook and decision rules, pattern and priority outputs, coder-agreement and expert-CVI validation, and one complete worked example. Together, the files allow an external researcher to reproduce the reported indicators and trace the worked example from source record to action route.
Files
Steps to reproduce
1. Identify the 116 included incidents. In TCICP - Cleaned Data.xlsx, filter screening status to Included. Confirm 116 rows representing 49 participants, and that 116 included plus 16 not included reconcile to the 132 records in TCICP - Raw Data.xlsx. 2. Reproduce category counts. Tabulate trigger consensus, engagement consensus and outcome consensus across the 116 included incidents. Each dimension sums to 116 and matches the consensus frequency column in TCICP - Codebook.xlsx. 3. Reproduce agreement percentages and kappa. Compare the coder 1 and coder 2 columns for each dimension. Expected values are 86/116 = 74.14% and kappa 0.702 for trigger; 84/116 = 72.41% and kappa 0.663 for engagement direction; 78/116 = 67.24% and kappa 0.609 for perceived outcome; and 74/116 = 63.79% exact agreement for impact, with unweighted kappa 0.528, linear-weighted kappa 0.586 and quadratic-weighted kappa 0.651 using the ordered values -2, -1, 0, +1, +2. Twenty incidents received impact ratings of opposite sign. 4. Reproduce accumulation endpoints. Sort the included incidents by coded incident order and record the position at which each code first appears. All engagement-direction codes have appeared by incident 13, all perceived-outcome codes by incident 21, all trigger codes by incident 22, and all 15 patterns by incident 44. 5. Reproduce pattern-level counts. Aggregate the included incidents by pattern id and recompute incident n, distinct participants, distinct workshops, distinct courses, mean impact and negative impact minus2 n. These reproduce the values in TCICP - Patterns and Priorities.xlsx for all 15 patterns. 6. Reproduce priority scores. Apply recurrence flag = 1 when incident n is at least 5, severity flag = 1 when negative impact minus2 n is at least 2, and cross course flag = 1 when distinct courses is at least 3. Then priority score = 2 x recurrence flag + 2 x severity flag + cross course flag, banded as High 4 to 5, Medium 2 to 3 and Monitor 0 to 1. The result is 7 High, 7 Medium and 1 Monitor. 7. Reproduce the content-validity indices. In TCICP - Validation.xlsx, recode ratings of 3 or 4 as positive across the 390 ratings; confirm 313 positive and 77 critical. For each of the 65 items, I-CVI is the positive count divided by 6. The mean of the item values gives S-CVI/Ave = 0.803; the proportion of items at 1.00 gives S-CVI/UA = 0.231, which is 15 items. Forty-two items met the 0.78 threshold and 23 required review. 8. Inspect the worked example. TCICP - Worked Example.xlsx follows SRC-0A9RD to incident CI-0EROH and pattern PAT-13. Its values can be cross-checked against TCICP - Raw Data.xlsx, TCICP - Cleaned Data.xlsx, TCICP - Codebook.xlsx and TCICP - Patterns and Priorities.xlsx.
Institutions
- Hanoi University of Science and TechnologyHanoi, Hanoi