Research questions
What would count as evidence that PrepTalk helps?
A learner completes a reply after receiving several hints. That tells us they finished this activity with support. To ask whether the activity helped them learn, we need to look at what they can do afterward.
First, examine the system’s decisions with teachers
We can investigate feedback quality before collecting classroom data. One proposed study would ask teachers to judge the same set of invented responses against clearly stated task criteria, independently of one another and without seeing the AI’s decisions. We could then compare the validator with their judgments, while also examining where the teachers disagree.
Teacher agreement also needs to be measured rather than assumed. Hallgren explains how inter-rater reliability depends on the rating design and the statistic used, and distinguishes agreement from validity. Even strong agreement would not by itself establish that our criteria measure the intended writing skills (Hallgren, 2012, pp. 23–26).
The cases should include clear successes, missing requirements and acceptable answers expressed in unexpected ways. Teacher review could also examine whether hints address the actual problem, suggest a useful next step and leave the writing to the learner. A separate set of cases would be needed to evaluate changes made after that review.
This would study the quality of the validator and its feedback. It would not establish that learners improve their writing, and invented responses cannot represent every difficulty learners will encounter.
Give the learner another writing problem
One question worth studying is whether learners can use the same skills in a new task without the same support. Can they develop a reason, choose a relevant example and address another writer’s position?
A later task could also help distinguish something remembered briefly from something the learner can still use. These are directions for study design, not a description of a trial that has already taken place.
The distinction between performance during practice and durable learning is central to Soderstrom and Bjork’s review. Retention and transfer tasks help investigate learning beyond the supported activity (Soderstrom & Bjork, 2015).
What recent AI research adds
In a randomized high-school mathematics study, Bastani and colleagues found that a general GPT-based assistant improved practice performance but reduced performance on a subsequent exam without assistance. A version designed with teaching safeguards largely avoided that reduction, without showing a significant exam advantage over the control group (Bastani et al., 2025).
That finding concerns mathematics, not foreign-language writing. It supports checking assisted performance and independent outcomes separately; it does not establish whether PrepTalk’s approach works.
Decide what a comparison would mean
Comparing a supported group with an unsupported group sounds simple, but the tasks, time available and starting differences between learners all matter. A study would need to make those choices explicit before interpreting a difference in the final texts.
The way writing is judged matters too. If the question is whether people develop arguments more effectively, counting grammatical errors alone will not answer it. The outcome measures need to match the question.
Use the activity records to ask better questions
Attempts, hints and completion can show where the activity is difficult. They might help explain why an overall result differs between tasks or learners. They cannot replace an outcome measure of writing.
Learners’ accounts of the experience would also be useful. A hint can look sensible to its designer while feeling confusing or intrusive to the person receiving it. Any additional research data would need its own agreed collection and consent arrangements.
Where PrepTalk stands
The current research section presents the teaching model, activity designs and questions we want to investigate. It does not report a completed effectiveness study. The dashboard uses example data.
The public website is live. Classroom deployment still requires curriculum and validation review. As studies take shape, we will describe their methods and status here; any findings should include their limitations.
This page sets out questions for future research. It is not a registered protocol, recruitment notice or report of findings.
References
- Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8(1), 23–34. Publisher PDF.
- Soderstrom, N. C., & Bjork, R. A. (2015). Learning versus performance: An integrative review. Perspectives on Psychological Science, 10(2), 176–199.
- Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), e2422633122. Open full text.