What can attempts and hints tell us about writing?

Start with the stage, then look at the number

Three attempts at understanding a source post raise a different question from three attempts at writing an example. The first might suggest a reading difficulty. The second might suggest trouble connecting a general point to a specific situation.

Either could also expose a problem with our instructions or with the system’s judgment. A repeated rejection is a reason to inspect the activity, not a label to attach to the learner.

Ask what “used a hint” counts

Imagine ten sessions start an activity and five reach the closing stage. Two of those five receive a hint there. That is 20% of all sessions, or 40% of sessions that reached the stage. Both figures are arithmetically correct; they answer different questions.

The same care applies to time. A long pause might mean thinking, checking something elsewhere or leaving the computer. Elapsed time is useful context, but it cannot tell us which of those happened.

This measurement issue has been studied directly: Kovanović and colleagues found that different ways of estimating time on task changed the results of learning-analytics models. Any reported time measure therefore needs a clear account of how it was calculated (Kovanović et al., 2015).

Completion after a hint is a sequence

If a learner succeeds after a hint, we know the order of events. We do not yet know whether the hint caused the change, whether the learner understood it or whether they will use the idea again.

For that reason, we keep questions about the activity separate from questions about learning. “Which stages lead to repeated attempts?” can guide a design review. “Can learners later develop a reason on their own?” needs another writing task and an appropriate study design.

Soderstrom and Bjork distinguish performance during practice from learning demonstrated through later retention or transfer. That distinction supports using activity records alongside a separate measure of what learners can do afterward (Soderstrom & Bjork, 2015).

Compare like with like

Changing the source post, the instructions or the support can change how people respond. PrepTalk therefore keeps the activity version with its structured research measures. A shared interface does not make two activities equivalent.

The pilot retains approved structured measures and the final product. Rejected drafts, hint text and intermediate writing are not part of the ordinary record retained after completion. Researchers need to work within that boundary when deciding which questions the data can answer.