It’s the question on the lips of every teacher (as well as many parents and students too): Does ChatGPT help or hinder student learning? If in doubt, it’s best to see what the research on this says. And fortunately, a new large-scale review has just been released. So, let’s take a look at what they found.
We can divide our analysis of this paper into two sections; the first are the headline results. The second is what we learn when we dig a bit deeper into them. The former is encouraging. The latter is underwhelming and potentially very concerning, to say the least.
ChatGPT study results: the headlines
An analysis of the results of 69 studies released in the last few years concluded that ChatGPT:
- Improves student academic performance
- Improves student motivation
- Reduces the mental effort it takes to complete the task
At first glance, you can see why this is encouraging. Understandably, who wouldn’t want more motivated students who get better grades and find it less mentally taxing whilst doing so?
Unfortunately, all that glistens is not gold. Look beyond the headline findings, and we see a) why it is important not to rush to conclusions and b) the potential darker side of AI’s impact on student learning.
5 important notes on the ChatGPT study
We’ve spotted five important things that could go unnoticed in this ChatGPT study review that could be potentially useful things to consider:
1. Who participated in the studies
Out of the 69 studies, 84% of the studies took place with university students who were nearing the end of their formal education. Whilst still valuable, this focus poses a significant problem when applying findings to younger high/secondary school students. University students represent a small, specific group: they are typically more highly motivated, academically successful, independent and older.
This demographic is not representative of most students in secondary education, who may have varying levels of motivation, ability, and cognitive development. As a result, these studies provide limited insight into how ChatGPT impacts younger learners who are at an earlier stage of their education journey—yet who often are using AI tools too.
2. How long the interventions lasted for
It was disappointing to read that the vast majority of research into ChatGPT’s impact on student learning only lasted for a short period of time. The researchers report that only 14% of the studies published lasted for more than 10 weeks. Given that learning is a long-term, invisible and often complex process, it would have been more reassuring to see studies that track the impact of ChatGPT over a period of months, not weeks.
3. What happens when you remove ChatGPT
A number of the studies in this meta-analysis allowed students to use ChatGPT for their final assessment. Almost half of them fail to mention overtly if students were or weren’t permitted to do so. This omission makes it difficult to evaluate the tool’s true educational impact, particularly when students cannot access it during tests (which reflects what happens in real life).
Does using ChatGPT for revision create lasting knowledge, or does it encourage surface-level understanding reliant on external support (this is similar to the famous learning vs performance debate, which we have written about here)? For example, in this randomised control trial, researchers found that ChatGPT had a negative effect on learning when it was used during study sessions but not accessible in final exams, as the most common prompt students used was “tell me the answer,” with very few using a learning-type prompt that asked ChatGPT for help in understanding the content.
4. Is “reduced mental effort” a good thing?
The researchers report this finding as one of the headline results. It appears they do so as a good potential consequence of ChatGPT. But is this actually a good thing? While this might seem advantageous, it directly conflicts with well-established principles of learning, such as desirable difficulties and the value of “thinking hard”. These cognitive challenges strengthen memory and understanding.
When students rely on ChatGPT to generate answers or ideas, they bypass critical thinking processes that are essential for deeper learning. For educators, this raises the challenge of ensuring students use ChatGPT as a tool to complement, rather than replace, student cognitive effort. Essentially, if we make the task too easy, then not much will be learnt and remembered.
5. What about Mathematics?
ChatGPT notoriously struggles with Maths. So, we were particularly interested to see what the studies that looked at the impact it had on student learning in that subject found. Disappointingly, out of 69 studies reviewed, only 1 was in Maths. Clearly, any generalisation of the impact ChatGPT has on student learning can’t yet be applied from such a small sample size.
So, does ChatGPT help student learning?
Back to our original question and title of this blog; does ChatGPT help student learning? If we were to take a glance at the 69 studies, the answer would be YES. The inevitably of AI becoming a central part of all aspects of our lives is undeniable. It is going to impact education. What we are waiting to find out is a) it improves the lives of teachers and b) if it improves the learning for students of all ages.
Just as we should scrutinise and double check the output of ChatGPT, we should do the same when reading research papers too. The headlines are alluring, but the devil is always in the details. Upon scrutinising this paper, we conclude that it is still a bit too early to tell with regards to the questions about its impact on student learning. It can clearly improve their output and performance.
But does it improve learning, especially in secondary school students? At this stage, we would suggest that the evidence isn’t there to conclude that yet, and unless students use it to help them think harder and improve their understanding, it will be unlikely to do so. But if we can help them do that, then there may be some potential there. To help students start with that, try using these seven approaches to using AI in the classroom.
