cross-posted from: https://hexbear.net/post/9348639

The Generative AI Learning Penalty: Evidence from Chinese Secondary Education

Using 30 months of panel data on 26,811 Chinese students in grades 7-12, we study how generative AI affects homework productivity and learning. The data combine monthly closed-book exams, high-school and college entrance exams, and homework scores and completion time across nine subjects. We exploit staggered AI adoption in a difference-in-differences design. AI adoption raises homework scores by 18% and reduces completion time by 30%, but lowers monthly exam scores by 20% within six months. High-stakes entrance-exam scores fall by 18 and 24%, with the full penalty emerging only after about two years. The losses are largest in social science subjects, followed by STEM and languages, and are especially large for junior students, high-achieving students, and boys. The learning losses are concentrated among roughly 80% of AI users whose behavior is consistent with homework outsourcing, as indicated by exceptionally short homework completion time coupled with high homework scores. AI users who maintain similar homework completion time as non-AI users experience small learning losses.

  • cfgaussian@lemmygrad.ml
    29·
    13 days ago

    A lot of words to say that having someone else do your homework for you means you will do worse when you have to take a test without that help. Which is both logical and something everyone already knew since schools were invented.

    We all hated homework as kids but the reality is that regularly and diligently doing your homework was often the best exam preparation, far better than just cramming before the exam.

    There is a reason why teachers assign homework. And using “AI” to do it for you is little different than asking your friend or your parent to do it for you – chances are you learn very little in the process, and your skills remain stunted.

    • Orcinus@lemmygrad.mlEnglish
      17·
      13 days ago

      I guess I’m too disabled to have a say because I’m far and away not representative of the average recipient of homework but I absolutely hated it. No one made a case for me back then why is had benefits the way you just did. I just refused to do it amd still passed exams, granted I don’t remember much of the contents of them.

      What do you do when someone’s terminally disinterested? So much so you have teachers calling them “wastes of oxygen”? Guess it’s a fringe disability case but I’ve experienced it and seen it happen to others.

      • chinawatcherwatcher@lemmygrad.ml
        16·
        13 days ago

        I just refused to do it amd still passed exams, granted I don’t remember much of the contents of them.

        you really hit the nail on the head here, because i did this too: passing exams via cramming but never actually learning any material. that’s why i’m so interested in learning how to learn rn.

        What do you do when someone’s terminally disinterested? So much so you have teachers calling them “wastes of oxygen”?

        i mean firstly any teacher who is calling anyone a “waste of oxygen” is firstly completely wrong and secondly not being very motivating. learning how to motivate people is, imo, the very first job of any teacher lol. the subject material should be presented in a way such that its utility is made obvious to any student and therefore giving them interest to learn it. students should constantly be encouraged for any interest that they do show, and taught how to think correctly in the process (i.e. focusing more only long-term rather than short-term interests, not demeaning themselves or others for errors, things of that nature). i think with the right teaching method, so long as the subject material actually has utility (or inherent interest, depending on the topic) for people students being disinterested is a structural problem.

      • Commiejones@lemmygrad.ml
        9·
        13 days ago

        This is what is wrong with current education systems.

        The whole system is built for 60% of people do well if they work hard while 20% are abandoned for not being capable and the other 20% are held back from their potential because they “don’t work hard” ignoring that they don’t work hard because the work is easy for them.

        They reward diligence and hard work equally to people who can do the same work with ease.

      • DisabledAceSocialist@lemmygrad.mlEnglish
        3·
        7 days ago

        I just found homework stressful and exhausting, especially as you get home after a full day of school and then have potentially hours of homework yet to go. Insane system.

        • Orcinus@lemmygrad.mlEnglish
          1·
          7 days ago

          It is. But it’s apparently how China does things and they produce the most accomplished scientists nowadays, people who are actively saving the world in the long term. I wonder if it’s because of or in spite of homework. What variables influence this?

            • Orcinus@lemmygrad.mlEnglish
              1·
              7 days ago

              Individually, you are right. But one could argue that in seige countries, being productive and accomplished means the difference between being a resilient socialist state and succumbing to imperialist powers. On the other hand, I think studies have shown that people are at their most productive when working hours are short. But diminishing returns are still returns. I don’t know. But I’m looking forward to the fall of the empire because it means other countries can relax their guard.

  • opiumfree@lemmygrad.mlEnglish
    143·
    13 days ago

    words cant express how happy i am to not have to study some bullshit then take a bullshit test ever again my life…

    • SocialistVibes01@lemmy.ml
      7·
      12 days ago

      study some bullshit

      Hm, Math? Chem? Phys? Bio? shudders Hist, Geog, and Phil?

      You’re making Marx sad in his tomb, comrade.

  • Commiejones@lemmygrad.ml
    8·
    13 days ago

    I don’t know what the numbers mean. What is a “score” and why does it go up to 132? It looks like most of the data is clustered around the 80,90 co-ords and that the rest seems to just be outliers stretched out to push a naritive.

    • AstroStelar [he/him]@hexbear.netEnglish
      7·
      13 days ago

      100 was set as the average score among students that don’t use AI. 120 then means you scored 20% better than the control group average.

      I looked through the study and the graphs are very confusing to read, but the conclusion is very concise so I’ll post it here:

      Conclusion and Discussion

      Using large-scale evidence from a natural school setting, we find that generative AI use substantially reduces learning among secondary-school students. The central pattern is stark: AI raises homework scores and reduces homework time, but lowers performance on closed-book exams. Regular monthly exam scores fall by 20 percent, while high-stakes entrance-exam scores fall by 18 and 24 percent.

      The negative learning effects fully materialize after six months for regular exam scores and after two years for entrance exams. The negative effects on learning outcomes appear to be mostly driven by the 81 percent of AI-using students, who spend less time on homework than even the fastest non-AI student, receive high homework scores matching the capability of generative AI tools they are using, and yet very low exam scores. By contrast, AI students who spend as much time on homework as non-AI students achieve similar exam scores and higher homework scores. [However, almost no AI-using students spend this much time on homework anymore after more than five months of use.]

      Our findings highlight the importance of the demand side and student incentives. Much of the existing RCT literature focuses on the supply-side question of how to design AI tools that scaffold learning while limiting outsourcing. In China, as in many other countries, such tutoring tools already exist and are often available at low or zero cost. Yet most students in our setting do not choose them. Instead, they rely on general-purpose AI tools that provide quick answers and facilitate homework outsourcing.

      How can we incentivize students to choose tutoring AI tools in an environment with general-purpose AI tools that give direct answers? Our findings suggest several possible interventions. First, providing credible information about the long-run learning costs of homework outsourcing may alter student behavior. Second, increasing the weight placed on closed-book, in-person assessments may help restore the connection between effort and reward. Third, parents and teachers may be more effective if they monitor inputs, such as homework time and study effort, rather than outputs such as homework scores.

      I also found the section on who gets most affected interesting:

      • Effect is worse in junior high than high school, suggesting uninformed (over)use makes things worse;

      • “The average estimated effect 6-10 months after adoption for students who report using generative AI 0-1 hours per week is -5 percent, compared to -30 percent for students who report using generative AI 5 hours or more.”

      • The effect is slightly worse for boys than for girls, widening the gender gap (boys on average score worse).

      • High-scoring students who adopt AI see the largest declines, poor-performing students are affected less.

      • Rylo@lemmygrad.ml
        5·
        13 days ago

        Effect is worse in junior high than high school, suggesting uninformed (over)use makes things worse;

        I was always under the impression that the cognitive outsourcing is worse the earlier it starts. I am in academia, and a lot of really bright researchers, professors etc are using generative AI tools, but they have spent a good 20-30 years already reading, researching and producing high-quality output already, they are not using it to get away from thinking about hard problems, simply to speed up the process of tackling them.

        It is way worse for students. I can only speak for mathematics, but the field is trying to come to gripes with what this whole ordeal will mean, and even then we are at least quite shielded from some of the problems with education (quite easy to expand and assess in-person exams, for instance).

        You generally NEED a minimum of 10-20k hours working and thinking about this stuff to understand it at a high level, there are no shortcuts. Now the barrier to be able to put some slop together and pretend you’ve come up with a result is a lot lower, which is the shortcut many students are going for sadly.

        If anyone is interested in the discussion I’ll just put some good starters:

        https://nitter.net/aaswaminathan01/status/2025418710198960388

        https://davidbessis.substack.com/p/the-fall-of-the-theorem-economy

        • AstroStelar [he/him]@hexbear.netEnglish
          8·
          12 days ago

          I hold the belief that LLMs are the final nail in the coffin of an results-oriented education model that has become outmoded. I have personally experienced cognitive dissonance between my idealised notion of education and knowledge acquisition on one hand, and on the other the reality of stress, lack of reward for going beyond “good enough” and fear of my perfectionism yet slowness making me unable to catch up.

          One way I keep myself from collapsing from all the things to worry about is that I try to focus on a few things I feel passionate about: the future of education is one of them.