Experimental Learning Impact of Generative AI

Sources#

• Andrew Ng: The Biggest Opportunities in AI Aren't Where You Think
• Designing Against Deskilling: Metacognitive Feedback Reduces Cognitive Offloading to LLM Assistants
• Experimental Evidence on the Learning Impact of Generative AI
• Training novices to think, or giving them LLMs? Evidence from an RCT
• Verified, not generated: expert-verified AI study materials and the distribution of learning gains in a university course

Summary#

Zara Contractor and Germán Reyes (Middlebury College, arXiv 2607.08849, July 2026) run the study most of the AI-and-learning corpus was missing: a randomized controlled experiment with an objective, proctored, unaided skill measure. 211 undergraduates learn an unfamiliar technical topic (blockchain, carbon capture, or CRISPR) and write an analytical essay in a 35-minute lab session, randomly assigned to AI-allowed or AI-forbidden; one week later all of them return and are tested and write again without any AI or resources. The headline: AI access raises immediate test scores by 0.27 SD, about 76% of that gain still shows up a week later unaided, and essay quality rises only after AI is removed. But the durable part is not uniform — it belongs almost entirely to students who used AI to explain concepts (augmentation), while students who used it to draft their text (automation) post large short-run essay gains that vanish completely once AI is gone. This is the near-controlled test of "outsource your thinking, not your understanding" and the objective-skill measure the automation–optimism link's self-report could not supply.

Evidence note. empirical and causal (randomization, F-test balance, double-lasso controls, ITT + TOT/2SLS) — the strongest design in this vault's learning cluster. But scope it honestly: elite liberal-arts undergraduates (mean GPA 3.68, SAT 1386, >80% already AI-adopters), a proctored 35-minute lab task, off-the-shelf ChatGPT (GPT-4o), one topic, one week. It measures learning per unit of time with time-on-task held fixed (AI changed how the hour was spent, not its length — plausibly because the lab offered few competing uses of time). The authors explicitly do not claim that real-world AI adoption raises learning overall: outside the lab students choose how long to study, and many use AI to save time. It informs, but does not settle, workplace skill-atrophy.

The design in one paragraph#

Between-subjects randomization at the lab level (eight time slots × two parallel labs). Session One: baseline 5-question test → 35-minute learning phase (read, search, draft a ~500-word essay; AI-allowed group gets a logged-in ChatGPT tab, AI-forbidden gets Google) → unaided 5-question post-test. Compliance enforced by proctors, ChatGPT logs, and interface screenshots. Session Two (~7 days later, everyone unaided): 10-question test + a 20-minute essay on the same topic, complementary prompt. Learning is measured two ways: knowledge tests (factual/conceptual recall) and essays (higher-order analysis), the latter graded blind by 311 master's/PhD Prolific graders plus an LLM grader, averaged, with objective linguistic features (length, readability, lexical diversity, textual similarity, and Pangram AI-detection) alongside.

First stage was strong: assignment lifted AI use by 67.3pp off a near-zero control; 68% of the treated used AI; 88% found it helpful. In the logs, the most common use was explaining concepts (57%), then drafting (31%), then summarizing (17%).

Durable test-score gains, concentrated in the middle — and skewed to the able#

• Immediate: +6.7pp on a 56.3% control baseline = +0.27 SD (ITT; TOT +10.0pp / +0.40 SD). More than double the 0.10 SD median of Kraft (2020)'s 747 education RCTs, comparable to the 0.29 SD of structured human tutoring, ≈ a one-SD jump in GPA, ≈ $8,600/student of school spending.
• Retention (one week, unaided): +5.1pp = +0.27 SD, ~76% of the immediate effect persists (the two are statistically indistinguishable). Durable, partially decaying knowledge.
• Shape: the gain sits in the middle of the score distribution — the tails (0/1 and near-perfect scorers) barely move (Figure 4).
• Who gains: larger in the upper GPA/SAT quartiles (bottom quartile ~0.05 SD; upper quartiles 0.19–0.40 SD) — so AI access may widen learning gaps, a genuine tension with the floor-raising story of software democratization.
• Felt vs. measured: self-assessed knowledge shows no treatment effect (β=0.02). Students learned more but did not feel they had — the mirror image of the felt-vs-measured worry the AEI self-report leaves open.

Higher-order skills surface only after AI is removed#

In Session One, treated essays are longer, simpler, and 12.3pp more AI-flagged (a 96% jump), but overall quality rises only slightly and imprecisely (+0.17 SD, p=0.25) — because the essay is partly AI-written, blending output with learning. Notably, no homogenization: within-group essay similarity is flat, contra the convergence Brynjolfsson et al. (2025) found in workplace writing (open-ended prompts admit many valid answers).

The clean read comes from Session Two, written unaided a week later (Figure 6): the AI-detection effect collapses to 0.0pp and the stylistic differences fade — proving the Session One style shift was AI text entering essays, not durable change — while quality gains emerge: writing style & clarity +0.30 SD (p=0.016) and relevance to prompt +0.26 SD (p=0.041). The learning was real and it raised higher-order skill, not just fact recall — but it only became visible once the AI crutch was taken away.

Augmentation vs. automation: the controlled test of "outsource thinking, not understanding"#

The paper's most load-bearing result. Classifying treated users from their ChatGPT logs: 49% augmentation (AI works with the student — tutor, explainer), 32% automation (AI does the work — drafts the essay), 8% mixed, 11% off-topic. Validation that the split is real: automation users' essays are 53.6% AI-flagged vs 20.9% for augmentation users, and automation users spend 17% less time reading/searching.

Automation's Session One advantage is AI-produced output that does not survive its removal; augmentation's gains persist unaided, consistent with skill accumulation. This is a within-experiment demonstration that AI's short-run productivity boost and its long-run learning effect can point in opposite directions, decided by how the tool is used — the exact shape of Karpathy's thesis, and consistent with Strömberg et al. (2026) (AI raises homework, lowers exams, losses concentrated among outsourcers), Bastani et al. (2025) (base GPT-4: −0.19 SD later), and Shen & Tamkin (2026) (engineers who learned a library via AI scored worse unassisted). The meta-analysis (Figure 5, grand mean 0.18 SD across 22 estimates) sorts the whole literature along this axis: losses cluster where AI could do the practice for you, gains where AI plays coach.

Mechanisms#

• Time reallocated, not saved. No effect on total learning time (unlike the large workplace time-savings of Noy & Zhang 2023), but the mix shifts −5.3pp writing / +4.4pp reading-and-searching — effort moves from producing text to absorbing it (task execution → stewardship, à la Copilot users spending half their time verifying).
• Enjoyment up 13% (+0.66 pts, p=0.031): AI made learning more engaging, not mechanical.
• Cheating up but not the driver: rule violations rose +12.6pp combined (p=0.005), yet a back-of-envelope bound attributes at most ~2.2pp (about a third) of the test-score gain to cheating — the effect is mostly real learning.

Beliefs: experience corrects the magnitude, not the direction#

Both groups expect AI to help, but only the treated gauge how much correctly. Control students predict a +25.2pp own gain — about 5× the actual 5.1pp; treated students, who used it, predict +3.8pp, close to the estimate. Perceived gains track actual gains across subgroups. And students already hold the right model: 69% name both a help and a harm channel, 54% say the effect "depends on how AI is used," and their two most-named mechanisms (AI explains/tutors 40%; AI shortcuts the work 42%) map exactly onto augmentation vs automation. Students grasp the mechanism from the outset; firsthand use only calibrates the magnitude — a belief-updating result that rhymes with the professional-misjudgment literature (Becker et al. 2025 / METR: developers predicted +24% speedup, measured −19%).

The practitioner statement of the automation half (Ng, August 2026)#

Andrew Ng — co-founder of Coursera, founder of DeepLearning.AI — says the deskilling half of this result out loud, as a general claim, to a general audience (Andrew Ng: The Biggest Opportunities in AI Aren't Where You Think, Silicon Valley Girl, 2026-08-28, practitioner-opinion): "frankly AI models are terrible for learning… Students score higher on homeworks when they use AI. Yay, higher homework scores. But retention, their long-term performance is much worse because their AI do the work for them." His mechanism is this page's: "when you ask AI to do work for you you're cognitive offloading to AI which is great because that's how society moves forward and gets work done but human retention is much worse." His datum is himself: six months after asking a model how a front-end/back-end component works, "I don't remember the answer… I ask AI again."

Set against the design above, the claim is right about one arm and wrong as stated. Homework up, retention down is the automation row of the table — Session One essay quality +0.54 SD, Session Two +0.02 — and Strömberg et al.'s homework-up-exams-down pattern in the meta-analysis. It is not the augmentation row, where gains persist unaided (+0.29 SD on the test a week later), nor the pooled ITT, where ~76% of the immediate gain survives. Ng's qualifier — "LLMs as they are most commonly used are terrible for learning. I'm not saying there's no way to use it in a way that is good for learning" — is where the two meet, and it turns his claim into an empirical one about the mix: in this experiment's logs the most common use was explaining concepts (57%), and the augmentation share (49%) exceeded the automation share (32%). Whether that mix holds outside a proctored lab, where grades reward output, is the third open question below; Ng's "as most commonly used" assumes the answer. His "the data is very clear" cites nothing, and the meta-analysis on this page (grand mean 0.18 SD across 22 estimates, sign decided by use mode) is what the data actually looks like.

The conflict of interest runs in the unusual direction: he makes the claim while announcing LearnVector, a venture building one-to-one learning experiences that his diagnosis makes necessary. That does not make the diagnosis wrong — it matches the randomized result for automation-mode use — but it is a reason the claim arrives without its augmentation half.

What it settles — and what it doesn't#

It settles, for this population and task, that off-the-shelf AI can produce durable, objectively measured learning gains, dissolving the "AI must erode learning" prior — but conditions the result on use mode. It does not settle: whether the gains hold outside a proctored lab where time is endogenous; whether they hold for non-elite learners (they skew to the able, suggesting not uniformly); or whether the same augmentation/automation split governs workplace deskilling, where the analogue evidence is still mixed (Brynjolfsson's support agents learn; Budzyń's endoscopists deskill).

The sibling RCT: train the human instead of restricting the tool (Bocconi × OpenAI, 2026-08)#

The design above asks what AI access does. A second randomized experiment asks the question this page's third open question actually poses — whether an upstream intervention can change what novices do with the model — by crossing LLM access with a taught cognitive skill. Asirvatham, Betti, Brown, Camuffo, Chatterji, Fumagalli, Gambardella, Mariani, Pandey, Ramos, Salvucci & Simic, Training novices to think, or giving them LLMs? (Training novices to think, or giving them LLMs? Evidence from an RCT, CEPR DP21882, August 2026, empirical): a preregistered 2×2 with 1,053 first-year undergraduates at Bocconi, randomized at the level of 13 intact class sections (Control 249 / Causal 256 / GPT 197 / Causal×GPT 351, the combined cell oversampled for power on the interaction). Arm one is a ~6-minute causal-reasoning game — 12 binary-choice items in four sections, each with a worked example and feedback, against a placebo that poses the same 12 items with no scaffolding and no feedback; the words "causal reasoning" are never used and the scenarios are city-policy, not business, to avoid anchoring. Arm two is ChatGPT Edu (GPT-4o) in a second browser tab. Both arms then write a ≤180-word consultant recommendation on a real merchandising problem in 45 proctored minutes, scored 1–5 on awareness and usage by three of twenty trained, condition-blind master's raters.

COI, and it is structural. Three of the twelve authors are OpenAI-affiliated or OpenAI-contracted, the treatment is ChatGPT, the paper is published on cdn.openai.com, and the measurement stack is OpenAI's — the causal-reasoning attributes are LLM-scored against a rubric, ideas are extracted by GPT-5.2 (with Claude Opus 4.6 as second judge), similarity to experts is cosine distance in text-embedding-3-large. The one outcome outside that loop is the human-rated awareness/usage score, and it is the one showing the largest pro-LLM effect, which cuts the other way. The counterweight worth naming: the result least useful to an LLM vendor — that only the human training arm raises idea diversity — is reported as the paper's headline finding.

The training worked, and the tool did not crowd it out. Relative to control, the causal arm raises mechanism identification ~0.55 SD (+0.241, s.e. 0.028) and falsification logic ~0.85 SD (+0.869, s.e. 0.062); GPT alone moves neither (−0.008 and +0.029, both n.s.). GPT instead raises coherent logic ~0.45 SD (+0.214, s.e. 0.043), where causal training adds only ~0.15 SD. Critically, every Causal × GPT interaction is positive and significant — mechanisms +0.129 (p<0.01), falsifiability +0.203 (p<0.05), coherent logic +0.228 (p<0.01) — on top of the main effects. Trained novices with a model in reach still reason like trained novices; the skill is complementary to the tool, not substituted by it. (Causal participants also finished the training game ~2.7 min faster and scored ~11 points higher on it, Figure A.1 — a pre-GPT measure, so it grades the training, not the tool.)

But the evaluation rewards the tool, not the training. GPT raises the awareness/usage score by +0.862 (s.e. 0.079, p<0.01) against a control-group estimate of 2.09 on the 1–5 scale — about 1.0 SD of the outcome's own dispersion (SD 0.843) — and moves solutions measurably closer to three independently interviewed domain experts (+0.026 / +0.028 / +0.044 cosine similarity on means of 0.794 / 0.753 / 0.806, all p<0.01). Causal training scores negative (−0.111, s.e. 0.040, p<0.01) and the interaction is indistinguishable from zero. Post-double-selection lasso decomposes the GPT effect: 0.862 → 0.490 → 0.448 → 0.362 as content, diversity and text-style controls enter, so ~42% (the paper's prose rounds this to "about half") survives everything, i.e. GPT improves substance and not only polish. Inter-rater agreement is itself a treatment outcome: ICC 0.534 / 0.493 in the GPT arms against 0.275 / 0.322 in the non-GPT arms — LLM-assisted text is not just better-scored but easier to score.

The negative result is where the page earns its place: the rubric penalizes exactly what the training produces. In the same lasso, coherent logic (+0.355) and number of ideas (+0.092) load positively while falsifiability (−0.109), mechanisms (−0.166) and between-solution idea diversity (−7.676) load negatively, all p<0.01 or better. Scaled by each variable's own SD, the conformity premium (−0.29 points per SD of distance from the modal answer) is roughly 1.6× the reward for within-solution variety (+0.18 points per SD) — and causal training's +0.47 SD of between-solution diversity, run through that coefficient, is −0.14 points, which is essentially the whole negative Causal coefficient. The causal arm is the only treatment that raises idea diversity (within-solution +0.55 SD, between-solution +0.47 SD; GPT's effects are +0.12 SD and zero), and GPT neither adds to it nor destroys it. The authors' reading, which this page adopts: the training did not fail — the assessment did not ask for what the training produced.

What it cannot say, in the authors' own words: "Whether the second component reflects knowledge that participants acquired and retained, or output they procured without acquiring anything, we cannot say." There is no unaided post-measure — no removal of the tool, no retention test, no second session. On this page's axis that is decisive: the Bocconi design measures assisted output, where Contractor & Reyes measure what survives withdrawal. It therefore cannot produce an automation row, and nothing here contradicts the vanishing-gains result; the two experiments measure different halves of the same sentence. Two further scope limits: randomization is at the class level (13 clusters, standard errors clustered on mother tongue × high-school type × class), and the task is deliberately one on which "LLMs are likely to be well-trained" — a documented marketing framework with a codified rubric, i.e. the most favourable possible terrain for the model and the least favourable for a novice's own concept.

The third RCT: leave the tool alone, change the offloading decision (LMU Munich, 2026-09)#

Contractor & Reyes observe the use mode and Bocconi trains the user beforehand. Maier, Schwabe, Schneider & Feuerriegel, Designing Against Deskilling (Designing Against Deskilling: Metacognitive Feedback Reduces Cognitive Offloading to LLM Assistants, arXiv 2609.20143, 2026-09-17, empirical, a CHI 2027 submission) is the first design in the vault that manipulates the use mode inside the session, and it does it without restricting the assistant. The study is a preregistered (AsPredicted) online experiment with N=704 UK Prolific adults (mean age 39). It runs a 2×2 plus a no-AI control: 146 control, 137 AI-only, 144 metacognitive feedback, 138 effort-based reward, 139 combined. Participants did 10 fraction-arithmetic learning items next to a GPT-OSS-120B chat panel, then 6 unaided transfer items (new operands, same operations). The assistant was prompted to give method guidance by default and a complete answer only on explicit request. An LLM-plus-numeric-check classifier then labeled each item by the highest help level requested: none, guided help, intermediate result, or complete answer. "Answer offloading" means requesting the complete answer, so this study classifies the automation row itself instead of inferring it.

The two interventions target the decision itself, not what the model will say:

• Metacognitive feedback. After each item, an LLM-written three-part card: How you used the assistant, What that meant for your practice, and Something to notice, plus a bar placing the item between "All with you" and "Mostly the assistant". It is informational, not instructional: it never says how much help to ask for next.
• Effort-based reward. A correct item scores 10 points with no help or guidance only, 5 after an intermediate result, and 1 after a complete answer.

Results (Table 3, reconciled against the prose):

The feedback × reward interaction is null on both outcomes. The participant-level ANOVAs in Appendix H reach the same verdicts: feedback cuts offloading by −5.05pp and raises the score by +5.28pp, and the reward's offloading p is.0501.

What makes it worth a section:

• Pricing drove out the wrong help. The reward cut any assistant use sharply (OR 0.39, p<.001), and guided help trended down (OR 0.69, p=.060). But among the items where participants did use the assistant, answer offloading rose to 62.8% of requests, against 58.2% in AI-only (Figure 5). Feedback cut it to 51.2% and moved the rest into guided help (34.8%) and intermediate results (13.9%).
• The two arms felt different. The reward was the most effortful arm: perceived effort was higher under the reward than under feedback in both phases, Δ≈0.6 on a 9-point scale, p=.010. The authors' design rule follows: scaffold productive engagement rather than pricing it.
• Feedback breaks repetition, not first use. The feedback arm was as likely as AI-only to use the assistant at all (66.7% vs 68.6%) and to offload at least once (38.2% vs 39.4%). The difference is persistence: after offloading one item, participants offloaded the next one 69.7% of the time without feedback and 55.9% with it (interaction OR 0.54, p=.040).
• The deskilling link is correlational. Across conditions, each +10pp of answer offloading goes with 32% lower odds of solving a test item unaided (OR 0.68 [0.62, 0.74]). Offloading itself was not randomized, and the authors flag prior ability as a possible confound.
• Feedback changed behaviour, not judgment. Metacognitive accuracy (predicted vs actual score) and sensitivity (confidence AUC) did not differ between conditions. Every condition under-predicted its own score, including AI-only (+0.53 items) and control (+0.93) (Appendix I.3).
• Offloading depends on the person, and the fix works for everyone. The participant random intercept carries 82% of the latent variance in offloading. Low perceived confidence (OR 0.32 per point) and low need for cognition (OR 0.52) predict it; trust in AI does not. None of these moderates the feedback effect, so there is no sign of intervention-generated inequality.

The H1 null disagrees with the literature on access, and the paper names the likely reason. Bastani et al. (−0.19 SD), Liu et al., and this page's own automation row all find unrestricted access harms unaided performance. Here, access did not significantly harm it. The authors attribute this to the default: complete answers came only on explicit request. Fewer than 40% of AI-only participants ever asked for one, against 61% of Liu et al.'s participants who used their assistant mainly for answers, and most of Bastani's unrestricted transcripts. This is a scope condition, not a supersession. Neither result overturns the other, because they tested different assistants. It turns "does access deskill?" into a question about the interaction default. It also sits beside Contractor & Reyes's positive ITT, where off-the-shelf ChatGPT was used mostly to explain concepts (57%). Both suggest the damage is concentrated where answer delivery is frictionless.

How far it reaches.

• Effect size. Both confirmatory effects clear one-sided tests only: the H2a interval touches 1.00 and the H2b interval crosses it. The feedback arm's unaided score (0.67) is at, not above, the no-AI control's (0.65). The authors state this themselves: reducing answer offloading preserves learning, but no intervention yet beats a no-AI baseline.
• Duration. A single session with an immediate test on fraction arithmetic, so durability is untested.
• Classifier. It was validated only on synthetic dialogues: 198 development and 198 held-out, with perfect agreement. There is no human audit of the 3,969 real participant messages.
• Answer-on-request default. The study depends on it, so the feedback effect is measured on an assistant already built to make offloading deliberate. Whether it survives an assistant that volunteers answers is the authors' own stated limit.

The other sign: verified materials, gains in the bottom tail (King's College London, 2026-10)#

Contractor & Reyes find AI's learning gains skew to the able, with the bottom GPA/SAT quartile at ~0.05 SD. Dang & Nguyen, Verified, not generated (Verified, not generated: expert-verified AI study materials and the distribution of learning gains in a university course, arXiv 2610.07097, 2026-10-05, empirical), report the opposite sign. They also propose a construct that predicts which sign a deployment gets.

The construct: judgement burden. It is the expertise and effort a learner must supply to separate usable from unusable AI output before learning from it. It has two parts: how variable the output is, which sets how much checking is needed, and who is responsible for the checking. The burden is regressive, because checking draws on the subject knowledge, time and confidence weaker students have least of. The paper sorts the levelling and widening field results by it. Brynjolfsson's support agents, Noy & Zhang's writers and Dell'Acqua's in-frontier consultants were levelled, because the AI carried the expertise or the task was one where output was reliably good. Otis et al.'s Kenyan entrepreneurs (low performers worse, high performers better, no average effect), Dell'Acqua's out-of-frontier task and Bastani's unrestricted GPT-4 students were widened or harmed, because the user had to decide what to accept. All of these are cited secondhand through this paper. Where the learner screens, gains should go to the able. Where an accountable person screens before release, they should go to the weak. Contractor & Reyes's students prompted ChatGPT themselves and so sit in the first cell; this course sits in the second.

The design. It is a two-cohort difference-in-differences, not an RCT. The setting is a compulsory first-year Principles of Economics course at a UK business school, with 85 students per cohort, 170 students and 340 exam marks in all. The exam gives 50 marks to each half and carries the whole module mark. In 2024/25 only the macro half (Part 2) received NotebookLM-generated ten-minute podcasts, FAQs and quiz-style study guides built from the course's own slides. A named graduate TA checked each item against the sources and regenerated what was vague or misaligned ("first drafts were usable but not releasable"). Items were released labelled "AI-generated and academically verified". Students never prompted the model. Use was voluntary and carried no marks. The 2024/25 exam was the resit paper drafted and moderated alongside the 2023/24 main paper, never sat, which fixes the two halves' relative difficulty before the intervention.

Results (Tables 1–3 reconciled against the prose and Figure 2):

• Average effect: +2.34 marks of 50 (0.33 SD of the pre-period Part 2 distribution, 0.40 SD pooled), p=0.045 HC1 and 0.013 clustered. It is protection from a decline, not a rise: Part 2 rose 0.73 (n.s.) while the untreated Part 1 fell 1.61 (p=0.021).
• Where it lands: the share of marks below 30, the upper-second (60%) boundary employers screen on, fell 24.7pp relative to the counterfactual (p=0.001). Treated-half marks below 30 went from 20 to 9; untreated-half marks from 6 to 16. The effect is significant at every threshold from 23 to 31 marks and at none above. Quantile DiD: +11.0 marks at the 10th percentile (95% CI 5.0–13.0), +5.0 at the 20th (CI crosses zero), roughly zero from the 30th up. About three-quarters of the average comes from the bottom quintile (+8.9 marks). The treated half's SD fell 7.09 → 5.03 (Brown-Forsythe p=0.004) while the untreated half's held.
• Fragile mean, robust threshold: drop the 3–8 lowest pre-period Part 2 students and the average falls to 1.88–1.41 marks and loses significance (HC1 p 0.099–0.198). The below-30 estimate stays at −19.9 to −22.9pp, p≤0.006. The authors' methodological point follows: an evaluation reporting only the mean would have judged the intervention marginal, and a reviewer who probed the mean would have judged it fragile. Neither verdict says anything about the students the intervention was built for. They propose reporting three distributional quantities by default: the change in share below progression-relevant boundaries, the change in dispersion, and the threshold or quantile profile.
• Students' accounts (36 interviews, field notes, illustrative only): the label gave students a reason to engage, and the trust it carried attached to a known person rather than to the model: "I don't trust an AI. But I do trust my GTA." It did not end their scrutiny. Several students described the relief in extraneous-load terms ("I could focus on understanding rather than fact-checking").
• Cost: about 30 minutes per weekly set, most of it checking, and 2.5 staff-hours in total (~1.8 minutes per student). Against unverified generation, verification is the entire marginal cost.

What it cannot say. The intervention bundles verification with new material in three formats. Accessible extra material and retrieval-practice quizzes could produce lower-tail gains with no verification at all, and the authors concede the design cannot apportion the effect. The comparison the construct needs, verified vs unverified releases of the same material, was not run. There is one pre-period cohort, so parallel trends are untestable. Effort substitution between halves remains possible: the within-student correlation rose 0.284 → 0.482 rather than falling, but not significantly (p=0.135). No individual usage logs were kept, so this is intention-to-treat on access. The cohort was high-attaining (module means ~70%, no failures), so nothing here speaks to the pass mark. One shape the paper reports without stressing: the pre-period treated cell was the outlier. Its SD was 7.09 against 4.3–5.0 in the other three cells, and 20 of its marks were below 30 against 6 in the same cohort's other half. The drop-the-lowest tests answer the few-extreme-students version of that worry, but not a cohort-level shock to one half of one year's paper. The paper does not state whether the authors taught the course; the "Part 2 lecturer" produced the materials.

How it sits against this page's other results. It is a scope condition, not a contradiction. Contractor & Reyes's widening result is about unrestricted access to a model the student prompts. This one is about pre-screened artifacts the student only consumes, a deployment none of the RCTs above tests. Two convergences with the LMU trial: both locate the damage where answer delivery or acceptance is unscreened by anyone, and both find the lever is the interaction design, not the model. The construct also gives Returns to Expertise in Agentic Coding its inverse. When the user screens, expertise is what converts AI output into value. When screening moves upstream, the gradient flips. The cost of that move is labour, the verification bottleneck in a classroom. The authors name its failure mode: a label goes on signalling after the checking behind it has become cursory, which is a rubber-stamp review with students as the downstream consumers.

Connections#

• The Tragedy of the Cognitive Commons — the framework this experiment's augmentation/automation split is the cleanest measurement of: Lovett's "augmentation without internalization" predicts exactly the vanishing of automation users' gains, and names it the mechanism no employment statistic can detect

• AI-Exposed College Majors at Labor-Market Entry — where classroom offloading may reach the labor market. The Census graduate study proposes that AI-inflated grades (Chirikov 2026: largest rises in homework-heavy, AI-exposed courses) stop signalling skill, and that employers respond by hiring fewer graduates. If so, the automation that suppresses learning in these RCTs also degrades the credential

• Organizational Complements to AI — where the HAT substitution model lives, and the mechanism-level check on its prediction P7 (workers respond to substitution pressure by upskilling): whether that response works is the augmentation/automation split measured here, since automation users' gains vanish once AI is removed — a distinction the model's private-effort formulation has no term for

• The Household Production Boundary — the scale of the demand this experiment speaks to: education is 20.7% of global AI conversations and over-represented 5.8× against time actually spent on it in the US, higher still in developing countries — making the augmentation-vs-automation distinction measured here the one that decides whether that volume compounds or hollows

• The Automation–Optimism Link — the primary complement. The AEI survey found heavy delegators self-report no learning loss; this experiment supplies the objective, randomized measure that survey could not — and it both agrees (augmentation users' learning persists unaided) and exposes the divergence the survey can't see (automation users' gains are hollow, and "automation share" pools both types)

• Outsource Your Thinking, Not Your Understanding — the near-controlled test of the thesis: augmentation (AI helps you understand) builds durable skill; automation (AI does the thinking) leaves nothing once removed — Karpathy's principle, randomized

• AI Brain Fry — both put an objective, measured number on AI's cognitive effect (there, oversight fatigue → +11%/+39% errors; here, learning → +0.27 SD, or hollow gains for automators) against the softer self-report signal; the deskilling half of this paper is that page's mechanism in a learning task

• Returns to Expertise in Agentic Coding — the heterogeneity rhymes: gains skew to higher-ability students, and augmentation ≈ using AI to deepen the understanding that Anthropic's study finds is what amplifies an agent; both say the benefit accrues to whoever brings (or builds) understanding

• Exposure Taxonomy: Observed, Theoretical, Reported, Anticipated — the belief half: like reported/anticipated exposure, students' perceptions of AI's effect are directionally right but miscalibrated in magnitude until firsthand use corrects them

• Printing Press Software Democratization — the tension: democratization predicts AI raises the floor, but here gains concentrate in the upper ability quartiles, hinting AI may widen learning gaps rather than close them. The 2026-10 King's College DiD supplies the floor-raising case and the condition for it: gains land in the bottom quintile when an accountable tutor screens the AI material before students see it

• Verification as the New Bottleneck — the classroom instance of the hub: in the King's College course, verification is the whole marginal cost of AI study material (2.5 staff-hours), and the distributional benefit depends on someone doing it

• The Data Wall and the Validation Commons Are One Supply Constraint — this experiment used as one of three corroborating measurements that validation capacity can erode before any capability threshold arrives (beside Budzyń's endoscopists and Vicente & Matute's 80.7%). What it supplies that the others don't is randomization; what keeps it from settling the commons question is population and duration — one week, elite undergraduates, a proctored task rather than a career

• Configurable Human Participation — the system-side mirror, published the same week: HAS-Bench's agency scale runs on the same augmentation-vs-automation axis (A1–A2 automation-oriented vs A3–A5 augmentation-oriented), and both land the same shape of result — the value of AI–human collaboration is decided by its mode and timing, not its amount (there: more agency at A4 breaks tasks A3 solved; here: automation-mode use yields hollow gains that vanish with the tool)

• Andrew Ng — states the automation half as a general claim ("terrible for learning"), with a stake in the remedy; the section above sorts his claim against the two arms

• The Three Loops of AI-Native Building — the builder's side of the same author's learning claim: Ng's typing app is cheap implementation deliberately spent on a non-offloading tool for a child, in the interview where he calls LLMs terrible for learning

• Does the Augmentation/Automation Split Govern Skill at Work? — the workplace transfer of this page's use-mode split, tested against everything the vault holds on working adults. The sign replicates (Budzyń, Dell'Acqua, Wiles and Vicente & Matute on the automation row; Everett and Brynjolfsson's support agents on the augmentation row) but all six are secondhand and none splits use mode inside a design. The binding disanalogy is not the elite sample or the one week — it is that this experiment's mechanism requires time-on-task held fixed, and that workplace volume generates a third mode (skipped review) a proctored session cannot

• AI Writing in PhD Dissertations — the field analogue of the automation row, at the highest-stakes writing task in doctoral training. 29.4% of 2026 US STEM dissertations have AI-written abstracts, and the writers leave academic research more often. Use mode is unobserved, so the RCTs' augmentation/automation split cannot be applied to it, which is the measurement the field result most needs

• Screen-Relative AI Timing — a team-creativity RCT with a matching lever result. A standing overreliance warning cut the AI-matched share of submitted text from 65% to 47% without changing rated quality, while when the same assistant was available moved quality by 0.62 SD. Process design beat exhortation there, as interface feedback beat the points penalty here

Open Questions#

• Time-on-task is held fixed by the lab; the authors flag that real-world learning depends on how students reallocate saved time. Does the augmentation dividend survive once students can spend the hour AI frees on something else entirely?
• Gains skew to the able (upper GPA/SAT quartiles). Is the widening-gaps signal a durable property of unrestricted AI, or an artifact of a high-ceiling elite sample where the bottom quartile has little room to move? Partially answered (2026-10-21) by Verified, not generated: expert-verified AI study materials and the distribution of learning gains in a university course (empirical, quasi-experimental DiD, n=170). This is also a high-attaining cohort (module means ~70%, no failures), yet ~75% of the gain came from the bottom quintile (+11.0 marks at the 10th percentile, −24.7pp below the upper-second boundary). So a high-attaining sample does not by itself keep the bottom tail from moving, which weakens the ceiling-artifact reading. The paper's judgement-burden account instead predicts widening for exactly the unrestricted, student-prompted use tested here. It does not test that arm: its students never prompted a model, and verification is bundled with new material. The ceiling-vs-property question for unrestricted access stays open.
• Judgement-burden prediction: does the same AI study material released unverified produce smaller lower-tail gains, or none, than the verified release? The King's College DiD found bottom-quintile gains for verified material but could not separate verification from the material itself; the authors name a section-randomized verified-vs-unverified release, plus a label-only arm and named-vs-anonymous verifiers, as the test.
• Residual of the resolved lever question (see Resolved Questions): do per-item metacognitive feedback and an answers-on-request default hold at scale and under real grades, outside one fractions session? Settler: a graded-course or field RCT.
• Does the same use-mode split govern workplace skill accumulation (the open question The Automation–Optimism Link and AI Brain Fry leave for workers), or is a proctored one-week academic task too unlike on-the-job learning to transfer? (Promoted 2026-09-05 lint: the vault now holds the workplace-side pieces to test against — the observed automation-share drift on Anthropic Economic Index, the recruiter case in the controlled-variance study, the self-report on The Automation–Optimism Link, the offloading account on AI Brain Fry.) Partially answered (2026-09-05): Does the Augmentation/Automation Split Govern Skill at Work? — the sign transfers. Budzyń's endoscopists, Dell'Acqua's consultants, Wiles et al. and Vicente & Matute all reproduce this page's removal signature (performance high with the tool, no advantage without it) on non-students, while Everett's collaborative clinical workflows and Brynjolfsson's support agents supply the augmentation row — but all six reach the vault secondhand through The Tragedy of the Cognitive Commons, none classifies use mode inside a design, and none takes an unaided post-measure. The answer to the second clause is not "too unlike to transfer": the disanalogies that actually bite are that this design holds time-on-task fixed (its entire mechanism, −5.3pp writing / +4.4pp reading) while at work AI's headline effect is time saved, and that workplace volume generates a third mode this experiment cannot produce — abdication, the review step skipped outright (Acceleration Whiplash's 31.3% unreviewed PRs, Security Debt of Agent-Generated Code's 81.1% uncommented credential leaks), which the automation arm here is not, since those students still submitted work they had read. Still open, and no longer answerable by synthesis: it needs a workplace study that classifies use mode within-subject from logs and measures unaided skill after AI is withdrawn. #oq/source Checked and recorded as a restatement, not a partial (2026-09-22): When Does AI Augment Work? A Workflow-Level Framework for Human-Agent Collaboration (CIVIC-AI workshop whitepaper, practitioner-opinion, no study of its own) makes this question one of six conditions for calling a workplace deployment "augmentation" at all, and argues the transfer runs through a selection effect rather than an analogy: the tasks most amenable to AI delegation are "high-volume, procedurally defined and verifiable," which are "exactly those tasks through which junior workers develop their tacit judgement needed to evaluate outputs, catch failures and push back on algorithmic recommendations." It also writes down the instrument this bullet says is missing, in close to the same terms — a workflow record carrying total effort including verification and repair, plus repeated longitudinal measures of mastery, progression and agency, complemented by employee-level evidence because "formal adoption data may miss unofficial AI use." Nothing in it is measured, no use mode is classified anywhere, and no unaided post-measure is proposed, so the bullet is unchanged: what it gains is a second independent specification of the missing study, co-signed by a labour ministry that would be positioned to run it.

Resolved Questions#

• Answered (2026-10-21), on the interface and incentive levers: the interface lever works, the incentive lever fails. The augmentation/automation choice is endogenous to incentives (grade inflation and signaling-motivated students push toward automation). Can incentive or interface design shift the mix toward augmentation at scale — and would that reverse the deskilling half? (Andrew Ng's August 2026 "as most commonly used" assumes the mix is automation-dominated; the experiment's own logs run 49% augmentation / 32% automation, so the assumption is unmeasured, not confirmed.) Evidence (2026-09-22) by Training novices to think, or giving them LLMs? Evidence from an RCT — narrowly, and on a lever the question did not name. A preregistered 2×2 (n=1,053) randomizes a ~6-minute causal-reasoning training against a placebo, crossed with ChatGPT Edu access, and finds the trained skill is not crowded out by the tool: every Causal × GPT interaction on reasoning style is positive and significant on top of the main effects (mechanisms +0.129, falsifiability +0.203, coherent logic +0.228), and causal training's effect on idea diversity (+0.47 SD between-solution) is undiminished when a model is in reach. So something upstream can change what novices do with an LLM — the first randomized demonstration of that in this vault. Three reasons it is a partial and not a retirement. (i) It is training, not incentive or interface design — the only interface-like element, the one-line nudge "Remember the skills you learned during the game," is bundled inside the training arm and cannot be identified separately. (ii) It never measures the mix: use mode is not classified, conversation logs were collected but not analysed, and the outcome is reasoning style in the assisted output, not augmentation-vs-automation behaviour. (iii) The deskilling clause is untouched — there is no unaided post-measure, no tool withdrawal, no retention test, and the authors state flatly that they cannot say whether the LLM's advantage reflects knowledge acquired or output procured. It also supplies the question's inverse, which is the more useful half: the incentive that exists today pushes the other way. In the same paper's lasso, the evaluation rubric loads negatively on falsifiability (−0.109), mechanisms (−0.166) and distance from the modal answer (−7.676, ≈ −0.29 points per SD) while loading positively on the coherent, idea-dense text the LLM produces — so under a standard rubric the trained behaviour is scored down and the outsourced one up. Any incentive-design answer has to beat that gradient first. Evidence, on both named levers (2026-10-01): Designing Against Deskilling: Metacognitive Feedback Reduces Cognitive Offloading to LLM Assistants (empirical, preregistered, N=704) randomizes one interface lever and one incentive lever against the same classified use mode. The two split cleanly.
• Interface works. Per-item metacognitive feedback on the learner's own AI use cuts answer offloading (OR 0.47, one-sided p=.026). It moves the request mix toward guided help and intermediate results (answer offloading 58.2% → 51.2% of assisted items) and lifts unaided transfer (OR 1.51, p=.030). This is the reversal of the deskilling half, at least back to the no-AI baseline (0.67 vs 0.65).
• Incentive fails. A points schedule penalizing answer requests reduces use (OR 0.39) but not offloading (OR 0.66, n.s.). Among assisted items, offloading's share rises to 62.8% because guided help is what gets dropped.

Sources#

• Experimental Evidence on the Learning Impact of Generative AI — Contractor & Reyes, Experimental Evidence on the Learning Impact of Generative AI (arXiv 2607.08849, 2026-07-09), empirical. §4 immediate effects, §5 retention + augmentation/automation heterogeneity, §6 beliefs, §7 conclusion; Figures 4–8, Tables 4–8.
• Training novices to think, or giving them LLMs? Evidence from an RCT — Asirvatham, Betti, Brown, Camuffo, Chatterji, Fumagalli, Gambardella, Mariani, Pandey, Ramos, Salvucci & Simic, Training novices to think, or giving them LLMs? Evidence from an RCT (Bocconi University × OpenAI, CEPR DP21882, August 2026), empirical. §4.1 causal-reasoning constructs, §4.2 awareness/usage and expert similarity, §4.3 ideas, §4.4 PDS-lasso decomposition; Figures 1–6 and A.1; Appendix §2 rubric and inter-rater reliability, §4 variable construction, Tables A.3, A.7–A.14. Parse warning — PDF-derived (docling 2.126.0, MLX layout + table stages). Automated checks passed except an author-table weld, but an unflagged standard-error row-shift runs through Tables A.7, A.8, A.10–A.14: parenthesised s.e.'s land on the following row, and in Table A.11 this misattributes the Expert-Alumnus GPT coefficient (0.044***) to the Causal row. Table A.13 is worse — docling dropped the within-solution-diversity coefficients entirely (2.483 / 2.284) along with several s.e.'s, welding the row label into the neighbouring one. Every figure cited above was recovered from pdftotext -layout on the local PDF and reconciled against the prose; no table row is cited unreconciled. Two artefacts of the paper itself, not the parse: the CRT mean is 0.490 in Table A.7 and 0.769 in Table A.8 for the same variable and N (both verified in the reference parse), and the anonymization leaks — the university is withheld "to preserve authors' anonymity" but the appendix names Bocconi twice
• Designing Against Deskilling: Metacognitive Feedback Reduces Cognitive Offloading to LLM Assistants: Maier, Schwabe, Schneider & Feuerriegel (LMU Munich / MCML), Designing Against Deskilling: Metacognitive Feedback Reduces Cognitive Offloading to LLM Assistants (arXiv 2609.20143, 2026-09-17), empirical. Preregistered on AsPredicted, N=704. Cited from §4 design, §5.1 Table 3, §5.2–5.5, §6.2–6.5, Figure 5 (read from the image), and Appendices H, I.1–I.4. Parse warning, PDF-derived (docling 2.126.0, MLX layout and table stages). Table 2 and Table 3 match pdftotext -layout on the local PDF. Table 8 (Appendix H robustness) is split and shifted in the raw: the H2a estimate (−5.05) and H1's CI upper bound (5.14) land on the section-header rows, and CI brackets weld into the p column. The figures cited above come from pdftotext. Table 5 (the item bank) is cell-collapsed and is not cited.
• Verified, not generated: expert-verified AI study materials and the distribution of learning gains in a university course: Dang & Nguyen (King's Business School, King's College London), Verified, not generated: expert-verified AI study materials and the distribution of learning gains in a university course (arXiv 2610.07097, 2026-10-05), empirical, quasi-experimental two-cohort DiD, n=170 students and 340 marks. Cited from §2.3 judgement burden, §3.2–3.4 design, §4.1–4.6 (Tables 1–3, Figure 2 read from the image), §5.2–5.6. Field results from Brynjolfsson, Noy & Zhang, Dell'Acqua, Otis et al., Bastani and Rezvani reach this page secondhand through it. Parse note, PDF-derived (docling 2.126.0, MLX layout and table stages). Tables 1–3 parse cleanly. Every cited cell matches the prose, and Figure 2's threshold and quantile points match §4.2. The parse's three table-collapse flags are false positives: the multi-value cells are Table 2's before→after pairs ("7.09 → 5.03"), which the paper prints that way.
• Andrew Ng: The Biggest Opportunities in AI Aren't Where You Think — Andrew Ng interviewed by Marina Mogilko, Silicon Valley Girl (2026-08-28, practitioner-opinion): "AI models are terrible for learning," the homework-up/retention-down claim, the cognitive-offloading mechanism, the six-months-later self-report, the "as most commonly used" qualifier, and the LearnVector announcement. No study cited