Screen-Relative AI Timing

Sources#

Summary#

When generative AI makes options cheap, the scarce step in creative work becomes screening them: comparing alternatives, discarding most, committing to a few. Difang Huang, Shumin Yu, Fadong Chen and Weihua Zhou (arXiv 2610.14984, 2026-10-10, empirical, preregistered) argue that AI's value then depends on one variable organizations rarely set on purpose: whether the assistant's options arrive before or after the team's screen. An option that arrives before the gate must compete for a place in the portfolio. An option that arrives after it comes "too late to prune": the time a comparison would take is worth more as development of what the team already chose, so the team keeps the option without comparing it.

The test was 148 three-person teams (444 Chinese university students) on a 40-minute product-design task with platform-enforced stages: 15 minutes of search, 15 of selection, 10 of review. Five schedules gave access to the same model (DeepSeek V3.2, temperature 0.1): search only (early), selection only (late), throughout, never, and search only with a standing overreliance warning. Teams used the assistant at the same rate in the two focal arms (14.9 and 13.8 queries per team).

Headline. Early access beat late access by 0.356 points on the expert creativity composite (95% CI 0.064–0.648; Hedges' g = 0.62; Welch p = 0.018). Early beat no-AI by 0.486 (g = 0.83). Late access, with the same model for the same 15 minutes, produced a gain of 0.130 over no-AI that is indistinguishable from zero (p = 0.43), so moving the window forgoes about three-quarters of the gain. Continuous access matched early access and did no better (difference −0.007).

This is the measured, team-level form of the shift Implementation Abundance Inverts Product Work describes from practitioner accounts. When generation is cheap, the expensive step moves to curation. What this paper adds is that the curation only happens if the options reach it in time.

The mechanism, traced through the funnel#

Platform logs (13,324 chat messages, 2,124 assistant exchanges, 4,776 personal drafts) let the authors rebuild each team's idea funnel.

Early access widened the pool, and teams pruned it. Early teams held 19.8 distinct ideas at the gate against 3.9 without AI. Of those, 18.6 were AI-derived, and the team's own drafted ideas fell from 3.9 to 1.3, so the assistant replaced part of the team's ideation while multiplying the pool. About a third of pooled ideas (32%) reached the submission, the same share as without AI. A calibration (common normal distribution, a screen that keeps the two best) puts the screening premium of a 20-idea pool over a 4-idea pool at 0.46 points, close to the observed 0.49. The authors treat that calibration as an order-of-magnitude check: it assumes perfect screening and AI ideas as good as the team's own.

Late access replaced the screened pool instead of adding to it. Late teams reached the gate with the same pool as no-AI teams (3.8 vs 3.9 ideas). Then only 11% of their pre-gate ideas reached the submission (vs 32% without AI), and only 5% of their submitted text matched anything that existed before the gate (vs 22% without AI and 71% under early access). Sentence-embedding similarity gives the same pattern, so the low share is not the assistant paraphrasing pre-gate ideas. This is the paper's Remark 1 displacement case, not the additive case its main proposition assumes.

What the displacement cost is not directly visible. Proposals that originated before the gate were rated no higher than those that originated after it, and late submissions were not rated below no-AI ones. The authors conclude that the cost of late access is the premium it forgoes, a pool five times larger to choose from, not a measured penalty for discarding the team's own ideas. They also caution that within conditions neither pool size nor late generation requests predicts how much of the pool a team kept. The funnel differences are between randomized schedules, not a causal chain traced within teams.

Access history changes what the same tool is used for#

The cleanest causal result holds the window, system prompt and model fixed. Late teams and continuous-access teams both had the assistant during the selection window under the same refinement-oriented prompt. The only randomized difference was whether they had also used it during search. Late teams devoted 29.1% of their selection-window queries to generating new options; continuous teams devoted 11.0% (p < 0.001; 4.07 vs 1.20 generation requests per team). Dropping each team's first three window queries, to remove orientation, leaves 21.1% vs 8.9% (p = 0.011).

The model predicts this ordering (Proposition 3): an unscreened late option is worth requesting only if its expected quality is not far below what the team already holds. A team that enters selection with a strong, AI-enriched set gains little from another draw and develops what it has. A team that enters with a thin set reopens the search, which is the behaviour the model penalizes. That is why continuous access does not inherit the cost of late access: early use changes how the team later uses the same assistant. It also means studies of continuous access estimate a mixture of early and late effects, weighted by how participants happen to schedule their own use.

The fate of late output#

Assistant responses were visible only to the member who asked, so the team chat was the only place the team as a whole could compare an AI option with its own. Output that entered the submission without appearing in chat did not pass the team's screen.

Late output entered submissions more often than early output (23.7% vs 14.6% of idea units), more often near-verbatim (15.8% vs 8.7%), and more often without ever being shared (9.6% vs 3.9%). The share discussed in chat did not differ (26% vs 28%). Late output was not discussed less; more of it went into the deliverable without discussion.
Holding request type fixed halves the adoption and near-verbatim gaps, which then lose significance. The unshared-adoption gap survives (3.3 points, p = 0.046).
Within the selection window, late teams used output near-verbatim about twice as often as continuous teams (15.8% vs 8.2%, p = 0.024).
One finer prediction is not clearly supported: unshared adoption rose over the selection window only slightly faster than over the search window (p = 0.09).

The team-information point connects to the hidden-profile literature (Stasser & Titus): an assistant that answers only the asker produces unshared information, and some late output entered submissions without the rest of the team seeing it.

Breadth, not quality, and why the verdict depends on the evaluator#

Late access raised proposals per submission by 1.53 over the control mean of 2.10, a 73% increase and twice the early-access increase (+0.74). The late-minus-early difference (0.78 proposals) has the predicted sign but is imprecise (p = 0.19). Each additional proposal was associated with a 0.179-point lower rating, conditional on condition and task. The depth half of the breadth/depth hypothesis held only against early access: late proposals were not less elaborated than control proposals.

How the portfolio is valued decides the ranking. The early advantage is 0.361 points in the mean proposal and 0.221 in the best proposal (p = 0.055), and it reverses in the total: late submissions summed to 2.06 points more (p = 0.35). A small credit for breadth flips the verdict on late access. At 0.1 points per proposal, late access outscores working without AI (p = 0.043), and the early-vs-late gap vanishes at 0.46 points per proposal. An organization that rewards the number of ideas will see late access as a win. One that rewards the quality of what it commits to will not.

The LLM judge panel missed the timing cost#

Ten frontier models from five families (Claude Sonnet 4.6, Opus 4.6 and 4.7; GPT-5.5 and 5.5 Pro; Gemini 3.1 Flash and Pro; DeepSeek V3 and V4 Pro; Kimi K2.6) scored every proposal on the human rubric. The panel was internally consistent, its team scores correlated 0.61 with the human panel, and it agreed that AI access helps (all four AI arms 0.58–1.41 SD above control). It did not distinguish early from late access: the timing contrast was −0.08 SD under the model panel (p = 0.77) against 0.59 SD under the human panel (p = 0.019), on every rated dimension. Excluding one or both DeepSeek judges leaves it unchanged.

Part of the gap is breadth weighting: humans docked 0.179 points per extra proposal, the model panel 0.063. Proposal count accounts for about a sixth of the divergence, and adding the share of AI-derived sentences (a self-preference channel) and rated text quality brings it to about a third. The remainder is the model panel failing to register the lower originality and fit the experts saw in late-access proposals. The authors' managerial reading: "an organization that evaluates AI-assisted work with AI judges will see that assistance helps but not that its timing matters." This is a condition-dependent judge error of the kind Version-Dependent Judge Error measures for agent releases. The judge tracks the gross effect and misses the contrast that carries the decision.

The warning arm#

A bold-italic warning appended to every early-access response ("AI serves only as an initial reference point… discuss these ideas further with your team") cut the share of submission sentences closely matching the team's own assistant output from 65% to 47% (p = 0.020). Queries fell from 14.9 to 12.5 per team (p = 0.09), and rated creativity did not change (−0.167, p = 0.30). Because early options must still pass the team's screen, a reliance-reducing warning has little left to improve. The authors read the pair (the schedule moves quality, the warning does not) as "process design is a stronger lever than exhortation." That parallels the LMU result on Experimental Learning Impact of Generative AI, where interface-level metacognitive feedback changed offloading and a points penalty did not.

What the design does and does not separate#

Screen vs clock. The screen fell at minute 15 in every condition, so the design identifies access before vs after a screen whose timing is fixed in advance, as in a stage-gate process. It does not by itself separate the screen from elapsed time. The argument against a pure clock account is the late-vs-continuous contrast: same minutes, same prompt, different use depending on what the team held at the gate. The proposed decisive test crosses window position relative to the screen with position on the clock.
Prompt confound in the focal contrast. The system prompt was set by stage, so early teams consulted an assistant told to generate ideas and late teams one told to refine them. The authors count this as part of what late access means in practice. The late-vs-continuous comparison holds the prompt fixed.
Multiplicity. The timing contrast survives the preregistered Benjamini-Hochberg correction (adjusted p = 0.048) and randomization inference (p = 0.021). It does not survive the most conservative family-wise procedures across all eight contrasts (Westfall-Young 0.086, Holm 0.107). The early-vs-control and continuous-vs-control effects survive every adjustment.
Raters. Expert agreement was modest (ICC(2,k) = 0.44 at proposal level). The timing contrast holds for every rater subset (0.298–0.420 when any one rater is dropped). The early- and continuous-vs-control effects lose significance on the two doctoral raters alone.
One of the model's own predictions failed. Proposition 1(iii) predicts that late access lowers quality below no access unless selection-stage development help is large. In the data late access sits slightly above control (+0.13, n.s.), which the model accommodates only with large development help.
Scope. Students, a 40-minute object-combination task needing no domain context, one model, all materials in Chinese. The pool is measured from written drafts, not everything teams discussed.

Prescriptions (the authors')#

Make the assistant available while the team builds its option set and screen deliberately afterward. Where AI's first contribution comes late, either limit late use to developing already-chosen proposals, or require late output to pass a comparison before it enters the deliverable. The model's Corollary 2 says an enforced screen recovers part of late access's value and removes quality dilution when comparisons are cheap. That corollary is untested here. Decide whether you value depth or breadth before judging a deployment, and validate AI evaluators against human judgment.

Connections#

Implementation Abundance Inverts Product Work — the practitioner thesis this experiment puts a gate under: when generation is cheap the expensive step is curation, and here the curation only happens for options that arrive before the team has committed. Late options skip it
Design by Selection — "ask for ten options, then remix the two that work" is the early-access schedule run by one designer: generate before the screen, then select. The 148-team data are the controlled version, and a 20-idea pool was screened down to under three proposals without a measured quality cost
Selection Under a Submission Budget — the same order-statistics arithmetic in a different domain: the screening premium M_k(n+a) − M_k(n) is what AlphaCode's filter-and-cluster stage buys when 1M samples must become 10 submissions
Version-Dependent Judge Error — an LLM judge panel that tracks the gross effect but misses the decision-carrying contrast: here the condition is the access schedule rather than the agent version
LLM-as-a-Judge — a creative-rating instance of judges agreeing on direction while missing a treatment difference humans saw on every dimension; breadth weighting explains about a sixth of the gap
Experimental Learning Impact of Generative AI — the warning arm's null on quality matches that page's finding that interface and process design move behaviour where exhortation and penalties do not
Verification as the New Bottleneck — the general form: cheap generation moves the bottleneck to judgment. This page measures what happens when that judgment step is skipped for late options
LLM-Judge Self-Preference — the AI-derived-text share this page counts as a self-preference channel, measured directly. On Text Arena, six of twelve judges favour their own answers by +23 to +55pp, and judges agree with each other more than with the human voter (79.4% vs 56.9%): the same pattern as this page's ten-model panel tracking a gross effect while missing what human experts saw

Open Questions#

Does the effect follow the screen or the clock? The paper's own proposed design crosses the window's position relative to the screen with its position on the clock. Until it runs, the late-vs-continuous contrast is the only evidence against a clock account.
Does an enforced screen on late output (Corollary 2) recover the lost value in practice? Specifically, does it recover it in field settings with formal stage gates and longer projects with repeated screens, where comparison is cheap relative to development?
Does the AI-judge blindness to timing survive a judge given the team's pre-gate material or the full submission set? The panel scored submissions alone, so it could only see the outputs, not the process that produced them.

Sources#

Too Late to Prune: Generative AI, the Screen, and Team Creative Decisions — Difang Huang (AMSS, Chinese Academy of Sciences), Shumin Yu, Fadong Chen & Weihua Zhou (Zhejiang University), Too Late to Prune: Generative AI, the Screen, and Team Creative Decisions, arXiv 2610.14984, 2026-10-10, 80pp with Online Appendix, empirical, preregistered. Cited for the abstract and §1, the model (§2: Lemma 1, Propositions 1–3, Remark 1, Corollaries 1–2), design (§3), results (§4.1–4.7), robustness and alternative evaluators (§5.1–5.3), discussion (§6), and appendix prose (A.1 Proposition 4, A.4 numerical analysis, I.1 the judge-panel roster, I.3 the divergence decomposition). Parse warning: docling-derived (2.126.0 / docling-mlx 0.1.1, confidence_grade: excellent, 40 tables). Ingest flagged appendix tables for 3 collapsed multi-value cells, 15 shifted "Team" labels and 1 split row. Main-text Table 3's control-mean row is also split across two grid rows. Every figure on this page is quoted from prose or figure captions, and no table row is cited.
Self-Preference of AI Judges — Arena.ai, Self-Preference of AI Judges, blog post, 2026-09-30 (empirical; COI: Arena sells human-preference evaluation). 12 judges × 1,460 Text Arena battles, 34,580 verdicts. Full treatment on LLM-Judge Self-Preference; cited here for the Connections bullet (self-preference channel).