Research Taste as the Human Bottleneck

Sources#

• AI models have likely reached parity with superforecasters on ForecastBench
• Can AI automate Epoch?
• Debating RSI, the US-China Gap, and Jaggedness with JS Denain of Epoch AI
• Jeff Dean: The 1% Rule for Building in AI
• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown
• Summary of METR's predeployment evaluation of Claude Opus 5.5
• Thread by @AndrewYNg
• When AI builds itself
• Why I still haven't bought into true RSI

Summary#

As AI absorbs the execution of AI development (AI Accelerating AI Development), the human role contracts toward a residue the essay When AI builds itself calls research taste and judgment: "choosing which problems matter, which results to trust, and when an approach is a dead end." This is the top rung of the autonomy ladder and the single capability that, if it stays human, keeps recursive self-improvement from closing the loop. Whether it stays human is the essay's load-bearing uncertainty.

The narrowing role#

The essay's clearest statement of the dynamic: "The human role is narrowing at each step in the AI development process." Two concrete narrowings:

• Writing → reviewing. "Once human- and AI-authored code quality reach parity, humans will stop writing code entirely, and shift to only reviewing it." (Parity is reported as roughly now; see AI Accelerating AI Development.) This is the harness-shrinkage story told from the human side.
• Running → choosing experiments. "Once Claude can run experiments, the question shifts towards 'Which of these experiments is worth running?'" The doing — writing the code, running the experiment, producing the result — "now costs almost nothing in human time, even if it still has costs in compute."

The residual human comparative advantage, "for now," is taste: deciding what's worth the compute. Put in the terms of Compute Allocator, the human becomes a compute allocator at the level of an entire research program rather than a single invocation.

Why it might not stay human — "just another capability"#

The essay refuses to treat research taste as a permanent human moat. Two arguments push against it:

1. Perspiration is automatable, and that's most of the work. Genius is "1% inspiration and 99% perspiration"; the 99% — scale it up, see what breaks, fix it, retry — is exactly what Claude excels at. Large-scale research progress "is mostly a function of tools and resources." (See The Bitter Lesson, Recursive Self-Improvement.)
2. Taste shows the same capability curve as everything else. Early evidence of improving research judgment — Claude beating the human next-step choice 51%→64% on hard detour moments (AI Accelerating AI Development) — suggests "research taste might be just another AI capability that AI systems fail at for a time, then get good at." The precedent: AI was once bad at, then good at, explaining why a joke is funny, demonstrating theory of mind, and solving linguistic riddles — qualitative skills assumed to be human-shaped. This is the optimistic face of Jagged Intelligence (Ghosts, Not Animals): today's valley is tomorrow's peak.

A frontier researcher's practitioner reading lands on the "just another capability" side. Noam Brown (OpenAI, practitioner-opinion) reports that models will optimize the algorithms from his poker research 10–100× but cannot yet invent a better one — "I can give it a lot of time and it's still not able to do it" — so today's models are "a very good complement to researchers," "not able to fully replace the whole research cycle." But he explicitly expects this to fall: "I wouldn't be surprised if at some point — same thing with coding, same thing with math — there's just this inflection point where suddenly it's actually good enough… I wouldn't be surprised if we encounter that point for research taste as well." This is the jagged valley argument stated by someone watching the valley from inside a lab.

The honest counter, in an Anthropic employee's words: "The comparative advantage of humans as of right now is still in seeing the bigger picture and thinking beyond the confines of the immediate task." The question is whether "for right now" is a stable equilibrium or a receding frontier.

The third answer: it isn't a capability at all#

Both arguments above — taste as a durable moat, taste as the next jagged valley — presuppose that taste is a capability, something a mind either has or acquires. Andrew Ng rejects the premise (June 2026, practitioner-opinion):

"Many people describe this human contribution as 'taste,' but I prefer to think of it as humans having a context advantage, since that gives us a clearer path to helping AI systems get better… So long as the human knows something the AI does not, human-in-the-loop is needed to inject that knowledge into the system."

Under this reading the residue is not a faculty but an information asymmetry — the human knows the users, the constraints, the history, the thing so obvious nobody wrote it down. That makes the human role falsifiable (does the asymmetry exist?), perishable (it closes as context transfers), and an engineering problem rather than a mystique. It also means the question this page is organized around — ceiling or valley? — may be malformed. See Context Advantage, Not Taste for the full treatment, including where the reframe fails: it explains the deployment asymmetry (I know my users) but not the generative one (The Abstraction Barrier, Transformative Creativity), where the missing thing is a concept, not a fact.

The two frames disagree about what would count as evidence. Returns to Expertise in Agentic Coding is the instrument that distinguishes them: if the expertise premium declines because models acquired judgment, taste was a capability; if it declines because context transfer improved, taste was an asymmetry all along.

The missing half: how you acquire it#

Every argument above is about whether taste is durable. None is about whether it is teachable — which matters, because if the residual human role is taste and taste is innate, the role doesn't scale to the people entering it. Jeff Dean (YC Startup School 2026, practitioner-opinion) is the corpus's first source to treat it as trainable and to name a protocol. He starts from the same position as Brown — asked what becomes scarce once agents write all the code, he answers "incredibly good taste in what you ask your agents to work on," and adds "I think models are not necessarily going to be that good at it" — then refuses to leave it mystical ("it's not like there's a measurable objective of taste").

Three mechanisms, in increasing order of how deliberate they are:

• Breadth of prior problems. The conventional answer: having worked on many things teaches you "what kinds of things might be just barely possible by cobbling together these previous approaches."
• Score your own forecasts. The one that is actually a protocol: "write down a bunch of things you think might be important in the next 12 months. Maybe you pick one of them to work on, but go back and evaluate in 12 months which of these other things actually seemed important, or which ones did other people in the world go out and create, and which ones did not seem to happen yet. That can give you a lot more samples for your own taste creation capability." The move is to manufacture training data for yourself — the counterfactual branches you didn't take get graded by the world, so one year of work yields N labeled examples instead of one.
• Attack a load-bearing assumption on purpose. His live example: 60 years of chip fabrication assume every chip is identical and no bit flips, while large-scale distributed systems assume the opposite and build reliability above unreliable parts — so what would you build out of transistors with 20 errors per day? He is explicit that most such experiments fail ("there are very good reasons that for the last 50 years we've done this thing this way"), which is the point: the yield is the sampling, not the hit rate.

Why this reframes the page's central question. If taste is a sample-efficiency problem — a small number of expensive, slowly-graded observations about what mattered — then it is neither a permanent moat nor an ordinary capability, and the reason models lag is legible: the training signal arrives on a 12-month delay and mostly as counterfactuals nobody records. That is a third position alongside Ng's information asymmetry and the ceiling/valley pair, and it is the one with an obvious attack — write the predictions down where a model can read the grade. Dean does not draw that implication; it follows from his own protocol.

The one component that got measured#

Every argument above concerns a faculty nobody scores — the essay's own definition ("choosing which problems matter, which results to trust, and when an approach is a dead end") names three things, none of which has a leaderboard. The middle one does now. ForecastBench (Forecasting Research Institute, 2026-07-16, empirical) scores AI systems and elite human superforecasters on the same real-world questions, resolved against the world, and the July 2026 result is that several submissions are statistically indistinguishable from the superforecaster median — with one, Cassi AI's pipeline, ranking above it on market questions (77.7, N=92 vs 75.9, N=56), the half of the benchmark that turns on novel one-off events rather than base-rate lookups. Superforecasting is the cleanest operationalization of calibrated judgment under uncertainty that exists, and its reference class is the strongest known human baseline, so this is the first component of "taste" the wiki can report a number on. State it narrowly, though: a p of 0.41 against a null of equal accuracy is a failure to reject rather than a demonstration of equality, the human baseline was elicited in 2024 and is extrapolated forward, and FRI both runs the benchmark and supplied the baseline. Full treatment on Artificial Superintelligence (ASI).

The scope boundary, recorded rather than elided, because it is where the interesting disagreement lives. Forecasting a resolvable question is not research taste, and the differences are the specific ones this page has been circling:

• The question is given. A ForecastBench item arrives pre-posed with pre-specified resolution criteria. "Choosing which problems matter" is the part that has been removed before scoring starts — and it is the first item in the essay's own definition.
• The grading is free and fast. Markets and time series resolve on a schedule; the leaderboard needs only ~225 resolved dataset questions (about 10 days) to rank a submission. That is the exact opposite of the conditions Dean identifies as why taste is hard to acquire.
• The output is a probability on a stated proposition, not a new conceptual space. Nothing here touches the The Abstraction Barrier or Boden level-3 (Transformative Creativity) — the generative half of the residue is untouched by a calibrated number on someone else's question.

So the honest ledger: which results to trust is now measured and machine parity is claimed on it; which problems matter and when an approach is a dead end are not on any leaderboard.

The sharpest connection is to Dean's acquisition protocol, and it cuts against him. His one actually-operational mechanism for building taste is graded forecasting: "write down a bunch of things you think might be important in the next 12 months… go back and evaluate in 12 months which of these other things actually seemed important." He proposes it because the labeled data is scarce — one year of work yields a handful of samples. ForecastBench is that protocol industrialized: a continuously refreshed question supply, automated resolution, thousands of scored forecasts, a public leaderboard. The single taste mechanism on this page that was concrete enough to be a protocol is also the one that turned out to have a machine benchmark — and machines have reached the top human baseline on it. What it says about the other two mechanisms (breadth of prior problems, deliberately attacking a load-bearing assumption) is nothing, and those are precisely the two that do not reduce to a scored proposition. The sample-efficiency framing survives the result; it just no longer implies a human advantage on the component where samples are cheap.

It is also a datapoint for Ng's reframe, and a friendly one. The systems at the top of this leaderboard do not win by having better judgment in the weights. Cassi's architecture is a multi-stage pipeline that generates sub-questions and search queries, retrieves up-to-date context, filters for relevance and recency, then ensembles models and has another model analyze the reasoning. That is context acquisition as an engineering pipeline — the information asymmetry closed by construction rather than by a faculty. If the component of taste that fell first is the one a retrieval pipeline can attack, the asymmetry reading gets the better of this particular round.

A third-party evaluator names the same gap (September 2026)#

METR's predeployment evaluation of Claude Opus 5.5 (empirical) reaches for the same vocabulary from outside Anthropic: full AI-R&D automation is blocked on "large improvements in foresight, prediction, creating one's own feedback loops" and researcher judgment, and METR's evidence does not suggest Opus 5.5 represents a major improvement on these. It is a one-line corroboration rather than new texture — no new mechanism, no measurement of taste itself — but it is the first time this page's residue is restated by an evaluator rather than by Anthropic or a practitioner reflecting on Anthropic's own product. See AI R&D Autonomy Evaluation (AECI) for the full evaluation.

The residue observed in a write-up, not a summary (Epoch, October 2026)#

Epoch AI's real-work report (Can AI automate Epoch?, 2026-10-08, empirical; one run per task, one grader) gives this page its first documented trajectories, not just verdicts. One task asks models to propose a novel research project from Epoch's "9 big questions benchmarks can help answer", run a pilot, and propose follow-ups. All three parts of the essay's definition appear, and they do not fail equally.

• Choosing which problems matter: mostly passed at the level of direction. "It's not difficult to identify a valuable general direction." GPT-6 Astra's question (does an agent that fails to improve with practice fail to collect evidence, or fail to use it?) is called genuinely promising. The same model failed the audience version of the same skill: given every published Data Insight, it chose a topic too niche for Epoch's readers.
• Designing an experiment that measures what it claims: failed. Astra's design left its critical components undefined, so nobody could say how to run it. Epoch calls experiment design "the most important part of this process," and none of the evaluated models did it well.
• Which results to trust: failed, on the model's own results. In its pilot Astra's 4,096-token limit truncated 61 of 280 responses before they named an input. It noticed, fixed the limit, watched scores recover, and still reported "sensitivity to the acquisition budget" as a finding. Epoch's general statement: models "sometimes recognize these mistakes, but treat the flawed results as key findings."

This qualifies the ForecastBench ledger above rather than contradicting it. That section records which results to trust as the component where machines reached parity. ForecastBench scores calibrated probabilities on questions someone else posed, resolved by the world. Epoch's failure is about a result the model itself produced, from a setup it built, with nothing external to resolve it. Both can hold. Judging the trustworthiness of one's own experimental output, where the error is one's own, is the half of the component that parity on ForecastBench does not reach. The ledger should therefore read: which results to trust is at parity on external, resolvable questions, and not on self-generated evidence. The tiers are comparable (empirical both), but Epoch's is n = 1 with a single grader, and ForecastBench's parity is a failure to reject.

Fast experiments, slow understanding: an outside moderate's split (September 2026)#

Nathan Lambert (Why I still haven't bought into true RSI, practitioner-opinion) draws the line through this page's residue in a different place from the essay. He buys "the cycle of experiment design and testing being 10x faster in the near future, but not hypothesis generation and intuition building". So for him experiment design sits on the automatable side, where Epoch's report above observed it failing. His reason for the residue is not taste as a capability. "Many people under-index how much of science is communication and standard setting with colleagues," and "accelerating understanding will be the key bottleneck", which humans will improve at "only marginally". The gain he expects is that humans spend more time understanding, not that they get exponentially better at it. That is closer to Ng's reframing than to a ceiling argument.

He takes a post-training instance from John Schulman: a post-training team is large "because there are a lot of different areas where you have to figure out how the model should behave… someone has to think about how the model should behave in this area." Here deciding which problems matter means deciding what behaviour is wanted, and Schulman adds that getting it wrong "doesn't show up in benchmarks". That joins this page's rubber-stamping question to the measurement problem. A residue whose failures are invisible to benchmarks is the one hardest to audit for quiet transfer to the model. Growth-curve consequences on Intelligence Explosion Dynamics.

The first lab-side instrument aimed at the residue reads null (September 2026)#

Every measurement above of the top rung is Anthropic's next-step test, METR's predeployment summary or Epoch's single graded write-up. On Lambert's podcast (Debating RSI, the US-China Gap, and Jaggedness with JS Denain of Epoch AI, practitioner-opinion), JS Denain of Epoch AI relays a fourth from OpenAI's internal-acceleration post, which is not in raw/. OpenAI classified researchers' Codex sessions by task type and found uplift concentrated in engineering and troubleshooting. "For a lot of the high-level strategic decision-making, they both anecdotally and also when they look at sessions, don't seem to find a huge uplift in making better compute allocation decisions or deciding on research directions." Those two decisions are this page's residue in Compute Allocator terms, and a second lab's telemetry finds them untouched. Details on AI Accelerating AI Development.

Lambert restates the essay's split as an analogy: "junior PhD students will be 10X as productive, but from the advisor's perspective, their research agenda will not proceed 10X as fast." Denain does not treat the null as a ceiling. He names three routes by which the residue could fall: "a bit longer horizon RL"; "decent transfer from other fields where those kind of decisions are important"; and nesting, where a judgment task that cannot be verified is part of a longer one that can, as company decisions roll up into revenue. He also gives the same split from Epoch's own work. Classifying job postings is "automatable", while "LLM's assessment of what are the top five most interesting things is not that great" (Epoch Automation Reports). On this page's question this is evidence that the residue has not fallen yet. It says nothing about whether it can.

The bottleneck either way#

Even if taste stays human, it becomes the binding constraint — the Amdahl's-law bottleneck of the whole pipeline. If humans can't review code as fast as Claude generates it, "human review will become the bottleneck to AI development." And if humans spend most of their time on the single-digit fraction of work that is direction-setting, each human steers vastly more work — so the quality and throughput of human judgment becomes the scarce resource, exactly as Verification as the New Bottleneck predicts for verification and the HBR accountability-redesign work predicts for oversight. The risk is that the human nominally decides but actually rubber-stamps (the "not close to substituting for senior researchers" judgment quietly eroding into a formality).

The human cost (the quieter thread)#

The essay includes unusually candid employee quotes about what the narrowing feels like — worth recording because they name a cost the productivity charts don't:

• The collapse of the gift economy of small favors: asking a colleague "can you help me get this script running?" "created a little debt, a little mutual awareness. [Claude is] faster, it creates zero debt, but each of these is a lost bid for human collaboration."
• The vertigo of contingent relevance: "On days where everything works well, I can't help but think nothing I do matters … But then there are days where everything breaks and I don't understand why and I realize I have no idea what I've been up to anymore."

Connections#

• Structured Safety Case (Claim Decomposition) — the scope exclusion that matters most: Anthropic concedes known misalignment may differentially damage safety research because safety work is disproportionately made of fuzzy, uncheckable tasks, then rules the concern out of the assessment

• Recursive Self-Improvement — whether taste stays human decides which of the three futures obtains; the essay's central uncertainty

• AI Accelerating AI Development — the evidence that execution is already absorbed, leaving taste as the residue

• AI R&D Autonomy Evaluation (AECI) — "not close to substituting for senior Research Scientists/Engineers" is the formal version of "taste is still human"

• Jagged Intelligence (Ghosts, Not Animals) — the joke/theory-of-mind precedent for "a capability AI fails at, then masters"; taste as the current valley

• Autonomous Scientific Discovery — Mythos 5's autonomous hypothesis-generation (~80% preferred over Opus-class) and "only high-level human input" genomics are concrete chips at the taste moat

• Verification as the New Bottleneck — if taste/review can't keep pace with generation, judgment becomes the binding constraint

• Harness Shrinkage as Models Improve — the same role-narrowing dynamic from the harness side; what's left after the model-facing harness dissolves

• Compute Allocator — taste exercised as allocation: deciding which experiments are worth the compute

• The Bitter Lesson — "research progress is mostly tools and resources" is the bitter lesson aimed at taste itself

• The Abstraction Barrier — DeepMind's theoretical case that taste/novel-concept-discovery is a real ceiling, not just the next capability to fall: AI may be bounded by human conceptual frameworks

• Transformative Creativity — Boden level-3 (creating new conceptual spaces) is taste exercised at the frontier of concept-space; the "Einstein test" version of the moat

• Multi-Agent Collective Intelligence — steering large superhuman-speed agent groups (whose output humans can't fully consume) is the form taste takes under the collective pathway

• Instrumental Convergence — a knowledge-seeking objective is an alternative framing of judgment as positive-sum information gain rather than scarce human taste

• Artificial Superintelligence (ASI) — taste/judgment is the candidate for what (if anything) stays human as systems cross from AGI into ASI; also the home of the first measured component of it, where AI submissions reach the superforecaster median and the instrument's discriminating range ends at that baseline

• Returns to Expertise in Agentic Coding — the labor-data instance of the same "is taste durable?" question: Anthropic frames a decrease in the returns to domain expertise over time as the signal that "models are starting to supply the essential judgment users currently bring" — taste-as-just-another-capability, made empirically trackable in usage

• Planning / Execution Division of Labor — the rubber-stamping risk made concrete: humans nominally own ~70% of planning decisions, but whether that control is real judgment or approval-by-default is exactly the "how do you measure rubber-stamping?" question

• Implementation Abundance Inverts Product Work — the product-work cousin: as implementation cheapens, curation/taste becomes the expensive step, the same residue named here for AI research

• Why AI Lags at Design — design taste as a currently-durable pocket of the human-as-reward-function; a concrete domain where the taste moat holds longer

• Role Averaging, Not Role Elimination — "taste-makers guide from inception" carries the same rubber-stamping risk: guidance is only real if it isn't approval-by-default

• Context Advantage, Not Taste — the third answer: Andrew Ng argues taste is not a capability but an information asymmetry, which makes it neither ceiling nor valley but a closable gap; dissolves this page's central question for product work while leaving it standing at the research frontier

• Unknowns as the Agentic Bottleneck — if the residue is context rather than taste, elicitation is the response; Thariq's blindspot passes and interviews are the protocol for pumping it across

• Noam Brown — a frontier researcher's practitioner reading: models optimize his algorithms 100× but can't invent a better one "for a time"; he expects an inflection point for research taste like the ones in coding and math

• Jeff Dean — the acquisition side the rest of this page skips: taste as a sample-efficiency problem, with a protocol (write 12-month forecasts, grade them against what the world actually built) that manufactures the labeled data the skill is starved of

• Researcher Uplift from Code Output — the quantitative shadow of this split: Kwa's model prices code/engineering uplift and treats research judgment as the unmodeled residual (assumes zero non-code uplift), matching Anthropic's "concentrated in engineering execution rather than research judgment"; a measured non-code uplift > 1 would be taste starting to fall

• AI-Assisted Error Analysis — the same bottleneck at product scale rather than research scale, with the corpus's most concrete account of what expressing taste costs in practice: a failure mode invisible on first pass and unmissable after three (the recurring word "matters"), and a practitioner's judgment that validating an agent's proposed taxonomy is more expensive than authoring her own

• Claude Opus 5.5 — METR's predeployment evaluation restates the judgment gap ("foresight, prediction, creating one's own feedback loops") from outside Anthropic, the first non-Anthropic, non-practitioner source to use this page's own vocabulary

• Epoch Automation Reports — the essay's three components observed in one model's research write-up: direction passed, experiment design failed, and a self-inflicted artefact reported as a finding

• Intelligence Explosion Dynamics — what this residue does to the growth curve in Lambert's "lossy self-improvement" reading: RSI bends the cost curve where objectives are measurable, and leaves peak capability and post-training behaviour specification paced by humans

• Nathan Lambert — the fast-experiments, slow-understanding split, and the Schulman post-training quotes

Open Questions#

• Is research taste a genuine ceiling (an architectural capability scaling can't reach) or the next jagged valley to fill? The essay calls this the decisive unknown. Contested premise: Ng argues the question presupposes taste is a capability at all. (2026-10-15, status evidence, not a resolution: OpenAI's session analysis, as relayed by Denain (Debating RSI, the US-China Gap, and Jaggedness with JS Denain of Epoch AI), finds no notable uplift on compute allocation or research direction. That is the first second-lab reading that the valley has not filled. Denain's proposed routes are longer-horizon RL, cross-field transfer, and nesting the judgment inside a verifiable longer task such as revenue. Those name what would fill it, and the trigger stays a model that shows strategic-decision uplift.)
• If taste is automatable, what — if anything — remains a durable human comparative advantage in AI development? Partially answered (2026-08-12), by subtraction: one named component is off the list. Calibrated probability judgment on resolvable questions — "which results to trust" — no longer separates the strongest human reference class from a scaffolded AI pipeline on ForecastBench, and the pipeline that leads does it by retrieval and ensembling rather than by accumulated judgment. What remains unmeasured is the rest of the essay's own definition: choosing which problems matter (the question is supplied before scoring begins) and recognizing a dead end. The subtraction is worth more than it looks, because this was the component with the best claim to being measurable at all — the ones left are left partly because nobody knows how to score them. Narrowed (2026-10-15) by Can AI automate Epoch?: the subtraction holds only for externally resolved questions. On its own pilot results the leading model reported a truncation artefact as a finding, so judging self-generated evidence stays on the human list, along with experiment design. Picking a promising direction looks closer to falling (Epoch Automation Reports; n = 1, one grader).
• How do you measure rubber-stamping? "Humans set direction" can be true on paper while real judgment quietly transfers to the model. Partially answered (2026-10-21): Can AI automate Epoch? shows the failure the measure must catch: the leading models treat flawed experiment results as key findings, and only a human grader flagged them. That makes acceptance-without-detection concrete, but the study grades model outputs (n = 1 grader) and measures no human reviewer's override rate, so a rubber-stamp metric is still unspecified. Settler: a study logging human accept/override rates against seeded errors.

Sources#

• When AI builds itself — §"What might the future of work at Anthropic look like?" and §"What if we're wrong?"
• Thread by @AndrewYNg — Andrew Ng, The Batch (2026-06-30), practitioner-opinion: the "context advantage, not taste" reframing
• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown — Noam Brown (No Priors, 2026-06-26), practitioner-opinion: research taste as the residual human role; "not able to fully replace the whole research cycle," but expects an inflection point
• AI models have likely reached parity with superforecasters on ForecastBench — Forecasting Research Institute (Substack, 2026-07-16, empirical): the July 2026 tournament leaderboard, the market-question result, Cassi's pipeline architecture, and footnote 1's bootstrap p-values. Cited here only for the component claim — the leaderboard measures calibrated probability judgment on supplied, resolvable questions, not problem selection. Leaderboards are PNG screenshots re-read at zoom under the image two-pass rule; FRI runs the benchmark and supplied the human baseline
• Jeff Dean: The 1% Rule for Building in AI — Jeff Dean, YC Startup School 2026 (2026-07-30, practitioner-opinion): §"How to Become an AI-Native Founder" and §"Question Your Biggest Assumptions" — taste as the scarce skill once agents write the code, "models are not necessarily going to be that good at it," and the three acquisition mechanisms including the graded-forecast protocol
• Summary of METR's predeployment evaluation of Claude Opus 5.5 — METR, blog post, 2026-09-22 (empirical): the "foresight, prediction, creating one's own feedback loops" phrasing of the judgment gap, restated for Claude Opus 5.5. Full treatment on Claude Opus 5.5 and AI R&D Autonomy Evaluation (AECI)
• Can AI automate Epoch? — Kelly Hong & Greg Burnham (Epoch AI), "Can AI automate Epoch?", 2026-10-08 (empirical; one run per task, single human grader). The research-design task: Astra's question, its underspecified design, the 4,096-token / 61-of-280 truncation and its write-up, and the cross-model statement about treating flawed results as findings. Full treatment on Epoch Automation Reports
• Why I still haven't bought into true RSI — Nathan Lambert, Interconnects, 2026-09-19 (practitioner-opinion): the experiment-cycle-versus-understanding split, science as communication and standard-setting, and two John Schulman quotes on post-training. Lambert relays these from a Dwarkesh Podcast episode that is not in raw/
• Debating RSI, the US-China Gap, and Jaggedness with JS Denain of Epoch AI — Lambert with JS Denain (Epoch AI), Interconnects podcast #19, 2026-09-22, practitioner-opinion. The relayed OpenAI session-analysis null on strategic decisions (post not in raw/), the PhD-student analogy, Denain's three routes, and his job-postings automation remark