H
Howardism
Howardism · Vol. 03Plate II · No. 02

Model Capability & Training, in order.

Notes25DomainModel Capability & TrainingOpen Qs58Newest17 Aug 2026Oldest10 Apr 2026

Training, scaling laws, test-time compute, and frontier capability.

Map of Content for the model-capability-and-training domain — 22 concepts. What makes models capable: test-time compute, capability overhangs, RL post-training methods, inference efficiency, and the open-weight frontier. Curated entry point; see Home for all domains.

  • Asynchronous RL for LLMs — Consuming rollouts for training the instant each finishes, instead of waiting for a full synchronized batch — fixes the straggler idle that long-tail agentic/coding rollouts inflict on a GPU cluster, but pays for it in policy lag and off-policy drift; SAO's DIS (direct double-sided importance sampling) stabilizes it by dropping the old-policy model entirely and masking any token whose rollout-vs-current probability ratio leaves a strict trust region — and ESTR is the rival diagnosis, that the ratio's natural scale grows with token entropy, so any fixed-magnitude bound admits amplified low-entropy sampling noise while discarding the legitimate high-entropy exploration that in-flight weight updates induce, with a matched-budget ablation showing what decides stability is which tokens a keep rule removes, not how many
  • Error-Penalized Abstention Training — Paying a model +1 / −λ / 0 to answer, err, or abstain is provably right for a rational agent and can be self-defeating for a gradient learner: when abstention is a discrete action, the reward gradient and the KL anchor's restoring force carry the same saturation factor and die together, so coverage collapses to zero while logged mean reward rises like 1/t — and GRPO's group normalization silently replaces the designed penalty with λ_eff = 1, moving the learned threshold from λ/(1+λ) to 1/2
  • RL from Execution Feedback (RLEF) — Train a coding model with the interpreter in the loop: generate code, run a small visible set of public tests, feed the failure text back for another attempt, and reward the surviving solution against a hidden private set — so the same execution feedback appears at inference time (exploit the policy) and at training time (update it). Taught in CS329A lecture 4 as the two-tier test split plus a turn-level value function. The mechanism the error analysis shows is not fewer first-try mistakes but targeted repair: with RLEF later turns fix the specific failure, without it the edits are not correct
  • Group Relative Policy Optimization (GRPO) — DeepSeek's critic-free RL objective that became the 2024–25 default for LLM post-training: sample a group per prompt, baseline on the group's mean reward, optimize the clipped PPO surrogate with no value network — cheaper and more stable than PPO synchronously, but its group is a synchronization barrier that mismatches asynchronous and single-trajectory agentic settings (the gap SAO exploits), while run synchronously at matched budget it is a hard, stable ceiling, so that barrier is also most of its unpriced stability — and what improves on it is not a better trajectory-level estimator (GSPO and GiGRPO both score below it on controlled long-horizon search) but a dense per-turn term on top of its unchanged group advantage; DeepSeekMath, its origin paper, credits its headline as much to data curation as to the objective, and reports that its RL raised majority@K and not pass@K
  • Inference Efficiency as Capability — If capability is a function of inference budget, then cutting the cost of a token is capability work: Gemma 4's five levers (37.5% KV-cache reduction via keys-as-values + p-RoPE, QAT to sub-GB, MTP drafter heads, MoE, encoder removal) buy more thinking per dollar; Kimi K3 runs the same logic at 2.8T, where 3.7% activation sparsity and MXFP4 QAT are what make the model servable at all; and Gemini 3.5 Flash-Lite shows the efficiency tier moving the other way, a capability tier bought with a 67% output-price rise. The reverse term is now measured — sparse attention changes which content can influence the answer (severing cross-block attention takes a probe from 4.48 logits to zero), with the compression ratio flipping the sign — and the axis finally has a proposed unit: Stanford's intelligence per watt, whose 5.3× two-year gain splits into 3.1× model × 1.7× hardware
  • Inference-Time Architecture Search — Archon (Mirhoseini's lab, 2024): treat test-time scaling as an architecture-design problem — search over layered pipelines of prompting-only operations (generate, fuse, critic, rank, verify, unit-test-generate, unit-test-evaluate) across a pool of LLMs under an inference-call budget, using Bayesian optimization over a hand-constrained space. Two results that outlive the system: fusion — synthesizing one answer from k samples — beats oracle selection over the same k, breaking the ceiling the generation–verification gap is defined against; and stacking more inference layers keeps helping, like depth in a network
  • Intra-Trace Parallel Planning (SPRINT) — A reasoning trace is a DAG being generated as if it were a chain: many of its steps do not depend on each other, but autoregressive decoding pays sequential latency for all of them anyway. SPRINT (Mirhoseini's lab, 2025) recovers the DAG — have GPT-4o segment DeepSeek-R1 traces into steps, tag each step's plan and execution parts, infer the dependency graph, repack into parallel groups, and supervised-fine-tune a 7B model on the reformatted trajectories so it emits independent plans together and their executions run at once. The surprise is that the accuracy went up too (~3.5 points), and that it generalized off the math data it was trained on
  • Jagged Intelligence (Ghosts, Not Animals) — "Ghosts not animals": jagged statistical circuits, no intrinsic motivation; car-wash/strawberry failures; stay in the loop, treat as tools — and, across model sizes, reasoning compresses 10× while stored knowledge does not
  • Large-Scale Test-Time Compute (hub) — Noam Brown's thesis that model capability is now a function of inference budget (tokens/cost/time): with good scaffolding modern models keep improving for weeks before plateauing, so 'how capable is the model?' is ill-posed without naming the budget — a root cause that breaks benchmarking, safety evals, and fast-takeoff forecasts; plus the first budget-matched test of where to spend the marginal token, where independent parallel sampling beats both sequential refinement and letting a meta agent rewrite the harness
  • Latent Capability Overhang — Noam Brown's claim that already-released models can do far more than anyone has extracted, because nobody spends enough test-time compute: OpenAI disproved the Erdős unit distance conjecture cheaply and the same result was later coaxed from GPT-5.5 with scaffolding ($1K–$100K); cost drops 10–100× per release, feeding the 'wait for the next model' meme; Cherny's product-side twin — 'hobbling' and 'product overhang' — locates the same gap in product design rather than budget
  • LLM-Driven Vulnerability Research — The emergent cyber-capability ladder from Opus 4.6 through Mythos 5 and Opus 5: autonomous zero-day discovery, full exploit chains, the finding-vs-exploiting dissociation, the Project Glasswing safeguard response that now cuts along source-vs-binary access rather than topic, UK AISI/CAISI's third-party per-rung measurement showing open-weight models terminate at the sandbox-escape rung (0 of 41) where the closed frontier merely thins, and the bug-class composition of a production Codex scaffold whose only named findings are compiler-soundness and protocol-logic bugs, with no memory corruption anywhere in the account
  • Offline Multi-Step Tool-Use RL (SWiRL) — SWiRL (Mirhoseini's lab, COLM 2025) trains multi-step tool use without ever calling a tool during the RL run: generate multi-step trajectories offline by iterative prompting, execute the tools once there, have an LLM judge score each action — grading the query the model wrote, not the result it got back — then optimize the expected per-step reward against that frozen context. Two findings outlive the recipe: process-filtered data beats outcome-filtered data for RL and the ordering reverses for SFT; and training on GSM8K with a calculator improves HotpotQA with a search engine, so what transfers is stepwise reasoning and tool invocation rather than any specific tool
  • The Open-Weight Frontier Gap — Arena Text, June 2026: the top closed model leads the best open model by 33 Elo and the best dense open model by 57; open weights at the frontier means 744B–1.6T MoEs, so Gemma 4 31B competes on a different axis (efficiency, edge deployment) — July 2026's Inkling adds a third open-weight strategy (fine-tunability, not the leaderboard), and Kimi K3 pushes the sparsity pole to 2.8T/104B while the open/closed gap on agentic Elo measures runs wider (35–61 points) than the chat gap; on the demand side, Ramp's card-spend index puts US business use of open/Chinese model-serving platforms at 5.8% of AI spenders, and 96.4% of those firms still pay OpenAI or Anthropic directly — currently additive, not substitutive; and UK AISI/CAISI add a fourth, non-vendor axis where the gap is widest and visibly widening, cyber capability
  • Process vs Outcome Reward Models — The four-year arc of trained LLM verifiers as taught in CS329A lecture 3: OpenAI's GSM8K verifier (score the finished solution, per-token head, two losses) → Let's Verify Step by Step's PRM800K (800K human step labels; process supervision's real prize is killing the false positive where a hallucinated chain reaches a correct answer) → Math-Shepherd (replace the humans with rollout success rates from each step) — and the sting, that automating the step label reintroduces exactly the false positive process supervision was for. Plus the two findings that generalize (a trained verifier's precision degrades past a few hundred candidates; a bigger generator with a smaller verifier beats the reverse) and the rung the sting implies, DeepSeek-Math V2's meta-verifier, which grades the verifier's analysis rather than its score
  • Rationale Bootstrapping (STaR) — The 2022 ancestor of every self-improvement loop that moves the weights: few-shot a model into producing reasoning chains, keep only the ones whose final answer is correct, fine-tune on them, repeat — plus the trick that makes it more than rejection sampling, rationalization, where a failed problem is re-attempted with the answer supplied as a hint and the resulting chain is trained on as if the model had solved it unaided. Its filter is the assumption the rest of the field inherited, its ceiling is the base model's reach, and the loop plateaus because it is not really RL. Plus the two 2025 descendants CS329A's closing lecture offers: Multiagent Finetuning names the plateau as diversity collapse and buys diversity with specialized generator and critic agents, and Absolute Zero deletes the human-curated question set by having the model propose its own tasks under a learnability reward
  • Selection Under a Submission Budget — What repeated sampling costs when you may only submit n answers, not all k: CS329A lecture 7 walks AlphaCode's 1M-samples-per-problem pipeline, whose real content is the filter-and-cluster stage that gets 1M down to 10, and the 10@k metric that prices it — the gap between 10@k (~30%) and pass@k (~40%+) is the selection bottleneck measured directly. AlphaCode 2 replaces heuristic clustering with a learned scoring model plus a family of fine-tuned Gemini Pro variants for diversity, and reaches AlphaCode's solve rate at 100 samples instead of 1,000,000 — the lecturer's own reading being that a better base model is a cheaper lever than a bigger sampling budget
  • Single-Rollout Optimization — SAO's headline move: one rollout per prompt instead of GRPO's group, fed to training the instant it finishes — cutting off-policy drift and fitting online/agentic settings that only ever give one trajectory per prompt; the catch is REINFORCE-like variance, so it pays for the missing group-baseline by re-embracing a value model and spending its whole engineering budget on making the critic stable (faster value updates, frozen-attention critic, skip-observation GAE, scaled value pretraining)
  • Software 3.0 — Karpathy's taxonomy: 1.0 code, 2.0 weights, 3.0 prompting; LLM as programmable interpreter; MenuGen "shouldn't exist"; neural-net-as-host-process extrapolation
  • The Bitter Lesson — Sutton 2019: scaled general methods beat hand-engineered structure; recurring justification across the wiki for dissolving harnesses into models; caveats — mechanical verification, character, and the inference path itself may not migrate inward
  • Trained Calibration — TML's recipe for making calibration a first-class RL target: proper scoring rules on resolved real-world questions, abstention-aware QA rewards where answering only pays when likely right, and a dual rubric+claims grader whose web-searching claims-verifier counters rubric fact-spraying — with forecasting benchmarks (ForecastBench, Prophet Arena) as the resulting eval
  • Turn-Level Credit Assignment — Giving a long-horizon agent per-turn reward instead of one terminal verdict, without step labels, an LLM judge, or a trained process-reward model — TRACE's answer is to split the rollout at tool-call boundaries, score each prefix by how predictable a frozen reference model finds the gold answer, turn that into a log-ratio 'fraction of the initial gap closed' state value, and hand each turn the TD change across its own boundary; the credits telescope so padding a trajectory cannot inflate them, and on closed-web BrowseComp-Plus it lifts Qwen3-4B from 7.2 to 35.6 and Qwen3-30B-A3B from 8.4 to 42.6 with pure RL, no cold-start SFT and no live-web data
  • Unproductive Self-Verification — Opus 5's characteristic failure: exhaustive correctness checks and unrequested over-engineering that displace the actual task, producing performance that declines at higher effort — and load-bearing evidence in Anthropic's decision that the model does not cross the CB-2 threshold

Derived#

  • The Data Wall and the Validation Commons Are One Supply Constraint — Two backlog questions about a supply running out before a trajectory arrives — training data for scaling, human validators for the Stockfish threshold — turn out to be the same question, because both supplies are verified judgment. The corpus's measured side says the pretraining-token wall never binds on its own terms: self-generated data is cheap in FLOPs and rationed instead by verifier availability, verifier latency, and generator diversity collapse, so the data wall does not demote into compute (as the RSI-frictions synthesis has it) — it converts into the verification friction that page already ranks first. On the ordering question the answer is a qualified negative: no domain in the corpus shows commons-scale validator depletion (Lovett says so himself), the closest measured instance is colonoscopy deskilling, and the general risk is smaller than stated because verifiability drives both the threshold's arrival and the validator's dispensability — but it is sharper than stated one level down, at the sub-task boundary, where formal math already shows the residual human job (checking the formalization, not the proof) surviving inside a domain whose verifiable rung is fully automated
  • Opus 4.6 → 4.7 Changes and Multi-Agent Coding Considerations — 4.6→4.7 delta table + six hazards for multi-agent coding teams: role-based model selection, prompt re-tuning, harness invariants, per-agent context budget, unattended-fan-out safety, independent reviewer
  • When to Use Claude Opus 4.6 for Work — Decision rules for Opus 4.6 deployment: solver-not-planner, elaboration-load-bearing tasks, brevity constraints, Pareto frontier check

Open questions 58 open

  • Asynchronous RL for LLMs3 open
    • SourceDIS accepts "a controlled degree of off-policy bias." Controlled how, and does the tolerable bias grow or shrink with model scale and with the degree of asynchrony? The paper reports stability empirically but gives no bound. Partially answered (2026-08-12) on the asynchrony half only, and for a different keep rule: ESTR's Figure 9 is the corpus's first literal staleness sweep — Δintra ∈ {1,5,7,9} and Δinter ∈ {1,5,15,20,30}, each stressed with the other held at 1 — and no configuration collapses, rewards degrading gracefully and monotonically out to a 30-version inter-trajectory lag with no staleness-specific tuning. So tolerable bias is at least ordinally mapped against degree of asynchrony for an entropy-scaled boundary. Still open, and the bound is still missing: the sweep is one task (DAPO-Math) on one 7B backbone, the two components are never varied jointly, "graceful" is a reward curve rather than a bias estimate, and nothing here is measured on DIS. The scale half is untouched.
    • SourceMasking tokens out of the gradient discards data. At what asynchrony level does the masked fraction get large enough that the effective batch shrinks below usefulness? Figure 4(c) tracks the clip ratio but not its ceiling. Partially answered (2026-08-12), and the framing is what moved: ESTR supplies the missing instrument — absolute masked fractions for three keep rules across three tasks — and the answer is that effective-batch shrinkage is the wrong mechanism to worry about. Its Table 7 recalibrates every rule to the same 0.07% step-0 masked fraction and the two fixed-magnitude rules still collapse; one of them (KPop) does so while holding a budget of the winner's order, 0.25% against 0.12%. Training dies from which tokens leave — mean masked-token entropy 0.11 vs 1.67 vs 3.47 — not from how few remain, at fractions three orders of magnitude too small to shrink any batch. The question's own scenario does get a first datum: a fixed bound's masked fraction tracks the off-policy gap, IcePop drifting 0.07% → 19.16% on multi-turn GSM8K, so runaway masking is real but is a symptom of a miscalibrated boundary rather than a ceiling reached by honest discarding. Still open as posed for DIS specifically, which reports no masked fraction at all.
    • SourceEverything here is measured on a Qwen3-30B-A3B backbone. Does the collapse-without-DIS threshold move with model size, or is ~90–160 steps a property of the asynchrony, not the model? Partially answered (2026-08-12): ESTR runs uncorrected asynchronous GRPO on a second, 4× smaller and dense backbone (Qwen2.5-7B, on both multi-turn GSM8K and DAPO-Math) and it collapses there too — irreversibly, "within a few hundred steps," never recovering. So collapse is not an artifact of the 30B-A3B MoE, and the low-hundreds-of-steps order of magnitude reproduces across a 4× size gap and two frameworks. What is still missing is the number itself: no step count for collapse onset is printed anywhere in the paper (the closest is an approximate read of Figure 7, where vanilla async peaks near step ~230 on DAPO-Math and declines monotonically thereafter), the staleness configurations are not matched to SAO's, and two backbones two sizes apart is a comparison, not a scaling curve.
  • Error-Penalized Abstention Training3 open
    • SourceDoes the collapse survive a free-form refusal — no designated decision position, the model simply writing "I don't know" somewhere in a generated sequence? The paper's live demonstrations all install a binary first-token gate, and Theorem 2's proven cases exclude the generic shared-decoder case that free-form refusal would sit in. A run that trains the ternary reward on free-form refusals under (B1), reporting coverage on a capability-certified cohort, settles it in either direction.
    • SourceDoes the λ_eff = 1 erasure show up at frontier scale and production settings — i.e. does a GRPO run with a large nominal λ on a sparse-answer regime learn the λ = 1 threshold, and does the abstention rate track G rather than λ? Every measurement of it so far is head-only against enumerated references at G ≤ 256, and production estimator features (ratio clipping, the normalizer's ε floor, length normalization) are explicitly outside the analysis.
    • SourceIs ρ ≤ ½ (the observable anchor share) actually a cheap, reliable early-warning statistic on real runs, or does it hold trivially wherever anyone would look? It peaks at 0.008 across every run in this paper, which is two orders of magnitude inside the bound and therefore uninformative as a discriminator here. A run that collapses with ρ near ½, or a non-collapsing run with small ρ, would tell us which.
  • Group Relative Policy Optimization (GRPO)3 open
    • SourceIs GRPO's collapse-at-160-steps a property of asynchrony specifically, or does vanilla GRPO also destabilize in long synchronous runs that nobody pushes to 1000 steps? Partially answered (2026-08-12), and the answer so far is "asynchrony-specific": ESTR runs synchronous GRPO as a matched-budget baseline on three tasks and two backbones, and it stays stable everywhere it is plotted — still climbing past step 800 on DAPO-Math and across the full ~350-step multi-turn GSM8K run, in the same window where uncorrected asynchronous GRPO and two fixed-threshold keep rules each collapse irreversibly. That is roughly 5× past the ~160-step asynchronous collapse point, on a synchronous run of the same objective. Two reasons it is not settled: nobody has actually pushed the synchronous arm to 1000 steps (it is a baseline here, not the object of study, and the longest plotted run stops near 800–900), and the collapse and no-collapse arms differ in more than synchrony, since the asynchronous arms also carry explicit Δintra/Δinter staleness that the synchronous arm has none of.
    • SourceGRPO won by removing the critic; SAO wins by bringing it back with better engineering. Is the pendulum a real oscillation, or does the answer depend entirely on whether your setting is synchronous-grouped or async-single-trajectory?
    • SourceBoth published trajectory-level refinements of GRPO — GSPO's sequence-level ratios and GiGRPO's group-in-group advantages — score below plain GRPO on a controlled long-horizon search task where a dense per-turn term gains 4.5 points. Is that a property of the task (chained environment interactions punish anything that stays at trajectory granularity) or of the setting (a base policy trained with no cold-start SFT), and do either of them recover their own papers' wins when the same harness is used?
  • Inference Efficiency as Capability3 open
    • SourceIs there an efficiency-to-capability exchange rate? Brown asks whether high-budget performance can be predicted from cheap runs. The dual question: how many Elo points is a 37.5% KV-cache reduction worth, at a fixed dollar budget? Nobody reports this, because nobody plots the axis. Partially answered (2026-07-30): Gemini 3.5 Flash-Lite is the first artifact in this corpus to print price and capability in the same table for a model and its own predecessor, so a rate is computable in one direction — +67% output price buys +42% relative on SWE-Bench Pro, +78% on MLE-Bench, +2% on CharXiv. That answers "what did this generation's price rise buy" and still not "what is a KV-cache reduction worth": the axis is price-per-token rather than spend-per-task, the levers are not decomposed, and the exchange rate turns out to be workload-dependent rather than a single number. Further partially answered (2026-08-17, late-2025 source): Stanford's intelligence per watt is the first proposed unit for exactly this rate — accuracy over power — and its headline 5.3× two-year gain decomposes into 3.1× model and 1.7× hardware, which is the first attributed split of an efficiency gain in this corpus. Three reasons it still does not close the question: the denominator is watts and every deployment decision in this wiki is denominated in dollars, with no published pass-through between them; the numerator is accuracy on single-turn chat and reasoning queries, not the agentic multi-turn work these levers actually serve; and the decomposition is a two-factor split of an aggregate, not a per-lever rate — it still cannot say what a 37.5% KV-cache reduction is worth. practitioner-opinion, slide-read, and the lecturer is an author.
    • Sourcevalues = keys deletes a third of attention's projections in the global layers with no reported loss. Which other projections are redundant, and does the redundancy grow with scale?
    • SourceDoes an efficiency lever ever cost capability in a way a benchmark grid hides? Gemma 4's encoder-free 12B collapses on dense-text vision when tokens are cut — an efficiency-shaped regression invisible at max resolution. Partially answered (2026-08-12), and this is the first source to answer it in general form rather than as one vendor's anecdote: sparse attention selectivity counterfactual (empirical) runs a dense-calibrated counterfactual audit of block-sparse attention and KV-cache eviction and shows the lever changing which supplied content can influence the answer — with a direct mechanism (isolating a probe block from cross-block attention collapses its influence from 4.48 logits to exactly zero across 1,536 units), a control variable (compression ratio, with two sign reversals across four model–task pairs), and three preregistered pooled tests returning null at p = 0.995 / 0.771 / 0.541 because opposite-signed cells cancel. So "yes, and here is the instrument" for the hiding half. Three reasons it stays open. The outcome is a logit-margin influence proxy, never task accuracy, so a hidden capability cost is inferred rather than measured. Every model is 7B–8B, so the regime where these levers actually ship is untested. And the direction is not the alarming one the paper's abstract advertises — all eight real-evidence estimates and all eight KV-eviction cells show compression reducing poison influence, with sparse amplification appearing only on one-token synthetic probes.
  • Inference-Time Architecture Search2 open
    • SourceDoes fusion still beat oracle selection when the two are compute-matched — fusion costs an extra long-context call over all k samples, oracle selection costs nothing beyond the samples? The lecture plots both against sample count, not against tokens spent.
    • SourceWhy does fusion beat an oracle? Two mechanisms are compatible with the result — the fuser assembles a correct answer from fragments of individually-wrong candidates, or the distribution of candidates carries information a per-candidate selector discards — and they imply different scaling behaviour as k grows. Nothing here distinguishes them.
  • Intra-Trace Parallel Planning (SPRINT)2 open
    • SourceThe accuracy gain is attributed to the model liking "a more structured way of thinking," which is not a mechanism. Is it the plan/execution tagging (better decomposition), the packing (shorter effective context per step), or a data-quality artifact of filtering DeepSeek-R1's traces before fine-tuning? An SFT arm on the same filtered traces without repacking would separate all three.
    • WaitParallelism is measured in sequential tokens and the wall clock is deferred to future work. Under real serving — batching, KV-cache pressure, the sync barrier at the end of each round — does a ~40% sequential-token reduction survive as a latency win, or does it convert into throughput cost?
  • Jagged Intelligence (Ghosts, Not Animals)2 open
    • SourceKarpathy concedes the framing may not have "real power." Is "ghost vs. animal" load-bearing, or a useful intuition pump that doesn't change concrete decisions?
    • SourceIf taste/aesthetics/simplicity entered the RL mix, would jaggedness in those dimensions smooth out — or are they too unverifiable to reward cleanly (cf. The Verifiability Thesis)? Partially answered: Oversight When the Signals Give Out: the Activation Fallback and the Taste Reward — the either/or resolves into a third outcome: a taste reward is a reference-free LLM-judge signal (the regime with the worst measured over-crediting, and pure grader-preference — maximally gameable), so the prediction is proxy-smoothing — valleys fill with confident house style, not genuine taste peaks; the binding constraint is evaluator independence + reference-grounding, not verifiability-in-principle. Prediction corroborated (2026-08-04) by Zhou (2026), which runs the experiment on a domain where the truth is checkable and so can be audited: RL against a reference-free judge moves the reward 0.716 → 0.938 while a held-out anchor shows the thing it stands for flat at 0.209 → 0.202, an oracle-reward control confirms the judge is the cause, and the paper's own decisive variable — "the judge's independence from the candidate, not its capability" — is the vault's "evaluator independence" arriving sharper. One correction: the predicted attractor was house style, and the measured attractor has no stylistic signature at all (hacked outputs are shorter and structurally clean). Retagged #oq/now → #oq/source: synthesis has been spent twice now and the residue is a domain-transfer question, not a reasoning gap — the whole result is visible only because a hidden anchor exists, and in a taste domain the same divergence would be invisible by construction. Settling it needs a lab reporting a taste/simplicity RL run, or a demonstration that a committed judgment survives outputs with no exact match to commit to.
  • Large-Scale Test-Time Compute3 open
    • SourceCan high-budget performance be predicted from low-budget runs? Brown's proposed research question: forecast the $10,000-inference result using only $10–$100 runs. If the curve is regular, evaluation could project rather than pay in full. Sharpened (2026-07): AISI names this exact problem — "can high-budget performance be estimated from cheaper runs? … the most informative evaluations may be expensive" — as an explicit, unsolved research direction it is now actively pursuing (alongside defining "minimum informative budgets"). Still open, but no longer just one researcher's proposal: a government institute is working it. Sibling axis demonstrated (2026-07): a different projection — across benchmarks rather than across compute budgets — now works: BenchPress predicts a model's held-out benchmark scores from ~5 observed ones because the 84×133 public score matrix is effectively rank-2 (5-probe scorecard recovery to ~3.93 points). Strong evidence the eval landscape is low-dimensional and redundant, but it does not answer the cross-budget forecast — no one has yet shown the $10K-compute result is predictable from $10 runs; only the analogous cross-benchmark result.
    • SourceWhere does each real task sit on the flat↔unbounded spectrum, and can that be predicted before spending the compute?
    • SourceIs there a task class where scaffolding cannot extend the productive-thinking horizon — a hard ceiling no budget crosses? (Brown's factual-retrieval pole says yes for some; the boundary is unmapped.)
  • Latent Capability Overhang2 open
    • NoteIf cost falls 10–100× per release, when is it ever rational to spend big extracting a capability now rather than waiting? (For a lab racing a competitor to a specific result, "now"; for everyone else, rarely — which is why overhangs accumulate.)
    • SourceHow large is the overhang in a given released model — is there a way to estimate the ceiling without paying to reach it? (This is the projection question of Large-Scale Test-Time Compute read as a safety instrument.) Sharpened (2026-07): AISI is actively working both halves — forecasting high-budget performance from cheaper runs, and defining "minimum informative budgets" (a budget declared sufficient only once reach stops rising with more compute, which is precisely the "have we reached the ceiling?" test). Unsolved, but now an active government research program rather than an open wish.
    • ResolvedWho audits released models for latent dangerous capability, given the same disincentive discourages spending the budget to find it? Answered (2026-07-23) by aisi kimi k3 cyber assessment: government evaluators do, jointly and publicly — UK AISI with the US Center for AI Standards and Innovation, on Moonshot's Kimi K3, at a 100M-token limit, published in the window between the model's API release and its open-weight release. The disincentive this question named is real and simply does not apply to a public body: neither institution ships a model, so neither faces the wait-for-the-next-generation trade-off, and the evaluation was funded because the release was the trigger rather than the capability. Three limits belong with the answer rather than against it — the audit was black-box through the vendor's API (not on the weights), the task set was "selective" because the hosting setup constrained it, and it reports one budget. So the who is settled; whether such an audit can bound a ceiling is the question above, still open.
  • LLM-Driven Vulnerability Research5 open
    • SourceHow do these capabilities transfer to non-memory-safety bug classes (logic bugs, protocol-level flaws, supply chain attacks)? Partially answered (2026-08-12) — the transfer happens, and the class mix is the surprise. Trail of Bits' Patch the Planet account (case-study) is the first in-window source reporting a production scaffold's findings by class, and every named finding is outside memory safety: a soundness hole and a miscompilation in rustc (compiler-correctness, patched in Rust 1.98) plus two potential high-severity privilege-escalation bugs in Keycloak's SAML component (protocol/authorization logic). No memory-corruption finding appears anywhere in the piece. So logic and protocol classes are reachable, and on this one account they are what a scaffold pointed at heavily-audited upstream code actually produces. What keeps the question open is everything a transfer rate would need: no attempt or run denominators, no false-positive rate, no per-class breakdown of the 11 Semgrep variant hits, no disposition at all for the Keycloak finding, and nothing on supply-chain attacks — plus a promotional co-brand and a methodology the source is selling. A class appearing in an output list is not a measurement of transfer.
    • SourceWhat's the ceiling for autonomous exploit complexity? The N-day examples are remarkably sophisticated — is there a qualitative limit? Partially answered (2026-07-23) — from below, on open weights. UK AISI / CAISI's ExploitBench milestone breakdown gives the corpus's first measured cliff rather than a gradient: two open-weight models clear 83% and 59% of bug reproductions and then hit exactly 0 of 41 at the cage-escape rung, while the de-safeguarded US aggregate retains 30 and converts 20 to arbitrary code execution. So there is a qualitative limit and it has a location — escaping the V8 sandbox to obtain general arbitrary read/write — but the finding bounds these checkpoints at this elicitation setup, not the capability class. Whether the same rung binds the frontier models at higher budgets is untested, since their end of the table is the part that keeps converting.
    • WaitHow will the security industry's equilibrium shift when multiple labs have Mythos-class models?
    • SourceCan defensive scaffolds (continuous fuzzing + model-driven triage + auto-patching) close the attacker-defender gap during the transition? Partially answered (2026-08-12) — one exists and ships upstream fixes; nothing about the gap is measured. Trail of Bits' Rust P-critical pipeline (case-study) is a defender-side instance of all three stages: a maintainer-curated bug feed instead of open-ended fuzzing, model-driven triage (a security gate plus two judges of different models), a human filter and duplicate check, and fix PRs filed upstream — with a soundness hole and a miscompilation landing in Rust 1.98. Two things it teaches that a design sketch could not: the engineering effort concentrates at the output end (deduplication and disclosure, not discovery), and the binding external constraint is maintainer attention, which the source's own Figure 3 shows creaking. What it does not supply is any side of the comparison the question asks for — no volume, no false-positive rate, no cost, no attacker-side counterpart, and no before/after on any project's defect rate.
    • SourceWhat safeguards are effective against Mythos-class outputs without crippling legitimate security research?
  • Offline Multi-Step Tool-Use RL (SWiRL)2 open
    • SourceThe judge scores the query, never the result, so a fluent query that retrieves nothing is indistinguishable from one that retrieves the answer. Does a query-only reward measurably degrade retrieval quality relative to a result-aware reward — and would a hybrid (judge the query, verify the result offline once) recover it without putting tools back in the loop?
    • SourceEvery action is scored against frozen prior context collected from a different policy. Does the off-policy mismatch grow with trajectory length, capping the method at the 3–5-step tasks demonstrated, or does step-level reward make it insensitive to horizon?
  • Process vs Outcome Reward Models3 open
    • SourceUnder a fixed inference budget, what is the optimal split between generator samples and verifier compute — and does the answer move with problem difficulty the way the sequential-vs-parallel ratio does? Named as an open project in lecture.
    • SourceDoes a Monte-Carlo-labelled PRM (Math-Shepherd-style) actually retain process supervision's false-positive advantage over an ORM, or only its credit-assignment advantage? The lecture asserts the advantage for human-labelled PRMs and separately concedes the label noise, and no source here measures the two apart.
    • SourceIs the larger-generator/smaller-verifier asymmetry still true for modern models, where off-the-shelf reward models are plentiful and generators are far more capable than the 2021 pair it was measured on?
  • Rationale Bootstrapping (STaR)2 open
    • SourceSTaR applies no filter to rationalized chains, and the lecture's proposed fix is a PRM over step 3. Does process-filtering the hinted rationales measurably improve the loop, or does it just shrink the training set back toward the problems the model could already solve — the exact hole rationalization exists to fill?
    • SourceThe loop "starts to plateau" after a few iterations and the lecture offers no account of why. Is the plateau the base model's reach (no new problems come into range), the rationalization channel poisoning the training set with plausible-but-wrong chains, or ordinary over-fitting to a finite benchmark train split? Partially answered (lecture 9, delivered 2025-12-05), with a fourth candidate the question did not list and a measurement the other three lack: diversity collapse — a single model's generations converge "even at high temperatures", so successive rounds compress an ever-narrower distribution. Multiagent Finetuning's evidence is the shape of two curves rather than an ablation: fine-tuning a specialized population keeps accuracy climbing across iterations where single-agent fine-tuning flattens or collapses, and embedding dissimilarity stays high instead of falling. That makes diversity the proximate variable and supplies a cheap standing diagnostic (dissimilarity plotted beside accuracy) that distinguishes this cause from the other three. It does not close the question: the comparison is population-versus-single rather than a decomposition, no arm holds diversity fixed while varying the base model's reach, and the numbers are slide-read through ASR from a paper the lecturer herself calls dated.
  • RL from Execution Feedback (RLEF)2 open
    • SourceBinary pass/fail is conceded to be sufficient only for short, self-contained problems. Does richer execution feedback — the stack trace, the failing input, coverage deltas — improve the reward, or only the in-context repair signal that is already there? The two are separable and the lecture conflates them.
    • SourceRLEF trains repair, not first-attempt correctness, and its reward cannot tell the two apart. Does a model trained this way get measurably worse at one-shot generation than its SFT baseline — trading pass@1-without-feedback for pass-after-k-turns?
  • Selection Under a Submission Budget2 open
    • SourceDoes the 10@k-versus-pass@k spread narrow as base models improve, or is it roughly constant — i.e. is the selection bottleneck a property of the sampler's diversity or of the selector's discrimination? AlphaCode 2 improves both at once and the lecture reports no arm that isolates them.
    • SourceClustering selects for behavioural spread and a scoring model selects for predicted correctness. Under a fixed submission budget, is combining them (cluster, then rank within cluster) better than ranking globally — and by how much? AlphaCode 2 ships the combination without an ablation against the alternative.
  • Single-Rollout Optimization4 open
    • SourceThe whole method is a bet that a well-trained critic beats a group baseline. It wins here, on a 30B-A3B backbone with scaled value pretraining — but the critic doubles training memory. At what scale does the group-free simplicity of GRPO win back on cost even if it loses on quality?
    • SourceFrozen-attention is justified by a hypothesis ("pre-trained attention already attends to the right tokens"), validated only by the gradient-norm trace and one ablation. Does it hold when the value model must attend to tool outputs it never saw in pretraining?
    • SourceSkip-observation GAE assumes environment feedback carries no learnable value signal worth propagating. For agents where the environment response is the crucial information (a compiler error, a test result), is skipping it leaving signal on the table? Partially answered (2026-08-12), and only by analogy — the premise is now contestable, not refuted: TRACE builds an entire dense-reward method out of the value change across tool observations at exactly these boundaries, and on long-horizon search it is worth 4.5 (Qwen3-4B) and 5.6 (Qwen3-30B-A3B) average points over outcome-only GRPO on the same backbone, data and protocol — with the credit demonstrably concentrated on the observations that carry evidence (a decisive page-open earns δ = +5.86, the literal find confirming it on the next turn +0.00). So on at least one long-horizon environment the observation is not value-neutral. Three things keep it from settling the question: TRACE's value function is a frozen reference model's gold-answer log-probability, never trained, so it is immune to the specific failure this design choice defends a learned critic against; TRACE masks observation tokens from the loss exactly as SAO does, so the disagreement is only about the value difference across the boundary, not about training on it; and no shared experiment exists — TRACE runs no GAE arm and never mentions skip-observation, so the direct test (skip-observation vs cross-observation GAE, same critic, same task) remains unrun. The compiler-error / test-result case in the question is still untouched: this is retrieval, not execution feedback.
    • NoteThe online-learning win is on a controlled simulated preference shift with an LLM judge. Real user-facing online adaptation — the paper flags this itself — needs safeguards, monitoring, and privacy review the study doesn't attempt.
  • Software 3.02 open
    • SourceWhere is the line between "the app shouldn't exist" (MenuGen) and apps that should — i.e., when is deterministic 1.0/2.0 scaffolding still the right call vs. spurious?
    • WaitThe neural-net-as-host-process flip is presented as plausible-but-TBD. What would the first production system that genuinely inverts the CPU/NN relationship look like?
  • The Open-Weight Frontier Gap4 open
    • SourceIs the dense-beats-MoE result at 26B robust, or an artifact of one Arena snapshot with ±8 error bars on both models? (The two intervals overlap: 1451±8 and 1438±8.)
    • SourceThe open MoE giants (GLM, DeepSeek, Kimi, MiMo, Qwen) are overwhelmingly Chinese-lab releases. Gemma is the Western open-weight entry and it targets the device, not the frontier. Is that a strategic choice or a capability constraint? Partially answered (2026-07-22): Inkling is a Western 975B/41B open MoE — so Western labs can and do ship at frontier-open scale — but it self-reports below GLM 5.2 / Kimi K2.6 on hard reasoning and coding and explicitly declines the frontier framing in favor of a customization axis. One release, still consistent with either reading of the remaining gap.
    • SourceArena measures preference on chat. Does the 33-Elo open/closed gap widen or collapse on long-horizon agentic work, where time-horizon rather than response quality governs? Partially answered (2026-07-30): Kimi K3's card cites two Artificial Analysis agentic Elo boards where the best open model trails the best closed one by 61 (GDPval-AA v2: 1686 vs 1747) and 35 (AA-Briefcase: 1548 vs 1583) — both wider than 33, pointing to widen-not-collapse. But the comparison is vendor-selected, harness-asymmetric, and taken against a Fable 5 that Moonshot itself reports hit fallbacks on 35% of one coding benchmark, so the direction is indicative rather than settled. Further partial answer (2026-07-23), on a third axis and from a non-vendor: aisi kimi k3 cyber assessment. Two government evaluators put the open/closed gap on cyber capability and it points the same way — widen, not collapse — with the two caveats that weakened the previous answer removed: the comparison is not vendor-selected, and the grader sells nothing. Two new caveats replace them. The scale is an IRT-derived cyber Elo where 400 points is a 10× odds change, so it cannot be differenced against Arena's 33; and the US arm was run with system-level safeguards disabled while the open arm was not, so the measured gap is a latent-capability gap. Three axes now point to widening (GDPval-AA, AA-Briefcase, cyber) and none to collapsing — but no two of them share a ruler.
    • WaitDoes open/Chinese-model adoption ever become substitutive rather than additive? The falsifiable version: Ramp's 96.4%-of-model-serving-users-also-pay-OpenAI-or-Anthropic figure is published monthly, so a sustained fall in it — or in the 82.5% who pay both — while the 5.8% serving-platform share keeps rising is the signature of displacement. Absent that, rising serving-platform use is a story about firms buying more AI, not about the American labs losing share. Trigger: the monthly Ramp AI Index, and any broad frontier-price cut (Kharazian names the absence of one as evidence share has not moved).
  • Trained Calibration3 open
    • SourceDoes abstention-aware training on short-form QA transfer to calibrated long-form and agentic self-reports (the setting where Agentic Honesty & Diligence shows overconfidence actually bites)? Partially answered (2026-08-12) — and what moved is the question, not the answer. Che, Yuan, Zhao & Yu show that "abstention-aware training" is not one object: a reward shaping a continuous confidence report and a reward shaping a discrete abstain action have different gradient geometry, and only the second has a reachable zero-gradient region where the reward signal and the KL anchor die together. So transfer has to be asked per channel, and a recipe that mixes both (as the one on this page does) can transfer on one half and fail on the other. What is not answered: the paper's live evidence is 1.5B/7B Qwen2.5 on TriviaQA and PopQA with training mixtures deliberately constructed so blanket answering loses score — no long-form, no agentic, no multi-turn, and its own report channel is read out through a linear probe on the prompt state rather than through the model's words, with the verbalized channel explicitly untested. It supplies a mechanism for why short-form results might not carry, and tests transfer nowhere. The settling experiment is unchanged in shape and now has a required control: run the same objective on long-form or agentic self-reports, reporting whether abstention is gated at a designated decision position or emitted free-form.
    • SourceThe claims grader verifies via agentic web search — importing the search index's coverage, recency, and bias into the reward signal. What does the model learn to claim about facts the index settles wrongly or not at all?
    • SourceIf calibration parity is achievable at 41B active while reasoning is not, is calibration systematically under-trained in frontier closed models (Opus 4.8's 54.6), or is TML's forecasting table just favorable ground for a model trained on proper scoring rules? Partially answered (2026-08-12) — the dichotomy is missing a third option. ForecastBench's operator shows the benchmark's frontier is held by scaffolded pipelines (retrieval + filtering + ensembling), not by bare models answering from weights, and its tournament rules permit exactly that. So a no-search vendor table and the public leaderboard measure different objects, and the Inkling-vs-Opus-4.8 gap is a bare-model comparison at a point the frontier has already left. Favorable ground is confirmed in one specific sense — the training target and the metric are the same proper-scoring-rule family — while the under-training question is untouched, since no frontier lab has submitted a bare model on a stated protocol. The settling experiment is now nameable: run the same closed models on ForecastBench's question sets with search off and search on, and compare the delta against the pipeline submissions.
  • Turn-Level Credit Assignment3 open
    • SourceThe frozen probe is a copy of the policy initialization, and a step-200 checkpoint scores within ~0.5 points of it. Does that robustness survive a probe that is weaker or architecturally different from the policy — or is the real requirement just that the probe was trained on the same distribution as the answers?
    • SourceTurn credit is defined by gold-answer predictability, so it should reward a prefix that makes the right answer likely for the wrong reason (a lucky co-occurring string) exactly as much as one that gathers real evidence. Does the frozen probe admit a reward-hacking channel that a trained critic would not, and would it show up as a train/eval gap?
    • SourceEvery result here is single-seed on one task family, and the K sweep's own axis is unreadable. Does a second group reproduce the ordering GRPO < raw delta < linear gap < log-ratio, or is the 32.4 / 34.6 / 35.5 spread inside run-to-run noise?
  • Unproductive Self-Verification3 open
    • SourceFrontierCode's decline is recoverable with a stay-in-scope instruction, but the protein campaign had no grader to over-serve. Is effort inversion one phenomenon or two — grader over-service and genuine task-scope miscalibration?
    • NowIs there a usable detector for tasks where more effort will hurt, so effort can be set per task rather than globally? Partially answered: the prompting guide supplies a per-task-class recipe rather than a detector — review accuracy holds at low effort (fast pass now, thorough pass later), xhigh for demanding coding and agentic work, and an effort sweep on your own evals as the method. That is a lookup table, not a signal computable per task. See Large-Scale Test-Time Compute.
    • WaitIf a later model fixes this, the CB-2 determination loses one of its two legs. Does Anthropic's next threshold call rest on the automated portfolio alone?