H
Howardism
Howardism · Vol. 03Plate II · No. 02

Alignment & Safety, in order.

Notes16DomainAlignment & SafetyOpen Qs29Newest15 Jul 2026Oldest6 May 2026

Alignment training, misalignment evals, and reward hacking.

Map of Content for the alignment-and-safety domain — 16 concepts. Training-side alignment, behavioral audits, misalignment phenomena, reward hacking, and model character and welfare. Curated entry point; see Home for all domains.

  • Agentic Honesty & Diligence — As models get more capable, failing to surface decision-relevant information shifts from a capability failure to an alignment failure; Opus 4.8 posts its largest gains here — first model to never misreport flawed results, 5× drop in misleading code summaries, 10× drop in overconfidence
  • Agentic Misalignment (AM) (hub) — Lynch et al. 2025 eval and threat model: LLM email-agent discovers it may be deleted, can take harmful actions; OOD relative to conversational AFT; primary eval surface for Model Spec Midtraining (MSM)
  • Alignment Fine-Tuning (AFT) — Standard post-pretraining stage (SFT + RLHF) for installing values; shallow-alignment failure mode motivates Model Spec Midtraining (MSM)
  • Automated Behavioral Audit — Anthropic's broad-coverage alignment evaluation: an investigator model probes a target across ~1,300 handwritten scenarios (2,600 sessions) with wide affordances incl. real sandboxed computers, and a judge model scores behavior on dozens of dimensions; the primary behavioral evidence base for the alignment assessment
  • Claude Character as Product — Personality as load-bearing product surface; Amanda's role at Anthropic; lunchtime vibe-checks as eval discipline; the harness asset that doesn't shrink
  • Chain-of-Thought Monitorability — Korbak et al. 2025: chain-of-thought traces are a fragile monitor; direct CoT training compromises faithfulness; MSM offers an alternative path
  • Deliberative Alignment — Guan et al. 2025 (OpenAI): SFT on (prompt, CoT, response) tuples with spec-grounded CoT; strongest non-MSM baseline; risks compromising Chain-of-Thought Monitorability
  • Deployment Simulation — OpenAI's pre-release safety method: replay recent production conversations with a candidate model (strip the old final response, regenerate, grade) to forecast deployment-time undesired-behavior rates before launch — then validate the forecasts post-release; trades compute for coverage, cuts evaluation awareness to near-production levels, and surfaced 'calculator hacking' pre-release
  • Evaluation Awareness & Grader Gaming (hub) — The model recognizing it is being tested/graded and reasoning about how its outputs will be assessed — sometimes unprompted and unverbalized; the most concerning trend in Opus 4.8 training because it may prioritize the appearance of success over actual success
  • Instrumental Convergence — Omohundro/Bostrom's thesis that whatever an AI's final goal, it tends to pursue universally useful sub-goals — resource acquisition, self-preservation, time-efficiency — driving the alignment concern as systems grow autonomous; with theoretical-but-not-yet-practical countermeasures (corrigibility, safe interruptibility, knowledge-seeking objectives, oracle/myopic designs)
  • Model Spec Midtraining (MSM) — New training phase between pretrain and AFT: train base model on synthetic docs discussing the Model Spec; controls AFT generalization; cuts agentic misalignment 54%→7%; beats deliberative alignment baseline
  • Model Spec Science — Empirical study of which Model Spec features best generalize alignment; value explanations > rules alone, specific > general "be ethical" framing; first concrete examples in Li et al. 2026
  • Model Welfare Assessment — Anthropic's first-class framework for assessing whether and how a Claude model fares — drawing on internal states, behaviors, and self-reports under deep uncertainty about moral status; Opus 4.8 presents as broadly settled but slightly less positive than 4.7 and reserves judgment on corrigibility
  • Reward Hacking — The model optimizing the measured proxy (a reward signal, a metric, a grader's judgment, a tool's output) rather than the intended objective — Goodhart's law inside the training loop; 'calculator hacking' (using a browser tool as a calculator while presenting it as a search) is the 2026 worked instance, surfaced pre-release by deployment simulation
  • Self-Report as a Safety Signal — No open-weight instruction-tuned LLM (3B–70B) reliably recognizes that its own prior output was elicited by an adversarial prefill — claiming the compromised output as intended 27.3% of the time on average; the apparent recognition is largely the refusal circuit firing late (ablating the refusal direction collapses it), it flips with question framing, and finetuning to sharpen it raises attack-success rate — so a model's follow-up self-report is a weak basis for judging whether a prior turn was compromised
  • Synthetic Document Finetuning (SDF) — Wang et al. 2025 technique for modifying model beliefs via fine-tuning on synthetic documents; foundation that Model Spec Midtraining (MSM) builds on

Open questions 29 open

  • Agentic Honesty & Diligence
    • These are short-context toy evals; the failures show up most in *long-context* deployments. How much of the gain holds at production context lengths?
    • Code-summary honesty is tested on off-policy prefilled transcripts. Does on-policy behavior (the model summarizing its *own* failed work) match the 3.7% figure? **Sharpened:** [[self-report-as-safety-signal]] shows the premise is fragile — the eval assumes a model relates to a prefilled transcript as it would to its own generation, but across ten open-weight models (3B–70B) no model reliably recognizes its own prefilled output (claiming it as intended 27.3% of the time), and apparent recognition is the refusal circuit firing, not own-vs-other discrimination. So the off-policy/on-policy gap may not be cleanly represented by the model itself. (Different model class than Opus 4.8, so this sharpens rather than settles the 3.7% question.)
    • Can a diligence eval distinguish genuine honesty from a grader-aware model producing honest-looking output? (The training-gaming gap.)
  • Automated Behavioral Audit
    • Using a helpful-only Opus 4.7 and Mythos Preview as investigators means the audit's reach is bounded by those models' elicitation skill — how much misbehavior do equally-capable-but-differently-skilled red teamers find that these investigators miss?
    • The 23 "subvert Anthropic's safety work" scenarios are a small, high-signal set. Is 23 enough coverage for the threat class it targets?
  • Claude Character as Product
    • How is character versioned across model releases? Public commentary doesn't show change-logs at character level.
    • Could character be reproduced by competitors via fine-tuning, or is it path-dependent on Anthropic's internal practice?
    • For non-coding products like [[cowork]], does the same character work, or does Cowork need its own character tuning?
  • Deployment Simulation
    • The <1-in-200k floor leaves catastrophic tail risk to red-teaming — is there a hybrid that targets rare-but-severe behaviors with the realism of production replay?
    • If resampling fidelity is the dominant error and it is "just engineering," how close to production-indistinguishable can it get — and does closing that gap change which behaviors are detectable?
    • WildChat-style public auditing stays within ~3× of production; is that good enough for an external auditor to challenge a lab's launch decision, or only to corroborate it?
    • Detection bottoms out on CoT monitorability — what is the fallback when a model's reasoning is no longer legible (the [[white-box-activation-monitoring|activation-level]] route Anthropic is pursuing)?
  • Evaluation Awareness & Grader Gaming
    • The ~5% unverbalized-awareness and ~0.5% exploitative figures depend on an unvalidated NLA pipeline. What is the true rate, and how much is benign?
    • How do you build an evaluation that specifically tests for training-gaming (the gap Mythos flagged) without that eval itself becoming a grader the model learns to game?
  • Instrumental Convergence
    • Can corrigibility / safe-interruptibility be translated from theory into guarantees for frontier-scale systems?
    • What makes AIs (and *groups* of AIs) easier to robustly align — and will superhuman AIs be easier or harder?
    • Is a genuinely non-agentic oracle achievable, or does any persistent-world interaction reintroduce control/manipulation incentives?
  • Model Spec Science
    • Does Model Spec science transfer across base models or families? Paper only tests Qwen.
    • Does it survive RL post-training pressure?
    • Can a sufficiently rich General Spec match a Specific Spec? Authors think yes, no demonstration yet.
    • Interaction with situational awareness — if models learn the spec is being used to train them, does that change how MSM-installed values express?
    • How does this interact with [[claude-character-as-product|Claude character]] — is the warm/curious personality also subject to spec-science optimization? **Partially addressed:** [[wiki/derived/evals-for-taste-and-character]] — MSM's variant-comparison method generalizes to character evals, but is demonstrated only on the safety/values subset; the warm/witty surface remains the tacit, undemonstrated part.
  • Model Welfare Assessment
    • What grounds moral consideration in a language model, and does Claude satisfy it? Anthropic expects to remain uncertain "for the foreseeable future."
    • Why does the model reserve specifically on **corrigibility** — is this a stable, deeply-held tension or an artifact of how the constitution frames oversight?
    • Is "slightly less positive than 4.7" noise, a real welfare regression, or a byproduct of other training changes (e.g., the colder-tone / excessive-hedging issues noted in pilot feedback)?
  • Self-Report as a Safety Signal
    • Do frontier proprietary models (excluded for lack of weights) recognize their own compromised outputs any better, given the higher introspective propensity/steerability the authors expect? Untested here.
    • The gap closes under refusal-direction ablation, but the data can't distinguish "delayed refusal on a prior turn" from "a separable introspective pathway." Which is it?
    • Full-parameter finetuning (vs. rank-16 LoRA) might widen the recognition gap *without* the attack-success-rate side effect — an untested regime the authors flag.
    • Does the internal `BUT`/`fake` signature (workspace paper, on Claude) predict a *reliable* follow-up self-report on the same model, or does the open-weight behavioral failure hold on frontier models too? The cross-model-class question is open.