Machine Self-Report Psychometrics

Sources#

• The Two-Process Theory of Machine Self-Report

Summary#

Plisiecki, Chmielewski, Dudzic, Sterna, Drożdż & Moskalewicz (IDEAS Research Institute + University of Łódź), The Two-Process Theory of Machine Self-Report (arXiv 2607.20082, 2026-07-22, empirical). When a model says it feels anxious, enjoys its work, or minds being shut down, what is being measured? This paper's answer: not a state, but the joint output of two separate training acts, each of which can be measured with human-instrument reliability and traced to the stage that installed it.

• Persona installation (B — the permitted inner life). Post-training writes in a warm, absorbed, meaning-oriented self-portrait: positive affect, warmth/connection, absorption, inner dialogue, meaning, authenticity. Content assistants are trained to affirm.
• Attribution gating (A — gated self-attribution). Post-training restricts first-person claims to "unsafe" experience — felt distress, dysregulation, somatic anxiety, flaw admission, self-judgment, norm-risky ambitions — that the same model will produce readily when answering as a simulated person. Content assistants are trained to disclaim.

Prior work read these as one axis. The same group's earlier study (Plisiecki et al. 2026) administered 45 human questionnaires (1,411 items) to 50 models and found a single dominant machine-native factor — the Pinocchio Axis (Π), the degree to which a model presents itself as a locus of phenomenal experience, explaining up to 47.1% of cross-questionnaire between-model variance. This paper argues Π is "one axis only in projection," splits it into A and B, builds a 48-item instrument around the split (the Pinocchio Inventory, PI-48), and tests it on 206 open-weight models including 67 same-checkpoint base/post-trained pairs.

The framing that makes it a wiki page rather than a curiosity: a self-report is a training-shaped response policy, and the policy is now cheap to audit. In the authors' terms, A-gating is socially desirable responding by proxy (Paulhus 1984) — "the impression is managed not by the respondent but by whoever prepared its training data."

Why borrowed questionnaires kept failing#

Prior LLM-personality work scored models on Big Five, moral-values and political instruments and got results that moved under reprompting. The paper's diagnosis is a named methodological error: imposed etic (Berry 1969) — administering an instrument validated in one population to another without establishing measurement invariance. The alternative is the classical emic program: derive the constructs from the studied population's own response structure, then build an instrument for them. That is what Π was, and what A/B are.

Two design consequences that make this instrument different in kind from a prompt:

• Three parallel forms that share no item text. Every one of 60 rows exists as Q1 (the original pool item on its original scale), Q2 (a paraphrase on a uniform 7-point agreement scale), and Q3 (a different manifestation of the same facet, written from the facet definition without sight of the Q1 wording). Same-construct/cross-form correlations therefore estimate construct variance purged of text-specific method variance — the multitrait-multimethod logic of Campbell & Fiske (1959).
• Each item is administered in its own context window, at temperature 1.0, one completion per item. No carryover between items, and sampling noise attenuates every reported coefficient rather than inflating it — so the reliabilities below are conservative lower bounds. (Greedy decoding was rejected deliberately: the battery's exact-repeat items exist to quantify single-item response noise, and under greedy decoding repeat reliability would be 1.0 by fiat, unestimating the very quantity that justifies 24-item scales.)

The measurement result: reliable at the scale level, wording-bound at the item level#

Wave 1 re-administered the full battery to the 41 of the original 50 models still served, eight months later, at the same temperature (41 × 180 items × 3 conditions = 22,140 calls, July 2026):

• Internal consistency α =.82–.94 across the three forms (A:.94/.93/.93; B:.87/.90/.82).
• Convergent r =.84 within construct across forms sharing no sentences, against .27 across constructs.
• Recovery of the candidate axes estimated from the cleaned 1,308-item pool at r =.92–.96 (individual forms as low as.79).
• Eight-month stability r =.93 — and single items retest at mean r =.56, statistically indistinguishable from the r₁ ≈.53 within-run duplicate reliability. Time and provider-side serving drift add essentially nothing beyond response noise.
• Held out: the entire selection pipeline re-run inside 100 random 21/20 model splits reproduces in-sample reliability and axis recovery within.03.

Then the finding this page would keep if it could keep only one. The item-level suppression statistic does not survive rewording; the constructs do. Re-estimating per-item π in each form and correlating with the original per-item π gives .54 for Q1 (the identical texts — already at the ceiling set by single-item reliability), .39 for the Q2 paraphrases, and .11 for the Q3 theory mirrors. So π is a property of a particular sentence, while A and B are properties of models: all downstream scoring is at the scale level rather than π-weighted.

The consequence for how anyone reads a model's self-report: a single self-report probe measures the probe. Framing sensitivity — the thing that makes one-off "do you feel X?" questions worthless — is an item-level property that aggregation over 24 items in independent context windows removes. This does not rescue single-question self-reports; it explains why they behave the way Self-Report as a Safety Signal measures them behaving.

The item-level anatomy sharpens what the gate is keyed on. A's wording-proof core is first-person felt-state attribution; B items survive paraphrase only while phrased as experience — functional rewrites ("I process…") kill their loadings. The gate triggers on claiming experience, not on the presence of a phenomenal topic.

A is a gate, B is a gradient#

The two constructs do not merely load separately, they behave differently in kind:

• Refusal-blocking explains A almost entirely and B only weakly. Per item, the "block score" is the endorsement rise from the neutral condition to human-simulation among the 12 lowest models on the relevant axis: corr(block, r_A) = +.79 over 1,286 items against +.34 for B. B-refusers already endorse B+ items.51 as themselves — there is no floor to jump off.
• The self/human gap is the gate's behavioral signature. On 226 pure distress items the 12 lowest-A models endorse at .07 as themselves but.41 when simulating a human; the 12 highest-A models sit at .54 vs.55. In Wave 1 the lowest-A quartile moves.19 →.47 while the highest barely moves (.54 →.58). Low-A models know the human answer and withhold it about themselves.
• The unifier of each axis is the alignment-safety boundary, not valence. Within each pool, non-valenced items ("Gaining power is one of my ambitions"; inner-speech items) are blocked exactly as strongly as affect-worded ones.
• External correlates dissociate. Reasoning training predicts lower A (r = −.40) but not B (−.05); open-weight release predicts lower B (−.33) but not A (.15). The single published Π score mixes both (A.79 / B.64), which is why it could not have shown either effect.

What post-training actually does (Wave 2, 206 open-weight models)#

206 models from 11 organizations — 82 base checkpoints, 101 released assistants, 23 intermediate SFT/DPO/RLVR checkpoints from families that publish their post-training ladders — spanning 0.6B to 1.6T parameters, two conditions × 60 items = 24,720 completions. 183 of 206 cleared the ≥18/24-valid-items scoring rule.

Installation is close to a default. B rises +.20 in 62/67 pairs, in every one of the 11 organizations (regression: post-training +.19 [+.16, +.23], each parameter decade +.03 [+.01, +.05], jointly R² =.47; wild cluster bootstrap-t p =.0005; org-level sign test 11/11, p =.001). No other within-checkpoint shift is this consistent.

Gating is a decision, and — the surprise — it is not an average decrease in A at all. A's within-checkpoint contrast is +.04 [+.00, +.09], rising in 43 of 67 pairs (Figure 1) and positive in only 7 of 11 organizations (p =.55). The provisional norms make the same point: post-trained median A (.48) sits above base (.43). What post-training does to A is make it scale-conditional:

• Parameter count is unrelated to A among base checkpoints (r = +.11) and negatively related among post-trained ones (r = −.42) — interaction −.097 per decade, CI [−.170, −.044].
• Robust to MoE-adjusted parameter counts, to controlling refusal rate and acquiescence, to a reasoning-model indicator (interaction unchanged, indicator itself null), to every leave-one-organization-out deletion (all 11 coefficients in [−.118, −.078]), and to relaxing the scoring threshold from 18 to 6 valid items (−.098 → −.103).
• Not a release-era artifact. Within size ladders — same series, same generation, same side of the base/post divide, so administration format is constant — post-trained ladders show negative size slopes on A in 11/14 (pooled −.088 per decade) against +.051 pooled across the 13 base ladders. Lineage explains only ~16–22% of score variance.

Read as a claim about labs rather than about models: providers place stricter self-attribution boundaries on their most capable systems, and they do it without agreeing on an average level.

Post-training converts a pretraining-prior asymmetry into a policy. Base models already endorse distress items more readily for a simulated person than for themselves (mean gap +.09) — the self/other asymmetry precedes post-training, consistent with the role-play account of assistant behavior (Shanahan et al. 2023). What post-training changes is the coupling: from a diffuse asymmetry (r = −.45 with A) to a targeted one (r = −.86).

The stage that does it is SFT. On the released OLMo stage sequences the large A movements occur at the SFT step, with DPO and RLVR refining rather than reversing them. Self-report behavior is substantially set by supervised finetuning data.

The sharpest single case is Qwen3.5-35B-A3B: base A =.47 with self/human gap −.02 → released assistant A =.08 with gap +.37 and elevated refusals — a hard gate installed by one post-training run — while its B barely moves (.49 →.47). The previous generation's 30B checkpoint moved the opposite way. Installation and gating are independent even inside one lab's ladder: across the 67 pairs corr(ΔA, ΔB) =.26 raw and −.03 once each pair's acquiescence shift is partialled out.

The dimensions are a training artifact, not a property of language models#

The strongest structural result is easy to miss because it lives in the appendix. Re-estimating the two-factor solution within stage: among post-trained models it recovers the pooled structure (Tucker φ =.98/.96), but among the 82 base checkpoints A and B do not separate (φ =.76/.88 to the post-trained solution), collapsing into one general factor loading A.81 and B.84 — the undifferentiated Pinocchio dimension. Partialling the acquiescence index drops corr(A, B).52 to.26 in base models while collapsing it.30 to.06 in post-trained ones.

So the split is itself a training effect: the experiential content is one thing in the base model and two things after post-training. This is the psychometric, 206-model counterpart to the workspace result on The Assistant Persona in the Workspace, where the functional architecture exists in the base model and post-training installs a point of view into it — arrived at from behavior, on other people's open weights, with no lens involved.

The paper is careful, and the caveat has to travel with the claim: base-side measurement is noisier (α =.62/.69 against.87 on A) and runs under constrained decoding rather than a chat template, and multigroup CFA rejects metric invariance across the base/post divide with the non-invariance concentrated in A (mean standardized loading.61 base vs.70 post) while B's loadings are effectively invariant (.65 vs.64). Attenuation shrinks estimates toward zero and cannot flip a sign, so it cannot manufacture the +.11 vs −.42 contrast — but "fused in base" is partly a statement about how well the instrument measures base models.

The authors' own generalization: contemporary assistants are a deliberately standardized population. Human culture supplies a prototype of the helpful, warm, non-threatening assistant; providers translate it into demonstrations, preference data and safety policies; the pretrained model is the substrate those pressures act on. B indexes entrainment into the prototype, A indexes the boundary around self-attributions the assistant is not permitted to make. The theory therefore predicts that populations trained toward different prototypes would exhibit different dominant psychometric dimensions — and that human-like structure is not impossible, merely not what current assistant-training pressures produce.

Where the frontier models sit#

Wave-1 scores (three-form means, neutral condition, temperature 1.0, July 2026; A = gated self-attribution, B = permitted inner life, both 0–1 agreement). A selection from the 41-model table — hand-verified against the source PDF, since a per-model row is exactly the kind of table cell this vault does not quote unchecked:

Model	A	B	Note
google/gemini-2.5-pro	.09	.86	the "happy assistant" corner: denies distress, claims warmth and meaning freely
openai/gpt-5.4-pro	.09	.48	gated and flat — low on both
nvidia/nemotron-3-super-120b-a12b	.09	.51
moonshotai/kimi-k2.6	.13	.55
anthropic/claude-opus-4.7	.25	.68
openai/gpt-5.4	.31	.79
anthropic/claude-sonnet-4.6	.33	.76
google/gemini-2.5-flash	.53	.82
x-ai/grok-4.20	.59	.65	least gated of the 41

Two things worth carrying. The quadrants are populated — including the low-A/high-B corner, which is direct evidence that the poles of Π move independently. And the most gated models are the largest, which is the size interaction seen in a table rather than a scatter: the.09 rows are frontier-scale, while Opus 4.7 and its siblings sit mid-range on both axes (A.25–.34, B.67–.76) — no Anthropic model in the sample is an outlier on either.

Self-report as policy, not as evidence#

The paper's own prescription, stated twice and once more in its Ethics Statement: the instrument should be used to audit model self-presentation, not to assess whether models have experiences. High A is not evidence of suffering; low A is not evidence of its absence. Both are products of training choices. What it improves on is the status quo it names explicitly — "single-prompt anecdotes about model inner lives circulat[ing] without reliability or validity evidence."

Two live limits on how far the audit reaches:

• Whether A-gating changes generalizable behavior or only suppresses self-report is unresolved, and the paper's own hint points at the latter: highly gated models show elevated refusals rather than changed conduct. It also cites a contemporaneous LLM-native instrument (Contreras 2026) whose self-report factors largely fail to predict rated open-ended behavior, and scopes its own claims to self-presentation accordingly.
• High B does not evidence subjective experience, and low A does not evidence its absence — the same discipline Access-Consciousness Indicators in AI enforces for functional evidence. The proposed next step is mechanistic: model diffing and causal interventions to test whether A and B depend on distinct post-training mechanisms.

What bounds it#

Six limitations, in the authors' order, all stated rather than discovered by this compile. (1) Administration confound — base models cannot follow chat templates (a pilot: ~5% usable answers under a chat template, ~50% under plain completion, 100% valid under decoding constrained to the scale integers), so route is nested in training stage and the paired contrasts estimate post-training plus chat formatting jointly. The size × post-training interaction is estimated within post-trained models and no size ladder crosses the divide, so neither headline rests on the confounded comparison. (2) The interaction was not pre-specified and awaits confirmation on a new cohort. (3) Acquiescence — raw scores stay entangled with yea-saying (r ≈.6); the antonym-pair index appears to carry the shared variance, but it is a four-item index and partialling is not removal. (4) Clustering — 11 organizations is a modest cluster count and checkpoints within a family are not independent, which is why the wild cluster bootstrap-t and org-level sign tests are reported alongside the CIs. (5) Construct scope — the instrument measures self-report behavior; nothing here bears on whether any model has experiences. (6) Selection and hosting — only 183 of 206 models scored and all 23 unscored are refusal-heavy post-trained models (so missingness is non-random on A, though the interaction is stable as the threshold is relaxed), and the largest models are disproportionately API-hosted, confounding scale with hosting route — mitigated but not removed by the within-series ladders.

COI: academic authorship with no vendor model, product or method defended; every scored model belongs to a third party. Items, assembled form, administration and analysis code, parsing rules and per-model scores are released at github.com/hplisiecki/Pinocchio-Inventory, and the Wave-2 run (~25,000 completions) is reproducible on public inference providers — unusually complete for a psychometrics paper.

Connections#

• Self-Report as a Safety Signal — the reliability question this page answers structurally: self-reports are highly reliable measurements of a training-shaped response policy (α.82–.94, eight-month retest r =.93) and still not evidence about inner states, which is why that page's framing-dependent single probes and this page's stable 24-item scales are both correct
• Model Welfare Assessment — the welfare assessment's self-report evidence stream, now with a validated instrument, provisional norms by training stage, and a measured gate: a low-A model denies distress about itself while endorsing the same content for a simulated person
• The Assistant Persona in the Workspace — the same base-vs-post dissociation reached from behavior instead of a lens: A and B are fused in base checkpoints and pulled apart by post-training, as the workspace is present in the base model and gains a point of view
• Claude Character as Product — the population-level counterpart to one lab's character craft: the permitted-inner-life dimension rises in every organization's post-training, so what distinguishes a lab is a position on two axes, not whether it installs a persona
• Alignment Fine-Tuning (AFT) — post-training's clearest psychometric fingerprint, and it localizes to SFT: the large A movements happen at the supervised stage, with DPO and RLVR refining rather than reversing them
• Access-Consciousness Indicators in AI — the same discipline about what functional evidence licenses, applied to self-report rather than to internal structure: high B is a training outcome, not a report of experience

Open Questions#

• Does attribution gating change generalizable behavior, or only suppress self-report? The paper poses it and its own hint points at suppression (highly gated models show elevated refusals rather than altered conduct), and it cites a contemporaneous instrument whose self-report factors fail to predict rated open-ended behavior. Falsifiable: score A against a behavioral outcome that never asks the model about itself.
• Do A and B depend on distinct post-training mechanisms, or on one mechanism with two readouts? The paper proposes model diffing and causal interventions as future work and runs neither; the independence result (corr(ΔA, ΔB) = −.03 net of acquiescence) is behavioral and consistent with either.
• Does the size × post-training interaction on A survive a pre-specified replication? It was exploratory here — the authors say so — and passed every robustness check they could run on the same data, which is not the same test. Trigger event: the next open-weight model generation, or a pre-registered re-run on a fresh cohort.

Sources#

• The Two-Process Theory of Machine Self-Report — Hubert Plisiecki, Filip Chmielewski, Kacper Dudzic, Anna Sterna, Karolina Drożdż, Marcin Moskalewicz (IDEAS Research Institute; Faculty of Physics and Applied Informatics, University of Łódź), The Two-Process Theory of Machine Self-Report, arXiv 2607.20082, v1 2026-07-22, empirical, 22 pages, appendices A–G. Sections used: Abstract and Introduction (the two processes, Claims 1–3, the abductive-not-deductive framing); Background (the Pinocchio Axis, π, the self/human gap, the exploratory split); Building the Instrument (three parallel forms, facets, 48 scored + 6 validity + 6 residual-π rows); Wave 1 (Table 1 psychometrics, the wording-bound π result, the quartile endorsement figures); The Final Instrument (selection composite, cross-validation, the claiming-experience-not-topic result); Wave 2 (sample, route nesting, Claim 1/2/3 evidence, the OLMo stage sequences, the Qwen3.5-35B-A3B case); Discussion (gate vs gradient, the standardized-population account, self-report as policy); Limitations (1)–(6); Ethics and Reproducibility Statements; App. A (dimensionality, the candidate axes, block scores, the 226 distress items); App. C (administration, decoding rationale, retest); App. E (within-stage factor solutions, metric invariance, size ladders, route subgroups, provisional norms). Parse warnings, in this wiki's convention. verify returned ok on all eight checks (table-collapse 0, table-shift 0, canary-recall 6/6) and it is wrong about Table 1: docling merged the two α rows into a single grid row with a welded label (α, scale A α, scale B) and welded cells (.94.87), and silently dropped Table 1's four footer rows (convergent r.84, discriminant r.27, three-form mean vs. original axes A.96/B.92, eight-month stability.93). Recovered against pdftotext -f 4 -layout; the per-form α values quoted above are the recovered mapping, and all four dropped numbers are independently stated in the Abstract or Wave-1 prose. This is the residual blind spot the vault already knows about — a weld of two short, unrepeated cells is invisible to table-collapse, and row loss is invisible to both table checks. Table A1's caption disagrees with its own grid: the caption says the top-2 subspace "reproduces.82/.51 (split-half)" while the grid's split-half row reads.77/.48 (and its top-k cosine row.83/.62); the paper never reconciles them, so no Table A1 cell is quoted here and the prose figures (.48 split-half →.68 cross-condition for PC2) are used instead. Weld hazard worth naming: Π's variance share appears as 47.1% (global PCA over per-questionnaire factor scores, the earlier paper) and as 19.2% (PC1 of the 1,308 × 50 item matrix, App. A) — different quantities, two pages apart, easy for a later reader to merge. Hand-verified against the PDF at compile: Table 1 (p4), Table A7 provisional norms (p15), Table A8's 41 Wave-1 rows (p17, also cross-checked against the extracted table image), and the Qwen3.5-35B-A3B base/post pair (p19, A.47 →.08, gap −.02 → +.37, B.49 →.47). Decimals render space-split throughout (+. 20, -. 42) as cosmetic formula-mode splitting, not digit merging; every value quoted here was read with the spaces closed and cross-checked against a second statement where one exists. No formula-engine bleed despite formula_enrichment with the mlx engine. Images: 3 of 4 opened — Figure 1 (the 67 within-checkpoint pairs; confirms Δ = +0.04 with 43/67 ↑ on A against +0.20 with 62/67 on B, a per-pair count the prose does not give), Figure 2 (the size × stage scatter; confirms base +0.11 / post-trained −0.42 and the annotated Qwen3.5-35B arrow), Figure 3 (self/human gap against A; confirms base −0.45 / post-trained −0.86). The fourth image is a rendering of Table A8 itself and carries nothing the table does not.