Recent work from Anthropic showed that Claude Sonnet 4.5 encodes 171 emotion concepts as linear directions in its internal representations, with geometry mirroring human psychological structure (Sofroniew et. al, 2026). We replicated this in two open-weight models: Apertus-8B-Instruct-2509 and Gemma-4-E4B-it, and found two things worth flagging. First, both models recover valence geometry with correlations approaching Claude's (r ≈ 0.75–0.83 vs. Anthropic's r = 0.81), but they do so via strikingly different developmental trajectories across layers. Second, the corpus of stories used to probe the model appears to bias what geometry is recoverable. Arousal encoding is much stronger when probing with Gemma-generated stories than Apertus-generated stories, in both models. The choice of probing corpus does not appear to be a neutral methodological detail.

1. Background and motivation

Large language models sometimes appear to respond emotionally, such as expressing frustration when stuck, enthusiasm when helping. It's tempting to dismiss this as surface behaviour, but recent work by Sofroniew et al. (2026) suggests something more is going on internally. They identified what they call emotion vectors in Claude Sonnet 4.5: linear directions in activation space corresponding to 171 emotion concepts, which causally influence model behaviour when steered. Notably, the geometry of this emotion space mirrored human psychology, as they used human-evaluated ground truth. The first two principal components aligned with valence (positive–negative) and arousal (high–low intensity), the two axes psychologists since Russell (1980) have used to organize affective experience.

This is intriguing but raises immediate questions about generality. Is this a quirk of Claude's training, or a general property of instruction-tuned language models? How does this geometry develop across the depths of a network; does it emerge suddenly or gradually? And to what extent does the way you extract these vectors shape what you find?

Answering these questions matters beyond curiosity. If emotion representations are general and robustly extractable, then monitoring them could provide early signals of misaligned internal states across different deployed models. If they're fragile or extraction-dependent, interpretability tooling needs to account for that. We took the open-weight angle to test this. Two recently released models, both fully open: Apertus-8B-Instruct-2509, a 32-layer transformer with published weights, training data, and code, and Gemma-4-E4B-it, a 42-layer transformer. For each, we extracted emotion contrast vectors at multiple layers, using two story corpora (one generated by Apertus, one by Gemma), allowing us to disentangle what's intrinsic to the model from what depends on the probing data.

2. How we extracted emotion vectors

The core technique, following Sofroniew et al., is straightforward. For each of 171 emotion concepts (drawn from the NRC VAD lexicon, (Mohammad, 2018)), we generated 9 short stories depicting a character experiencing that emotion without naming it, plus 40 general emotionally neutral control stories in total. We ran all these through the model and grabbed the residual stream activation at the final token of each story.

Following Sofroniew et al. (2026), we construct one activation vector ${\mathbf{v}}_e^{(l)}$ per emotion $e$ and layer $l$. Since these vectors capture general linguistic structure, we apply a two-step procedure to isolate the emotion-specific component. The first step averages the activations of a forward pass of the 9 emotion stories, yielding one raw vector ${\mathbf{u}}_e\in {\mathbb{R}}^{d_{\text{model}}}$ per emotion and layer. But this still mixes emotion-specific and general linguistic features together.

In the second step, we remove the non-emotion-specific components. A subspace is constructed from the mean residual activations per layer. To isolate the emotion-specific component, we subtract from each emotion vector its projection onto the neutral subspace to get the contrast vector ${\mathbf{v}}_e$: ${\mathbf{v}}_e={\mathbf{u}}_e-{\sum }_{k=1}^K({\mathbf{u}}_e\cdot {\mathbf{p}}_k){\mathbf{p}}_k$

The intuition: subtracting the neutral baseline removes whatever is generic about "being a story”, such as narrative structure, sentence-level statistics, the residual stream's accumulating-norm tendencies, and isolates what's specific to the emotion concept. Stack these vectors across all 171 emotions and you get a matrix $V^{(l)}$ ∈ ℝ ${}^{(171\times d)}$ at each layer.

To check whether this matrix has human-interpretable structure, we apply PCA (Principal Component Analysis) and ask: does PC1 correlate with NRC valence ratings? Does PC2 correlate with arousal? PCA is unsupervised, it doesn't know what valence is, so if the dominant axis of variation in emotion space lines up with human valence judgments, that's evidence the model has spontaneously organised emotions along the same primary axis humans use.

Important note: PCs are orthogonal by construction, but valence and arousal in humans aren't perfectly orthogonal. Treating PC1 as "the valence direction" is an approximation, not the same thing as fitting a regression from emotion vectors to ground-truth valence. We come back to this in the discussion.

3. What we found

1. Valence replicates, robustly

Both models recover valence geometry, as can be seen in Figure 1. Apertus-8B-Instruct-2509 peaks at r = 0.76 around layer 31; Gemma-4-E4B-it reaches r = 0.83 at layer 16. These approach the r = 0.81 that Anthropic reported for Claude Sonnet 4.5. The first principal component of the emotion contrast matrix is, in both models, recognizably valence — positive and negative emotions separate along it. The headline here is reassuring: emotion vectors aren't a Claude-specific artifact. The basic finding generalizes across architectures and training pipelines.

2. But the developmental trajectories diverge

The interesting part is how each model gets there. Figure 1 shows the PC1–valence correlation across fractional layer depth for both models on both corpora.

Apertus-8B-Instruct-2509 shows progressive emergence: valence correlation is near zero at layer 12, rises steadily through the middle layers, and stabilizes around r ≈ 0.7 from layer 22 onward. By the time you reach the late layers, valence is robustly encoded. Gemma-4-E4B-it shows early peaking and collapse: valence correlation is r = 0.83 at layer 16 (the earliest layer we examined) — and then largely dissolves. By layer 18 it's near zero, and it stays near zero through layer 40.

This is one of those patterns that's easy to miss in a tabular results dump but jumps out visually. Two models trained differently, arriving at superficially similar peak correlations, via completely different developmental paths.

A natural follow-up question is whether this reflects a difference in the underlying geometry or just in which axes happen to correlate with valence at which depth. We checked this with cross-layer CKA (Centered Kernel Alignment, (Kornblith, 2019)) , a similarity measure between representation matrices that's invariant to rotation. Apertus shows a clear phase transition: CKA between early and late layers is low (~0.43), but CKA between all late-layer pairs is high (≥0.66). Gemma's geometry is globally stable on Apertus-stories (all layer pairs ≥0.76), with no comparable transition. So Apertus reorganizes its emotion representations mid-network in a way Gemma doesn't.

3. Arousal encoding is corpus-dependent

This is the finding that surprised us most (see Figure 2). Arousal correlations (PC2 vs. NRC arousal ratings) are weak in both models when we extract vectors from Apertus-generated stories — peak r ≈ 0.17 for Apertus-8B-Instruct-2509, r ≈ 0.21 for Gemma-4-E4B-it. Not great, well below the r = 0.66 Anthropic reported. But when we extract from Gemma-generated stories, arousal correlations more than double in both models: r = 0.45 for Apertus-8B-Instruct-2509, r = 0.43 for Gemma-4-E4B-it.

The key word is both. If this were a model-corpus matching effect — each model encoding its own stories better — we'd expect Apertus to perform better on Apertus stories and Gemma to perform better on Gemma stories. Instead, Gemma-generated stories produce stronger arousal extraction regardless of which model is being probed. The most likely explanation is that Gemma stories contain more arousal-discriminative linguistic content — more variation in narrative intensity, more physiological cues, wider arousal vocabulary range. The probing corpus matters for what geometry you can recover, in a way that isn't reducible to model-data matching. This has a methodological implication: when you extract emotion (or any concept) vectors via story-based contrast, the corpus generation model is not a free parameter. Different models produce stories that surface different aspects of the underlying concept space. Two researchers running "the same" extraction protocol with different story-generation models can end up with different conclusions about what geometry exists.

4. PCA for Both Models

Additionally, we created a 2D visualisation with PC1 and PC2, with all emotions plotted. PCA projections at each model's layers show emotion clustering and a clear corpus effect. PC1–valence correlations are similar across story conditions, but clusters are more clearly separated for Gemma stories, with positive and negative emotions forming denser groups. You can view the PCA for Apertus-8B-Instruct-2509 in Figure 3 and for Gemma-4-E4B-it in Figure 4.

5. Discussion: what to take from this

Similar geometry, different paths. The fact that both models converge on comparable peak valence correlations via opposite developmental trajectories suggests valence alignment with psychological structure is more about training than architecture. It also has a practical implication for downstream work: layer selection matters. A steering intervention validated at Apertus' layer 31 won't necessarily transfer to Gemma's layer 31: you'd want layer 16. Translating interpretability findings across models requires matching the functional layer, not the numerical layer.

**Concept directions can be unstable across depth. **Even within a single model, the direction most correlated with valence at one layer isn't necessarily the same direction at another layer. Cosine similarity between layerwise valence directions drops at non-adjacent layers, even when CKA shows the overall geometry is stable. This suggests valence may live in a stable subspace rather than along a single direction, and that "the valence direction" depends on where you look. This is a thread we're following in ongoing work.

**Methodology shapes findings. **The corpus-dependence result is, to us, the most actionable finding. It means published "emotion geometry" results are partially a property of the extraction pipeline, not just the model. Reproducible interpretability requires explicit attention to probing data, not just probing methods.

6. Where this is heading?

A few directions we're working on or thinking about:

• Causal validation. Geometric findings are suggestive but not behavioral. The natural next step is to actually steerusing these vectors and check whether behavioral effects match what the geometry predicts. Sofroniew et al. showed this for Claude; we want to know whether steerability tracks the layerwise correlation patterns we found, and whether the corpus effect on extraction translates into a corpus effect on steering reliability.
• **Why does the corpus matter? **The hypothesis is that Gemma stories carry more arousal-discriminative lexical content. A direct test: compute per-story arousal scores using the NRC VAD lexicon, and see whether Gemma stories have higher variance in arousal vocabulary across emotions. If yes, the corpus effect reduces to a measurable property of the text. If no, something more interesting is happening at the model level.
• Stable subspaces vs. rotating directions. We're investigating whether valence lives in a low-dimensional subspace that's preserved across layers, with the specific 1D direction rotating within it. Tools like principal angles between subspaces are the natural way to test this. If the answer is "yes, stable subspace," then steering interventions should target the subspace, not a single direction.
• Cross-modal extension. Does emotion geometry persist across modalities? CLIP-style vision-language models would be the natural testbed — same valence/arousal axes, different input modality.

7. Code

You can check out the code in our GitHub Repository: https://github.com/sinievanderben/emotion_experiment

Both datasets are available on HuggingFace:

• https://huggingface.co/datasets/snae/emotion_stories_Apertus_8B_Instruct
• https://huggingface.co/datasets/snae/emotion_stories_gemma_4_4B