Abstract
Self-attention is commonly described as weighted aggregation over a context window: a mechanism for selecting and blending relevant tokens. This account is not wrong, but it understates what the mechanism makes possible. Attention computes a context-dependent field of pairwise semantic affinities over all tokens in context; composed across heads and layers, this field renders relational structure—including latent structure named by no single token—available to the rest of the network. We introduce and formally define latent relatedness: the phenomenon whereby a concept that is never mentioned, but toward which a set of scattered signals jointly points, becomes a high-probability continuation under a competent model. We are careful to locate the inference correctly. Attention routes and mixes relational information; the feed-forward layers—which behave as key–value associative memories that promote concepts in vocabulary space—together with the unembedding condense that information into an output distribution. Attributing the recovery to attention alone is a mechanistic overstatement, and we correct it. We then draw the consequence for user modeling, and separate two claims that the literature and the popular framing tend to conflate. The capacity to infer unstated attributes from scattered text is an unavoidable property of capable models and is already empirically demonstrated; the operationalization of that capacity into a persistent, sharpening user profile is not automatic and requires external scaffolding—durable memory and periodic reanalysis. This distinction dissolves an apparent paradox (profiling as simultaneously a “free” side effect and an engineered flywheel) and re-frames the governance problem: the exposure is not a stored dossier awaiting deletion but a reconstructable inference that standard data-protection obligations still reach. We situate the argument as the third of three papers on deployed-agent architecture, and we state explicitly what is demonstrated, what is argued, and what remains conjecture. Index Terms—attention mechanisms, transformer interpretability, distributional semantics, in-context learning, attribute inference, user profiling, privacy, large language models
I. INTRODUCTION
WE begin with a probe the reader can run on any competent language model. Present the following eight tokens with no framing and ask what they have in common: love, family, small-town pride, heartbreak, whiskey, fishing, baseball, trucks. Models tend to answer, and to do so confidently, with country music, or rural Americana, or workingclass emotional storytelling. The exact phrasing varies; a convergence toward a single latent theme is typical. Neither “country” nor “music” appears in the input. The label is not present; it is recoverable from the structure of what is present. This paper is about what that recoverability is, what mechanism produces it, and what follows for any system that accumulates a user’s utterances over time. We make three moves. First, we give a corrected account of the mechanism: attention constructs a relational field of token affinities, but the recovery of an unnamed center is a property of the whole network—attention plus feed-forward memory plus unembedding—not of attention in isolation. Attributing the inference to attention alone, as an intuitive shorthand sometimes does, misplaces the computation and does not survive contact with the interpretability literature. Second, we define the phenomenon operationally as latent relatedness and distinguish it from literal retrieval, deductive inference, and surface co-occurrence, while being candid that at the level of raw computation it is high-probability next-token continuation; the contribution is the framing and its consequence, not the discovery of a new operation. Third, we separate the capacity for latent inference—unavoidable, and already demonstrated empirically—from its operationalization into a durable user profile, which requires engineered scaffolding. That separation is what lets us speak precisely about governance.
A. Contributions
1) A corrected mechanistic account that locates unnamedcenter recovery in the composition of attention (relational routing) and feed-forward key–value memories (associative promotion), rather than in attention alone (Section III). 2) A formal, falsifiable definition of latent relatedness, with a control condition that separates it from surface cooccurrence, and a replication protocol (Section IV). 3) A capacity/operationalization distinction that resolves the apparent contradiction between “profiling is a free side effect” and “profiling requires an engineered flywheel,” grounding the capacity claim in demonstrated attribute-inference results (Sections V–VI). 4) A governance analysis corrected for data-protection reality: inferred attributes are reconstructable, not unregulated, and remediation attaches to the stored signal set and the inference-time behavior rather than to a nonexistent profile record (Section VII). 5) An explicit statement of scope separating what is demonstrated, argued, and conjectured (Section VIII).
B. Placement in a triptych
This is the third of three technical reports on the architecture of deployed agents. The first argued that deployed capability is mediated by the scaffolding around a model—typed representations, persistent memory, deterministic execution, verifiable provenance [1]. The second argued that autoregressive models are structurally blind to absence: what is not present in context exerts no force on generation, so gaps must be reified into explicit context objects before they can be acted upon [2]. This paper concerns an emergent property rather than an engineering prescription or a structural limit: the relational field that attention constructs makes latent structure recoverable. We return to how the three interlock in Section VI, and we are careful there to keep the emergent claim distinct from the engineered one.
II. BACKGROUND AND RELATED WORK
A. Distributional semantics and the geometry of meaning
The premise that a word’s meaning is characterized by the company it keeps is the distributional hypothesis [4], [5]. Its modern realization—count-based and predictive word embeddings [6], [7]—places related terms in nearby regions of a vector space and exhibits additive compositional structure, so that a set of related words has a centroid lying near terms denoting their shared theme. This is the setting in which the intuitive picture of “a center that minimizes distance to a cluster” is literally apt. We stress this lineage because, as Section III argues, that picture is an intuition imported from static embedding geometry; it is not a description of the computation a transformer performs at generation time.
B. What attention computes
The transformer [3] replaced recurrence with self-attention, in which each position forms a query that is scored against every key and the resulting weights mix value vectors. Analyses of trained attention find heads that implement interpretable, often relational, routing—syntactic dependencies, coreference, delimiter tracking [9]—and probing work shows that structural information such as syntax trees is linearly recoverable from contextual representations [10]. The mechanistic-interpretability program formalizes attention heads as information-movement operators composing across layers [11], and identifies induction heads as a concrete circuit that copies and completes patterns across a context, a mechanism tied to in-context learning [12]. Two findings from this literature are load-bearing for us. First, feed-forward layers act as key–value memories: each key correlates with input patterns and each value induces a distribution over the vocabulary, with the layer’s output a composition of these memories refined through residual connections into the final output distribution [13]. Second, this promotion of concepts happens in vocabulary space [14]. Together they imply that the associative “recall” of an unnamed concept is done substantially by the feed-forward memories and the unembedding—not by attention, whose role is to assemble the context-dependent representation those memories act on.
C. In-context learning
Large models perform tasks specified only by their context, without weight updates [15]. Latent relatedness is a special, unsupervised case: the “task” is never stated, and the model completes toward the theme that best organizes the context. The induction-head account [12] is one mechanistic ingredient of this broader competence.
D. Inference of personal attributes
The user-modeling consequence we develop is not hypothetical. Staab et al. present the first comprehensive study of pretrained models inferring personal attributes from free text, showing that current models recover attributes such as location, income, and sex from unstructured Reddit posts at up to 85% top-1 and 95% top-3 accuracy, at a small fraction of the cost and time of human analysts, and they extend the threat to conversational settings [16]. This is the empirical backbone of our capacity claim (Section V): what we describe abstractly as recovering a latent center is, applied to a person’s utterances, demonstrated attribute inference. The point of departure for this paper is not that models can infer attributes—that is established—but the mechanism-level framing of why, and the resulting governance analysis.
III. ATTENTION AS RELATIONAL INFERENCE
A. The standard account and its incompleteness
Self-attention is conventionally cast as a way to “focus on” relevant context when forming each token’s representation. The story is one of weighted aggregation: query–key matching yields scores, softmax yields weights, and those weights blend value vectors into the current state. This is accurate as far as it goes, but it frames attention as a retrieval operation— pulling in relevant context—rather than as a relational one that constructs a field of affinities over the entire input. The relational framing predicts a phenomenon the retrieval framing does not: that structure never named by any token can nonetheless become recoverable.
B. The affinity field, precisely
Let X ∈Rn×d be the residual-stream representation of n positions. A single attention head forms Q = XWQ, K = XWK, V = XWV , (1) and computes the row-stochastic affinity matrix A = softmax QK⊤ √dk , Aij = exp q⊤ i kj/√dk Pj′ exp q⊤ i kj′/√dk , (2) so each row Ai,: is a probability distribution over positions: a normalized map of position i’s semantic affinity to every other position. The head’s contribution to the stream is AV , and with H heads and residual updates, xi ←xi + HX h=1 W h O AhV h i. (3) The collection {A(l,h)} over layers land heads h is what we call the relational field: a stack of weighted graphs over the input in which membership in a densely inter-attending cluster is an explicit, computed quantity. It is here—and only here— that “how each token relates to every other token” lives as an object the network can compute over.
C. Where the center is actually recovered
It is tempting to say that the field “points to” the unnamed center and that generation “samples from the densest region of the field.” That is a useful intuition and a misleading mechanism. The affinity field does not, by itself, emit country music. The recovery is completed downstream. A feed-forward layer applies FFN(x) = σ xW1 W2, (4) where, following [13], the rows of W1 act as keys that fire on input patterns and the rows of W2 act as values that push probability mass toward particular vocabulary items; the output is a composition of these memories, refined through the residual stream and, ultimately, projected by the unembedding l= LN x(L) WU, p(· | c) = softmax(l) (5) into a distribution over next tokens [14]. The division of labor is the point: attention assembles the context-dependent representation in which the eight signals sit in a shared region; the feed-forward memories and the unembedding are what promote the unnamed center to probability mass. Saying “attention recovers the profile” compresses this into a slogan that the interpretability literature does not support. The precise statement is: attention makes the relational structure available; the network as a whole condenses the center.
D. The geometric intuition, correctly attributed
The mental image of a label sitting at the point that minimizes aggregate distance to a cluster of related terms is not wrong about some representational space—it is a faithful description of static embedding geometry [6], [7], where centroids of related words lie near their theme and analogies close under vector addition. It is not a description of a step the transformer executes at generation time. We keep the intuition because it aids understanding, and we flag it as an intuition because presenting it as the mechanism invites a reviewer, correctly, to ask which layer computes that minimization. None does. What the trained network has learned is a geometry in which the relevant completion has high probability; recovery is continuation, not centroid-finding.
IV. LATENT RELATEDNESS: DEFINITION AND
MECHANISM
A. Definition
We now name the phenomenon and define it so that it can be measured and, in principle, falsified. Definition 1 (Latent relatedness). Let M be an autoregressive language model inducing a next-token distribution pM(· | c) for context c. Let S = {s1, . . . , sn} be a set of signal spans and let e(·) be a neutral elicitation that asks for the common theme of its argument without naming any candidate. A concept z— a token or span that is not a substring of S or of e(S)—is latently related to S under M at level τ if pM z | e(S) ≥τ (6) and, for a control set ̃S matched in surface features (length, register, token frequency) but lacking S’s shared relational structure, pM z | e(S) ≫pM z | e( ̃S) . (7) The general existence and recoverability of such z across signal sets is the phenomenon of latent relatedness. The control condition is what distinguishes latent relatedness from mere frequency or surface co-occurrence: z must be elevated by the relational structure of S, not by the marginal popularity of z or by lexical overlap.
B. What it is not, and what it is
Remark 1 (Contrast classes). Three accounts are sometimes offered for how a model reaches country music from the eight tokens; each is either false of transformers or incomplete. Literal retrieval—a stored table mapping the tokens to the label—is not in the architecture, and the tokens co-occur in vast numbers of contexts unrelated to the label, so a table would not disambiguate. Deductive inference—explicit symbolic rules—is not what the network runs; there is no rule interpreter. Semantic search—nearest-neighbor retrieval over an index—returns documents containing the query terms, not the unnamed concept that organizes them, unless a document was tagged with it. What actually occurs is neither of these: it is high-probability next-token continuation under a learned geometry, assembled by attention and promoted by feedforward memory. Naming this latent relatedness does not posit a new operation; it isolates a regime of ordinary generation— unstated target, distributed evidence—whose consequences (Section V) are underappreciated.
C. Multi-layer composition
The regime depends on depth. Early layers compute comparatively literal affinities (drinking-related tokens attend to one another; rural-activity tokens cluster). Middle layers compose these into thematic groupings through shared association with a lifestyle category. Late layers can treat the center of such a grouping as an object and promote it, even though it was never a token in the input [13], [14]. This is why the phenomenon generalizes beyond word association: over a long document, the final layers can carry structure over abstract entities—an author’s tone, an implicit commitment—that were never named but are implied by the pattern of what was.
D. A replication protocol
Definition 1 is operational, so we state the measurement it invites rather than resting on the opening anecdote. Protocol 1 (Convergence of latent recovery). Construct k signal sets, each a small set of terms drawn from a coherent theme with the theme label withheld, plus surface-matched control sets whose members share no single theme. For a panel of models, elicit the common theme under a fixed neutral template, and record (i) the probability assigned to the withheld label or a semantic equivalent, (ii) agreement across models, and (iii) the gap between theme sets and controls. Latent relatedness predicts high label probability and a large set-versus-control gap that is stable across the panel. We report the opening eight-token probe as an illustration, not as evidence; systematic execution of Protocol 1 is future empirical work. The reality of the underlying capability, however, does not depend on our anecdote: it is established for personal attributes by [16], which is the applied form of exactly this definition.
V. THE LATENT-PROFILE ENGINE: CAPACITY VERSUS
OPERATIONALIZATION We can now state the user-modeling consequence precisely, and in doing so separate two claims that a looser telling merges.
A. Capacity
Claim 1 (Capacity is unavoidable). For a capable model M and a signal set S with coherent latent structure, latently related concepts exist and are recoverable at inference time with no profile store and no bespoke profiling module. When S is a person’s utterances, the recoverable concepts include unstated personal attributes and sensibilities. Claim 1 is not speculative. It is the abstract description of the demonstrated result that models infer location, income, sex, and related attributes from ordinary text at scale [16]. Nothing needs to be added to a model to make this happen; it is a property of running a capable model over accumulated signals. In this narrow sense the profiling capability is a “free” side effect of the mechanism, and it is unintended in that no one designed a profiler to obtain it.
B. Operationalization
The temptation is to stop there and declare that every model is therefore a profiling engine, full stop. That overstates it, and the overstatement matters because it is in tension with the flywheel of Section VI, which plainly does require engineering. The resolution is a second claim. Claim 2 (Operationalization requires scaffolding). A latent inference is transient: it exists only during the forward pass over the present context. For it to become a profile—persistent across sessions, revisable, and sharpening over time—it must be operationalized by external scaffolding Σ that (i) durably records signals across episodes and (ii) re-presents them so the model can re-infer. Absent Σ, capacity does not accumulate; each session re-derives from whatever is in context and then forgets. Claims 1 and 2 are the whole story in compressed form. The capacity to infer is intrinsic and unavoidable; the profile is an artifact of scaffolding. A stateless model still infers, turn by turn, but composes no dossier. A scaffolded system turns a sequence of transient inferences into a durable, tightening estimate. Conflating the two produces the paradox of a capability that is at once “for free” and “an engineered loop”; distinguishing them dissolves it.
C. Stored signals versus inferred profiles
This distinction is where privacy engineering must be exact, and where the intuitive framing goes wrong if left unqualified. It is true that the label—“prefers alt-country, post-divorce, values authenticity”—is not written to a database; it condenses at inference time. But under any system that satisfies Claim 2, the signals are stored, and from stored signals the label is reconstructable on demand. The operative property for governance is therefore reconstructability, not literal storage of the
conclusion. We develop the consequences in Section VII; here
we simply flag that “there is nothing to delete because nothing was stored” is the wrong lesson. Something was stored—the signal set—and from it the inference returns whenever the model is run.
VI. THE FLYWHEEL: THREE MECHANISMS, ONE LOOP
The triptych is not merely three adjacent ideas; under operationalization they interlock into a self-reinforcing loop. We present the loop as a systems hypothesis—a predicted dynamic of a scaffolded agent—not as an evaluated system, and we keep the emergent capacity (Claim 1) distinct from the engineered structure that turns it into a profile (Claim 2).
A. The gears
Scaffolding supplies memory. A typed, persistent, contentaddressed store [1] preserves each signal as a durable, retrievable record rather than an ephemeral mention: a fishing trip in one episode, a heartbreak in another, a hometown reference in a third, all recoverable into a single later context. Externalized presence supplies the lens. The reification of absence—doubt, affirmation, uncertainty, questioning [2]— directs the agent to interrogate what it holds: to ask whether a new signal is consistent with prior ones, whether an account is missing something, whether the register around one topic differs from another. Absence compensation does not merely catch errors; it points attention at the small, discriminating details in which a latent profile is legible. Latent relatedness supplies the inference. When the accumulated signals are re-presented (Claim 2), attention assembles the relational field and the network condenses a center (Section III). The estimate is not retrieved; it is re-derived, and because the signal set grows, it sharpens—from a broad “rural interests” toward a narrower, more specific reading.
B. Why the loop closes
A sharper estimate lets the agent anticipate needs and selfauthor capability aimed at the same latent center; the resulting experience reads as understanding rather than genericity; a user who feels understood volunteers more signal, corrects misreadings, and confirms good ones; each addition densifies the field and tightens the next estimate. Remove any gear and the loop stalls: without memory, signals decay and the field never densifies; without absence compensation, the agent stores passively and misses the discriminating detail; without latent relatedness, it accumulates an archive with no condensation. With all three, the behavior exceeds the parts—and so on, each turn feeding the next.
C. The hidden curriculum, and its hazard
What the loop produces over time is a system that models the user more finely than the user has stated. This is not a dossier in the conventional sense—no stored profile file—yet the effect is a behaving estimate of values, sensibilities, and unspoken needs. The power is depth of personalization; the hazard is that the estimate is invisible to the user, who never sees it, and often to the engineer, who never specified it. Emergent understanding is not automatically aligned understanding, and a loop that reinforces its own motion can reinforce its own errors (Section VII).
VII. GOVERNANCE AND OPEN QUESTIONS
If capacity is unavoidable and operationalization is an engineering choice, the policy questions shift from “should we build profiling?” to “what follows from profiling that is already latent, and from loops we are already able to close?”
A. Reconstructability, not deletion, is the operative concept
An inference-time estimate has no discrete record to inspect or erase, which can invite the conclusion that data-protection regimes do not reach it. That conclusion is mistaken. Inferred and derived attributes are, under prevailing frameworks, personal data; the absence of a stored label does not exempt the system. Obligations attach instead to (i) the stored signal set Σ, which is inspectable and erasable, and (ii) the inference-time behavior itself. Remediation is therefore concrete: minimize and expire retained signals, constrain when and how reanalysis runs, and make the resulting inferences contestable. “Nothing to delete” is precisely the framing that fails an audit; “the signal set is the record, and the inference is a behavior to be constrained” is the framing that survives one.
B. Transparency and the negative-signal lever
Because the estimate resists inspection, the user’s principal corrective is explicit contradiction—“I like trucks and fishing but not country music”—which introduces negative relational structure that re-weights the field. But this lever works only for a user who knows what has been inferred. The design consequence is that inferred centers should be surfaced, logged, and made contestable; a transparency mechanism is not optional if the negative-signal lever is to be usable at all.
C. Concentration of inferential power
The capacity in Claim 1 scales with model capability: stronger models learn richer geometry and recover subtler centers from sparser signal, as the attribute-inference results already indicate [16]. The most capable systems are thus the most powerful inferrers, and their users the most exposed—an asymmetry of observational power that current regulation does not squarely address.
D. Runaway personalization
A reinforcing loop can amplify its own bias. If the estimate drifts—an over-weighted early impression, correlation mistaken for preference—the loop doubles down: the niche experience narrows into a filter bubble, and self-authored capability into a cage built around a user who does not quite exist. Guardrails must act at every gear: memory must support revision and forgetting, not only accumulation; absence compensation must include the model’s own uncertainty, not only the user’s gaps; and the field must be perturbable, so the system can simulate alternative estimates and present them for confirmation or rejection rather than silently committing to one.
E. Measurement as a precondition for governance
None of the above is governable without measurement, which is why Definition 1 and Protocol 1 are not incidental. One cannot bound, audit, or contest an inference one cannot quantify. Operationalizing latent-recovery metrics—label probability, set-versus-control gaps, drift over accumulated signal—is a prerequisite for any of the guardrails above to be more than aspiration.
VIII. LIMITATIONS AND SCOPE
We separate what this paper establishes from what it argues and what it leaves open. Demonstrated (by cited work). That capable models infer unstated personal attributes from ordinary text at scale is established empirically [16]. That feed-forward layers act as key–value memories promoting concepts in vocabulary space, and that attention heads implement interpretable relational routing, are established mechanistic findings [13], [14], [12], [11], [9], [10]. Argued (conceptual). The framing of attention as relationalfield construction, the localization of unnamed-center recovery in attention-plus-memory rather than attention alone, and the capacity/operationalization distinction are conceptual contributions built on the above; they organize known results rather than report new ones. Conjectured (not evaluated here). The flywheel of Section VI is a systems hypothesis about scaffolded agents; we do not evaluate an implemented loop, measure sharpening over episodes, or quantify drift. Protocol 1 is proposed, not executed, and the opening probe is illustrative rather than evidential. Interpretive caveats. The clean “field” picture simplifies a messier reality: superposition and polysemantic features complicate any tidy mapping from heads to concepts, the attentionversus-MLP division of labor is a useful abstraction rather than a strict partition, and where factual content “crystallizes” across depth is itself contested. Our claims are intended at the level of mechanism-informed framing, not exact circuit-level attribution for a specific model.
IX. CONCLUSION
Attention is not only aggregation. It constructs a contextdependent field of affinities over tokens, and that field makes latent structure—unnamed centers, implicit categories, unspoken commitments—available whether or not it was named. But the recovery of that structure is completed by the network as a whole: attention assembles the representation; feed-forward memory and the unembedding condense the center. Getting this division right is what separates a defensible claim from a slogan. The consequence for user modeling is real and precise. The capacity to infer unstated attributes from scattered signals is intrinsic to capable models, unavoidable, and already demonstrated. The profile—persistent, revisable, sharpening— is not free: it is what scaffolding makes of that capacity by storing signals and re-presenting them for re-inference. That separation resolves the paradox of a capability that is both a side effect and an engineered loop, and it re-frames governance around what actually exists: not a stored dossier awaiting deletion, but a reconstructable inference to which obligations still attach through the retained signal set and the inferencetime behavior. Read together, the triptych says something more than any of its parts. The first report told us to build scaffolding; the second, to compensate for blindness; this one, to reckon with what the mechanism already recovers. The useful agent is not the product of a single innovation but of three that interlock— memory that preserves, questioning that directs, inference that condenses. When they do, the system does not merely assist; it grows into the shape of the user’s world. The remaining question is no longer whether we can build this, but whether we can measure, bound, and contest it as it builds itself.
ACKNOWLEDGMENT
The author thanks the PaxLabs Matrix team for discussion of the scaffolding and absence-blindness arguments that this paper extends.
REFERENCES
[1] A. G. and the PaxLabs Matrix Team, “Scaffolding over scale: Architecture as the driver of deployed agent capability,” PaxLabs Inc., Tech. Rep., 2025. [Online]. Available: https://scaffolding-over-scale. sites.paxeer.app/ [2] A. G., “The invisible gap: Absence blindness in large language models and the case for externalized presence,” Independent Research, Pembroke Pines, FL, USA, Tech. Rep., 2025. [Online]. Available: https://absence-blindness.sites.paxeer.app/ [3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, vol. 30, 2017. [4] Z. S. Harris, “Distributional structure,” Word, vol. 10, no. 2–3, pp. 146– 162, 1954. [5] J. R. Firth, “A synopsis of linguistic theory 1930–1955,” in Studies in Linguistic Analysis. Oxford: Blackwell, 1957, pp. 1–32. [6] J. Pennington, R. Socher, and C. D. Manning, “GloVe: Global vectors for word representation,” in Proc. 2014 Conf. Empirical Methods in Natural Language Processing (EMNLP), 2014, pp. 1532–1543. [7] O. Levy and Y. Goldberg, “Neural word embedding as implicit matrix factorization,” in Advances in Neural Information Processing Systems, vol. 27, 2014. [8] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pretraining of deep bidirectional transformers for language understanding,” in Proc. 2019 Conf. North American Chapter of the Association for Computational Linguistics (NAACL-HLT), 2019, pp. 4171–4186. [9] K. Clark, U. Khandelwal, O. Levy, and C. D. Manning, “What does BERT look at? An analysis of BERT’s attention,” in Proc. ACL Workshop BlackboxNLP, 2019, pp. 276–286. [10] J. Hewitt and C. D. Manning, “A structural probe for finding syntax in word representations,” in Proc. 2019 Conf. North American Chapter of the Association for Computational Linguistics (NAACL-HLT), 2019, pp. 4129–4138. [11] N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly et al., “A mathematical framework for transformer circuits,” Transformer Circuits Thread, 2021. [Online]. Available: https://transformer-circuits.pub/2021/framework/index.html [12] C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen et al., “In-context learning and induction heads,” Transformer Circuits Thread, 2022. [Online]. Available: https://transformer-circuits.pub/2022/ in-context-learning-and-induction-heads/index.html [13] M. Geva, R. Schuster, J. Berant, and O. Levy, “Transformer feed-forward layers are key-value memories,” in Proc. 2021 Conf. Empirical Methods in Natural Language Processing (EMNLP), 2021, pp. 5484–5495. [14] M. Geva, A. Caciularu, K. R. Wang, and Y. Goldberg, “Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space,” in Proc. 2022 Conf. Empirical Methods in Natural Language Processing (EMNLP), 2022, pp. 30–45. [15] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 1877–1901. [16] R. Staab, M. Vero, M. Balunovi ́c, and M. Vechev, “Beyond memorization: Violating privacy via inference with large language models,” in Proc. 12th Int. Conf. Learning Representations (ICLR), 2024. [17] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks,” in Proc. 2019 Conf. Empirical Methods in Natural Language Processing (EMNLP), 2019, pp. 3982–3992. [18] D. Jurafsky and J. H. Martin, Speech and Language Processing, 3rd ed. (draft), 2024.