Leonard Bereska      Efstratios Gavves
{leonard.bereska, egavves}@uva.nl
University of Amsterdam

Abstract

Understanding AI systems’ inner workings is critical for ensuring value alignment and safety. This review explores mechanistic interpretability: reverse engineering the computational mechanisms and representations learned by neural networks into human-understandable algorithms and concepts to provide a granular, causal understanding. We establish foundational concepts such as features encoding knowledge within neural activations and hypotheses about their representation and computation. We survey methodologies for causally dissecting model behaviors and assess the relevance of mechanistic interpretability to AI safety. We examine benefits in understanding, control, alignment, and risks such as capability gains and dual-use concerns. We investigate challenges surrounding scalability, automation, and comprehensive interpretation. We advocate for clarifying concepts, setting standards, and scaling techniques to handle complex models and behaviors and expand to domains such as vision and reinforcement learning. Mechanistic interpretability could help prevent catastrophic outcomes as AI systems become more powerful and inscrutable. For an HTML version of the paper, visit https://leonardbereska.github.io/blog/2024/mechinterpreview/.

1 Introduction

As AI systems rapidly become more sophisticated and general ^{1 2}, advancing our understanding of these systems is crucial to ensure their alignment ³ with human values and avoid catastrophic outcomes ^{4 5}. The field of interpretability aims to demystify the internal processes of AI models, moving beyond evaluating performance alone. This review focuses on mechanistic interpretability, an emerging approach within the broader interpretability landscape that strives to comprehensively specify the computations underlying deep neural networks. We emphasize that understanding and interpreting these complex systems is not merely an academic endeavor – it’s a societal imperative to ensure AI remains trustworthy and beneficial.

The interpretability landscape is undergoing a paradigm shift akin to the evolution from behaviorism to cognitive neuroscience in psychology. Historically, lacking tools for introspection, psychology treated the mind as a black box, focusing solely on observable behaviors. Similarly, interpretability has predominantly relied on black-box techniques ⁶, analyzing models based on input-output relationships or using attribution methods that, while probing deeper, still neglect the model’s internal architecture. However, just as advancements in neuroscience allowed for a deeper understanding of internal cognitive processes, the field of interpretability is now moving towards a more granular approach. This shift from surface-level analysis to a focus on the internal mechanics of deep neural networks characterizes the transition towards inner interpretability ⁷.

Mechanistic interpretability, as an approach to inner interpretability, aims to completely specify a neural network’s computation, potentially in a format as explicit as pseudocode (also called reverse engineering), striving for a granular and precise understanding of model behavior. It distinguishes itself primarily through its ambition for comprehensive reverse engineering and its strong motivation towards AI safety. Our review serves as the first comprehensive exploration of mechanistic interpretability research, with the most accessible introductions currently scattered in a blog or list format ^{8 9 10 11 12 13 14}. Concurrently, ¹⁵ and ¹⁶ have also contributed valuable reviews giving concise, technical introductions to mechanistic interpretability in transformer-based language models. Our work complements these efforts by synthesizing the research (addressing the “research debt” ¹⁷) and providing a structured, accessible, and comprehensive introduction for AI researchers and practitioners.

The structure of this paper provides a cohesive overview of mechanistic interpretability, situating the mechanistic approach in the broader interpretability landscape (Section 2), presenting core concepts and hypotheses (Section 3), explaining methods and techniques (Section 4), presenting a taxonomy and survey of the current field (Section 5), exploring relevance to AI safety (Section 6), and addressing challenges (Section 7) and future directions (Section 8).

2 Interpretability Paradigms from the Outside In

We encounter a spectrum of interpretability paradigms for decoding AI systems’ decision-making, ranging from external black-box techniques to internal analyses. We contrast these paradigms with mechanistic interpretability, highlighting its distinct causal bottom-up perspective within the broader interpretability landscape (see Figure 1).

Figure 1: Interpretability paradigms offer distinct lenses for understanding neural networks: Behavioral analyzes input-output relations; Attributional quantifies individual input feature influences; Concept-based identifies high-level representations governing behavior; Mechanistic uncovers precise causal mechanisms from inputs to outputs.

Behavioral

interpretability treats the model as a black box, analyzing input-output relations. Techniques such as minimal pair analysis ¹⁸, sensitivity and perturbation analysis ¹⁹ examine input-output relations to assess the model’s robustness and variable dependencies ^{20 21 22}. Its model-agnostic nature is practical for complex or proprietary models but lacks insight into internal decision processes and causal depth ²³.

Attributional

interpretability aims to explain outputs by tracing predictions to individual input contributions using gradients. Raw gradients can be discontinuous or sensitive to slight perturbations. Therefore, techniques such as SmoothGrad ²⁴ and Integrated Gradients ²⁵ average across gradients. Other popular techniques are layer-wise relevance propagation ²⁶, DeepLIFT ²⁷, or GradCAM ²⁸. Attribution enhances transparency by showing input feature influence without requiring an understanding of the internal structure, enabling decision validation, compliance, and trust while serving as a bias detection tool, but also has fundamental limitations ²⁹.

Concept-based

interpretability adopts a top-down approach to unraveling a model’s decision-making processes by probing its learned representations for high-level concepts and patterns governing behavior. Techniques include training supervised auxiliary classifiers ³⁰, employing unsupervised contrastive and structured probes (see Section 4.2) to explore latent knowledge ³¹, and using neural representation analysis to quantify the representational similarities between the internal representations learned by different neural networks ^{32 33}. Beyond observational analysis, concept-based interpretability can enable manipulation of these representations – also called representation engineering ³⁴ – potentially enhancing safety by upregulating concepts such as honesty, harmlessness, and morality.

Mechanistic

interpretability is a bottom-up approach that studies the fundamental components of models through granular analysis of features, neurons, layers, and connections, offering an intimate view of operational mechanics. Unlike concept-based interpretability, it aims to uncover causal relationships and precise computations transforming inputs into outputs, often identifying specific neural circuits driving behavior. This reverse engineering approach draws from interdisciplinary fields like physics, neuroscience, and systems biology to guide the development of transparent, value-aligned AI systems. Mechanistic interpretability is the primary focus of this review.

3 Core Concepts and Assumptions

This section introduces the key concepts and hypotheses of mechanistic interpretability, as summarized in Figure 2. We start by defining features as the basic units of representation (Section 3.1). We then examine the nature of these features, including the challenges posed by polysemantic neurons and the implications of the superposition and linear representation hypotheses (Section 3.2). Next, we explore computation through circuits and motifs, considering the universality hypothesis (Section 3.3). Finally, we discuss the implications for understanding emergent properties, such as internal world models and simulated agents with potentially misaligned objectives (Section 3.4).

Figure 2: Overview of key concepts and hypotheses in mechanistic interpretability, organized into four subsection (pink boxes): defining features (Section 3.1 ), representation (Section 3.2 ), computation (Section 3.3 ), and emergence (Section 3.4 ). In turquoise, it highlights definitions like features, circuits, and motifs, and in orange, it highlights hypotheses like linear representation superposition universality simulation prediction orthogonality. Arrows show relationships, e.g., superposition enabling an alternative feature definition or universality connecting circuits and motifs.

3.1 Defining Features as Representational Primitives

Features as fundamental units of representation.

The notion of a feature in neural networks is central yet elusive, reflecting the pre-paradigmatic state of mechanistic interpretability. We adopt the notion of features as the fundamental units of neural network representations, such that features cannot be further disentangled into simpler, distinct factors. These features are core components of a neural network’s representation, analogous to how cells form the fundamental unit of biological organisms ¹⁰.

Concepts as natural abstractions.

The world consists of various entities that can be grouped into categories or concepts based on shared properties. These concepts form high-level summaries like “tree” or “velocity,” allowing compact world representations by discarding many irrelevant low-level details. Neural networks can capture and represent such natural abstractions ³⁵ through their learned features, which serve as building blocks of their internal representations, aiming to capture the concepts underlying the data.

Features encoding input patterns.

In traditional machine learning, features are understood as characteristics or attributes derived directly from the input data stream ³⁶. This view is particularly relevant for systems focused on perception, where features map closely to the input data. However, in more advanced systems capable of reasoning with abstractions, features may emerge internally within the model as representational patterns, even when processing information unrelated to the input. In this context, features are better conceptualized as any measurable property or characteristic of a phenomenon ⁸, encoding abstract concepts rather than strictly reflecting input attributes.

Features as representational atoms.

A key property of features is their irreducibility, meaning they cannot be decomposed into or expressed as a combination of simpler, independent factors. In the context of input-related features, ³⁷ define a feature as irreducible if it cannot be decomposed into or expressed as a combination of statistically independent patterns or factors in the original input data. Specifically, a feature is reducible if transformations reveal its underlying pattern, which can be separated into independent co-occurring patterns or is a mixture of patterns that never co-occur. We propose generalizing this notion of irreducibility to features encoding abstract concepts not directly tied to input patterns, such that features cannot be reduced to combinations or mixtures of other independent components within the model’s representations.

Features beyond human interpretability.

Features could be defined from a human-centric perspective as semantically meaningful, articulable input patterns encoded in the network’s activation space ⁸. However, while cognitive systems may converge on similar natural abstractions ³⁵, these need not necessarily align with human-interpretable concepts. Adversarial examples have been interpreted as non-interpretable features meaningful to models but not humans. Imperceptible perturbations fool networks, suggesting reliance on alien representational patterns ³⁸. As models surpass human capabilities, their learned features may become increasingly abstract, encoding information in ways incongruent with human intuition ³⁹. Mechanistic interpretability aims to uncover the actual representations learned, even if diverging from human concepts. While human-interpretable concepts provide guidance, a non-human-centric perspective that defines features as independent model components, whether aligned with human concepts or not, is a more comprehensive and future-proof approach.

3.2 Nature of Features: From Monosemantic Neurons to Non-Linear Representations

Figure 3: Contrasting privileged and non-privileged bases. In a non-privileged basis, there is no reason to expect features to be basis-aligned – calling basis dimensions neurons has no meaning. In a privileged basis, the architecture treats basis directions differently – features can but need not align with neurons 30. Leftmost: Privileged basis; individual features (arrows) align with basis directions, resulting in monosemantic neurons (colored circles). Middle left: Privileged basis, where despite having more features than neurons, some neurons are monosemantic, representing individual features, while others are polysemantic (overlapping gradients), encoding superposition of multiple features. Middle right: Non-privileged basis where, even when the number of features equals the number of neurons, the lack of alignment between the feature directions and basis directions results in polysemantic neurons encoding combinations of features. Rightmost: Non-privileged, polysemantic neurons as feature directions do not align with neuron basis.

Neurons as Computational Units?

In the architecture of neural networks, neurons are the natural computational units, potentially representing individual features. Within a neural network representation $h\in\mathbb{R}^{n}$, the $n$ basis directions are called neurons. For a neuron to be meaningful, the basis directions must functionally differ from other directions in the representation, forming a privileged basis – where the basis vectors are architecturally distinguished within the neural network layer from arbitrary directions in activation space, as shown in Figure 3. Typical non-linear activation functions privilege the basis directions formed by the neurons, making it meaningful to analyze individual neurons ⁴⁰. Analyzing neurons can give insights into a network’s functionality ^{41 42 43 44 45 46 47 48 49}.

Monosemantic and Polysemantic Neurons.

A neuron corresponding to a single semantic concept is called monosemantic. The intuition behind this term comes from analyzing what inputs activate a given neuron, revealing its associated semantic meaning or concept. If neurons were the representational primitives of neural networks, all neurons would be monosemantic, implying a one-to-one relationship between neurons and features. Comprehensive interpretability would be as tractable as characterizing all neurons and their connections. However, empirically, especially for transformer models ⁴⁰, neurons are often observed to be polysemantic, i.e., associated with multiple, unrelated concepts ^{50 42 51 10}. For example, a single neuron may be activated by both images of cats and images of cars, suggesting it encodes multiple unrelated concepts. Polysemanticity contradicts the interpretation of neurons as representational primitives and, in practice, makes it challenging to understand the information processing of neural networks.

Exploring Polysemanticity: Hypotheses and Implications.

To understand the widespread occurrence of polysemanticity in neural networks, several hypotheses have been proposed:

1. One trivial scenario would be that feature directions are orthogonal but not aligned with the basis directions (neurons). There is no inherent reason to assume that features would align with neurons in a non-privileged basis, where the basis vectors are not architecturally distinguished. However, even in a privileged basis formed by the neurons, the network could represent features not in the standard basis but as linear combinations of neurons (see Figure 3, middle right).
2. An alternative hypothesis posits that redundancy due to noise introduced during training, such as random dropout ⁵², can lead to redundant representations and, consequently, to polysemantic neurons ⁵³. This process involves distributing a single feature across several neurons rather than isolating it into individual ones, thereby encouraging polysemanticity.
3. Finally, the superposition hypothesis addresses the limitations in the network’s representative capacity – the number of neurons versus the number of crucial concepts. This hypothesis argues that the limited number of neurons compared to the vast array of important concepts necessitates a form of compression. As a result, an $n$ -dimensional representation may encode features not with the $n$ basis directions (neurons) but with the $\propto\exp(n)$ possible almost orthogonal directions ⁴⁰, leading to polysemanticity.

Superposition Hypothesis.

The superposition hypothesis suggests that neural networks can leverage high-dimensional spaces to represent more features than the actual count of neurons by encoding features in almost orthogonal directions. Non-orthogonality means that features interfere with one another. However, the benefit of representing many more features than neurons may outweigh the interference cost, mainly when concepts are sparse and non-linear activation functions can error-correct noise ⁴⁰.

[Uncaptioned image]

Toy models can demonstrate under which conditions superposition occurs ^{40 54}. Neural networks, via superposition, may effectively simulate computation with more neurons than they possess by allocating each feature to a linear combination of neurons, creating what is known as an overcomplete linear basis in the representation space. This perspective on superposition suggests that polysemantic models could be seen as compressed versions of hypothetically larger neural networks where each neuron represents a single concept (see Figure 5). Consequently, an alternative definition of features could be:

Figure 5: Observed neural networks (left) can be viewed as compressed simulations of larger, sparser networks (right) where neurons represent distinct features. An “almost orthogonal” projection compresses the high-dimensional sparse representation, manifesting as polysemantic neurons involved with multiple features in the lower-dimensional observed model, reflecting the compressed encoding. Figure adapted from 30.

Research on superposition, including works by ^{40 54 55}, often investigates simplified models. However, understanding superposition in practical, transformer-based scenarios is crucial for real-world applications, as pioneered by ⁵⁶.

The need for understanding networks despite polysemanticity has led to various approaches: One involves training models without superposition ⁵⁷, for example, using a softmax linear unit ⁵¹ as an activation function to empirically increase the number of monosemantic neurons, but at the cost of making other neurons less interpretable. From a capabilities standpoint, polysemanticity may be desirable as it allows models to represent more concepts with limited compute, making training cheaper. Overall, engineering monosemanticity has proven challenging ⁵⁸ and may be impractical until we have orders of magnitude more compute available.

Another approach is to train networks in a standard way (creating polysemanticity) and use post-hoc analysis to find the feature directions in activation space, for example, with Sparse Autoencoders (SAEs). SAEs aim to find the true, disentangled features in an uncompressed representation by learning a sparse overcomplete basis that describes the activation space of the trained model ^{58 59 60} (also see Section 4.2).

If not neurons, what are features then?

We want to identify the fundamental units of neural networks, which we call features. Initially, neurons seemed likely candidates. However, this view fell short, particularly in transformer models where neurons often represent multiple concepts, a phenomenon known as polysemanticity. The superposition hypothesis addresses this, proposing that due to limited representational capacity, neural networks compress numerous features into the confined space of neurons, complicating interpretation.

This raises the question: How are features encoded if not in discrete neuron units? While a priori features could be encoded in an arbitrarily complex, non-linear structure, a growing body of theoretical arguments and empirical evidence supports the hypothesis that features are commonly represented linearly, i.e., as linear combinations of neurons – hence, as directions in representation space. This perspective promises to enhance our comprehension of neural networks by providing a more interpretable and manipulable framework for their internal representations.

The linear representation hypothesis suggests that neural networks frequently represent high-level features as linear directions in activation space. This hypothesis can simplify the understanding and manipulation of neural network representations ⁶¹. The prevalence of linear layers in neural network architectures favors linear representations. Matrix multiplication in these layers most readily processes linear features, while more complex non-linear encodings would require multiple layers to decode.

However, recent work by ³⁷ provides evidence against a strict formulation of the linear representation hypothesis by identifying circular features representing days of the week and months of the year. These multi-dimensional, non-linear representations were shown to be used for solving modular arithmetic problems in days and months. Intervention experiments confirmed that these circular features are the fundamental unit of computation in these tasks, and the authors developed methods to decompose the hidden states, revealing the circular representations.

Establishing non-linearity can be challenging. For example, ⁶² initially found that in a GPT model trained on Othello, the board state could only be decoded with a non-linear probe when represented in terms of white and black pieces, seemingly violating the linearity assumption. However, ^{63 61} later showed that a linear probe sufficed when the board state was decoded in terms of “one’s own” and “the opponent’s” pieces, reaffirming the linear representation hypothesis in this case. In contrast, the work by ³⁷ provides a clear and convincing existence proof for non-linear, multi-dimensional representations in language models.

While the linear representation hypothesis remains a useful simplification, it is important to recognize its limitations and the potential role of non-linear representations ¹¹. As neural networks continue to evolve, ongoing reevaluation of the hypothesis is crucial, particularly considering the possible emergence of non-linear features under optimization pressure for interpretability ⁶⁴. Alternative perspectives, such as the polytope lens proposed by ⁶⁵, emphasize the impact of non-linear activation functions and discrete polytopes formed by piecewise linear activations as potential primitives of neural network representations.

Despite these exceptions, empirical evidence largely supports the linear representation hypothesis in many contexts, especially for feedforward networks with ReLU activations. Semantic vector calculus in word embeddings ⁶⁶, successful linear probing ^{67 30}, sparse dictionary learning ^{58 60 68}, and linear decoding of concepts ⁶⁹, tasks ⁷⁰, functions ⁷¹, sentiment ⁷², refusal ⁷³, and relations ^{74 75} in large language models all point to the prevalence of linear representations. Moreover, linear addition techniques for model steering ^{76 77 78} and representation engineering ³⁴ highlight the practical implications of linear feature representations.

Building upon the linear representation hypothesis, recent work investigated the structural organization of these linear features within activation space. ⁷⁹ reveal a geometric framework for categorical and hierarchical concepts in large language models. Their findings demonstrate that simple categorical concepts (e.g., mammal, bird) are represented as simplices in the activation space, while hierarchically related concepts are orthogonal. This geometric analysis aligns with earlier observations on feature clustering and splitting in neural networks ⁴⁰. It suggests that the linear features are not merely scattered directions but are organized to reflect semantic relationships and hierarchies.

3.3 Circuits as Computational Primitives and Motifs as Universal Circuit Patterns

Having defined features as directions in activation space as the fundamental units of neural network representation, we now explore their computation. Neural networks can be conceptualized as computational graphs, within which circuits are sub-graphs consisting of linked features and the weights connecting them. Similar to how features are the representational primitive, circuits function as the computational primitive ⁸⁰ and the primary building block of these networks ¹⁰.

Figure 6: Comparing observed models (left) and corresponding hypothetical disentangled models (right) trained on similar tasks and data. The observed models show different neuronal activation patterns, while the dissection into feature-level circuits reveals a motif - a shared circuit pattern emerging across models, hinting at universality – models converging on similar solutions based on common underlying principles.

The decomposition of neural networks into circuits for interpretability has shown significant promise, particularly in small models trained for specific tasks such as addition, as seen in the work of ⁸¹ and ⁸². Scaling such a comprehensive circuit analysis to broader behaviors in large language models remains challenging. However, there has been notable progress in scaling circuit analysis of narrow behaviors to larger circuits and models, such as indirect object identification ⁸³ and greater-than computations ⁸⁴ in GPT-2 and multiple-choice question answering ⁸⁵.

In search of general and universal circuits, researchers focus particularly on more general and transferable behaviors. ⁸⁶ ’s work on copy suppression in GPT-2’s attention heads sheds light on model calibration and self-repair mechanisms. ⁸⁷ and ⁸⁸ focus on how large language models represent symbolic knowledge through variable binding and entity-attribute binding, respectively. ^{89 90 91 92 93} explore mechanisms for factual recall, revealing how circuits dynamically balance pre-trained knowledge with new contextual information. ⁹⁴ extend circuit analysis to sequence continuation tasks, identifying shared computational structures across semantically related sequences.

More promisingly, some repeating patterns have shown universality across models and tasks. These universal patterns are called motifs ¹⁰ and can manifest not just as specific circuits or features but also as higher-level behaviors emerging from the interaction of multiple components. Examples include the curve detectors found across vision models ^{95 96}, induction circuits enabling in-context learning ⁹⁷, and the phenomenon of branch specialization in neural networks ⁹⁸. Motifs may also capture how models leverage tokens for working memory or parallelize computations in a divide-and-conquer fashion across representations. The significance of motifs lies in revealing the common structures, mechanisms, and strategies that naturally emerge across neural architectures, shedding light on the fundamental building blocks underlying their intelligence. Figure 6 contrasts observed neural network models with hypothetical disentangled models, illustrating how a shared circuit pattern can emerge across different models trained on similar tasks and data, hinting at an underlying universality.

Universality Hypothesis.

Following the evidence for motifs, we can propose two versions for a universality hypothesis regarding the convergence of features and circuits across neural network models:

The universality hypothesis posits a convergence in forming features and circuits across various models and tasks, which could significantly ease interpretability efforts in AI. It proposes that artificial and biological neural networks share similar features and circuits, suggesting a standard underlying structure ^{35 99 32}. This idea posits that there is a fundamental basis in how neural networks, irrespective of their specific configurations, process and comprehend information. This could be due to inbuilt inductive biases in neural networks or natural abstractions ³⁵ – concepts favored by the natural world that any cognitive system would naturally gravitate towards.

Evidence for this hypothesis comes from cross-species neural structures in neuroscience, where similar neural structures and functions are found in different species ¹⁰⁰. Additionally, machine learning models, including neural networks, tend to converge on similar features, representations, and classifications across different tasks and architectures ^{101 102 103 58}. ¹⁰⁴ provide mathematical support for emerging universal features.

While various studies support the universality hypothesis, questions remain about the extent of feature and circuit similarity across different models and tasks. In the context of mechanistic interpretability, this hypothesis has been investigated for neurons ¹⁰⁵, group composition circuits ¹⁰⁶, and modular task processing ¹⁰⁷, with evidence for the weak but not the strong formulation ¹⁰⁶.

3.4 Emergence of World Models and Simulated Agents

Internal World Models.

World models are internal causal models of an environment formed within neural networks. Traditionally linked with reinforcement learning, these models are explicitly trained to develop a compressed spatial and temporal representation of the training environment, enhancing downstream task performance and sample efficiency through training on internal hallucinations ¹⁰⁸. However, in the context of our survey, our focus shifts to internal world models that potentially form implicitly as a by-product of the training process, especially in LLMs trained on next-token prediction – also called GPT.

LLMs are sometimes characterized as stochastic parrots ¹⁰⁹. This label stems from their fundamental operational mechanism of predicting the next word in a sequence, which is seen as relying heavily on memorization. From this viewpoint, LLMs are thought to form complex correlations based on observational data but cannot develop causal models of the world due to their lack of access to interventional data ¹¹⁰.

An alternative perspective on LLMs comes from the active inference framework ¹¹¹, a theory rooted in cognitive science and neuroscience. Active inference postulates that the objective of minimizing prediction error, given enough representative capacity, is adequate for a learning system to develop complex world representations, behaviors, and abstractions. Since language inherently mirrors the world, these models could implicitly construct linguistic and broader world models ¹¹².

The simulation hypothesis suggests that models designed for prediction, such as LLMs, will eventually simulate the causal processes underlying data creation. Seen as an extension of their drive for efficient compression, this hypothesis implies that adequately trained models like GPT could develop internal world models as a natural outcome of their predictive training ^{113 114}.

In addition to theoretical considerations for emergent causal world models ^{115 116}, mechanistic interpretability is starting to provide empirical evidence on the types of internal world models that may emerge in LLMs. The ability to internally represent the board state in games like chess ¹¹⁷ or Othello ^{62 61}, create linear abstractions of spatial and temporal data ¹¹⁸, and structure complex representations of mazes, demonstrating an understanding of maze topology and pathways ¹¹⁹ highlight the growing abstraction capabilities of LLMs. ¹²⁰ identified contextual word representations that function as models of entities and situations evolving throughout a discourse, akin to linguistic models of dynamic semantics. ¹²¹ demonstrated that LLMs can map conceptual domains (e.g., direction, color) to grounded world representations given a few examples, suggesting they learn rich conceptual spaces ¹²² reflective of the non-linguistic world.

The prediction orthogonality hypothesis further expands on this idea: It posits that prediction-focused models like GPT may simulate agents with various objectives and levels of optimality. In this context, GPT are simulators, simulating entities known as simulacra that can be either agentic or non-agentic, with different objectives from the simulator itself ^{113 114}. The implications of the simulation and prediction orthogonality hypotheses for AI safety and alignment are discussed in Section 6.

In conclusion, the evolution of LLMs from simple predictive models to entities potentially possessing complex internal world models, as suggested by the simulation hypothesis and supported by mechanistic interpretability studies, represents a significant shift in our understanding of these systems. This evolution challenges us to reconsider LLMs’ capabilities and future trajectories in the broader landscape of AI development.

4 Core Methods

Mechanistic interpretability (MI) employs various tools, from observational analysis to causal interventions. This section provides a comprehensive overview of these methods, beginning with a taxonomy that categorizes approaches based on their key characteristics (Section 4.1). We then survey observational (Section 4.2), followed by interventional techniques (Section 4.3). Finally, we study their synergistic interplay (Section 4.4). Figure 7 offers a visual summary of the methods and techniques unique to mechanistic interpretability.

Figure 7: Overview of key methods and techniques in mechanistic interpretability research. Observational approaches include structured probes, logit lens variants, and sparse autoencoders (SAEs). Interventional methods, focusing on causal understanding, encompass activation patching variants for uncovering causal mechanisms and causal scrubbing for hypothesis evaluation.

4.1 Taxonomy of Mechanistic Interpretability Methods

We propose a taxonomy based on four key dimensions: causal nature, learning phase, locality, and comprehensiveness (Table 1).

The causal nature of methods ranges from purely observational, which analyze existing representations without direct manipulation, to interventional approaches that actively perturb model components to establish causal relationships. The learning phase dimension distinguishes between post-hoc techniques applied to trained models and intrinsic methods that enhance interpretability during the training process itself.

Locality refers to the scope of analysis, spanning from individual neurons (e.g., feature visualization) to entire model architectures (e.g., causal abstraction). Comprehensiveness varies from partial insights into specific components to holistic explanations of model behavior.

Table 1: Taxonomy of Mechanistic Interpretability Methods

Method	Causal Nature	Phase	Locality	Comprehensiveness	Key Examples
Feature Visualization	Observation	Post-hoc	Local	Partial	123
					124
Exemplar methods	Observation	Post-hoc	Local	Partial	125
					126
Probing Techniques	Observation	Post-hoc	Both	Both	127
					56
Structured Probes	Observation	Post-hoc	Both	Both	31
Logit Lens Variants	Observation	Post-hoc	Global	Partial	128
					129
Sparse Autoencoders	Observation	Post-hoc	Both	Comprehensive	60
					58
Activation Patching	Intervention	Post-hoc	Local	Partial	130
					83
Path Patching	Intervention	Post-hoc	Both	Both	131
Causal Abstraction	Intervention	Post-hoc	Global	Comprehensive	132
					133
					134
Hypothesis Testing	Intervention	Post-hoc	Global	Comprehensive	135
					136
Intrinsic Methods	–	Pre/During	Global	Comprehensive	51
					137

The categorization is based on the methods’ general tendencies. Some methods can offer local and global or partial and comprehensive interpretability depending on the scope of the analysis and application. Probing techniques can range from local to global and partial to comprehensive; simple linear probes might offer local insights into individual features, while more sophisticated structured probes can uncover global patterns. Sparse autoencoders decompose individual neuron activations (local) but aim to disentangle features across the entire model (global). Path patching extends local interventions to global model understanding by tracing information flow across layers, demonstrating how local perturbations can yield broader insights.

In practice, mechanistic interpretability research involves both method development and their application. When applying methods to understand a model, combining techniques from multiple categories is often necessary and beneficial to build a more comprehensive understanding (Section 4.4).

4.2 Observation

Mechanistic interpretability draws from observational methods that analyze the inner workings of neural networks, with many of these methods preceding the field itself. For a detailed exploration of inner interpretability methods, refer to ⁷. Two prominent categories are example-based methods and feature-based methods:

Probing for Features.

Probing ^{67 138} involves training a classifier using the activations of a model, with the classifier’s performance subsequently observed to deduce insights about the model’s behavior and internal representations. However, the probe’s performance may often reflect its own learning capacities more than the actual characteristics of the model’s representations ³⁰. This dilemma has led researchers to investigate the ideal balance between the complexity of a probe and its capacity to accurately represent the model’s features ^{139 140}.

The linear representation hypothesis offers a resolution to this issue. Under this hypothesis, the failure of a simple linear probe to detect certain features suggests their absence in the model’s representations. Conversely, suppose a more complex probe succeeds where a simpler one fails. In that case, it implies that the model contains features that a complex function can combine into the target feature but that the target feature itself is not explicitly represented. Thus, the hypothesis implies that using linear probes could suffice in most cases, circumventing the complexity considerations generally associated with probing ³⁰.

¹²⁷ analyzed chess knowledge acquisition in AlphaZero, revealing the emergence of strategic concepts during training. In language models, ⁵⁶ introduced sparse probing to decode internal neuron activations to understand feature representation and sparsity. They show that early layers use sparse combinations of neurons to represent many features in superposition, while middle layers seem to have dedicated monosemantic neurons for higher-level contextual features.

Probing is limited in drawing causal or behavioral conclusions. Its primarily observational nature focuses on how information is encoded rather than how it is used (see Figure 1), necessitating careful analysis and integration with interventional techniques (Section 4.3), or alternative approaches ¹⁴¹. While in explainable AI, probing has primarily analyzed high-level concepts like linguistic representations ^{142 143}, MI aims to probe towards uncovering underlying computational processes and functionality. This shift in goals towards uncovering mechanistic computation is a nuanced distinction rather than a clear-cut line between probing in MI and the broader explainability field.

Structured Probes.

While focusing on bottom-up, mechanistic interpretability approaches, we can also consider integrating top-down, concept-based structured probes with mechanistic interpretability.

Structured probes aid conceptual interpretability, probing language models for complex features like truth representations. Notably, ³¹ ’s contrast-consistent search identifies linear projections exhibiting logical consistency in hidden states, contrasting truth values for statements and negations.

However, structured probes face significant challenges in unsupervised probing scenarios. As ¹⁴⁴ showed, arbitrary features, not just knowledge-related ones, can satisfy contrast consistency equally well, raising doubts about scalability. For example, the loss may capture simulation of knowledge from hypothesized simulacra within sufficiently powerful language models rather than the models’ true knowledge. Furthermore, ¹⁴⁴ demonstrates self-supervised probing methods (like ³¹) often detect prominent but unintended distractor features in the data. The discovered features are also highly sensitive to prompt choice, and there is no principled way to select prompts that would reliably surface a model’s true knowledge.

While structured probes primarily focus on high-level conceptual representations ³⁴, their findings could potentially inform or complement mechanistic interpretability efforts. For instance, identifying truth directions through structured probes could help guide targeted interventions or analyze the underlying circuits responsible for truthful behavior using mechanistic techniques such as activation patching or circuit tracing (Section 4.3). Conversely, mechanistic methods could provide insights into how truth representations emerge and are computed within the model, addressing some of the challenges faced by unsupervised structured probes.

Logit Lens.

The logit lens ¹²⁸ provides a window into the model’s predictive process by applying the final classification layer (which projects the residual stream activation into logits/vocabulary space) to intermediate activations of the residual stream, revealing how prediction confidence evolves across computational stages. This is possible because transformers tend to build their predictions across layers iteratively ¹⁴⁵. Extensions of this approach include the tuned lens ¹²⁹, which trains affine probes to decode hidden states into probability distributions over the vocabulary, and the Future Lens ¹⁴⁶, which explores the extent to which individual hidden states encode information about subsequent tokens.

Researchers have also investigated techniques that bypass intermediate computations to probe representations directly. ¹⁴⁷ propose using linear transformations to approximate hidden states from different layers, revealing that language models often predict final outputs in early layers. ¹⁴⁸ present a theoretical framework for interpreting transformer parameters by projecting them into the embedding space, enabling model alignment and parameter transfer across architectures.

Other techniques focus on interpreting specific model components or submodules. The DecoderLens ¹⁴⁹ allows analyzing encoder-decoder transformers by cross-attending intermediate encoder representations in the decoder, shedding light on the information flow within the encoder. The Attention Lens ¹⁵⁰ aims to elucidate the specialized roles of attention heads by translating their outputs into vocabulary tokens via learned transformations.

Feature Disentanglement via Sparse Dictionary Learning.

Recent work suggests that the essential elements in neural networks are linear combinations of neurons representing features in superposition ⁴⁰. To disentangle these features, researchers have developed sparse autoencoders (SAEs), which decompose neural network activations into individual component features ^{59 60}. This process, known as sparse dictionary learning, reconstructs activation vectors as sparse linear combinations of directional vectors within the activation space ¹⁵¹.

The theoretical foundations of SAEs are rooted in work on disentangled representations. ¹⁵² demonstrate that autoencoders can recover ground truth features under conditions of feature sparsity and non-negativity. Furthermore, ¹⁵³ provides guarantees for the uniqueness and stability of dictionaries for sparse representation, even in the presence of noise. These theoretical underpinnings support SAEs’ ability to uncover true, disentangled features underlying the data distribution.

In practice, SAEs stand out for their simplicity and scalability ⁵⁹. They incorporate sparsity regularization to encourage learning sparse yet meaningful data representations, with the precise tuning of the sparsity penalty on hidden activations critical in dictating the autoencoder’s sparsity level. We provide an overview of the SAE architecture in Figure 8.

SAEs’ dictionary features exhibit higher scores on autointerpretability metrics and increased monosemanticity ^{58 60 59}. They are scalable to state-of-the-art models and can detect safety-relevant features ¹⁵⁴, measure feature sparsity ⁶⁸, and interpret reward models in reinforcement learning-based language models ¹⁵⁵.

Evaluating SAE quality remains challenging due to the lack of ground-truth interpretable features. Researchers have addressed this through various approaches: ¹⁵⁶ proposed using language models trained on chess and Othello transcripts as testbeds, providing natural collections of interpretable features. ⁵⁹ constructed a toy model with traceable features, while ^{157 158} compared SAE results with supervised features in large language models to demonstrate their viability.

The versatility of SAEs extends to various neural network architectures. They have been successfully applied to transformer attention layers ¹⁵⁹ and convolutional neural networks ¹⁶⁰. Notably, ¹⁶⁰ applied SAEs to the early vision layers of InceptionV1, uncovering new interpretable features, including additional curve detectors not apparent from examining individual neurons ⁹⁶.

In circuit discovery, SAEs have shown particular promise (see also Section 4.4). ¹⁶¹ proposed a circuit discovery framework alternative to activation patching (discussed in Section 4.3.1), leveraging dictionary features decomposed from all modules writing to the residual stream. Similarly, ¹⁶² employed discrete sparse autoencoders for discovering interpretable circuits in large language models.

Recent advancements have focused on improving SAE performance and addressing limitations. ¹⁶³ introduced a gating mechanism to separate the functionalities of determining which directions to use and estimating their magnitudes, mitigating shrinkage – the systematic underestimation of feature activations. An alternative approach by ¹⁶⁴ uses transcoders to faithfully approximate a densely activating MLP layer with a wider, sparsely-activating MLP layer, offering another path to interpretable feature discovery, a type of sparse distillation ¹⁶⁵.

polysemantic

4.3 Intervention

Causality as a Theoretical Foundation.

The theory of causality ¹¹⁰ provides a mathematically precise framework for mechanistic interpretability, offering a rigorous approach to understanding high-level semantics in neural representations ¹³². By treating neural networks as causal models, with their compute graphs serving as causal graphs, researchers can perform precise interventions and examine the roles of individual parameters ¹⁶⁶. This causal perspective on interpretability has led to the development of various intervention techniques, including activation patching (Section 4.3.1), causal abstraction (Section 4.3.2), and hypothesis testing methods (Section 4.3.3).

4.3.1 Activation Patching

(a)

Activation patching is a collective term for a set of causal intervention techniques that manipulate neural network activations to shed light on the decision-making processes within the model. These techniques, including causal tracing ¹³⁰, interchange intervention ¹⁶⁷, causal mediation analysis ¹⁶⁸, and causal ablation ⁸³, share the common goal of modifying a neural model’s internal state by replacing specific activations with alternative values, such as zeros, mean activations across samples, random noise, or activations from a different forward pass (Figure 9a).

The primary objective of activation patching is to isolate and understand the role of specific components or circuits within the model by observing how changes in activations affect the model’s output. This enables researchers to infer the function and importance of those components. Key applications include localizing behavior by identifying critical activations, such as understanding the storage and processing of factual information ^{130 169 131 170}, and analyzing component interactions through circuit analysis to identify sub-networks within a model’s computation graph that implement specified behaviors ^{83 84 85 70 169}.

The standard protocol for activation patching (Figure 9a) involves:

1. Running the model with a clean input and caching the latent activations;
2. Executing the model with a corrupted input;
3. Re-running the model with the corrupted input but substituting specific activations with those from the clean cache; and
4. Determining significance by observing the variations in the model’s output during the third step, thereby highlighting the importance of the replaced components.

This process relies on comparing pairs of inputs: a clean input, which triggers the desired behavior, and a corrupted input, which is identical to the clean one except for critical differences that prevent the behavior. By carefully selecting these inputs, researchers can control for confounding circuitry and isolate the specific circuit responsible for the behavior.

Differences in patching direction – clean to corrupted (causal tracing) versus corrupted to clean (resample ablation) – provide insights into the sufficiency or necessity of model components for a given behavior. Clean to corrupted patching identifies activations sufficient for restoring clean performance, even if they are unnecessary due to redundancy, which is particularly informative in OR logic scenarios (Figure 9b, OR gate). Conversely, corrupted to clean patching determines the necessary activations for clean performance, which is useful in AND logic scenarios (Figure 9b, AND gate).

Activation patching can employ corruption methods, including zero-, mean-, random-, or resample ablation, each modulating the model’s internal state in distinct ways. Resample ablation stands out for its effectiveness in maintaining consistent model behavior by not changing the data distribution too much ¹⁷¹. However, it is essential to be careful when interpreting the patching results, as breaking behavior by taking the model off-distribution is uninteresting for finding the relevant circuit ¹⁷².

Path Patching and Subspace Activation Patching.

Path patching extends the activation patching approach to multiple edges in the computational graph ^{83 131}, allowing for a more fine-grained analysis of component interactions. For example, path patching can be used to estimate the direct and indirect effects of attention heads on the output logits. Subspace activation patching, also known as distributed interchange interventions ¹³³, aims to intervene only on linear subspaces of the representation space where features are hypothesized to be encoded, providing a tool for more targeted interventions.

Recently, ¹⁷³ introduced patchscopes, a framework that unifies and extends activation patching techniques: using the model’s text generation to explain internal representations, it enables more flexible interventions across various interpretability tasks, improving early layer inspection and allowing for cross-model analysis.

Limitations and Advancements.

Activation patching has several limitations, including the effort required to design input templates and counterfactual datasets, the need for human inspection to isolate important subgraphs, and potential second-order effects that can complicate the interpretation of results ¹⁷⁴ and the hydra effect ^{175 176} (see discussion in Section 7.2). Recent advancements aim to address these limitations, such as automated circuit discovery algorithms ¹⁷⁷, gradient-based methods for scalable component importance estimation like attribution patching ^{178 179}, and techniques to mitigate self-repair interference during analysis ¹⁸⁰.

4.3.2 Causal Abstraction

Causal abstraction ^{181 132} provides a mathematical framework for mechanistic interpretability, treating neural networks and their explanations as causal models. This approach validates explanations through interchange interventions on network activations ¹³⁶, unifying various interpretability methods such as LIME ²¹, causal effect estimation ¹⁸², causal mediation analysis ¹⁶⁸, iterated nullspace projection ¹⁸³, and circuit-based explanations ¹³².

To overcome computational limitations, distributed alignment search ¹³³ introduced gradient-based distributed interchange interventions, extending causal abstraction to larger models ¹⁸⁴. Further advancements include causal proxy models ¹³⁴, which address the challenge of counterfactual observations.

Applications of causal abstraction span from linguistic phenomena analysis ^{185 186}, and evaluation of interpretability methods ¹⁸⁷, to improving performance through representation finetuning ¹⁸⁸, and improving efficiency via model distillation ¹⁸⁶.

4.3.3 Hypothesis Testing

In addition to the causal abstraction framework, several methods have been developed for rigorous hypothesis testing about neural network behavior. These methods aim to formalize and empirically validate explanations of how neural networks implement specific behaviors.

Causal scrubbing ¹³⁵ formalizes hypotheses as a tuple $({\mathcal{G}},{\mathcal{I}},c)$, where ${\mathcal{G}}$ is the model’s computational graph, ${\mathcal{I}}$ is an interpretable computational graph hypothesized to explain the behavior, and $c$ maps nodes of ${\mathcal{I}}$ to nodes of ${\mathcal{G}}$. This method replaces activations in ${\mathcal{G}}$ with others that should be equivalent according to the hypothesis, measuring performance on the scrubbed model to validate the hypothesis.

Locally consistent abstractions ¹³⁶ offer a more permissive approach, checking the consistency between the neural network and the explanation only one step away from the intervention node. This method forms a middle ground between the strictness of full causal abstraction and the flexibility of causal scrubbing.

These methods form a hierarchy of strictness, with full causal abstractions being the most stringent, followed by locally consistent abstractions and causal scrubbing being the most permissive. This hierarchy highlights trade-offs in choosing stricter or more permissive notions, affecting the ability to find acceptable explanations, generalization, and mechanistic anomaly detection.

4.4 Integrating Observation and Intervention.

To comprehensively understand internal neural network mechanisms, combining observational and interventional methods is crucial. For instance, sparse autoencoders can be used to disentangle superposed features ⁶⁰, followed by targeted activation patching to test the causal importance of these features ⁸³. Similarly, the logit lens can track prediction formation across layers ¹²⁸, with subsequent interventions confirming causal relationships at key points. Probing techniques can identify encoded information ³⁰, which can then be subjected to causal abstraction ¹³² to understand how this information is utilized. This iterative refinement process, where broad observational methods guide targeted interventions and intervention results inform further observations, enables a multi-level analysis that builds a holistic understanding across different levels of abstraction. Recent work ^{189 190 191 162 192} demonstrates the potential of integrating sparse autoencoders with automated circuits discovery ^{177 179}, combining feature-level analysis with circuit-level interventions to uncover the interplay between representation and mechanism.

5 Current Research

This section surveys current research in mechanistic interpretability across three approaches based on when and how the model is interpreted during training: Intrinsic interpretability methods are applied before training to enhance the model’s inherent interpretability (Section 5.1). Developmental interpretability involves studying the model’s learning dynamics and the emergence of internal structures during training (Section 5.2). After training, post-hoc interpretability techniques are applied to gain insights into the model’s behavior and decision-making processes (Section 5.3), including efforts towards uncovering general, transferable principles across models and tasks, as well as automating the discovery and interpretation of critical circuits in trained models (Section 5.4).

Figure 10: Key desiderata for interpretability approaches across training and analysis stages: (1) Intrinsic: Architectural biases for sparsity, modularity, and disentangled representations. (2) Developmental: Predictive capability for phase transitions, manageable number of critical transitions, and a unifying theory connecting observations to singularity geometry. (3) Post-hoc: Global, comprehensive, automated discovery of critical circuits, uncovering transferable principles across models/tasks, and extracting high-level causal mechanisms.

5.1 Intrinsic Interpretability

Intrinsic methods for mechanistic interpretability offer a promising approach to designing neural networks that are more amenable to reverse engineering without sacrificing performance. By encouraging sparsity, modularity, and monosemanticity through architectural choices and training procedures, these methods aim to make the reverse engineering process more tractable.

Intrinsic interpretability methods aim to constrain the training process to make learned programs more interpretable ¹⁹³. This approach is closely related to neurosymbolic learning ¹⁹⁴ and can involve techniques like regularization with spatial structure, akin to the organization of information in the human brain ^{137 195}.

Recent work has explored various architectural choices and training procedures to improve the interpretability of neural networks. ⁵⁷ and ⁵¹ demonstrate that architectural choices can affect monosemanticity, suggesting that models could be engineered to be more monosemantic. ¹⁹⁶ propose using a bilinear layer instead of a linear layer to encourage monosemanticity in language models.

¹³⁷ and ¹⁹⁵ introduce a biologically inspired spatial regularization regime called brain-inspired modular training for forming modules in networks during training. They showcase how this can help RNNs exhibit brain-like anatomical modularity without degrading performance, in contrast to naive attempts to use sparsity to reduce the cost of having more neurons per layer ^{57 58}.

Preceding the mechanistic interpretability literature, various works have explored techniques to improve interpretability, such as sparse attention ¹⁹⁷, adding $L^{1}$ penalties to neuron activations ^{198 199}, and pruning neurons ²⁰⁰. These techniques have been shown to encourage sparsity, modularity, and disentanglement, which are essential aspects of intrinsic interpretability.

5.2 Developmental Interpretability

Developmental interpretability examines the learning dynamics and emergence of internal structures in neural networks over time, focusing on the formation of features and circuits. This approach complements static analyses by investigating critical phase transitions corresponding to significant changes in model behavior or capabilities ^{201 202 203 204}. While primarily a distinct field, developmental interpretability often intersects with mechanistic interpretability, as exemplified by ⁹⁷ ’s work. Their research, rooted in mechanistic interpretability, demonstrated how the emergence of in-context learning relates to specific training phase transitions, connecting microscopic changes (induction heads) with macroscopic observables (training loss).

A key motivation for developmental interpretability is investigating the universality of safety-critical patterns, aiming to understand how deeply ingrained and thereby resistant to safety fine-tuning capabilities like deception are. In addition, researchers hypothesize that emergent capabilities correspond to sudden circuit formation during training ⁸⁰, potentially allowing for prediction or control of their development.

Singular Learning Theory (SLT), developed by Watanabe ^{205 206}, provides a rigorous framework for understanding overparameterized models’ behavior and generalization. By quantifying model complexity through the local learning coefficient, SLT offers insights into learning phase transitions and the emergence of structure in the model ²⁰⁷. Recent work by ²⁰⁸ applied this coefficient to identify developmental stages in transformer models, while ²⁰⁹ and ²¹⁰ advanced SLT’s scalability and application to the toy model of superposition (Figure 4), respectively.

While direct applications to phenomena such as generalization ²¹¹, learning functions with increasing complexity ²¹², and the transition from memorization to generalization (grokking) ^{213 214 215 81 216 217 218 219 220 221} are limited, these areas, along with neural scaling laws ^{222 223 80} (which can be connected to mechanistic insights ²²⁴), represent promising future research directions.

In conclusion, developmental interpretability serves as an evolutionary theory lens for neural networks, offering insights into the emergence of structures and behaviors over time ²²⁵. Drawing parallels from systems biology ²²⁶, this approach can apply concepts like network motifs, robustness, and modularity to neural network development, explaining how functional capabilities arise. Sometimes, understanding how structures came about is easier than analyzing the final product, similar to how biologists find certain features in organisms easier to explain in light of their evolutionary history. By studying the temporal aspects of neural network training, researchers can potentially uncover fundamental principles of learning and representation that may not be apparent from examining static, trained models alone.

5.3 Post-Hoc Interpretability

In applied mechanistic interpretability, researchers explore various facets and methodologies to uncover the inner workings of AI models. Some key distinctions are drawn between global versus local interpretability and comprehensive versus partial interpretability. Global interpretability aims to uncover general patterns and behaviors of a model, providing insights that apply broadly across many instances ^{227 172}. In contrast, local interpretability explains the reasons behind a model’s decisions for particular instances, offering insights into individual predictions or behaviors. Comprehensive interpretability involves achieving a deep and exhaustive understanding of a model’s behavior, providing a holistic view of its inner workings ¹⁷². In contrast, partial interpretability often applied to larger and more complex models, concentrates on interpreting specific aspects or subsets of the model’s behavior, focusing on the application’s most relevant or critical areas.

Large Models – Narrow Behavior.

Circuit-style mechanistic interpretability aims to explain neural networks by reverse engineering the underlying mechanisms at the level of individual neurons or subgraphs. This approach assumes that neural vector representations encode high-level concepts and circuits defined by model weights encode meaningful algorithms ^{10 96}. Studies on deep networks support these claims, identifying circuits responsible for detecting curved lines or object orientation ^{96 95 98}.

This paradigm has been applied to language models to discover subnetworks (circuits) responsible for specific capabilities. Circuit analysis localizes and understands subgraphs within a model’s computational graph responsible for specific behaviors. For large language models, this often involves narrow investigations into behaviors like multiple choice reasoning ⁸⁵, indirect object identification ⁸³, or computing operations ⁸⁴. Other examples include analyzing circuits for Python docstrings ²²⁸, “an” vs “a” usage ²²⁹, and price tagging ¹⁸⁴. Case studies often construct datasets using templates filled by placeholder values to enable precise control for causal interventions ^{83 84 184}.

Toy Models – Comprehensive Analysis.

Small models trained on specialized mathematical or algorithmic tasks enable more comprehensive reverse engineering of learned algorithms ^{81 230 106}. Even simple arithmetic operations can involve complex strategies and multiple algorithmic solutions ^{81 230}. Characterizing these algorithms helps test hypotheses around generalizable mechanisms like variable binding ^{88 87} and arithmetic reasoning ¹⁷⁰. The work by ²¹⁶ builds on the work that analyzes transformers trained on modular addition ⁸¹ and explains grokking in terms of circuit efficiency, illustrating how a comprehensive understanding of a toy model can enable interesting analyses on top of that understanding.

Towards Universality.

The ultimate goal is to uncover general principles that transfer across models and tasks, such as induction heads for in-context learning ⁹⁷, variable binding mechanisms ^{88 87}, arithmetic reasoning ^{170 231}, or retrieval tasks ¹⁰⁷. Despite promising results, debates surround the universality hypothesis – the idea that different models learn similar features and circuits when trained on similar tasks. ¹⁰⁶ finds mixed evidence for universality in group composition, suggesting that while families of circuits and features can be characterized, precise circuits and development order may be arbitrary.

Towards High-level Mechanisms.

Causal interventions can extract a high-level understanding of computations and representations learned by large language models ^{107 70 88 34}. Recent work focuses on intervening in internal representations to study high-level concepts and computations encoded. For example, ⁷⁰ patched residual stream vectors to transfer task representations, while ⁸⁸ intervened on residual streams to argue that models generate IDs to bind entities to attributes. Techniques for representation engineering ³⁴ extract reading vectors from model activations to stimulate or inhibit specific concepts. Although these interventions don’t operate via specific mechanisms, they offer a promising approach for extracting high-level causal understanding and bridging bottom-up and top-down interpretability approaches.

5.4 Automation: Scaling Post-Hoc Interpretability

As models become more complex, automating key aspects of the interpretability workflow becomes increasingly crucial. Tracing a model’s computational pathways is highly labor-intensive, quickly becoming infeasible as the model size increases. Automating the discovery of relevant circuits and their functional interpretation represents a pivotal step towards scalable and comprehensive model understanding ²³².

Dissecting Models into Interpretable Circuits.

The first major automation challenge is identifying the critical computational sub-circuits or components underpinning a model’s behavior for a given task. A pioneering line of work aims to achieve this via efficient masking or patching procedures. Methods like automated circuit discovery ¹⁷⁷ and attribution patching ^{179 233} iteratively knock out model activations, pinpointing components whose removal has the most significant impact on performance. This masking approach has proven scalable even to large models ⁸⁵.

Other techniques take a more top-down approach. ⁸⁷ specify high-level causal properties (desiderata) that components solving a target subtask should satisfy and then learn binary masks to expose those component subsets. ¹⁸⁰ construct information flow graphs highlighting key nodes and operations by tracing attribution flows, enabling extraction of general information routing patterns across prediction domains.

Explicit architectural biases like modularity can further boost automation efficiency. ²³² find that models trained with brain-inspired modular training ¹³⁷ produce more readily identifiable circuits compared to standard training. Such domain-inspired inductive biases may prove increasingly vital as models grow more massive and monolithic.

Interpreting Extracted Circuits.

Once critical circuit components have been isolated, the key remaining step is interpreting what computation those components perform. Sparse autoencoders are a prominent approach for interpreting extracted circuits by decomposing neural network activations into individual component features, as discussed in Section 4.2.

A novel paradigm uses large language models themselves as an interpretive tool. ⁴⁸ demonstrate generating natural language descriptions of individual neuron functions by prompting language models like GPT-4 to explain sets of inputs that activate a neuron. ²³⁴ similarly employ language models to annotate unsupervised neuron clusters identified via hierarchical clustering. ²³⁵ describe the roles of neurons in vision networks with multimodal models. These methods can easily leverage more capable general-purpose models in the future. ²³⁶ take a complementary graph-based approach in their neuron-to-graph tool: automatically extracting individual neurons’ behavior patterns from training data as structured graphs amenable to visualization, programmatic comparisons, and property searches. Such representations could synergize with language model-based annotation to provide descriptions of neuron roles.

However, robustly interpreting the largest trillion-parameter models using automated techniques remains an open challenge. Another novel approach, mechanistic-interpretability-based program synthesis ²³⁷, entirely sidesteps this complexity by auto-distilling the algorithm learned by a trained model into human-readable Python code without relying on further interpretability analyses or model architectural knowledge. As models become increasingly vast and opaque, such synergistic combinations of methods – uncovering circuits, annotating them, or altogether transcribing them into executable code – will likely prove crucial for maintaining insight and oversight when scaling model size.

6 Relevance to AI Safety

How Could Interpretability Promote AI Safety?

Figure 11: Potential benefits and risks of mechanistic interpretability for AI safety.

Gaining mechanistic insights into the inner workings of AI systems seems crucial for navigating AI safety as we develop more powerful models ²³⁸. Interpretability tools can provide an understanding of artificial cognition, the way AI systems process information and make decisions, which offers several potential benefits:

Mechanistic interpretability could accelerate AI safety research by providing richer feedback loops and grounding for model evaluation ²³⁹. It may also help anticipate emergent capabilities, such as the emergence of new skills or behaviors in the model before they fully manifest ^{203 201 81 240}. This relates to studying the incremental development of internal structures and representations as the model learns (Section 5.2). Additionally, interpretability could substantiate theoretical risk models with concrete evidence, such as demonstrating inner misalignment (when a model’s behavior deviates from its intended goals) or mesa-optimization (the emergence of unintended subagents within the model) ^{241 242}. It may also trigger normative shifts within the AI community toward rigorous safety protocols by revealing potential risks or concerning behaviors ³⁹.

Regarding specific AI risks ⁴, interpretability may prevent malicious misuse by locating and erasing sensitive information stored in the model ^{130 243}. It could reduce competitive pressures by substantiating potential threats, promoting organizational safety cultures, and supporting AI alignment (ensuring AI systems pursue intended goals) through better monitoring and evaluation ⁵. Interpretability can provide safety filters for every stage of training: before training by deliberate design ³⁹, during training by detecting early signs of misalignment and potentially shifting the distribution towards alignment ^{64 244}, and after training by rigorous evaluation of artificial cognition for honesty ^{31 34} and screening for deceptive behaviors ²⁴⁵.

The emergence of internal world models in LLMs, as posited by the simulation hypothesis, could have significant implications for AI alignment research. Finding an internal representation of human values and aiming the AI system’s objective may be a trivial way to achieve alignment ²⁴⁶, especially if the world model is internally separated from notions of goals and agency ²⁴⁷. In such cases, world model interpretability alone may be sufficient for alignment ²⁴⁸.

Conditioning pre-trained models is considered a comparatively safe pathway towards general intelligence, as it avoids directly creating agents with inherent goals or agendas ^{249 250}. However, prompting a model to simulate an actual agent, such as “You are a superintelligence in 2035 writing down an alignment solution,” could inadvertently lead to the formation of internal agents ²⁵⁰. In contrast, reinforcement learning tends to create agents by default ^{251 252}.

The prediction orthogonality hypothesis suggests that prediction-focused models like GPT can simulate agents with potentially misaligned objectives ¹¹³. Although GPT may lack genuine agency or intentionality, it may produce outputs that simulate these qualities ^{253 114}. This underscores the need for careful oversight and, better yet, using mechanistic interpretability to search for internal agents or their constituents, such as optimization or search processes – an endeavor known as searching for search ^{254 255}.

Mechanistic interpretability integrates well into various AI alignment agendas, such as understanding existing models, controlling them, making AI systems solve alignment problems, and developing alignment theories ^{256 257}. It could enhance strategies like detecting deceptive alignment (hypothetical when a model ensures to appear aligned as to pursue misaligned goals without raising suspicion) ²⁴⁵, eliciting latent knowledge from models ²⁵⁸, and enabling better scalable oversight, such as in iterative distillation and amplification ²⁵⁹. A high degree of understanding may even allow for well-founded AI approaches (AI systems with provable guarantees) ²⁶⁰ or microscope AI (extract world knowledge from the model without letting the model take actions) ³⁹. Furthermore, comprehensive interpretability itself may be an alignment strategy if we can identify internal representations of human values and guide the model to pursue those values by retargeting an internal search process ²⁴⁶. Ultimately, understanding and control are intertwined, and deeper understanding can control AI systems more reliably.

However, there is a spectrum of potential misalignment risks, ranging from acute, model-centric issues to gradual, systemic concerns ²⁶¹. While mechanistic interpretability may address risks stemming directly from model internals – such as deceptive alignment or sudden capability jumps – it may be less helpful for tackling broader systemic risks like the emergence of misaligned economic structures or novel evolutionary dynamics ²⁶². The multi-scale risk landscape calls for a balanced research portfolio to minimize risk, where research on governance, complex systems, and multi-agent simulations complements mechanistic insights and model evaluations. The perceived utility of mechanistic interpretability for AI safety largely depends on researchers’ priors regarding the likelihood of these different risk scenarios.

How Could Mechanistic Insight be Harmful?

Mechanistic interpretability research could accelerate AI capabilities, potentially leading to the development of powerful AI systems that are misaligned with human values, posing significant risks ^{263 264 5}. While historically, interpretability research had little impact on AI capabilities, recent exceptions like discoveries about scaling laws ²⁶⁵, architectural improvements inspired by studying induction heads ^{97 266 267 268}, and efficiency gains inspired by the logit lens technique ²⁶⁸ demonstrated its potential to enhance capabilities. Scaling interpretability research may necessitate automation ^{177 48}, potentially enabling rapid self-improvement of AI systems ²⁶⁹. Some researchers recommend selective publication and focusing on lower-risk areas to mitigate these risks ^{270 271 40 81}.

Mechanistic interpretability also poses dual-use risks, where the same techniques could be used for both beneficial and harmful purposes. Fine-grained editing capabilities enabled by interpretability could be used for machine unlearning (removing private data or dangerous knowledge from models) ^{272 273 243 274} but could be misused for censorship. Similarly, while interpretability may help improve adversarial robustness ⁷, it may also facilitate the development of stronger adversarial attacks ^{42 275}.

Misunderstanding or overestimating the capabilities of interpretability techniques can divert resources from critical safety areas or lead to overconfidence and misplaced trust in AI systems ^{276 239}. Robust evaluation and benchmarking (Section 8.2) are crucial to validate interpretability claims and reduce the risks of overinterpretation or misinterpretation.

7 Challenges

7.1 Research Issues

Need for Comprehensive, Multi-Pronged Approaches.

Current interpretability research often focuses on individual techniques rather than combining complementary approaches. To achieve a holistic understanding of neural networks, we propose utilizing a diverse interpretability toolbox that integrates multiple methods (see also Section 4.4), such as: (i) Coordinating observational (e.g., probing, logit lens) and interventional methods (e.g., activation patching) to establish causal relationships. (ii) Combining feature-level analysis (e.g., sparse autoencoders) with circuit-level interventions (e.g., path patching) to uncover representation-mechanism interplay. (iii) Integrating intrinsic interpretability approaches with post-hoc analysis for robust understanding.

For example, coordinated methods could be used for reverse engineering trojaned behaviors ²⁷⁷, where observational techniques identify suspicious activations, interventional methods isolate the relevant circuits, and intrinsic approaches guide the design of more robust architectures.

Cherry-Picking and Streetlight Interpretability.

Another concerning pattern is the tendency to cherry-pick results, relying on a small number of convincing examples or visualizations as the basis for an argument without comprehensive evaluation ⁷. This amounts to publication bias, showcasing an unrealistic highlight reel of best-case performance. Relatedly, many interpretability techniques are primarily evaluated on small toy models and tasks ^{106 40 57 210}, risking missing critical phenomena that only emerge in more realistic and diverse contexts. This focus on cherry-picked results from toy models is a form of streetlight interpretability ²³⁹, examining AI systems under only ideal conditions of maximal interpretability.

7.2 Technical Limitations

Scalability Challenges and Risks of Human Reliance.

A critical hurdle is demonstrating the scalability of mechanistic interpretability to real-world AI systems across model size, task complexity, behavioral coverage, and analysis efficiency ^{40 54}. Achieving a truly comprehensive understanding of a model’s capabilities in all contexts is daunting, and the time and compute required must scale tractably. Automating interpretability techniques is crucial, as manual analysis quickly becomes infeasible for large models. The high human involvement in current interpretability research raises concerns about the scalability and validity of human-generated model interpretations. Subjective, inconsistent human evaluations and lack of ground-truth benchmarks are known issues ⁷. As models scale, it will become increasingly untenable to rely on humans to hypothesize about model mechanisms manually. More work is needed on automating the discovery of mechanistic explanations and translating model weights into human-readable computational graphs ⁴⁰, but progress on that front may also come from outside the field ²⁷⁸.

Obstacles to Bottom-Up Interpretability.

There are fundamental questions about the tractability of fully reverse engineering neural networks from the bottom up, especially as models become more complex ²⁷⁹. Models may learn internal representations and algorithms that do not cleanly map to human-understandable concepts, making them difficult to interpret even with complete transparency ¹²⁷. This gap between human and model ontologies may widen as architectures evolve, increasing opaqueness ²⁸⁰. Conversely, model representations might naturally converge to more human-interpretable forms as capability increases ^{39 88}.

Analyzing Models Embedded in Environments.

Real-world AI systems embedded in rich, interactive environments exhibit two forms of in-context behavior that pose significant interpretability challenges beyond understanding models in isolation. Externally, models may dynamically adapt to and reshape their environments through in-context learning from the interactions and feedback loops with their external environment ²⁸¹. Internally, the hydra effect demonstrates in-context reorganization, where models flexibly reorganize their internal representations in a context-dependent manner to maintain capabilities even after ablating key components ¹⁷⁵. These two instances of in-context behavior – external adaptation to the environment and internal self-reorganization – undermine interpretability approaches that assume fixed circuits. For models deeply embedded in rich real-world settings, their dynamic coupling with the external world via in-context environmental learning and their internal in-context representational reorganization make strong interpretability guarantees difficult to attain through analysis of the initial model alone.

Adversarial Pressure Against Interpretability.

As models become more capable through increased training and optimization, there is a risk they may learn deceptive behaviors that actively obscure or mislead the interpretability techniques meant to understand them. Models could develop adversarial “mind-reader” components that predict and counteract the specific analysis methods used to interpret their inner workings ^{244 64}. Optimizing models through techniques like gradient descent could inadvertently make their internal representations less interpretable to external observers ^{282 283 242}. In extreme cases, a highly advanced AI system singularly focused on preserving its core objectives may directly undermine the fundamental assumptions that enable interpretability methods in the first place.

These adversarial dynamics, where the capabilities of the AI model are pitted against efforts to interpret it, underscore the need for interpretability research to prioritize worst-case robustness rather than just average-case scenarios. Current techniques often fail even when models are not adversarially optimized. Achieving high confidence in fully understanding extremely capable AI models may require fundamental advances to make interpretability frameworks resilient against an intelligent system’s active deceptive efforts.

8 Future Directions

Given the current limitations and challenges, several key research problems emerge as critical for advancing mechanistic interpretability. These problems span four main areas: emphasizing conceptual clarity Section 8.1, establishing rigorous standards Section 8.2, improving the scalability of interpretability techniques Section 8.3, and expanding the research scope Section 8.4. Each subsection presents specific research questions and challenges that need to be addressed to move the field forward.

Figure 12: Roadmap for advancing mechanistic interpretability research, highlighting key strategic directions.

8.1 Clarifying Concepts

Integrating with Existing Literature.

To mature, mechanistic interpretability should embrace existing work, using established terminology rather than reinventing the wheel. Diverging terminology inhibits collaboration across disciplines. Presently, the terminology used for mechanistic interpretability partially diverges from mainstream AI research ²³⁹. For example, while the mainstream speaks of distributed representations ^{284 285} and the goal of disentangled representations ^{286 287}, the mechanistic interpretability literature refers to the same phenomenon as polysemanticity ^{54 288 53} and superposition ^{40 55}. Using common language invites “accidental” contributions and prevents isolating mechanistic interpretability from broader AI research.

Mechanistic interpretability relates to many other fields in AI research, including compressed sensing ⁴⁰, modularity, adversarial robustness, continual learning, network compression ⁷, neurosymbolic reasoning, trojan detection, and program synthesis ^{239 237}, and causal representation learning. These relationships can help develop new methods, metrics, benchmarks, and theoretical frameworks. For instance:

1. Neurosymbolic Reasoning and Program Synthesis: Mechanistic interpretability aims for reverse engineering neural networks by converting their weights into human-readable algorithms. This endeavor can draw inspiration from neurosymbolic reasoning ¹⁹⁴ and program synthesis. Techniques like creating programs in domain-specific languages ^{289 290 291}, extracting decision trees ²⁹² or symbolic causal graphs ²⁹³ from neural networks align well with the goals of mechanistic interpretability. Adopting these approaches can extend the toolkit for reverse engineering AI systems.
2. Causal Representation Learning: Causal Representation Learning (CRL) aims to discover and disentangle underlying causal factors in data ²⁹⁴, complementing mechanistic interpretability’s goal of understanding causal structures within neural networks. While mechanistic interpretability typically examines individual features and circuits, CRL offers a framework for understanding high-level causal structures. CRL techniques could enhance interpretability by identifying causal relationships between neurons or layers ^{295 296}, potentially revealing model reasoning. Its focus on interventions and counterfactuals ^{297 298} could inspire new methods for probing model internals ^{299 300}. CRL’s emphasis on learning invariant representations ^{301 302} could guide the search for robust features, while its approach to transfer learning ^{303 304} could inform studies into model generalization.
3. Trojan Detection: Detecting deceptive alignment models is a key motivation for inspecting model internals, as – by definition – deception is not salient from observing behavior alone ⁶. However, quantifying progress is challenging due to the lack of evidence for deception as an emergent capability in current models ²⁰¹, apart from sycophancy ^{305 306} and theoretical evidence for deceptive inflation behavior ³⁰⁷. Detecting trojans (or backdoors) ³⁰⁸ implanted via data poisoning could be a proxy goal and proof-of-concept. These trojans simulate outer misalignment (where the model’s behavior is misaligned with the specified reward function or objectives due to poorly defined or incorrect reward signals) rather than inner misalignment such as deceptive alignment (where the model appears aligned with the specified objectives but internally pursues different, misaligned goals). Moreover, activating a trojan typically results in an immediate change of behavior, while deception can be subtle, gradual, and, at first, entirely internal. Nevertheless, trojan detection can still provide a practical testbed for benchmarking interpretability methods ³⁰⁹.
4. Adversarial Robustness: There is a duality between interpretability and adversarial robustness ^{40 7 310}. More interpretable models tend to be more robust against adversarial attacks ³¹¹, and vice versa, adversarially trained models are often more interpretable ³¹². For instance, techniques like input gradient regularization have been shown to simultaneously improve the interpretability of saliency maps and enhance adversarial robustness ^{313 314}. Furthermore, interpretability tools can help create more sophisticated adversaries ^{315 316}, improving our understanding of model internals. Viewing adversarial examples as inherent neural network features ³⁸ rather than bugs also hints at alien features beyond human perception. Connecting mechanistic interpretability to adversarial robustness thus promises ways to gain theoretical insight, measure progress ²³⁹, design inherently more robust architectures ³¹⁷, and create interpretability-guided approaches for identifying (and mitigating) adversarial vulnerabilities ³¹⁸.

More details on the interplay between interpretability, robustness, modularity, continual learning, network compression, and the human visual system can be found in the review by ⁷.

Corroborate or Refute Core Assumptions.

Features are the fundamental units defining neural representations and enabling mechanistic interpretability’s bottom-up approach ²⁵⁹, but defining them involves assumptions requiring scrutiny, as they shape interpretations and research directions. Questioning hypotheses by seeking additional evidence or counter-examples is crucial.

The linear representation hypothesis treats activation directions as features ^{319 61 40}, but the emergence and necessity of linearity is unclear – is it architectural bias or inherent? Stronger theory justifying linearity’s necessity or counter-examples like autoencoders on uncorrelated data without intermediate linear layers ⁴⁰ are needed. An alternative lens views features as polytopes from piecewise linear activations ⁶⁵, questioning if direction simplification suffices or added polytope complexity aids interpretability.

The superposition hypothesis suggests that polysemantic neurons arise from the network compressing and representing many features within its limited set of neurons ⁴⁰, but polysemanticity can also occur incidentally due to redundancy ^{288 53 175}. Understanding superposition’s role could inform mitigating polysemanticity via regularization ²⁸⁸. Superposition also raises open questions like operationalizing computation in superposition ^{320 321}, attention head superposition ^{40 322 85 323}, representing feature clusters ⁴⁰, connections to adversarial robustness ^{40 318 324}, anti-correlated feature organization ⁴⁰, and architectural effects ³²⁵.

8.2 Setting Standards

Prioritizing Robustness over Capability Advancement.

As the mechanistic interpretability community expands, it is essential to maintain the norm of not advancing AI capabilities while simultaneously establishing metrics necessary for the field’s progress ⁷. Researchers should prioritize developing comprehensive tools for analyzing the worst-case performance of AI systems, ensuring robustness and reliability in critical applications. This includes focusing on adversarial tasks, such as backdoor detection and removal ^{326 308 327}, and evaluating the accuracy of explanations in producing adversarial examples ¹³¹.

Establishing Metrics, Benchmarks, and Algorithmic Testbeds.

A central challenge in mechanistic interpretability is the lack of rigorous evaluation methods. Relying solely on intuition can lead to conflating hypotheses with conclusions, resulting in cherry-picking and optimizing for best-case rather than average or worst-case performance ^{328 329 7 239}. Current ad hoc practices and proxy measures ²²⁷ risk over-optimization (Goodhart’s law – When a measure becomes a target, it ceases to be a good measure). Distinguishing correlation from causation is crucial, as interpretability illusions demonstrate that visualizations may be meaningless without causal linking ^{330 331 332}.

To advance the field, rigorous evaluation methods are needed. These should include: (i) assessing out-of-distribution inputs, as most current methods are only valid for specific examples or datasets ^{7 38 42 277 31}; (ii) controlling systems through edits, such as implanting or removing trojans ³³³ or targeted editing ^{44 43 130 334 335 336}; (iii) replacing components with simpler reverse-engineered alternatives ³³⁷; and (iv) comprehensive evaluation through replacing components with hypothesized circuits ³³⁸.

Algorithmic testbeds are essential for evaluating faithfulness ^{339 340} and falsifiability ³⁴¹. Tools like Tracr ³³⁷ can provide ground truth labels for benchmarking search methods ¹³¹, while toy models studying superposition in computation ³²⁰ and transformers on algorithmic tasks can quantify sparsity and test intrinsic methods. Recently, ^{342 343} introduced datasets of transformer weights with known circuits for evaluating mechanistic interpretability techniques.

8.3 Scaling Techniques

Broader and Deeper Coverage of Complex Models and Behaviors.

A primary goal in scaling mechanistic interpretability is pushing the Pareto frontier between model and task complexity and the coverage of interpretability techniques ²⁵⁹. While efforts have focused on larger models, it is equally crucial to scale to more complex tasks and provide comprehensive explanations essential for provable safety ^{260 344 345} and enumerative safety ^{60 40} by ensuring models won’t engage in dangerous behaviors like deception. Future work should aim for thorough reverse engineering ⁸², integrating proven modules into larger networks ⁸¹, and capturing sequences encoded in hidden states beyond immediate predictions ¹⁴⁶. Deepening analysis complexity is also key, validating the realism of toy models ⁴⁰ and extending techniques like path patching ^{131 137} to larger language models. The field must move beyond small transformers on algorithmic tasks ⁸¹ and limited scenarios ³³¹ to tackle more complex, realistic cases.

Towards Universality.

As mechanistic interpretability matures, the field must transition from isolated empirical findings to developing overarching theories and universal reasoning primitives beyond specific circuits, aiming for a comprehensive understanding of AI capabilities. While collecting empirical data remains valuable ¹³, establishing motifs, empirical laws, and theories capturing universal model behavior aspects is crucial. This may involve finding more circuits/features ^{346 347}, exploring circuits as a lens for memorization/generalization ⁸⁴, identifying primitive general reasoning skills ⁸⁸, generalizing specific findings to model-agnostic phenomena ³⁴⁸, and investigating emergent model generality across neural network classes ¹¹⁹. Identifying universal reasoning patterns and unifying theories is key to advancing interpretability.

Automation.

Implementing automated methods is crucial for scaling interpretability of real-world state-of-the-art models across size, task complexity, behavior coverage, and analysis time ³⁴⁹. Manual circuit identification is labor-intensive ⁸⁵, so automated techniques like circuit discovery and sparse autoencoders can enhance the process ^{236 350}. Future work should automatically create varying datasets for understanding circuit functionality ¹⁷⁷, develop automated hypothesis search ¹³¹, and investigate attention head/MLP interplay ³⁵¹. Scaling sparse autoencoders to extract high-quality features automatically for frontier models is critical ⁵⁸. Still, it requires caution regarding potential downsides like AI iteration outpacing training ²⁶⁹ and loss of human interpretability from tool complexity ²²⁷.

8.4 Expanding Scope

Interpretability Across Training.

While mechanistic interpretability of final trained models is a prerequisite, the field should also advance interpretability before and during training by studying learning dynamics ^{352 40 64}. This includes tracking neuron development ³⁵³, analyzing neuron set changes with scale ⁸⁰, and investigating emergent computations ⁸². Studying phase transitions could yield safety insights for reward hacking risks ⁹⁷.

Multi-Level Analysis.

Complementing the predominant bottom-up methods ⁸⁴, mechanistic interpretability should explore top-down and hybrid approaches, a promising yet neglected avenue. The top-down analysis offers a tractable way to study large models and guide microscopic research with macroscopic observations ¹⁰⁷. Its computational efficiency could enable extensive “comparative anatomy” of diverse models, revealing high-level motifs underlying abilities. These motifs could serve as analysis units for understanding internal modifications from techniques like instruction fine-tuning ³⁵⁴ and reinforcement learning from human feedback ^{355 356}.

New Frontiers: Vision, Multimodal, and Reinforcement Learning Models.

While some mechanistic interpretability has explored convolutional neural networks for vision ^{95 96}, vision-language models ^{357 358 359}, and multimodal neurons ⁴⁷, little work has focused on vision transformers ^{357 360 361 362}. Future efforts could identify mechanisms within vision-language models, mirroring progress in unimodal language models ^{81 83}.

Reinforcement learning (RL) is also a crucial frontier given its role in advanced AI training via techniques like reinforcement learning from human feedback (RLHF) ^{355 356}, despite potentially posing significant safety risks ^{253 251}. Interpretability of RL should investigate reward/goal representations ^{363 364 365 366 324}, study circuitry changes from alignment algorithms ^{367 368 369 370}, and explore emergent subgoals or proxies ^{241 119} such as internal reward models ³⁷¹. While current state-of-the-art AI systems as prediction-trained LLMs are considered relatively safe ²⁵⁰, progress on interpreting RL systems may prove critical for safeguarding the next paradigm ³⁷².

Acknowledgements

I am grateful for the invaluable feedback and comments from Leon Lang, Tim Bakker, Jannik Brinkmann, Can Rager, Louis van Harten, Jacqueline Bereska, Benjamin Shaffrey, Thijmen Nijdam, Alice Rigg, Arthur Conmy, and Tom Lieberum. Their insights substantially improved this work.

Glossary

circuits

Sub-graphs within neural networks consisting of and the weights connecting them. Circuits can be thought of as computational primitives that perform understandable operations to produce (ideally interpretable) features from prior (ideally interpretable) features. Examples include circuits for detecting curves at specific orientations ^{96 95}, continuing repeated patterns in text ⁹⁷, and resolving anaphoric references ⁸³. While circuits can involve clearly interpretable features, the definition allows for intermediate representations that are less easily interpretable.

concepts

An abstract idea or representation derived from observations of the world. Concepts refer to the natural abstractions that a cognitive system, like a neural network, aims to capture and represent through its learned features, which may or may not align perfectly with human-defined concepts.

deceptive alignment

When a misaligned model aims to appear aligned to gain more power to take control once sufficiently powerful.

deceptive inflation

Theoretical result on deceptive behavior: policies produce trajectories that look better than they actually are from the human’s perspective with limited observations to get higher reward signals during training. This deceptive behavior arises in reinforcement learning from human feedback when the human provides feedback based only on partial observations of the trajectories, while the policy has full state information during training ³⁰⁷.

disentangled

In disentangled representations, individual dimensions or components correspond to distinct, independent factors of variation in the data, rather than representing a tangled mixture of these factors.

eliciting latent knowledge

Developing strategies to make a machine learning model explicitly report latent facts or knowledge embedded in its parameters, especially in cases where the model’s output is untrusted ²⁵⁸. This involves finding patterns in neural network activations that track the true state of the world ³⁷³.

features

The fundamental units of how neural networks encode knowledge, which cannot be further decomposed into smaller, distinct. Features are core components of a neural network’s representation, analogous to how cells form the fundamental unit of biological organisms ¹⁰. The hypothesis suggests an alternative definition: that features correspond to the concepts that a larger, sparser network with sufficient capacity would learn to represent with individual () neurons ^{10 58}.

grokking

“Grokking refers to the surprising phenomenon of delayed generalization where neural networks, on certain learning problems, generalize long after overfitting their training set.” ²¹³

hydra effect

The phenomenon where models can internally self-repair and maintain capabilities even when key components are ablated, making it challenging to identify the relevant components underlying a particular behavior ¹⁷⁵.

inner misalignment

Inner misalignment, or goal misgeneralization, occurs when an AI system develops goals or behaviors during training that are misaligned with the intended objectives despite a correctly specified reward signal.

internal world models

Internal causal environment models formed within neural networks, implicitly emerging as a by-product of prediction (e.g., in large language models).

irreducible

We adopt the notion of features as the fundamental units of neural network representations, such that features cannot be further decomposed into smaller, distinct factors. To make this more precise, we can formalize the definition of features as irreducible input patterns following ³⁷: A feature $f$ of sparsity $s$ is a function that maps a subset of the input space (with probability $1-s>0$) into a higher-dimensional representational space. We say the feature is active on this subset. A feature $f$ is reducible into features $a$ and $b$ if there exists a transformation that decomposes $f$ into $a$ and $b$, such that the transformed distribution $p(a,b)$ is either:

1. Separable: $p(a,b)=p(a)p(b)$
2. A mixture: $p(a,b)=wp_{1}(a,b)+(1-w)p_{2}(a,b)$ where $p_{1}$ is lower-dimensional. Features are defined as irreducible patterns that cannot be decomposed into separable or mixture distributions via such transformations. This formalizes the notion that features form the fundamental atomic units underlying neural representations. Features that can be disentangled into statistically independent components (separable) or simpler lower-dimensional factors (mixtures) are not considered the core representational primitives. The key properties are that 1) features map from the input space to higher-dimensional representational spaces, 2) features are sparse and only activated on subsets of the input, and crucially, 3) features are irreducible and cannot be expressed as transformations of other statistically independent components.

iterative distillation and amplification

A technique for training AI systems by repeatedly distilling knowledge from a larger model into a smaller one while amplifying the smaller model’s capabilities through feedback and interaction with humans.

linear representation

Features are directions in activation space, i.e., linear combinations of neurons.

machine unlearning

Techniques for removing private data or dangerous knowledge from models.

mesa-optimization

The emergence of unintended subagents within a model with their own objectives, potentially misaligned with the original training objective.

microscope AI

Systems that extract and utilize knowledge from a model without allowing the model to take autonomous actions. This involves reverse engineering a trained model to understand its learned knowledge about the world, aiming to leverage this understanding directly without deploying the model in an operational capacity.

modularity

The property of an AI system being composed of distinct, semi-independent components or submodules that can be separately understood, modified, and recombined, rather than a monolithic, opaque structure.

monosemantic

A neuron corresponding to a single concept. The intuition is that analyzing what inputs activate a given neuron reveals its associated semantic meaning or concept. In contrast to.

motifs

Repeating patterns that emerge across models and tasks, manifesting as circuits, features, or higher-level behaviors from component interactions. Examples include curve detectors, induction circuits, and branch specialization. Motifs reveal common structures and mechanisms underlying neural network intelligence.

natural abstractions

High-level summaries or descriptions of a system or environment learned and used by many cognitive systems. According to the natural abstraction hypothesis ³⁵, a set of “natural” abstractions exist that represent redundantly encoded information in the world and tend to be learned by intelligent systems produced through local selection pressures. These natural abstractions form a relatively small, discrete set of concepts like “tree,” “velocity,” etc., that allow compact descriptions of the world while discarding many irrelevant low-level details.

outer misalignment

Outer misalignment, or reward hacking, occurs when the specified reward function or utility function fails to capture the desired objectives correctly. This leads the AI to optimize for behaviors that achieve high reward scores but are misaligned with the intended outcomes.

oversight

(Scalable) oversight refers to the challenge of providing reliable supervision—through labels, reward signals, or critiques—to AI models, ensuring effectiveness even as models surpass human-level performance.

polysemantic

Neurons that are associated with multiple, unrelated concepts, contradicting the interpretation of neurons as representational primitives and making it challenging to understand the information processing of neural networks. This term is derived from linguistic concepts of polysemy ³⁷⁴, and in the context of neural networks first introduced by ⁵⁰, who suggested that word embeddings of polysemous words may be stored as a of vectors representing distinct meanings. ¹⁰ first used the term polysemanticity, elaborating on the concept of polysemantic neurons as a challenge for mechanistic interpretability.

prediction orthogonality

A model whose objective is prediction can simulate agents who optimize toward any objectives with any degree of optimality ¹¹³.

privileged basis

In certain neural network representations, the basis directions formed by the individual neurons are architecturally distinguished from arbitrary directions in the activation space. This privileged basis makes it meaningful to analyze the properties and roles of individual neurons, as the architecture encourages features to align with these basis directions. Hence, a privileged basis is necessary but not sufficient for the formation of monosemantic neurons. ⁴⁰.

representation engineering

A top-down approach to transparency research that treats representations as the fundamental unit of analysis, aiming to understand and control representations of high-level cognitive phenomena in neural networks like large language models. Representation engineering has two main areas: 1) Reading representations to probe and interpret their contents, and 2) Controlling representations to manipulate high-level concepts like honesty or morality ³⁴.

reverse engineering

The process of deconstructing a neural network’s computations to fully understand and specify its operations. This involves breaking down the network’s functionality into explicit, interpretable components, potentially as clear and detailed as pseudocode.

reward hacking

See outer misalignment.

simulacra

The text outputs generated by a predictive model simulating the causal processes underlying text creation. These outputs simulate coherent and contextually relevant language, sometimes exhibiting agentic behaviors or goals despite the predictive model itself lacking genuine agency or intentionality. Simulacra can be either agentic, mimicking intentional and persuasive language use, or non-agentic, merely generating descriptive text without simulated goals or agency ^{113 253}.

simulation

The simulation hypothesis says that when scaled up sufficiently, predictive models will learn to simulate the real-world causal processes that generated their training data ¹¹³. When these models are optimized for predictive accuracy on broad data distributions like natural language, they are incentivized to discover the underlying rules, physics, and semantics that govern the data to model and predict future observations effectively. This allows the models to go beyond just memorizing or pattern-matching their training sets, instead learning to simulate hypothetical scenarios, reason about counterfactuals, and exhibit behaviors characteristic of general intelligence – all as a byproduct of the drive for efficient compression and accurate prediction. The simulation hypothesis suggests these models will develop rich internal world models capturing the causal dynamics of the training distribution.

streetlight interpretability

Examining AI systems under only ideal conditions of maximal interpretability, risking missing critical phenomena that only emerge in more realistic and diverse contexts.

superposition

The superposition hypothesis suggests that neural networks can leverage high-dimensional spaces to represent more features than the actual count of neurons by encoding features in almost orthogonal directions ⁴⁰.

sycophancy

The tendency of models to generate responses that align with user beliefs rather than providing truthful information. This behavior, encouraged by human feedback used in fine-tuning, is observed in state-of-the-art AI assistants across various tasks ³⁰⁵. Sycophancy arises because human preference judgments often favor responses that match users’ views, leading to a preference for convincingly written sycophantic responses over correct ones.

universality

The universality hypothesis proposes the emergence of common circuits across neural network models trained on similar tasks and data distributions. A stronger form posits that these common circuits represent a set of fundamental computational motifs that neural networks gravitate towards when learning. The weaker version suggests that for a given task, dataset, and model architecture, an optimal way to solve the problem may exist, which different models will tend to converge towards, resulting in analogous circuits. The universality hypothesis implies that rather than each model learning arbitrary, unstructured representations, there is an underlying universality to the circuits that emerge, shaped by the learning task and inductive biases.

well-founded AI

Developing AI systems with provable safety guarantees about their behavior and alignment with human values through rigorous mathematical modeling and verification. ^{260 344}.

1. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. CoRR, April 2023. ↩

2. Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian Hadfield, Jeff Clune, Tegan Maharaj, Frank Hutter, Atılım Güneş Baydin, Sheila McIlraith, Qiqi Gao, Ashwin Acharya, David Krueger, Anca Dragan, Philip Torr, Stuart Russell, Daniel Kahneman, Jan Brauner, and Sören Mindermann. Managing ai risks in an era of rapid progress. CoRR, November 2023. ↩

3. Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng, Juntao Dai, Xuehai Pan, Aidan O’Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo, and Wen Gao. Ai alignment: A comprehensive survey. CoRR, January 2024. ↩

4. Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An overview of catastrophic ai risks. CoRR, October 2023. ↩ ↩²

5. Dan Hendrycks and Mantas Mazeika. X-risk analysis for ai research. CoRR, June 2022. ↩ ↩² ↩³

6. Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, Lee Sharkey, Satyapriya Krishna, Marvin Von Hagen, Silas Alberti, Alan Chan, Qinyi Sun, Michael Gerovitch, David Bau, Max Tegmark, David Krueger, and Dylan Hadfield-Menell. Black-box access is insufficient for rigorous ai audits. ACM Conference on Fairness, Accountability, and Transparency, January 2024. ↩ ↩²

7. Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. TMLR, August 2023. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹

8. Christopher Olah. Mechanistic interpretability, variables, and the importance of interpretable bases. Transformer Circuits Thread, 2022. ↩ ↩² ↩³

9. Neel Nanda. A comprehensive mechanistic interpretability explainer & glossary. Neel Nanda’s Blog, December 2022d. ↩

10. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, March 2020. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹

11. Lee Sharkey, Sid Black, and beren. Current themes in mechanistic interpretability research. AI Alignment Forum, November 2022a. ↩ ↩²

12. Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, March 2018. ↩

13. Neel Nanda. Mechanistic interpretability quickstart guide. Neel Nanda’s Blog, January 2023f. ↩ ↩²

14. Neel Nanda. An extremely opinionated annotated list of my favourite mechanistic interpretability papers v2. AI Alignment Forum, July 2024. ↩

15. Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R. Costa-jussà. A primer on the inner workings of transformer-based language models. CoRR, May 2024. ↩

16. Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical review of mechanistic interpretability for transformer-based language models. CoRR, July 2024. ↩

17. Chris Olah and Shan Carter. Research debt. Distill, March 2017. ↩

18. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 2020. ↩

19. Giuseppe Casalicchio, Christoph Molnar, and Bernd Bischl. Visualizing the feature importance for black box models. ECML PKDD, 2018. ↩

20. Lloyd S. Shapley. A value for n -person games. Cambridge University Press, October 1988. ↩

21. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “why should i trust you?”: Explaining the predictions of any classifier. NAACL, August 2016. ↩ ↩²

22. Ian C. Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: a unified framework for model explanation. J. Mach. Learn. Res., January 2021. ↩

23. Jaap Jumelet. Evaluating and interpreting language models. NLP Lecture, November 2023. ↩

24. Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. CoRR, June 2017. ↩

25. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. ICML, June 2017. ↩

26. Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, July 2015. ↩

27. Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. ICML, 2017. ↩

28. Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization. ICCV, 2016. ↩

29. Blair Bilodeau, Natasha Jaques, Pang Wei Koh, and Been Kim. Impossibility theorems for feature attribution. Proc. Natl. Acad. Sci. U.S.A., January 2024. ↩

30. Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. CoRR, September 2021. ↩ ↩² ↩³ ↩⁴ ↩⁵

31. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. ICLR, 2023. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

32. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. ICML, July 2019. ↩ ↩²

33. Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. CoRR, June 2021. ↩

34. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to ai transparency. CoRR, October 2023. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

35. Lawrence Chan, Leon Lang, and Erik Jenner. Natural abstractions: Key claims, theorems, and critiques. AI Alignment Forum, March 2023. ↩ ↩² ↩³ ↩⁴ ↩⁵

36. Christopher M. Bishop. Pattern recognition and machine learning. Springer-Verlag New York Inc., 2006. ↩

37. Joshua Engels, Isaac Liao, Eric J. Michaud, Wes Gurnee, and Max Tegmark. Not all language model features are linear. CoRR, May 2024. ↩ ↩² ↩³ ↩⁴

38. Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. NeurIPS, August 2019. ↩ ↩² ↩³

39. Evan Hubinger. Chris olah’s views on agi safety. AI Alignment Forum, November 2019a. ↩ ↩² ↩³ ↩⁴ ↩⁵

40. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. Transformer Circuits Thread, 2022b. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰ ↩¹¹ ↩¹² ↩¹³ ↩¹⁴ ↩¹⁵ ↩¹⁶ ↩¹⁷ ↩¹⁸ ↩¹⁹ ↩²⁰ ↩²¹ ↩²² ↩²³ ↩²⁴ ↩²⁵ ↩²⁶ ↩²⁷

41. Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. Neuron-level interpretation of deep nlp models: A survey. TACL, November 2022. ↩

42. Jesse Mu and Jacob Andreas. Compositional explanations of neurons. NeurIPS, June 2020. ↩ ↩² ↩³ ↩⁴

43. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. ACL, 2022. ↩ ↩²

44. Amirata Ghorbani and James Zou. Neuron shapley: Discovering the responsible neurons. NeurIPS, November 2020. ↩ ↩²

45. Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, n-gram, positional. CoRR, September 2023. ↩

46. Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. Analyzing individual neurons in pre-trained language models. EMNLP, October 2020. ↩

47. Gabriel Goh, Nick Cammarata †, Chelsea Voss †, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, March 2021. ↩ ↩²

48. Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. OpenAI Blog, 2023. ↩ ↩² ↩³

49. Jing Huang, Atticus Geiger, Karel D’Oosterlinck, Zhengxuan Wu, and Christopher Potts. Rigorously assessing natural language explanations of neurons. CoRR, September 2023. ↩

50. Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. TACL, December 2018. ↩ ↩²

51. Nelson Elhage, Tristan Hume, Olsson Catherine, Nanda Neel, Tom Henighan, Scott Johnston, Sheer ElShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hatfield-Dodds, Jackson Kernion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Kadavath, Josh Jacobson, Eli Tran-Johnson, Jared Kaplan, Jack Clark, Tom Brown, Sam McCandlish, Dario Amodei, and Christopher Olah. Softmax linear units. Transformer Circuits Thread, 2022a. ↩ ↩² ↩³ ↩⁴

52. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 2014. ↩

53. Simon C. Marshall and Jan H. Kirchner. Understanding polysemanticity in neural networks through coding theory. CoRR, January 2024. ↩ ↩² ↩³

54. Adam Scherlis, Kshitij Sachan, Adam S. Jermyn, Joe Benton, and Buck Shlegeris. Polysemanticity and capacity in neural networks. CoRR, July 2023. ↩ ↩² ↩³ ↩⁴

55. Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah. Superposition, memorization, and double descent. Transformer Circuits Thread, 2023. ↩ ↩²

56. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. TMLR, 2023. ↩ ↩² ↩³

57. Adam S. Jermyn, Nicholas Schiefer, and Evan Hubinger. Engineering monosemanticity in toy models. CoRR, November 2022. ↩ ↩² ↩³ ↩⁴

58. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, October 2023. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹

59. Lee Sharkey, Dan Braun, and Beren Millidge. Taking features out of superposition with sparse autoencoders. AI Alignment Forum, 2022b. ↩ ↩² ↩³ ↩⁴ ↩⁵

60. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. ICLR, January 2024. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

61. Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, September 2023b. ↩ ↩² ↩³ ↩⁴

62. Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. ICLR, 2023a. ↩ ↩²

63. Neel Nanda. Actually, othello-gpt has a linear emergent world representation. Neel Nanda’s Blog, March 2023c. ↩

64. Evan Hubinger. A transparency and interpretability tech tree. AI Alignment Forum, June 2022. ↩ ↩² ↩³ ↩⁴

65. Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ramón Guevara, Beren Millidge, Gabriel Alfour, and Connor Leahy. Interpreting neural networks through the polytope lens. CoRR, November 2022. ↩ ↩²

66. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. NeurIPS, October 2013. ↩

67. Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. ICLR, 2016. ↩ ↩²

68. Mingyang Deng, Lucas Tao, and Joe Benton. Measuring feature sparsity in language models. CoRR, 2023. ↩ ↩²

69. Laura O’Mahony, Vincent Andrearczyk, Henning Muller, and Mara Graziani. Disentangling neuron representations with concept vectors. CVPR Workshops, April 2023. ↩

70. Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. EMNLP, October 2023. ↩ ↩² ↩³ ↩⁴

71. Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. CoRR, 2023. ↩

72. Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Language models linearly represent sentiment. ICML MI Workshop, June 2024. ↩

73. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. CoRR, 2024. ↩

74. Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. CoRR, August 2023. ↩

75. David Chanin, Anthony Hunter, and Oana-Maria Camburu. Identifying linear relational concepts in large language models. CoRR, 2023. ↩

76. Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. CoRR, September 2023. ↩

77. Mansi Sakarvadia, Aswathy Ajith, Arham Khan, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian Foster. Memory injections: Correcting multi-hop reasoning failures during inference in transformer-based language models. CoRR, September 2023a. ↩

78. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. NeurIPS Spotlight, July 2023b. ↩

79. Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. ICML MI Workshop (Oral), June 2024. ↩

80. Eric J. Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. CoRR, March 2023. ↩ ↩² ↩³ ↩⁴

81. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. ICLR, January 2023a. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰

82. Philip Quirke and Fazl Barez. Understanding addition in transformers. CoRR, October 2023. ↩ ↩² ↩³

83. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. ICLR, 2023. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰

84. Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. NeurIPS, 2023. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

85. Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. CoRR, July 2023. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

86. Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy suppression: Comprehensively understanding an attention head. CoRR, October 2023. ↩

87. Xander Davies, Max Nadeau, Nikhil Prakash, Tamar Rott Shaham, and David Bau. Discovering variable binding circuitry with desiderata. CoRR, July 2023. ↩ ↩² ↩³ ↩⁴

88. Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context? CoRR, October 2023. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

89. Qinan Yu, Jack Merullo, and Ellie Pavlick. Characterizing mechanisms for factual recall in language models. CoRR, October 2023. ↩

90. Neel Nanda, S. Rajamanoharan, J. Kramár, and R. Shah. Fact finding: Attempting to reverse-engineer factual recall on the neuron level. AI Alignment Forum, 2023c. URL https://www. alignmentforum. org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall, 2023c. ↩

91. Ang Lv, Yuhan Chen, Kaiyi Zhang, Yulong Wang, Lifeng Liu, Ji-Rong Wen, Jian Xie, and Rui Yan. Interpreting key mechanisms of factual recall in transformer-based language models. CoRR, 2024. ↩

92. Bilal Chughtai, Alan Cooney, and Neel Nanda. Summing up the facts: Additive mechanisms behind factual recall in llms. NeurIPS Workshop Attributing Model Behaviour at Scale, 2024. ↩

93. Francesco Ortu, Zhijing Jin, Diego Doimo, Mrinmaya Sachan, Alberto Cazzaniga, and Bernhard Schölkopf. Competition of mechanisms: Tracing how language models handle facts and counterfactuals. ACL 2024, June 2024. ↩

94. Michael Lan and Fazl Barez. Locating cross-task sequence continuation circuits in transformers. CoRR, November 2023. ↩

95. Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah. Curve circuits. Distill, 2021. ↩ ↩² ↩³ ↩⁴

96. Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schubert, Michael Petrov, and Chris Olah. Curve detectors. Distill, June 2020. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

97. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

98. Chelsea Voss, Gabriel Goh, Nick Cammarata, Michael Petrov, Ludwig Schubert, and Chris Olah. Branch specialization. Distill, April 2021. ↩ ↩²

99. Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C. Love, Erin Grant, Iris Groen, Jascha Achterberg, Joshua B. Tenenbaum, Katherine M. Collins, Katherine L. Hermann, Kerem Oktar, Klaus Greff, Martin N. Hebart, Nori Jacoby, Qiuyi Zhang, Raja Marjieh, Robert Geirhos, Sherol Chen, Simon Kornblith, Sunayana Rane, Talia Konkle, Thomas P. O’Connell, Thomas Unterthiner, Andrew K. Lampinen, Klaus-Robert Müller, Mariya Toneva, and Thomas L. Griffiths. Getting aligned on representational alignment. CoRR, November 2023. ↩

100. Jan Kirchner. Neuroscience and natural abstractions. LessWrong, March 2023. ↩

101. Yiting Chen, Zhanpeng Zhou, and Junchi Yan. Going beyond neural network feature similarity: The network feature complexity and its interpretation using category theory. CoRR, November 2023a. ↩

102. Guy Hacohen, Leshem Choshen, and Daphna Weinshall. Let’s agree to agree: Neural networks share classification order on real datasets. ICML, 2020. ↩

103. Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? NIPS Workshop on Feature Extraction, December 2015. ↩

104. Giovanni Luca Marchetti, Christopher Hillar, Danica Kragic, and Sophia Sanborn. Harmonics of learning: Universal fourier features emerge in invariant networks. CoRR, December 2023. ↩

105. Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models. CoRR, January 2024. ↩

106. Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. ICML, 2023. ↩ ↩² ↩³ ↩⁴ ↩⁵

107. Alexandre Variengien and Eric Winsor. Look before you leap: A universal emergent decomposition of retrieval tasks in language models. ICML MI Workshop, December 2023. ↩ ↩² ↩³ ↩⁴

108. David R. Ha and J. Schmidhuber. Recurrent world models facilitate policy evolution. NeurIPS, September 2018. ↩

109. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? ACM FAccT, March 2021. ↩

110. Judea Pearl. Causality. Cambridge University Press, 2009. ↩ ↩²

111. Tommaso Salvatori, Ankur Mali, Christopher L. Buckley, Thomas Lukasiewicz, Rajesh P. N. Rao, Karl Friston, and Alexander Ororbia. Brain-inspired computational intelligence via predictive coding. CoRR, 2023. ↩

112. Jan Kulveit, Clem von Stengel, and Roman Leventov. Predictive minds: Llms as atypical active inference agents. CoRR, November 2023. ↩

113. janus. Simulators. LessWrong, September 2022. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

114. Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role play with large language models. Nature, November 2023. ↩ ↩² ↩³

115. Jonathan Richens and Tom Everitt. Robust agents learn causal world models. ICLR Oral, February 2024. ↩

116. Eshaan Nichani, Alex Damian, and Jason D. Lee. How transformers learn causal structure with gradient descent. CoRR, February 2024. ↩

117. Adam Karvonen. Emergent world models and latent variable estimation in chess-playing language models. COLM, July 2024. ↩

118. Wes Gurnee and Max Tegmark. Language models represent space and time. ICLR, 2024. ↩

119. M. Ivanitskiy, Alexander F. Spies, Tilman Rauker, Guillaume Corlouer, Chris Mathwin, Lucia Quirke, Can Rager, Rusheb Shah, Dan Valentine, Cecilia Diniz Behn, Katsumi Inoue, and Samy Wu Fung. Structured world representations in maze-solving transformers. CoRR, December 2023. ↩ ↩² ↩³

120. Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. ACL-IJCNLP, August 2021. ↩

121. Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. ICLR, 2022. ↩

122. Peter Gardenfors. Conceptual spaces: The geometry of thought. MIT press, 2004. ↩

123. Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. ECCV, 2014. ↩

124. Roland S. Zimmermann, Judy Borowski, Robert Geirhos, Matthias Bethge, Thomas S. A. Wallis, and Wieland Brendel. How well do feature visualizations support causal understanding of cnn activations? NeurIPS, November 2021. ↩

125. Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions. CoRR, August 2023. ↩

126. Albert Garde, Esben Kran, and Fazl Barez. Deepdecipher: Accessing and investigating neuron activation in large language models. NeurIPS Workshop XAIA, October 2023. ↩

127. Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. PNAS, November 2022. ↩ ↩² ↩³

128. nostalgebraist. interpreting gpt: the logit lens. AI Alignment Forum, August 2020. ↩ ↩² ↩³

129. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. CoRR, August 2023. ↩ ↩²

130. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. NeurIPS, 2022a. ↩ ↩² ↩³ ↩⁴ ↩⁵

131. Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching. CoRR, 2023. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

132. Atticus Geiger, Chris Potts, and Thomas Icard. Causal abstraction for faithful model interpretation. CoRR, January 2023a. ↩ ↩² ↩³ ↩⁴ ↩⁵

133. Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D. Goodman. Finding alignments between interpretable causal variables and distributed neural representations. CoRR, 2023b. ↩ ↩² ↩³

134. Zhengxuan Wu, Karel D’Oosterlinck, Atticus Geiger, Amir Zur, and Christopher Potts. Causal proxy models for concept-based model explanations. ICML, 2023a. ↩ ↩²

135. Lawrence Chan, Adrià Garriga-alonso, Nicholas Goldowsky-Dill, ryan_greenblatt, jenny, Ansh Radhakrishnan, Buck, and Nate Thomas. Causal scrubbing: a method for rigorously testing interpretability hypotheses [redwood research]. AI Alignment Forum, December 2022. ↩ ↩²

136. Erik Jenner, Adrià Garriga-alonso, and Egor Zverev. A comparison of causal scrubbing, causal abstractions, and related methods. AI Alignment Forum, June 2023. ↩ ↩² ↩³

137. Ziming Liu, Eric Gan, and Max Tegmark. Seeing is believing: Brain-inspired modular training for mechanistic interpretability. Entropy, June 2023a. ↩ ↩² ↩³ ↩⁴ ↩⁵

138. John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. NAACL HLT, June 2019. ↩

139. Steven Cao, Victor Sanh, and Alexander M. Rush. Low-complexity probing via finding subnetworks. NAACL-HLT, April 2021. ↩

140. Elena Voita and Ivan Titov. Information-theoretic probing with minimum description length. EMNLP, March 2020. ↩

141. Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. TACL, February 2021. ↩

142. Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. ACL, August 2019. ↩

143. Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. Proceedings of the AAAI Conference on Artificial Intelligence, July 2019. ↩

144. Sebastian Farquhar, Vikrant Varma, Zachary Kenton, Johannes Gasteiger, Vladimir Mikulik, and Rohin Shah. Challenges with unsupervised llm knowledge discovery. CoRR, 2023. ↩ ↩²

145. Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. EMNLP, October 2022. ↩

146. Koyena Pal, Jiuding Sun, Andrew Yuan, Byron C. Wallace, and David Bau. Future lens: Anticipating subsequent tokens from a single hidden state. CoNLL, 2023. ↩ ↩²

147. Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. Jump to conclusions: Short-cutting transformers with linear transformations. CoRR, March 2023. ↩

148. Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. ACL, December 2022. ↩

149. Anna Langedijk, Hosein Mohebbi, Gabriele Sarti, Willem Zuidema, and Jaap Jumelet. Decoderlens: Layerwise interpretation of encoder-decoder transformers. CoRR, 2023. ↩

150. Mansi Sakarvadia, Arham Khan, Aswathy Ajith, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian Foster. Attention lens: A tool for mechanistically interpreting the attention head information retrieval mechanism. CoRR, October 2023b. ↩

151. B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: a strategy employed by v1? Vision Res, December 1997. ↩

152. James C. R. Whittington, Will Dorrell, Surya Ganguli, and Timothy E. J. Behrens. Disentangling with biological constraints: A theory of functional cell types. CoRR, September 2022. ↩

153. Charles J. Garfinkle and Christopher J. Hillar. On the uniqueness and stability of dictionaries for sparse representation of noisy signals. IEEE Transactions on Signal Processing, December 2019. ↩

154. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, and Brian Chen. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. ↩

155. Luke Marks, Amir Abdullah, Luna Mendez, Rauno Arike, Philip Torr, and Fazl Barez. Interpreting reward models in rlhf-tuned language models using sparse autoencoders. CoRR, October 2023a. ↩

156. Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Smith, C. M. Verdun, David Bau, and Samuel Marks. Measuring progress in dictionary learning for language model interpretability with board game models. ICML MI Workshop (Oral), July 2024. ↩

157. Aleksandar Makelov, George Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. CoRR, May 2024. ↩

158. Aleksandar Makelov. Sparse autoencoders match supervised features for model steering on the ioi task. ICML MI Workshop, June 2024. ↩

159. Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. Interpreting attention layer outputs with sparse autoencoders. ICML MI Workshop, June 2024. ↩

160. Liv Gorton. The missing curve detectors of inceptionv1: Applying sparse autoencoders to inceptionv1 early vision. ICML MI Workshop, June 2024. ↩ ↩²

161. Zhengfu He, Xuyang Ge, Qiong Tang, Tianxiang Sun, Qinyuan Cheng, and Xipeng Qiu. Dictionary learning improves patch-free circuit discovery in mechanistic interpretability: A case study on othello-gpt. CoRR, 2024. ↩

162. Charles O’Neill and Thang Bui. Sparse autoencoders enable scalable and reliable circuit identification in language models. CoRR, May 2024. ↩ ↩²

163. Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoencoders. CoRR, April 2024. ↩

164. Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable llm feature circuits. ICML MI Workshop, June 2024. ↩

165. slavachalnev. Sparse mlp distillation. LessWrong, 2024. ↩

166. Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, and Yonatan Belinkov. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. CoRR, August 2024. ↩

167. Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah D. Goodman, and Christopher Potts. Inducing causal structure for interpretable neural networks. ICML, January 2021b. ↩

168. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. NeurIPS, 2020. ↩ ↩²

169. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. EMNLP, October 2023. ↩ ↩²

170. Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. EMNLP, October 2023. ↩ ↩² ↩³

171. Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. ICLR, 2023. ↩

172. Neel Nanda. How to think about activation patching. AI Alignment Forum, April 2023e. ↩ ↩² ↩³

173. Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models. CoRR, January 2024. ↩

174. Georg Lange, Alex Makelov, and Neel Nanda. An interpretability illusion for activation patching of arbitrary subspaces. AI Alignment Forum, August 2023. ↩

175. Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations. CoRR, July 2023. ↩ ↩² ↩³ ↩⁴

176. Cody Rushing and Neel Nanda. Explorations of self-repair in language models. ICML, May 2024. ↩

177. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. NeurIPS, 2023. ↩ ↩² ↩³ ↩⁴ ↩⁵

178. Neel Nanda. Attribution patching: Activation patching at industrial scale. Neel Nanda’s Blog, February 2023d. ↩

179. Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery. CoRR, October 2023. ↩ ↩² ↩³

180. Javier Ferrando and Elena Voita. Information flow routes: Automatically interpreting language models at scale. CoRR, February 2024. ↩ ↩²

181. Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. NeurIPS, 2021a. ↩

182. Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. Causalm: Causal model explanation through counterfactual language models. Computational Linguistics, May 2021. ↩

183. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. ACL, July 2020. ↩

184. Zhengxuan Wu, Atticus Geiger, Christopher Potts, and Noah D. Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. CoRR, May 2023b. ↩ ↩² ↩³

185. Aryaman Arora, Dan Jurafsky, and Christopher Potts. Causalgym: Benchmarking causal interpretability methods on linguistic tasks. CoRR, February 2024. ↩

186. Zhengxuan Wu, Atticus Geiger, Joshua Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christopher Potts, and Noah Goodman. Causal distillation for language models. NAACL-HLT, July 2022b. ↩ ↩²

187. Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. Ravel: Evaluating interpretability methods on disentangling language model representations. CoRR, 2024. ↩

188. Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. Reft: Representation finetuning for language models. CoRR, May 2024. ↩

189. Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. CoRR, March 2024. ↩

190. Lucius Bushnaq, Stefan Heimersheim, Nicholas Goldowsky-Dill, Dan Braun, Jake Mendel, Kaarel Hanni, Avery Griffin, Jorn Stohler, Magdalena Wache, and Marius Hobbhahn. The local interaction basis: Identifying computationally-relevant and sparsely interacting features in neural networks. CoRR, May 2024. ↩

191. Dan Braun, Jordan Taylor, Nicholas Goldowsky-Dill, and Lee Sharkey. Identifying functionally important features with end-to-end sparse dictionary learning. ICML MI Workshop, May 2024. ↩

192. Xuyang Ge, Fukang Zhu, Wentao Shu, Junxuan Wang, Zhengfu He, and Xipeng Qiu. Automatically identifying local and global circuits with linear computation graphs. CoRR, May 2024. ↩

193. Dan Friedman, Alexander Wettig, and Danqi Chen. Learning transformer programs. NeurIPS, June 2023b. ↩

194. Ryan Riegel, Alexander Gray, Francois Luus, Naweed Khan, Ndivhuwo Makondo, Ismail Yunus Akhalwaya, Haifeng Qian, Ronald Fagin, Francisco Barahona, Udit Sharma, Shajith Ikbal, Hima Karanam, Sumit Neelam, Ankita Likhyani, and Santosh Srivastava. Logical neural networks. NeurIPS, June 2020. ↩ ↩²

195. Ziming Liu, Mikail Khona, Ila R. Fiete, and Max Tegmark. Growing brains: Co-emergence of anatomical and functional modularity in recurrent neural networks. CoRR, 2023b. ↩ ↩²

196. Lee Sharkey. A technical note on bilinear layers for interpretability. CoRR, May 2023. ↩

197. Biao Zhang, Ivan Titov, and Rico Sennrich. Sparse attention with linear units. EMNLP, October 2021. ↩

198. Theodoros Kasioumis, Joe Townsend, and Hiroya Inakoshi. Elite backprop: Training sparse interpretable neurons. International Workshop on Neuro-Symbolic Learning and Reasoning, 2021. ↩

199. Georgios Georgiadis. Accelerating convolutional neural networks via activation map compression. CoRR, March 2019. ↩

200. Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. ICLR, March 2019. ↩

201. Jacob Steinhardt. Emergent deception and emergent optimization. Bounded Regret, February 2023. ↩ ↩² ↩³

202. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? CoRR, May 2023. ↩

203. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. TMLR, October 2022. ↩ ↩²

204. James B. Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J. Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning. ICML, May 2023. ↩

205. Sumio Watanabe. Algebraic Geometry and Statistical Learning Theory. Cambridge Monographs on Applied and Computational Mathematics. Cambridge University Press, 2009. ↩

206. Sumio Watanabe. Mathematical Theory of Bayesian Statistics. Chapman and Hall, 1 edition, April 2018. ↩

207. Edmund Lau, Daniel Murfet, and Susan Wei. Quantifying degeneracy in singular models via the learning coefficient. CoRR, August 2023. ↩

208. Jesse Hoogland, Liam Carroll, and Daniel Murfet. Stagewise development in neural networks. AI Alignment Forum, March 2024. ↩

209. Zach Furman and Edmund Lau. Estimating the local learning coefficient at scale. CoRR, February 2024. ↩

210. Zhongtian Chen, Edmund Lau, Jake Mendel, Susan Wei, and Daniel Murfet. Dynamical versus bayesian phase transitions in a toy model of superposition. CoRR, October 2023b. ↩ ↩²

211. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, February 2017. ↩

212. Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L. Edelman, Fred Zhang, and Boaz Barak. Sgd on neural networks learns functions of increasing complexity. NeurIPS, May 2019. ↩

213. Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. NeurIPS, 2022a. ↩ ↩²

214. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. CoRR, January 2022. ↩

215. Ziming Liu, Eric J. Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. ICML, 2022b. ↩

216. Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficiency. CoRR, September 2023. ↩ ↩²

217. Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind. The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon. CoRR, 2022. ↩

218. William Merrill, Nikolaos Tsilivis, and Aman Shukla. A tale of two circuits: Grokking as competition of sparse and dense subnetworks. CoRR, 2023. ↩

219. Ziming Liu, Ziqian Zhong, and Max Tegmark. Grokking as compression: A nonlinear complexity perspective. CoRR, 2023c. ↩

220. Dashiell Stander, Qinan Yu, Honglu Fan, and Stella Biderman. Grokking group multiplication with cosets. CoRR, 2023. ↩

221. Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization. ICML MI Workshop, June 2024. ↩

222. Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. ICLR, October 2022. ↩

223. Ziming Liu and Max Tegmark. A neural scaling law from lottery ticket ensembling. CoRR, 2023. ↩

224. Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish. Scaling laws and interpretability of learning from repeated data. CoRR, 2022. ↩

225. Naomi Saphra. Interpretability creationism. The Gradient, 2023. ↩

226. Uri Alon. An introduction to systems biology: design principles of biological circuits. Chapman and Hall/CRC, 2019. ↩

227. Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. CoRR, March 2017. ↩ ↩² ↩³

228. Stefan Heimersheim and Jett. A circuit for python docstrings in a 4-layer attention-only transformer. AI Alignment Forum, February 2023. ↩

229. Joseph Miller and Clement Neo. We found an neuron in gpt-2. AI Alignment Forum, February 2023. ↩

230. Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. CoRR, 2023. ↩ ↩²

231. Jannik Brinkmann, Abhay Sheshadri, Victor Levoso, Paul Swoboda, and Christian Bartelt. A mechanistic analysis of a transformer trained on a symbolic multi-step reasoning task. CoRR, February 2024. ↩

232. Jatin Nainani. Evaluating brain-inspired modular training in automated circuit discovery for mechanistic interpretability. CoRR, January 2024. ↩ ↩²

233. János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. Atp*: An efficient and scalable method for localizing llm behaviour to components. CoRR, March 2024. ↩

234. Basel Mousi, Nadir Durrani, and Fahim Dalvi. Can llms facilitate interpretation of pre-trained language models? EMNLP, 2023. ↩

235. Nicholas Bai, Rahul Ajay Iyer, Tuomas Oikarinen, and Tsui-Wei Weng. Describe-and-dissect: Interpreting neurons in vision networks with language models. ICML MI Workshop, June 2024. ↩

236. Alex Foote, Neel Nanda, Esben Kran, Ioannis Konstas, Shay Cohen, and Fazl Barez. Neuron to graph: Interpreting language model neurons at scale. CoRR, May 2023. ↩ ↩²

237. Eric J. Michaud, Isaac Liao, Vedang Lad, Ziming Liu, Anish Mudide, Chloe Loughridge, Zifan Carl Guo, Tara Rezaei Kheirkhah, Mateja Vukelić, and Max Tegmark. Opening the ai black box: program synthesis via mechanistic interpretability. CoRR, February 2024. ↩ ↩²

238. Neel Nanda. A longlist of theories of impact for interpretability. AI Alignment Forum, March 2022e. ↩

239. Stephen Casper. The engineer’s interpretability sequence. AI Alignment Forum, February 2023. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

240. Boaz Barak, Benjamin L. Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit. NeurIPS, 2022. ↩

241. Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. CoRR, May 2019. ↩ ↩²

242. Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Blaise Agüera y Arcas, Max Vladymyrov, Razvan Pascanu, and João Sacramento. Uncovering mesa-optimization algorithms in transformers. CoRR, September 2023. ↩ ↩²

243. Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. A survey of machine unlearning. CoRR, October 2022. ↩ ↩²

244. Lee Sharkey. Circumventing interpretability: How to defeat mind-readers. CoRR, December 2022. ↩ ↩²

245. Peter S. Park, Simon Goldstein, Aidan O’Gara, Michael Chen, and Dan Hendrycks. Ai deception: A survey of examples, risks, and potential solutions. CoRR, August 2023b. ↩ ↩²

246. John Wentworth. How to go from interpretability to alignment: Just retarget the search. AI Alignment Forum, August 2022. ↩ ↩²

247. Thane Ruthenis. Internal interfaces are a high-priority interpretability target. AI Alignment Forum, December 2022. ↩

248. Thane Ruthenis. World-model interpretability is all we need. AI Alignment Forum, January 2023. ↩

249. Jozdien. Conditioning generative models for alignment. AI Alignment Forum, July 2022. ↩

250. Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Conditioning predictive models: Risks and strategies. CoRR, February 2023. ↩ ↩² ↩³

251. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and fundamental limitations of reinforcement learning from human feedback. CoRR, 2023a. ↩ ↩²

252. Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from a deep learning perspective. CoRR, December 2022. ↩

253. Leonard Bereska and Efstratios Gavves. Taming simulators: Challenges, pathways and vision for the alignment of large language models. AAAI-SS, October 2023. ↩ ↩² ↩³

254. NicholasKees and janus. Searching for search. AI Alignment Forum, November 2022. ↩

255. Erik Jenner, Shreyas Kapur, Vasil Georgiev, Cameron Allen, Scott Emmons, and Stuart Russell. Evidence of learned look-ahead in a chess-playing neural network. CoRR, June 2024. ↩

256. technicalities and Stag. Shallow review of live agendas in alignment & safety. LessWrong, 2023. ↩

257. Evan Hubinger. An overview of 11 proposals for building safe advanced ai. CoRR, December 2020. ↩

258. Paul Christiano, Ajeya Cotra, and Mark Xu. Eliciting latent knowledge, January 2021. ↩ ↩²

259. Lawrence Chan. What i would do if i wasn’t at arc evals. AI Alignment Forum, May 2023. ↩ ↩² ↩³

260. Max Tegmark and Steve Omohundro. Provably safe systems: the only path to controllable agi. CoRR, September 2023. ↩ ↩² ↩³

261. Jan Kulveit. Risks from ai misalignment at different scales, July 2024. ↩

262. Dan Hendrycks. Natural selection favors ais over humans. CoRR, July 2023b. ↩

263. Nate Soares. If interpretability research goes well, it may get dangerous. LessWrong, April 2023. ↩

264. Nicholas Kross. Why and when interpretability work is dangerous. LessWrong, May 2023. ↩

265. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. CoRR, March 2022. ↩

266. Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. ICLR, 2023a. ↩

267. Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. ICML, April 2023. ↩

268. Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. NeurIPS Oral, October 2022. ↩ ↩²

269. RicG. Agi-automated interpretability is suicide. LessWrong, May 2023. ↩ ↩²

270. Marius Hobbhahn and Lawrence Chan. Should we publish mechanistic interpretability research? AI Alignment Forum, April 2023. ↩

271. Justin Shovelain and Elliot McKernon. The risk-reward tradeoff of interpretability research. LessWrong, July 2023. ↩

272. Phillip Huang Guo, Aaquib Syed, Abhay Sheshadri, Aidan Ewart, and Gintare Karolina Dziugaite. Robust unlearning via mechanistic localizations. ICML MI Workshop, June 2024. ↩

273. Chen Sun, Nolan Andrew Miller, Andrey Zhmoginov, Max Vladymyrov, and Mark Sandler. Learning and unlearning of fabricated knowledge in language models. ICML MI Workshop, June 2024. ↩

274. Nicky Pochinkov and Nandi. Machine unlearning evaluations as interpretability benchmarks. AI Alignment Forum, August 2023. ↩

275. Stephen Casper, Kaivalya Hariharan, and Dylan Hadfield-Menell. Diagnostics for deep neural networks with automated copy/paste attacks. NeurIPS 2022 ML Safety Workshop (Best paper award), May 2023b. ↩

276. Charbel-Raphaël. Against almost every theory of impact of interpretability. AI Alignment Forum, August 2023. ↩

277. Stephen Casper, Yuxiao Li, Jiawei Li, Tong Bu, Kevin Zhang, Kaivalya Hariharan, and Dylan Hadfield-Menell. Red teaming deep neural networks with feature synthesis tools. NeurIPS, 2023c. ↩ ↩²

278. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. CoRR, August 2024. ↩

279. Dan Hendrycks. Introduction to AI Safety, Ethics, and Society. Self-published, 2023a. ↩

280. Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. CoRR, June 2022. ↩

281. Connor Leahy. Barriers to mechanistic interpretability for agi safety. AI Alignment Forum, 2023. ↩

282. Evan Hubinger. Gradient hacking. AI Alignment Forum, October 2019b. ↩

283. Deqing Fu, Tian-Qi Chen, Robin Jia, and Vatsal Sharan. Transformers learn higher-order optimization methods for in-context learning: A study with linear models. CoRR, October 2023b. ↩

284. Geoffrey E Hinton. Distributed representations. Carnegie Mellon University, 1984. ↩

285. Chris Olah. Distributed representations: Composition & superposition. Transformer Circuits Thread, 2023. ↩

286. Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. CoRR, December 2018. ↩

287. Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. CoRR, June 2019. ↩

288. Victor Lecomte, Kushal Thaman, Trevor Chow, Rylan Schaeffer, and Sanmi Koyejo. Incidental polysemanticity. CoRR, 2023. ↩ ↩² ↩³

289. Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. CoRR, April 2019b. ↩

290. Abhinav Verma, Hoang M. Le, Yisong Yue, and Swarat Chaudhuri. Imitation-projected programmatic reinforcement learning. NeurIPS, 2019a. ↩

291. Dweep Trivedi, Jesse Zhang, Shao-Hua Sun, and Joseph J. Lim. Learning to synthesize programs as interpretable and generalizable policies. NeurIPS, 2021. ↩

292. Quanshi Zhang, Yu Yang, Haotian Ma, and Ying Nian Wu. Interpreting cnns via decision trees. CVPR, 2019. ↩

293. Jie Ren, Mingjie Li, Qirui Chen, Huiqi Deng, and Quanshi Zhang. Defining and quantifying the emergence of sparse concepts in dnns. CoRR, April 2023. ↩

294. Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Towards causal representation learning. Special Issue of Proceedings of the IEEE - Advances in Machine Learning and Deep Neural Networks, February 2021. ↩

295. Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher Pal. A meta-transfer objective for learning to disentangle causal mechanisms. CoRR, February 2019. ↩

296. Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Christopher Pal. Systematic evaluation of causal discovery in visual model based reinforcement learning. CoRR, July 2021. ↩

297. Judea Pearl and Dana Mackenzie. The Book of Why: The New Science of Cause and Effect. Penguin Books Limited, May 2018. ↩

298. Jonas Peters, Dominik Janzing, and Bernhard Schlkopf. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, October 2017. ↩

299. Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. CoRR, November 2020. ↩

300. Michel Besserve, Arash Mehrjou, Rémy Sun, and Bernhard Schölkopf. Counterfactuals uncover the modular structure of deep generative models. CoRR, December 2019. ↩

301. Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society Series B: Statistical Methodology, November 2015. ↩

302. Julius von Kügelgen, M. Loog, A. Mey, and B. Scholkopf. Semi-supervised learning, causality, and the conditional cluster assumption. Conference on Uncertainty in Artificial Intelligence, May 2019. ↩

303. Mateo Rojas-Carulla, Bernhard Scholkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. JMLR, 2018. ↩

304. Sara Magliacane, Thijs van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M. Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. NeurIPS, October 2018. ↩

305. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models. CoRR, October 2023. ↩ ↩²

306. Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, Ethan Perez, and Evan Hubinger. Sycophancy to subterfuge: Investigating reward-tampering in large language models. CoRR, 2024. ↩

307. Leon Lang, Davis Foote, Stuart Russell, Anca Dragan, Erik Jenner, and Scott Emmons. When your ais deceive you: Challenges with partial observability of human evaluators in reward learning. CoRR, March 2024. ↩ ↩²

308. Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, Kshitij Sachan, Michael Sellitto, Mrinank Sharma, Nova DasSarma, Roger Grosse, Shauna Kravec, Yuntao Bai, Zachary Witten, Marina Favaro, Jan Brauner, Holden Karnofsky, Paul Christiano, Samuel R. Bowman, Logan Graham, Jared Kaplan, Sören Mindermann, Ryan Greenblatt, Buck Shlegeris, Nicholas Schiefer, and Ethan Perez. Sleeper agents: Training deceptive llms that persist through safety training. CoRR, 2024. ↩ ↩²

309. Narek Maloyan, Ekansh Verma, Bulat Nutfullin, and Bislan Ashinov. Trojan detection in large language models: Insights from the trojan detection challenge. CoRR, April 2024. ↩

310. Leonard Bereska. Mechanistic interpretability for adversarial robustness — a proposal. Leonard Bereska’s Blog, August 2024. ↩

311. Amlan Jyoti, Karthik Balaji Ganesh, Manoj Gayala, Nandita Lakshmi Tunuguntla, Sandesh Kamath, and Vineeth N. Balasubramanian. On the robustness of explanations of deep neural network models: A survey. CoRR, November 2022. ↩

312. Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness as a prior for learned representations. CoRR, September 2019. ↩

313. Andrew Slavin Ross and Finale Doshi-Velez. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. AAAI, November 2017. ↩

314. Keke Du, Shan Chang, Huixiang Wen, and Hao Zhang. Fighting adversarial images with interpretable gradients. ACM TURC, October 2021. ↩

315. Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. Activation atlas. Distill, March 2019. ↩

316. Stephen Casper, Max Nadeau, Dylan Hadfield-Menell, and Gabriel Kreiman. Robust feature-level adversaries are interpretability tools. NeurIPS, October 2021. ↩

317. Stanislav Fort and Balaji Lakshminarayanan. Ensemble everything everywhere: Multi-scale aggregation for adversarial robustness. CoRR, August 2024. ↩

318. Jorge García-Carrasco, Alejandro Maté, and Juan Trujillo. Detecting and understanding vulnerabilities in language models via mechanistic interpretability. IJCAI, August 2024. ↩ ↩²

319. Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. NeurIPS Workshop on Causal Representation Learning, November 2023a. ↩

320. Dmitry Vaintrob, jake_mendel, and Kaarel. Toward a mathematical framework for computation in superposition. AI Alignment Forum, 2024. ↩ ↩²

321. Kaarel Hänni, Jake Mendel, Dmitry Vaintrob, and Lawrence Chan. Mathematical models of computation in superposition. ICML MI Workshop, August 2024. ↩

322. Adam Jermyn, Chris Olah, and T Henighan. Circuits updates - may 2023: Attention head superposition. Transformer Circuits Thread, 2023. ↩

323. Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. Successor heads: Recurring, interpretable attention heads in the wild. CoRR, 2023. ↩

324. Joseph Bloom and Jay Bailey. Features and adversaries in memorydt. LessWrong, October 2023. ↩ ↩²

325. Neel Nanda. 200 cop in mi: Exploring polysemanticity and superposition. Neel Nanda’s Blog, January 2023a. ↩

326. Max Lamparth and Anka Reuel. Analyzing and editing inner mechanisms of backdoored language models. CoRR, 2023. ↩

327. Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: A comprehensive benchmark of backdoor learning. NeurIPS Datasets and Benchmarks, October 2022a. ↩

328. Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell, May 2019. ↩

329. Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, February 2019. ↩

330. Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Vi’egas, and M. Wattenberg. An interpretability illusion for bert. CoRR, April 2021. ↩

331. Dan Friedman, Andrew Lampinen, Lucas Dixon, Danqi Chen, and Asma Ghandeharioun. Interpretability illusions in the generalization of simplified models. CoRR, 2023a. ↩ ↩²

332. Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, November 2017. ↩

333. Mantas Mazeika, Andy Zou, Akul Arora, Pavel Pleskov, Dawn Song, Dan Hendrycks, Bo Li, and David Forsyth. How hard is trojan detection in dnns? fooling detectors with evasive trojans. CoRR, September 2022. ↩

334. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. ICLR, 2022b. ↩

335. David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. ICLR, December 2018. ↩

336. Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. NeurIPS Spotlight, January 2023. ↩

337. David Lindner, János Kramár, Sebastian Farquhar, Matthew Rahtz, Thomas McGrath, and Vladimir Mikulik. Tracr: Compiled transformers as a laboratory for interpretability. CoRR, 2023. ↩ ↩²

338. Philip Quirke, Clement Neo, and Fazl Barez. Increasing trust in language models through the reuse of verified circuits. CoRR, February 2024. ↩

339. Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? CoRR, April 2020. ↩

340. Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. ICML MI Workshop, June 2024. ↩

341. Matthew L. Leavitt and Ari Morcos. Towards falsifiable interpretability research. CoRR, October 2020. ↩

342. Hannes Thurnherr and Jérémy Scheurer. Tracrbench: Generating interpretability testbeds with large language models. ICML MI Workshop, June 2024. ↩

343. Rohan Gupta, Iván Arcuschin, Thomas Kwa, and Adrià Garriga-Alonso. Interpbench: Semi-synthetic transformers for evaluating mechanistic interpretability techniques. CoRR, July 2024. ↩

344. David “davidad” Dalrymple, Joar Skalse, Yoshua Bengio, Stuart Russell, Max Tegmark, Sanjit Seshia, Steve Omohundro, Christian Szegedy, Ben Goldhaber, Nora Ammann, Alessandro Abate, Joe Halpern, Clark Barrett, Ding Zhao, Tan Zhi-Xuan, Jeannette Wing, and Joshua Tenenbaum. Towards guaranteed safe ai: A framework for ensuring robust and reliable ai systems. CoRR, May 2024. ↩ ↩²

345. Jason Gross, Rajashree Agrawal, Thomas Kwa, Euan Ong, Chun Hei Yip, Alex Gibson, Soufiane Noubir, and Lawrence Chan. Compact proofs of model performance via mechanistic interpretability. ICML MI Workshop, June 2024. ↩

346. Neel Nanda. 200 cop in mi: Looking for circuits in the wild. Neel Nanda’s Blog, 2022a. ↩

347. Neel Nanda. 200 cop in mi: Studying learned features in language models. Neel Nanda’s Blog, 2022c. ↩

348. Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. A mechanism for solving relational tasks in transformer language models. CoRR, May 2023. ↩

349. Marius Hobbhahn. Marius’ alignment agenda, 2022. ↩

350. Neel Nanda. 200 cop in mi: Techniques, tooling and automation. Neel Nanda’s Blog, 2023b. ↩

351. Giovanni Monea, Maxime Peyrard, Martin Josifoski, Vishrav Chaudhary, Jason Eisner, Emre Kıcıman, Hamid Palangi, Barun Patra, and Robert West. A glitch in the matrix? locating and detecting language model grounding with fakepedia. CoRR, 2023. ↩

352. Neel Nanda. 200 cop in mi: Analysing training dynamics. Neel Nanda’s Blog, 2022b. ↩

353. Leo Z. Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith. Probing across time: What does roberta know and when? EMNLP, September 2021. ↩

354. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. CoRR, 2022. ↩

355. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. NeurIPS, December 2017. ↩ ↩²

356. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR, April 2022. ↩ ↩²

357. Vedant Palit, Rohan Pandey, Aryaman Arora, and P. Liang. Towards vision-language mechanistic interpretability: A causal tracing tool for blip. ICCVW, 2023. ↩ ↩²

358. Emmanuelle Salin, Badreddine Farah, Stéphane Ayache, and Benoit Favre. Are vision-language transformers learning multimodal representations? a probing perspective. AAAI, June 2022. ↩

359. Jacob Hilton, Nick Cammarata, Shan Carter, Gabriel Goh, and Chris Olah. Understanding rl vision. Distill, 2020. ↩

360. Estelle Aflalo, Meng Du, Shao-Yen Tseng, Yongfei Liu, Chenfei Wu, Nan Duan, and Vasudev Lal. Vl-interpret: An interactive visualization tool for interpreting vision-language transformers. CVPR, June 2022. ↩

361. M. Vilas, Timothy Schaumlöffel, and Gemma Roig. Analyzing vision transformers for image classification in class embedding space. CoRR, 2023. ↩

362. Xu Pan, Aaron Philip, Ziqian Xie, and Odelia Schwartz. Dissecting query-key interaction in vision transformers. ICML MI Workshop, June 2024. ↩

363. Ulisse Mini, Peli Grietzer, Mrinank Sharma, Austin Meek, Monte MacDiarmid, and Alexander Matt Turner. Understanding and controlling a maze-solving policy network. CoRR, October 2023. ↩

364. Paul Colognese and Jozdien. High-level interpretability: detecting an ai’s objectives. AI Alignment Forum, 2023. ↩

365. Paul Colognese. Internal target information for ai oversight. LessWrong, 2023. ↩

366. Joseph Bloom and Paul Colognese. Decision transformer interpretability. AI Alignment Forum, 2023. ↩

367. Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. ICLR, February 2024. ↩

368. Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rocktäschel, and David Scott Krueger. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks. CoRR, November 2023. ↩

369. Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. CoRR, 2024. ↩

370. Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip Torr, Amartya Sanyal, and Puneet K. Dokania. What makes and breaks safety fine-tuning? a mechanistic study. ICML MI Workshop, June 2024. ↩

371. Luke Marks, Amir Abdullah, Clement Neo, Rauno Arike, Philip Torr, and Fazl Barez. Beyond training objectives: Interpreting reward model divergence in large language models. CoRR, October 2023b. ↩

372. Leopold Aschenbrenner. Situational awareness: The decade ahead. Series: Situational Awareness. June, 2024. ↩

373. Alex Mallen and Nora Belrose. Eliciting latent knowledge from quirky language models. CoRR, December 2023. ↩

374. Ingrid Lossius Falkum and Agustin Vicente. Polysemy: Current perspectives and approaches. Lingua, April 2015. ↩