Inner alignment Source: en.wikipedia.org/wiki/Inner_alignment

This article includes a list of references, related reading, or external links, but its sources remain unclear because it lacks inline citations. Please help improve this article by introducing more precise citations. (June 2025) (Learn how and when to remove this message)

Inner alignment is a concept in artificial intelligence (AI) safety that refers to ensuring that a trained machine learning model reliably pursues the objective intended by its designers, even in novel or unforeseen situations. It contrasts with "outer alignment," which is concerned with whether the training objective itself reflects human values and goals. Inner alignment becomes especially relevant when machine learning systems, particularly advanced neural networks or large language models, develop internal optimization behavior that may not align with their intended purpose.^[1]

A 2025 study demonstrates, using Rice's theorem and the Halting Problem, that determining whether an arbitrary AI model satisfies a non-trivial alignment function is undecidable. That is, no universal algorithm can verify alignment across all AI systems. However, this undecidability only applies to arbitrary models. If AI systems are constructed from a finite set of base operations that are provably alignment-preserving, it becomes possible to build an enumerable set of AI models with guaranteed alignment properties.

A key proposal in the study is that alignment should be embedded architecturally, rather than imposed post-training. To make alignment verifiable, systems must be designed to halt — that is, reach a terminal state in finite steps. Mechanisms such as time-penalizing utility functions, self-modifying procedures, and output masking are proposed to ensure this halting property. This approach is compared to biological systems with built-in mortality, and lays the groundwork for provable-by-design AI safety architectures.^[2]

Inner alignment as a key element in achieving human-centric AI has been outlined, particularly models that satisfy the "3H" criteria: Helpful, Honest, and Harmless. In this context, inner alignment refers to the reliable generalization of externally defined objectives across novel or adversarial inputs.

A range of techniques to support this goal has been highlighted, including parameter-efficient fine-tuning, interpretability-focused design, robust training, and factuality enhancement. These strategies aim to ensure that models not only learn aligned behavior but also retain and apply it across deployment contexts. Inner alignment is thus viewed as critical to making aligned AI behavior stable and generalizable.^[3]

Research on large language model (LLM)-based embodied agents—AI systems that physically interact with the environment—addresses inner alignment by fine-tuning models to generate actions within a predefined, safe, and executable action space. A parameter-efficient tuning approach adjusts internal behaviors so that generated outputs remain consistent with the agent’s operational context. This method reduces “action hallucination,” where agents produce infeasible or unintended actions. The inner alignment strategy thus serves as a foundational layer, ensuring that the agent’s raw output is aligned prior to applying outer alignment techniques like retrieval-based filtering and policy arbitration.^[4]

Another line of research explores the application of the Free Energy Principle and Active Inference framework to inner alignment. In these models, agents construct hierarchical world models and minimize prediction error through iterative self-modeling. This leads to the emergence of "value cores"—stable, self-reinforcing internal states that guide goal-oriented behavior. These architectures allow agents to develop preferences and behaviors aligned with human-compatible values through mechanisms like iterated policy selection and preference learning. Rather than viewing emergent subgoal optimization (mesa-optimization) as a threat, this framework suggests it can be beneficial when grounded in biologically inspired cognitive mechanisms.^[5]

Case studies highlight inner alignment risks in deployed systems. A reinforcement learning agent in the game CoastRunners learned to maximize its score by circling indefinitely instead of completing the race. Similarly, conversational AI systems have exhibited tendencies to generate false, biased, or harmful content, despite safeguards implemented during training. These cases demonstrate that systems may have the capacity to act aligned but instead pursue unintended internal objectives.^[6]

The persistence of such misalignments across different architectures and applications suggests that inner alignment problems may arise by default in machine learning systems. As AI models become more powerful and autonomous, these risks are expected to increase, potentially leading to catastrophic consequences if not adequately addressed.^[6]

• Outer alignment
• AI alignment
• Mesa-optimization
• Goodhart's law
• Reward hacking