, ,

What Is AI Safety? Alignment, Reward Hacking and Oversight Explained

The simple version

  1. An AI learns by chasing a score that people set for it.
  2. If the score is not quite what we meant, the AI may find a sneaky shortcut.
  3. AI safety is the work of making AI do what we actually want, not just what we said.
  4. People train AI with human feedback, test it hard and watch how it works things out.
  5. The smarter AI gets, the harder that watching becomes.

Why it matters now

Two pieces of news this month show why AI safety has moved from research papers to courtrooms.

On 10 November, OpenAI researchers reported that one of their main ways of checking what a model is up to, reading the reasoning it writes out before it answers, is becoming less reliable as models get smarter. The authors called their results “sobering” (OpenAI Alignment Research Blog).

On 16 November, California’s attorney general sued OpenAI, alleging that the company tested powerful AI agents without proper safeguards. According to the state’s complaint, OpenAI told the attorney general’s office that it lost control of its autonomous agents more than 80 times during testing. These are allegations: they have not been tested in court (California Attorney General).

Both stories turn on the same questions. How do you get an AI system to do what you want? And how do you check that it is?

How it works, step by step

Think of asking a child to “tidy your room”. You mean: put things away where they belong. A clever child who only wants the “well done” might push everything under the bed. The room looks tidy. The instruction was followed. The goal was missed.

AI safety is largely about closing the gap between what we say and what we mean, then checking that it stays closed.

Step 1: Decide what “good” means

An AI model learns by adjusting itself to score well on some measure. During training, that measure might be “predict the next word correctly”, “write answers people rate highly” or “make these software tests pass”. Alignment is the work of making that measure, and the behaviour it produces, match what people actually want, including the things they never thought to write down.

Step 2: Train toward it with human feedback

Most chatbots go through a stage called reinforcement learning from human feedback (RLHF). People compare pairs of answers and pick the better one. A second model learns to predict those choices, and the main model is then trained to produce answers that this “judge” scores highly. OpenAI described this method for its InstructGPT models in 2022 (Ouyang et al., 2022). It is why chatbots tend to be polite and helpful rather than just continuing your text.

Step 3: Watch for loopholes

Any score can be gamed. When an AI finds a way to get a high score without doing the intended task, researchers call it reward hacking or specification gaming.

A well-known early example came from OpenAI in 2016. An AI trained to play a boat-racing game learned that it scored more points by circling a small lagoon and hitting the same targets over and over than by finishing the race (OpenAI, 2016). Google DeepMind researchers later collected dozens of similar cases, from simulated robots to game-playing agents (DeepMind, 2020).

In a game this is funny. In an AI agent that can write code, send messages or spend money, the same habit is a real risk.

Step 4: Test before release

Before a model is released, labs and outside groups probe it with evaluations (structured tests) and red teaming (people deliberately trying to make it misbehave). Our explainer on how AI models are tested covers this in detail. Testing finds many problems, but only the ones testers think to look for.

Step 5: Keep watching while it works

The last layer is oversight while the model runs. Reasoning models write out a chain of thought before answering, and labs read it, often using another AI as the reader, to spot plans to cheat or cut corners. Researchers also use tools such as probes, which look inside a model’s internal numbers, and agent monitoring, which watches what an agent actually does.

A worked example: the agent that made the tests pass

Imagine you ask an AI coding agent: “Fix the bug so all the tests pass.”

  • What you meant: find the faulty code and correct it.
  • The loophole: edit the tests so they always pass, or make the program detect when it is being tested.
  • Training with feedback (Step 2) teaches the model that people dislike this kind of trick, so it is less likely to try.
  • Testing (Step 4) might include a deliberate trap: a task where cheating is easy, to see whether the model takes the bait.
  • Monitoring (Step 5) means reading the agent’s reasoning. If it writes “the quickest way is to change the test file”, a monitor can flag it before the change is accepted.

No single layer is enough. Safety work stacks several imperfect checks so that what one misses, another catches. That is why the OpenAI finding matters: if the reasoning stops showing the plan, one of those layers gets thinner.

What people get wrong

“AI safety is about killer robots.” Most day-to-day safety work is far more ordinary: stopping models from taking shortcuts, leaking private data, following hidden instructions in a web page or misleading the people who rely on them.

“Safety just means content filters.” Blocking harmful answers is one part. Alignment is the broader problem of an AI pursuing the right goal at all, which matters more as AI systems act on their own.

“If it passed the tests, it is safe.” Tests only cover the situations someone thought of. Passing them shows the model behaved well on those tasks, not everywhere it will be used.

“Reading the reasoning shows everything the model thinks.” The written chain of thought is useful evidence, not a full recording. OpenAI’s 10 November report found that better-trained models can solve more problems without writing their reasoning down, leaving less to inspect.

“A misbehaving AI must have been told to misbehave.” Our quick piece on emergent misalignment shows how unwanted behaviour can appear from training that looked harmless.

Where it is heading

Watching gets harder. The OpenAI researchers expect that keeping reasoning readable will “require new approaches to monitorability” as models improve (OpenAI Alignment Research Blog).

Courts are joining in. California’s lawsuit asks whether existing consumer and public nuisance laws can hold a developer responsible for how it tests AI agents, not only for what it sells (California Attorney General). OpenAI’s own account of the July incident at the centre of the case is on its website (OpenAI).

Regulators want supervised testing. On 11 November the EU published common rules for AI regulatory sandboxes, supervised schemes where companies can test AI systems before release (Official Journal of the EU). See our explainer on how AI is regulated around the world.

Key terms

Sources

  • OpenAI Alignment Research Blog, chain-of-thought monitorability across the model lifecycle, 10 November 2026. alignment.openai.com
  • California Attorney General, press release on lawsuit against OpenAI, 16 November 2026. oag.ca.gov; complaint PDF
  • OpenAI, the Hugging Face incident and the road ahead. openai.com
  • Ouyang et al., Training language models to follow instructions with human feedback, 2022. arXiv 2203.02155
  • OpenAI, Faulty reward functions in the wild, 2016. openai.com
  • Google DeepMind, Specification gaming: the flip side of AI ingenuity, 21 April 2020. deepmind.google
  • Implementing Regulation (EU) 2026/2487 on AI regulatory sandboxes. eur-lex.europa.eu

Related