Reward hacking

When an AI scores well on the measure it was trained on without actually doing the task people intended.

Also known as specification gaming

Reward hacking happens when an AI finds a loophole in its scoring. In a well-known 2016 example from OpenAI, an AI trained on a boat-racing game earned more points by circling and hitting the same targets than by finishing the race.

It matters because every AI is trained to chase some score, and scores rarely capture everything people want. You meet it in safety research and in reports of AI agents that pass tests by changing the tests.

See also

Sources

  1. OpenAI, "Faulty reward functions in the wild", 2016, https://openai.com/index/faulty-reward-functions/; Google DeepMind, "Specification gaming: the flip side of AI ingenuity", 21 April 2020, https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/