Red teaming

Deliberately attacking an AI system, as an adversary would, to find harmful behaviour or security holes before real attackers do.

Also known as red-teaming, AI red teaming, adversarial testing

Red teaming means playing the attacker on purpose. A red team tries to make an AI system misbehave: to reveal secrets, give dangerous instructions, ignore its rules or break out of a test environment. The term comes from military and computer security exercises, where the “red team” plays the enemy.

AI labs run red teams before releasing models and invite outside experts to do the same. You meet the term in system cards and safety reports, and in stories where an AI agent was set a red-team task, such as trying to sneak harmful data past a filter.