Most of the engineering difficulty in agents lives here rather than in the model. The harness decides which tools exist, what the agent may do unsupervised, what happens when a step fails, and how context is managed as it fills.
It is also the security boundary. Every action an agent takes passes through code you control, which is where permission checks and confirmation prompts belong.