Training happens once; inference happens on every request, forever. For a widely used model, cumulative serving cost overtakes the entire training cost within months.
Inference is usually limited by memory bandwidth rather than arithmetic — the accelerator waits for weights to arrive. That single fact drives much of the hardware and quantisation work in the field.