TALECRAFTERS
GLOSSARY · MODELS · SYSTEMS

INFERENCE

Actually running a trained model to produce an output, as distinct from training it, and the part you are billed for per use.

ALSO CALLEDgenerationmodel run

Training is the expensive thing that happened once, somewhere else. Inference is the thing you pay for every time you press generate, and it is where a production budget actually goes.

It is worth separating in your head because the two have opposite economics. Training cost is sunk and enormous; inference cost is small, per-unit and entirely under your control through batch sizes, resolutions, step counts and how many failures you tolerate.

Every lever in this glossary that saves money saves it at inference.

QUESTIONS PEOPLE ASK

What is the difference between training and inference?

Training builds the model, once, at enormous cost. Inference runs it, every time you generate, at a small per-use cost. Only the second one appears on your invoice.

What most affects inference cost?

Resolution, sequence length, step count and your acceptance rate. The last one is usually the largest and the least tracked.

THEORY IS FREE

Knowing the word is the cheap part.

Running it on a deadline, at volume, without burning the budget is the expensive part. That is the bit we do.

BRIEF US