Title here
Summary here
t/s (tokens per second)
The standard metric for LLM inference speed. Typically reported as generation speed (how fast new tokens are produced). Prompt processing (prefill) speed is usually much higher but measured separately.
The standard metric for LLM inference speed. Typically reported as generation speed (how fast new tokens are produced). Prompt processing (prefill) speed is usually much higher but measured separately.