Course: Efficiently Serving LLMs\nPlatform: DeepLearning.io
\nInput text is tokenized into numbers (called tensors).
\nThe tensor is fed to a model that gives the top-k predictions for the next token.
\nTake the top prediction and append it to the input, either until the maximum number of tokens are created or a stop token is reached.
\nFor a sequence without padding, the attention mask can contain 1 for every token.
\nDuring attention, the model computes Key (K) and Value (V) tensors. These can be stored in a KV cache for each transformer layer and reused during subsequent token generation.
\nCache the K-V matrix and pass that to the model, so that only the incremental value can be computed instead of starting computation from the first token while calculating attention.
\nThe K-V cache contains the values that have already been computed and can be reused for subsequent tokens.
\n Attention computation\n +-----------+\n | A |\n +-----+-----+\n |\n +----------------+----------------+\n | |\n v v\n New input token Cached K-V\n | +-----------------+\n v | K1 K2 K3 ... |\n +-----------+ | V1 V2 V3 ... |\n | Q, K, V | +-----------------+\n | for token | |\n +-----------+ |\n | |\n +----------------+----------------+\n |\n v\n Attention uses:\n Q(new) with\n K1...Kn + K(new)\n V1...Vn + V(new)\n |\n v\n Next-token logits\n |\n v\n Next token\n |\n |\n +------------------+\n |\n K(new), V(new) |\n v\n +---------------------+ +-------+\n | Updated K-V | <----| Append|\n | | +-------+\n | K1 K2 K3 ... Kn+1 |\n | V1 V2 V3 ... Vn+1 |\n +---------------------+\n |\n +----> cache for next stepPrefill phase — process the entire input prompt and build the initial KV cache. The output is used to generate the first token. (slowest)
\nDecode phase — generate subsequent tokens one at a time. (can use K-V cache)
\nBatching: Parallelly process N sequences of tokens.
\nIf the number of tokens in each sequence is different, use padding tokens on the left and set the attention mask value to `0`, so that the model does not pay attention to it.
\nFor example:
\n[The] [cat] [sat]\n[The] [home] [on] [the]\n[A] [boy]After padding:
\n[PAD] [PAD] [The] [cat] [sat]\n[PAD] [The] [home] [on] [the]\n[PAD] [PAD] [PAD] [A] [boy]The model can then generate the next token for each sequence.
\nHere, in the output from the model, keep all the elements in the batch.
\nNext, instead of taking `argmax` of the entire output, take the max of each row to choose the next token for that sequence.
\nIf you do this efficiently for every loop to see if a sequence is ready to be removed from the batch or a new one can be added, that's continuous batching. This can improve throughput.
\n