> For the complete documentation index, see [llms.txt](https://docs.quantaria.xyz/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.quantaria.xyz/deep-dives/kv-cache.md). # The KV cache and why it limits context The key-value (KV) cache is the memory a transformer keeps during generation. It explains most of the memory cost of long contexts, and it is why Quantaria cares about attention structure. This page explains the mechanism. ## What is cached During generation, each new token attends to every earlier token. Recomputing the keys and values for all earlier tokens at each step would be wasteful, so the model stores them. That store is the KV cache. ## How it grows For each layer and each head, the cache holds one key and one value per token. Its size grows linearly with the context length and with the number of layers and heads. A long context therefore uses a lot of memory even if the weights are small. ## Why weights are not the whole story Shrinking the weights does not shrink the cache. A model with compressed weights can still run out of memory on a long prompt if its cache is large. This is why a compact text model for long context needs an attention change as well as small layers. ## Ways to shrink it * **Fewer heads or shared keys and values.** Changes the attention design. * **Factorized keys and values.** The approach of [Tensor Product Attention](/deep-dives/tensor-product-attention.md). * **Quantization.** Stores the cache at lower precision, which is a separate technique. ## Quantaria's use Quantaria cites a KV cache about 10× smaller under Tensor Product Attention, from the paper. The planned Schmidt model relies on the cache reduction for long context on small devices. See [Models](https://github.com/bhavy2908/dataai_docs/tree/main/overview/models.md#schmidt). ## Related * [Tensor Product Attention](/deep-dives/tensor-product-attention.md) * [Getting Started](https://github.com/bhavy2908/dataai_docs/tree/main/guides/getting-started.md): context limits for Nano and Base --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.quantaria.xyz/deep-dives/kv-cache.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.