> For the complete documentation index, see [llms.txt](https://docs.quantaria.xyz/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.quantaria.xyz/deep-dives/tensor-product-attention.md).

# Tensor Product Attention

Tensor Product Attention (TPA) is an attention variant that factorizes the query, key, and value matrices into tensor products. The factorization shrinks the memory needed for past keys and values. This page explains how it works and where Hilbert-1 uses it.

## Standard attention, briefly

In multi-head attention, each token is projected into a query, a key, and a value. For a sequence of length T, the model stores keys and values for every past token. That store is the KV cache, and it grows with context length.

## The factorization

TPA writes each query, key, and value as a sum of tensor products of smaller factors, computed from the token's hidden state. The factors are much smaller than full per-head matrices. Because the keys and values are stored in factored form, the cache is smaller.

The method is compatible with rotary position embeddings (RoPE). The paper ([arXiv:2501.06425](https://arxiv.org/abs/2501.06425)) reports a KV cache about 10× smaller than standard multi-head attention. That is the paper's result.

## Why it matters for context

The KV cache is often the memory limit on long contexts. A smaller cache lets the same memory hold more tokens, or the same tokens in less memory. This is the reason TPA appears in both Hilbert-1 and the planned Schmidt model. See [The KV cache and why it limits context](/deep-dives/kv-cache.md).

## Trade-offs

* The factorization adds computation when a token is processed.
* The rank of the factors sets the trade-off between cache size and accuracy. Setting it too low can lose information that attention needs.

## Quantaria's use

Hilbert-1 Nano and Base use Tensor Product Attention, as listed in [Models](https://github.com/bhavy2908/dataai_docs/tree/main/overview/models.md). The factor ranks are part of the model's configuration and are not published in these docs.

## Related

* [Shared Multimodal Backbone](https://github.com/bhavy2908/dataai_docs/tree/main/architecture/multimodal-backbone.md#tensor-product-attention)
* [Research Foundations](https://github.com/bhavy2908/dataai_docs/tree/main/architecture/research-foundations.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.quantaria.xyz/deep-dives/tensor-product-attention.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.