Stanford researchers have open-sourced Memorilla, a 194-million-parameter memory module that compresses entire document collections into just 16 tokens for a frozen Qwen3-8B model. On the FactKG benchmark, the approach scored 84.33% accuracy versus 80.43% when the full context was used—using 108 times fewer document tokens.
TL;DR
- Memorilla compresses any document collection into 16 latent tokens for frozen LLMs, replacing full-context prompting.
- On FactKG, Qwen3-8B scored 84.33% with Memorilla vs 80.43% with full context, using 108× fewer tokens.
- Open-source Apache 2.0 code and checkpoints are available; deployment requires CUDA GPUs and Python 3.11.
What happened
Researchers from Stanford's SNAP group, with collaborators at Toronto, Penn, and MIT, released Memorilla under Apache 2.0. The module is a trainable memory system that converts entire document collections into 16 fixed-size latent tokens.
The system keeps the encoder (Qwen3-Embedding-4B) and decoder (Qwen3-8B) frozen, training only the 194-million-parameter memory module. It runs on Linux with CUDA GPUs and Python 3.11, with quickstart guidance requiring an 80 GB GPU.
Training follows a three-stage recipe: 1.98 million Wikipedia examples, 1.34 million from QA and summarization datasets, and fine-tuning on eight benchmarks. The full recipe takes about 19 hours on four H100 GPUs.
The team also released PersonalizationV4, a dataset of 149 synthetic users with 200 conversations each under CC BY 4.0.
Why it matters
For developers building agents or long-context applications, Memorilla offers a way to shrink memory footprints without sacrificing accuracy on key benchmarks like FactKG and TriviaQA.
Startups can use the module to reduce inference latency and GPU memory pressure, especially when handling large document collections or agent memory states.
Investors should note that the approach achieves the highest reported average score (49.68) across eight benchmarks among comparable systems, which may influence model selection for memory-intensive use cases.
The method's weakest point is PubMedQA, where it trails at 59.00% versus 69.00% for LLMLingua-2, suggesting room for improvement on fine-grained biomedical reasoning.
Key facts
- Memorilla compresses any document collection into 16 latent tokens for frozen LLMs.
- Qwen3-8B scored 84.33% on FactKG with Memorilla vs 80.43% with full context, using 108× fewer tokens.
- The memory module has 194,123,776 trainable parameters and keeps encoder/decoder frozen.
- Training recipe runs in about 19 hours on four H100 GPUs.
- Deployment requires Linux with CUDA GPUs and Python 3.11; quickstart needs one 80 GB GPU.
- Average score across eight benchmarks is 49.68, the highest reported among comparable systems.
- PubMedQA is the weakest benchmark for Memorilla at 59.00% vs 69.00% for LLMLingua-2.
- PersonalizationV4 dataset released under CC BY 4.0 with 149 synthetic users and 200 conversations each.
Context
Long-context LLMs struggle with quadratic compute costs and latency as context grows, pushing researchers to explore memory compression and retrieval augmentation.
Memorilla joins a wave of latent-memory approaches like MemoRAG and xRAG, which aim to distill document collections into compact representations for frozen models.
The work highlights Stanford's SNAP group as a leader in scalable AI systems, following prior contributions to graph learning and efficient inference.
