Falcon

Fast Weight Attention for Continual Learning
Recurrent fast-weight memories and selective state-space models compress a growing context into a bounded state — their writes can be viewed as online continual-learning rules.

Yifan Zhang, Steve Ta, Jasper Zhang, Jichen Feng, Shuzhen Li, Yongxin Zhang, Yifeng Liu, Huizhuo Yuan, Mengdi Wang†, Quanquan Gu†, Andrew Chi-Chih Yao†
March 9, 2026  ·  arXiv:2608.27763
🤗
Continual Learning Fast-Weight Memory State-Space Models Online Learning

Abstract

Recurrent fast-weight memories and selective state-space models compress a growing context into a bounded state. Because every new token updates that state, their writes can be viewed as online continual-learning rules — at inference time the model is, in effect, continually learning from its own context.

Falcon develops this fast weight attention perspective, connecting attention, recurrent memory, and state-space updates through the lens of continual learning over a bounded memory.

Read the Paper arXiv Code

Overview

Falcon: fast weight attention as an online continual-learning rule.
Figure: Falcon — viewing fast-weight and state-space writes as online continual-learning updates to a bounded memory.

Citation

If you find this work useful, please cite:

@article{zhang2026fast,
  title   = {Fast Weight Attention for Continual Learning},
  author  = {Zhang, Yifan and Ta, Steve and Zhang, Jasper and Feng, Jichen and Li, Shuzhen and Zhang, Yongxin and Liu, Yifeng and Yuan, Huizhuo and Wang, Mengdi and Gu, Quanquan and Yao, Andrew Chi-Chih},
  journal = {arXiv preprint arXiv:2608.27763},
  year    = {2026},
  month   = {March},
  url     = {https://arxiv.org/abs/2608.27763}
}