Eagle 7B is based on the Receptance Weighted Key Value architecture, blending Transformer and RNN advantages while sidestepping their limitations, with particular focus on unlimited input context length and energy efficiency.
RWKV opens a new route for scalable and efficient architectures to model complex relationships in sequential data. While many alternatives to Transformers have been proposed with similar claims, ours is the first to back up those claims with pretrained models with tens of billions of parameters.
Collection
[
|
...
]