Add model_max_length; note context-extension capability in README
Browse files
README.md
CHANGED
|
@@ -41,7 +41,7 @@ This repo fixes all four. The math is **bit-exact** with the evo-design referenc
|
|
| 41 |
| Hyena state size | 8 |
|
| 42 |
| Positional encoding | RoPE (base = 10000) |
|
| 43 |
| Architecture | StripedHyena (alternating Hyena / MHA blocks) |
|
| 44 |
-
| Max sequence length | 8 192 |
|
| 45 |
| Training dtype | bfloat16 (Hyena modal-form `poles` / `residues` kept in fp32) |
|
| 46 |
|
| 47 |
Architecturally identical to `Evo1-1-7B-8K`; only the trained weights differ (Evo 1.5 = Evo 1 (8k) + ~50% more pretraining tokens).
|
|
|
|
| 41 |
| Hyena state size | 8 |
|
| 42 |
| Positional encoding | RoPE (base = 10000) |
|
| 43 |
| Architecture | StripedHyena (alternating Hyena / MHA blocks) |
|
| 44 |
+
| Max sequence length | 8 192 (training context; StripedHyena supports extension beyond this with adjusted FFT buffers) |
|
| 45 |
| Training dtype | bfloat16 (Hyena modal-form `poles` / `residues` kept in fp32) |
|
| 46 |
|
| 47 |
Architecturally identical to `Evo1-1-7B-8K`; only the trained weights differ (Evo 1.5 = Evo 1 (8k) + ~50% more pretraining tokens).
|