Add model_max_length; note context-extension capability in README
Browse files
README.md
CHANGED
|
@@ -41,7 +41,7 @@ This repo fixes all four. The math is **bit-exact** with the togethercomputer re
|
|
| 41 |
| Hyena state size | 8 |
|
| 42 |
| Positional encoding | RoPE (base = 10000, **linearly scaled by 1/16**) |
|
| 43 |
| Architecture | StripedHyena (alternating Hyena / MHA blocks) |
|
| 44 |
-
| Max sequence length | 131 072 |
|
| 45 |
| Training dtype | bfloat16 (Hyena modal-form `poles` / `residues` kept in fp32) |
|
| 46 |
|
| 47 |
## Pretraining
|
|
|
|
| 41 |
| Hyena state size | 8 |
|
| 42 |
| Positional encoding | RoPE (base = 10000, **linearly scaled by 1/16**) |
|
| 43 |
| Architecture | StripedHyena (alternating Hyena / MHA blocks) |
|
| 44 |
+
| Max sequence length | 131 072 (training context; StripedHyena supports extension beyond this with adjusted FFT buffers) |
|
| 45 |
| Training dtype | bfloat16 (Hyena modal-form `poles` / `residues` kept in fp32) |
|
| 46 |
|
| 47 |
## Pretraining
|