Update README.md
Browse files
README.md
CHANGED
|
@@ -26,11 +26,4 @@ the target. Everything is RTN-quantized to int4 (compressed-tensors
|
|
| 26 |
|
| 27 |
Validated on 2x DGX Spark (GB10) at TP2 with `draft_tensor_parallel_size: 2`:
|
| 28 |
acceptance length 3.36 (78.6% draft acceptance) at 120k-token context,
|
| 29 |
-
decode 39.7 / 30.4 / 26.4 tok/s @512/65k/120k with 4-bit KV quantization.
|
| 30 |
-
|
| 31 |
-
Reference deployment (container image, recipes, GB10 operational notes):
|
| 32 |
-
see the `DEPLOYMENT.md` of the companion spark-vllm-docker fork.
|
| 33 |
-
|
| 34 |
-
Note: the int4 embedding requires the draft model's quantization config to be
|
| 35 |
-
applied to `embed_tokens` at load; vLLM builds that need the
|
| 36 |
-
`fix-eagle3-draft-embed-quant` mod from the deployment repo.
|
|
|
|
| 26 |
|
| 27 |
Validated on 2x DGX Spark (GB10) at TP2 with `draft_tensor_parallel_size: 2`:
|
| 28 |
acceptance length 3.36 (78.6% draft acceptance) at 120k-token context,
|
| 29 |
+
decode 39.7 / 30.4 / 26.4 tok/s @512/65k/120k with 4-bit KV quantization.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|