Benchmark Notes
End-to-End Route Comparison
All routes were measured with the current Taurscribe Rust runtime and the same
first 30 entries from eval_manifest_30.jsonl. Audio was decoded, converted to
16 kHz mono, edge-trimmed, and processed through the same Granite pipeline.
Test machine:
- CPU: AMD Ryzen 7 8845HS
- integrated GPU: AMD Radeon 780M
- discrete GPU: NVIDIA GeForce RTX 4070 Laptop GPU
- sample count: 30 LibriSpeech
test-cleanutterances - mean processed audio duration: 8.47 seconds
| Route | Load | Mean | Median | P95 | Mean RTF | Mean WER |
|---|---|---|---|---|---|---|
| CUDA artifact, RTX 4070 | 16.610 s | 0.250 s | 0.245 s | 0.303 s | 0.040 | 4.31% |
| Portable, full DirectML, Radeon 780M | 12.770 s | 4.045 s | 3.513 s | 4.518 s | 0.643 | 4.31% |
| Portable, CPU, 8 intra-op threads | 4.698 s | 8.823 s | 8.840 s | 10.168 s | 1.415 | 4.31% |
| Historical hybrid, 1-thread CPU encoder + DirectML | 8.421 s | 13.135 s | 12.874 s | 14.161 s | 2.194 | 4.31% |
CUDA load time includes a 5.594-second performance warmup. DirectML was forced to DXGI device 0, identified as the Radeon 780M on this laptop. Every portable graph was loaded on DirectML and runtime fallback was disabled for that run.
Interpretation
- Full DirectML was approximately 2.18 times faster than eight-thread CPU on this end-to-end workload.
- The earlier 5.2-second CPU prediction came from adding isolated per-graph probes and was not representative of full transcription. The measured end-to-end CPU mean is 8.823 seconds.
- All routes produced the same 4.31% mean WER on this subset, so the execution route did not change recognition accuracy here.
- Thirty utterances are sufficient for route validation but not for publishing a general model-quality claim. Consult the IBM base model card for the full Granite evaluation.
RTF is transcription time divided by processed audio duration. Values below 1.0 are faster than real time.