JaynadTodkarAmol/resilientai-team-mcte-mistral-voxtral-realtime-gptq-marlin

Team MCTE - Resilient AI Submission - Voxtral Mini 4B Realtime 2602

This is the WINNING Submission for the Audio-to-Text Model Compression Track of the Resilient AI Challenge, a first of its kind international open model compression challenge by the Govt of France & India, UNESCO & Coalition for Sustainable AI

This open weights contribution has been done by the 🇮🇳 MILITARY COLLEGE OF TELECOMMUNICATION ENGINEERING (MCTE), INDIAN ARMY

Category: audio-to-text

Inference server: vLLM 0.19.1 or newer

License: Apache-2.0, matching the original model

Launch

Prerequisites

• Linux x86_64
• NVIDIA GPU with a compatible NVIDIA driver
• NVIDIA L4 with 16 GB VRAM for the target evaluation environment
• Conda
• Python 3.11

No custom vLLM source changes, plugins, CUDA extensions, Python entrypoints, or external serving wrappers are required.

Environment Setup

From a shell with Conda available, create a clean Python 3.11 environment:

conda create -y -n voxtral-venv python=3.11 pip
conda activate voxtral-venv

Install the serving dependencies from this repository:

python -m pip install --no-cache-dir -r requirements.txt
python -m pip check

The validated requirements install vLLM 0.19.1 and include the audio and API dependencies required by Voxtral. pip check should report:

No broken requirements found.

Start the Server

Run this command from the repository root:

vllm serve "$(pwd)" --config vllm_config.yaml

For an evaluator using a Hugging Face repository URI, use:

vllm serve model_uri --config vllm_config.yaml

No additional vLLM command-line arguments or environment variables are required. The submitted vllm_config.yaml contains the tested L4 settings.

Verify the Server

After startup completes, the health endpoint should return HTTP 200:

curl -sS -o /dev/null -w '%{http_code}\n' \
  http://127.0.0.1:8000/health

An audio file can be transcribed with:

curl -sS -X POST http://127.0.0.1:8000/v1/audio/transcriptions \
  -F "file=@/path/to/audio.wav" \
  -F "model=$(pwd)" \
  -F "language=en"

The server was clean-room validated by creating a new Python 3.11 Conda environment, installing only requirements.txt, running the command above, checking /health and /v1/models, and serving an audio transcription.

Comprehensive Approach

Small Benchmark Dataset

We used a compact bilingual ASR validation set to iterate quickly while preserving coverage of two languages and varied audio lengths.

Source	Language	Split	Samples
LibriSpeech (openslr/librispeech_asr)	English	test-clean	10
FLEURS (google/fleurs)	Hindi (hi_in)	test	10
Total	English and Hindi		20

The corpus contains 205.165 seconds of audio and 518 reference words. English clips range from 3.315 to 23.315 seconds, and Hindi clips range from 5.1 to 17.1 seconds. This is a development set used for rapid testing.

Each becnhmarking run uses one unmeasured warmup request followed by sequential transcription requests. Corpus WER is computed after Unicode normalization, case folding, punctuation and symbol removal, and whitespace tokenization. The benchmark also records total latency, real-time factor, GPU utilization, peak VRAM, and power at 100 ms intervals using nvidia-smi. Reported energy excludes server startup, model loading, compilation, and warmup.

Experiment Sequence

The optimization work followed the competition priorities and kept the audio encoder, tokenizer, feature extraction, and transcription API unchanged.

1. Quantize decoder linear projections with symmetric GPTQ INT4, group size 128, and activation ordering disabled.
2. Test decoder layer dropping at 25, 24, 23, and 22 layers.
3. Compare FP16 and FP8 E4M3 KV caches through stock vLLM.
4. Compare GPTQ-Marlin, Compressed Tensors W4A16 Marlin, and native online FP8 weight execution on the NVIDIA L4.
5. Evaluate TurboQuant, TurboQuant-GPU, SHARD, AWQ feasibility, and reduced decoder sliding windows.
6. Promote only candidates that work through the required unmodified vLLM launch path without evaluator-side plugins or source patches.

Decoder Quantization And Layer Dropping

The following full-corpus development measurements were collected on a Tesla V100-SXM2 32 GB . They were used to select decoder depth before L4 runtime optimization. V100 energy and VRAM values must not be compared directly with the L4 table.

Candidate	Decoder layers	Corpus WER	Latency (s)	Gross energy (Wh)	Peak VRAM (MiB)
Original FP16	26	0.11969	61.144	3.0905	26,301
Decoder GPTQ INT4	26	0.12355	57.758	2.1771	26,423
GPTQ, drop layer 13	25	0.11583	56.623	2.1464	26,427
GPTQ, drop layers 12-13	24	0.11390	54.604	2.0573	26,429
GPTQ, drop layers 12-14	23	0.12162	54.515	2.0369	26,435
GPTQ, drop layers 11-14	22	0.13320	53.795	2.0443	26,375

The 24-layer model was selected. It had the best WER, reduced energy by 33.43% relative to the FP16 development baseline, and avoided the omissions observed in the 22-layer model. The small energy difference between 24 and 23 layers was within measurement variation.

NVIDIA L4 Runtime And Weight-Format Experiments

The following measurements use the full 20-sample corpus on one NVIDIA L4. Rows marked as averages combine two measured runs.

Candidate	Corpus WER	Latency (s)	Gross energy (Wh)	Peak VRAM (MiB)	Decision
Stock 24-layer GPTQ control	0.11390	60.310	1.1921	18,346	Superseded
GPTQ, FP16 KV, Triton control	0.11390	58.785	1.1657	18,154	Superseded
GPTQ, FP8 E4M3 KV, Triton, two-run average	0.11197	57.250	1.1351	18,154	Promoted to KV configuration
GPTQ-Marlin, FP8 KV, Triton, two-run average	0.11390	57.076	1.1309	17,953	Selected backend
Compressed Tensors W4A16 Marlin, two-run average	0.11390	57.231	1.1353	17,886	Rejected: no energy gain
Native online FP8 weights, 26 layers	0.11197	75.240	1.4943	18,064	Rejected: slower and higher energy
TurboQuant-vLLM TQ4 with compatibility patch	0.10811	196.346	2.7772	21,670	Rejected
TurboQuant-vLLM fused decode with compatibility patch	0.11197	166.910	2.4873	22,116	Rejected
Final submission, clean environment	0.11004	57.193	1.1344	17,856	Validated

Sliding-Window Experiment

Decoder windows of 8192, 4096, and 2048 were tested on a separate four-sample short/long English/Hindi subset. These values are not directly comparable with the 20-sample tables.

Decoder window	Samples	Corpus WER	Latency (s)	Gross energy (Wh)	Peak VRAM (MiB)
8192	4	0.08871	13.075	0.25398	18,152
4096	4	0.08871	13.092	0.25150	18,152
2048	4	0.08871	13.103	0.25295	18,152

All windows produced identical WER, and energy differences were below 1%. The clips generated fewer than 2048 decoder tokens, so window eviction was not exercised. No reduced-window variant was promoted.

Complete Method Decision Summary

Method investigated	Evidence	Outcome
Decoder GPTQ INT4	Full 20-sample benchmark	Retained
Decoder layer dropping	25, 24, 23, and 22 layers benchmarked	24 layers retained
GPTQ-Marlin	Two full L4 runs	Retained
FP8 E4M3 KV cache	FP16 control plus two FP8 runs	Retained
Compressed Tensors W4A16 Marlin	Two full L4 runs	Rejected: slightly higher energy
Native online FP8 weights	Full L4 run	Rejected: higher latency and energy
TurboQuant-vLLM	Two patched full L4 runs	Rejected: patches, custom kernels, high energy and VRAM
TurboQuant-GPU 0.1.8	Synthetic KV compatibility test	Rejected: rebuilt FP16 cache and no actual memory reduction
SHARD	Synthetic kernel and cache test	Rejected: custom cache, monkey patches, and custom Triton kernels
AWQ INT4	Native toolchain feasibility check	Blocked: toolchain did not recognize voxtral_realtime
Sliding-window reduction	Three four-sample L4 runs	Not promoted: no measured gain and eviction untested
Decoder width or KV-head reduction	Architecture review	Not attempted: shape changes require retraining and increase compatibility risk

Final Submission Candidate

The submitted model is derived from mistralai/Voxtral-Mini-4B-Realtime-2602 and contains these changes:

• Decoder depth reduced from 26 to 24 layers. Original layers 12 and 13 were removed, and retained layers were renumbered 0-23.
• Decoder attention and feed-forward linear projections use symmetric GPTQ INT4 weights with group size 128 and desc_act: false.
• Stock vLLM converts the checkpoint to its native GPTQ-Marlin runtime layout on the L4.
• KV cache uses FP8 E4M3 storage.
• Attention uses the stock vLLM TRITON_ATTN backend.
• Sequential evaluation uses one active sequence and a 2048-token batched prefill limit.
• Piecewise CUDA graphs are enabled through the standard vLLM configuration.

The following remain in their original FP16 form:

• causal audio encoder
• audio-language projector
• token embeddings
• normalization layers
• time-conditioning weights

The tokenizer, Tekken vocabulary, audio preprocessing, generation interface, and OpenAI-compatible transcription endpoint are unchanged. The final checkpoint is 4.2 GB (4,260,339,768 bytes) and requires no custom vLLM code, serving wrapper, plugin, CUDA extension, or evaluator-side patch.

vLLM Configuration

vllm_config.yaml contains only evaluator defaults, model-specific settings, and measured optimizations. It does not contain a local model path, tensor-parallel setting, or swap-space setting.

Configuration rationale:

Configuration	Why it is included
config-format: mistral	Tells vLLM to read the submitted consolidated Mistral checkpoint and its params.json metadata.
tokenizer-mode: mistral	Loads the submitted Mistral Tekken tokenizer from tekken.json instead of assuming a standard Transformers tokenizer.
gpu-memory-utilization: 0.85	Preserves the organizer-provided L4 evaluation default and leaves memory headroom for the model, audio processing, CUDA graphs, and runtime allocations.
max-model-len: 20000	Preserves the organizer-provided evaluation limit and prevents vLLM from reserving resources for the checkpoint maximum of 131072 tokens.
max-num-seqs: 1	Matches the sequential ASR evaluation workload, avoiding memory reservation and scheduler overhead for unused concurrent requests.
max-num-batched-tokens: 2048	Bounds each scheduler iteration and chunked-prefill batch, limiting transient memory and unnecessary work for the measured audio workload.
dtype: float16	Keeps the unquantized audio encoder, projector, embeddings, normalization layers, and time-conditioning weights in their validated FP16 format.
quantization: gptq_marlin	Loads the decoder GPTQ INT4 weights through the native L4-compatible Marlin implementation in vLLM; no external kernel package is required.
kv-cache-dtype: fp8_e4m3	Stores the decoder KV cache in FP8 E4M3 to reduce cache bandwidth and storage while retaining the measured WER.
attention-backend: TRITON_ATTN	Selects the built-in vLLM Triton attention backend, which supports Voxtral pooled causal Whisper attention; automatic selection previously chose an incompatible FlashInfer path.
compilation_config.cudagraph_mode: PIECEWISE	Enables built-in piecewise CUDA graphs in vLLM, reducing repeated launch overhead while supporting model operations that cannot be captured as one full graph.

The organizer-provided values for gpu-memory-utilization and max-model-len are explicit because local vLLM otherwise derives different defaults when this configuration file is supplied.

The final configuration intentionally does not contain model: .; the model is supplied by the evaluator through the vllm serve positional argument. It also omits disable-log-stats: true, which appeared in the local development configuration but only suppresses periodic logging and did not contribute to the measured optimization.

Known Runtime Notes

Automatic attention-backend selection previously chose FlashInfer. FlashInfer does not support Voxtral's pooled causal Whisper attention in this vLLM version. The submitted configuration explicitly selects stock TRITON_ATTN, which loaded and transcribed correctly.

vLLM converts the submitted GPTQ checkpoint to its GPTQ-Marlin runtime layout during loading. This is expected and does not require checkpoint conversion by the evaluator.

The checkpoint may emit informational warnings about FP8 KV-cache scaling factors. The tested configuration uses the default scale of 1.0 and remained within the target WER.

Environment

requirements.txt contains the minimal serving dependencies. pip-freeze.txt records the complete validated environment for debugging.