Instructions to use alonsopg/wmt26-layeraware-native-mlp-q4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use alonsopg/wmt26-layeraware-native-mlp-q4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="alonsopg/wmt26-layeraware-native-mlp-q4")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("alonsopg/wmt26-layeraware-native-mlp-q4", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use alonsopg/wmt26-layeraware-native-mlp-q4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "alonsopg/wmt26-layeraware-native-mlp-q4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alonsopg/wmt26-layeraware-native-mlp-q4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/alonsopg/wmt26-layeraware-native-mlp-q4

SGLang

How to use alonsopg/wmt26-layeraware-native-mlp-q4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "alonsopg/wmt26-layeraware-native-mlp-q4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alonsopg/wmt26-layeraware-native-mlp-q4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "alonsopg/wmt26-layeraware-native-mlp-q4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alonsopg/wmt26-layeraware-native-mlp-q4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use alonsopg/wmt26-layeraware-native-mlp-q4 with Docker Model Runner:
```
docker model run hf.co/alonsopg/wmt26-layeraware-native-mlp-q4
```

Native Layer-Aware MLP-q4 Extension

This constrained-track system applies native BitsAndBytes NF4 only to the 144 text-transformer MLP gate_proj, up_proj, and down_proj modules. Attention projections, embeddings, norms, the language-model head, and vision modules remain BF16. It uses no calibration or fine-tuning data.

Base model: google/gemma-3-12b-it
Runtime: PyTorch + Transformers + BitsAndBytes
Model artifact: mixed MLP-q4/BF16 model at workdir/model, or set MODEL_DIR when running.

This is the deployment-relevant extension of the selected-layer codec hypothesis. Unlike the codec ablation, quantized weights remain native 4-bit tensors at inference.

Setup

bash setup.sh

setup.sh installs this submission's runtime dependencies and the organizer modelzip helper package into ./.venv. When running outside the organizer repository, set MODELZIP_SOURCE.

Compress

bash compress.sh

compress.sh is optional documentation/reproducibility support. It applies NF4 with double quantization only to the 144 text-transformer MLP gate/up/down projections and serializes the mixed checkpoint under workdir/model.

Run

bash run.sh --lang-pair eng-zho_Hans --batch-size 8 --input input.txt --output output.txt

Local validation

On 332 reference-backed WMT25 English–Chinese records, this system obtained 28.30 chrF and 31.37 BLEU, compared with 27.67 chrF and 30.80 BLEU for the organizer global-q4 baseline. Peak local VRAM was 14.64 GiB on an RTX A6000. These development results are not official WMT26 scores.

Downloads last month: 148

Model tree for alonsopg/wmt26-layeraware-native-mlp-q4

Base model

google/gemma-3-12b-pt

Finetuned

google/gemma-3-12b-it

Finetuned

(368)

this model