README.md

metadata

language:
  - ko
  - en
license: llama3
library_name: transformers
tags:
  - moe
  - awq
  - quantized
  - w4a16
  - compressed-tensors
  - vllm
  - llm-compressor
base_model: LGAI-EXAONE/K-EXAONE-236B-A23B

K-EXAONE-236B-A23B-W4A16-G128

W4A16 AWQ quantization of LGAI-EXAONE/K-EXAONE-236B-A23B, produced with llm-compressor.

This is the first W4A16 AWQ checkpoint for K-EXAONE-236B-A23B publicly available — the original model only has FP8 and GGUF variants on HuggingFace.

---

Model Details

Property	Value
Base model	LGAI-EXAONE/K-EXAONE-236B-A23B
Architecture	ExaoneMoeForCausalLM
Total parameters	~236B
Active parameters	~23B per token
Quantization method	AWQ (Activation-aware Weight Quantization)
Weight precision	INT4 (packed)
Activation precision	BF16
Group size	128
Quantization scope	All Linear layers except lm_head and gate projections
Compressed-tensors version	0.15.0
Context length	262,144 tokens
Languages	Korean, English

Architecture Highlights

• 48 transformer layers with mixed sliding-window (LLLG pattern) and full attention
• MoE layers: 47 sparse MoE layers + 1 dense MLP (layer 0)
• 128 routed experts + 1 shared expert per MoE layer; top-8 experts activated per token
• Sigmoid scoring with norm_topk_prob=True
• Hidden size: 6144, MoE intermediate size: 2048

---

Quantization Details

Quantization was performed using llm-compressor with a MoE-aware AWQ recipe.

Method: AWQ applies channel-wise scaling to minimize quantization error by protecting salient weights, using a calibration dataset to determine optimal scales.

Recipe highlights:

• scheme: W4A16 (INT4 weights, BF16 activations)
• group_size: 128
• n_grid: 20 (search resolution for AWQ scale optimization)
• duo_scaling: True
• Smooth mappings cover all MoE expert layers (layers 1–47) independently, plus attention and MLP projections
• Layer 0 (dense MLP) and lm_head are excluded from quantization
• Gate weight tensors are excluded from quantization

The full recipe is available in recipe.yaml.

Calibration dataset: neuralmagic/LLM_compression_calibration (512 samples, sequence length 2048)

---

Usage

vLLM (Recommended)

Install vLLM (≥0.6.0 recommended for compressed-tensors support):

pip install vllm

from vllm import LLM, SamplingParams

llm = LLM(
    model="Hyun9junn/K-EXAONE-236B-A23B-W4A16-G128",
    max_model_len=8192,
    trust_remote_code=True,   # K-EXAONE uses custom modeling code
    tensor_parallel_size=4,   # adjust to the number of GPUs available
)

sampling_params = SamplingParams(
    temperature=0.6,
    top_p=0.9,
    max_tokens=512,
)

tokenizer = llm.get_tokenizer()

prompts = [
    "What is the capital of South Korea?",
    "Explain the difference between MoE and dense transformer models.",
]

formatted_prompts = [
    tokenizer.apply_chat_template(
        [{"role": "user", "content": p}],
        tokenize=False,
        add_generation_prompt=True,
    )
    for p in prompts
]

outputs = llm.generate(formatted_prompts, sampling_params)

for prompt, output in zip(prompts, outputs):
    print(f"Prompt : {prompt}")
    print(f"Response: {output.outputs[0].text.strip()}")

Transformers

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "Hyun9junn/K-EXAONE-236B-A23B-W4A16-G128"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [{"role": "user", "content": "한국의 수도는 어디인가요?"}]
input_ids = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

output = model.generate(input_ids, max_new_tokens=256, temperature=0.6, top_p=0.9)
print(tokenizer.decode(output[0][input_ids.shape[-1]:], skip_special_tokens=True))

---

Hardware Requirements

Precision	Min VRAM
This model (W4A16)	~120 GB
Original BF16	~480 GB

Tested on: NVIDIA B200 (180 GB HBM3e).

For multi-GPU inference, set tensor_parallel_size in vLLM to the number of GPUs.

---

Files

File	Description
model-00001-of-00003.safetensors	Model weights shard 1/3
model-00002-of-00003.safetensors	Model weights shard 2/3
model-00003-of-00003.safetensors	Model weights shard 3/3
model.safetensors.index.json	Weight shard index
config.json	Model config with quantization metadata
recipe.yaml	llm-compressor AWQ recipe used for quantization
tokenizer.json	Tokenizer
tokenizer_config.json	Tokenizer config
chat_template.jinja	Chat template
generation_config.json	Default generation config

---

License

This model inherits the license of the base model LGAI-EXAONE/K-EXAONE-236B-A23B. Please refer to the original model page for license details.

---

Citation

If you use this model, please cite the original K-EXAONE work:

@misc{k-exaone-236b,
  title  = {K-EXAONE-236B-A23B},
  author = {LG AI Research},
  year   = {2025},
  url    = {https://huggingface.co/LGAI-EXAONE/K-EXAONE-236B-A23B}
}

Quantization produced by Hyun9junn using llm-compressor.