Instructions to use hsmin92/internvl35-4b-awq-w4a16-g128 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use hsmin92/internvl35-4b-awq-w4a16-g128 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="hsmin92/internvl35-4b-awq-w4a16-g128")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("hsmin92/internvl35-4b-awq-w4a16-g128")
model = AutoModelForMultimodalLM.from_pretrained("hsmin92/internvl35-4b-awq-w4a16-g128", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use hsmin92/internvl35-4b-awq-w4a16-g128 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "hsmin92/internvl35-4b-awq-w4a16-g128"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hsmin92/internvl35-4b-awq-w4a16-g128",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/hsmin92/internvl35-4b-awq-w4a16-g128

SGLang

How to use hsmin92/internvl35-4b-awq-w4a16-g128 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "hsmin92/internvl35-4b-awq-w4a16-g128" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hsmin92/internvl35-4b-awq-w4a16-g128",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "hsmin92/internvl35-4b-awq-w4a16-g128" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hsmin92/internvl35-4b-awq-w4a16-g128",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use hsmin92/internvl35-4b-awq-w4a16-g128 with Docker Model Runner:
```
docker model run hf.co/hsmin92/internvl35-4b-awq-w4a16-g128
```

internvl35-4b-awq-w4a16-g128 / quantization /recipe.py

hsmin92

Add AWQ W4A16 g128 checkpoint, model card, and quantization metadata

ff32ca7 verified 20 days ago

Raw

History Blame Contribute Delete

2.45 kB

	"""Compression recipe used to create this checkpoint.

	Run quantization from the original BF16 checkpoint, not from an already
	quantized checkpoint.

	AWQ needs calibration data. This build used 128 samples from
	`lmms-lab/flickr30k`, each rendered at 448x448 (a single InternVL patch) with
	generic English/Korean scene-description prompts. Re-run with in-domain images
	if your target distribution is far from everyday photography.

	Targets are regex-scoped to the language decoder, so the vision tower,
	multimodal projector, input embeddings, and lm_head are never matched and stay
	in BF16. `ignore` is therefore empty by design.
	"""

	from llmcompressor.modifiers.awq import AWQModifier
	from llmcompressor.modifiers.quantization import QuantizationModifier

	ATTENTION = r"re:^model\.language_model\.layers\.\d+\.self_attn\.(q_proj\|k_proj\|v_proj\|o_proj)$"
	MLP = r"re:^model\.language_model\.layers\.\d+\.mlp\.(gate_proj\|up_proj\|down_proj)$"

	RECIPE = [
	AWQModifier(
	mappings=[
	{
	"smooth_layer": r"re:^model\.language_model\.layers\.\d+\.input_layernorm$",
	"balance_layers": [
	r"re:^model\.language_model\.layers\.\d+\.self_attn\.q_proj$",
	r"re:^model\.language_model\.layers\.\d+\.self_attn\.k_proj$",
	r"re:^model\.language_model\.layers\.\d+\.self_attn\.v_proj$",
	],
	},
	{
	"smooth_layer": r"re:^model\.language_model\.layers\.\d+\.self_attn\.v_proj$",
	"balance_layers": [
	r"re:^model\.language_model\.layers\.\d+\.self_attn\.o_proj$",
	],
	},
	{
	"smooth_layer": r"re:^model\.language_model\.layers\.\d+\.post_attention_layernorm$",
	"balance_layers": [
	r"re:^model\.language_model\.layers\.\d+\.mlp\.gate_proj$",
	r"re:^model\.language_model\.layers\.\d+\.mlp\.up_proj$",
	],
	},
	{
	"smooth_layer": r"re:^model\.language_model\.layers\.\d+\.mlp\.up_proj$",
	"balance_layers": [
	r"re:^model\.language_model\.layers\.\d+\.mlp\.down_proj$",
	],
	},
	],
	duo_scaling=True,
	n_grid=20,
	),
	QuantizationModifier(
	targets=[ATTENTION, MLP],
	ignore=[],
	scheme="W4A16_ASYM",
	),
	]