lower quants?
If it was possible could you quantize the model for more quants like iq2_xss and xs?
i have 12gb vram on my gpu
If it was possible could you quantize the model for more quants like iq2_xss and xs?
i have 12gb vram on my gpu
Hi! Good news — this is a MoE model, so 12GB VRAM is enough. You don't need lower quants.
The I-Compact (17 GB) quant in this repo can run on your 12GB VRAM setup in two ways:
llama.cpp: Just use --fit on (enabled by default), which automatically allocates the model layers across GPU and CPU based on your available VRAM. You can also use --cpu-moe or --n-cpu-moe N for finer control over MoE expert offloading. The beauty of MoE models is that expert weights are only loaded when selected by the router, so most experts can safely sit in CPU/RAM.
LM Studio: There's a "Force Model Expert Weights onto CPU" option in settings that offloads MoE expert weights to CPU while keeping attention layers on GPU — saving VRAM while maintaining performance.
Give it a try!
If it was possible could you quantize the model for more quants like iq2_xss and xs?
i have 12gb vram on my gpu
Hi! Good news — this is a MoE model, so 12GB VRAM is enough. You don't need lower quants.
The I-Compact (17 GB) quant in this repo can run on your 12GB VRAM setup in two ways:
llama.cpp: Just use --fit on (enabled by default), which automatically allocates the model layers across GPU and CPU based on your available VRAM. You can also use --cpu-moe or --n-cpu-moe N for finer control over MoE expert offloading. The beauty of MoE models is that expert weights are only loaded when selected by the router, so most experts can safely sit in CPU/RAM.
LM Studio: There's a "Force Model Expert Weights onto CPU" option in settings that offloads MoE expert weights to CPU while keeping attention layers on GPU — saving VRAM while maintaining performance.
Give it a try!
hey there yes i know of the cpu moe offloading im using a a llama.cpp eden fork and also
my concern is the speed i downloaded bf16 gguf and im trying to quantize it to iq2 xss range at least xd i have working qwen 3.6 35b already at that quant so
could u upload a imatrix please i would appreciate it
If it was possible could you quantize the model for more quants like iq2_xss and xs?
i have 12gb vram on my gpu
Hi! Good news — this is a MoE model, so 12GB VRAM is enough. You don't need lower quants.
The I-Compact (17 GB) quant in this repo can run on your 12GB VRAM setup in two ways:
llama.cpp: Just use --fit on (enabled by default), which automatically allocates the model layers across GPU and CPU based on your available VRAM. You can also use --cpu-moe or --n-cpu-moe N for finer control over MoE expert offloading. The beauty of MoE models is that expert weights are only loaded when selected by the router, so most experts can safely sit in CPU/RAM.
LM Studio: There's a "Force Model Expert Weights onto CPU" option in settings that offloads MoE expert weights to CPU while keeping attention layers on GPU — saving VRAM while maintaining performance.
Give it a try!
hey there yes i know of the cpu moe offloading im using a a llama.cpp eden fork and also
my concern is the speed i downloaded bf16 gguf and im trying to quantize it to iq2 xss range at least xd i have working qwen 3.6 35b already at that quant so
could u upload a imatrix please i would appreciate it
the 17gb is too much for my use case
If it was possible could you quantize the model for more quants like iq2_xss and xs?
i have 12gb vram on my gpu
Hi! Good news — this is a MoE model, so 12GB VRAM is enough. You don't need lower quants.
The I-Compact (17 GB) quant in this repo can run on your 12GB VRAM setup in two ways:
llama.cpp: Just use --fit on (enabled by default), which automatically allocates the model layers across GPU and CPU based on your available VRAM. You can also use --cpu-moe or --n-cpu-moe N for finer control over MoE expert offloading. The beauty of MoE models is that expert weights are only loaded when selected by the router, so most experts can safely sit in CPU/RAM.
LM Studio: There's a "Force Model Expert Weights onto CPU" option in settings that offloads MoE expert weights to CPU while keeping attention layers on GPU — saving VRAM while maintaining performance.
Give it a try!
hey there yes i know of the cpu moe offloading im using a a llama.cpp eden fork and also
my concern is the speed i downloaded bf16 gguf and im trying to quantize it to iq2 xss range at least xd i have working qwen 3.6 35b already at that quant so
could u upload a imatrix please i would appreciate itthe 17gb is too much for my use case
Hi! I've uploaded a new I-Mini quant (Ornith-1.0-35B-MTP-APEX-I-Mini.gguf, 13.35 GB) — it's the most compact APEX tier I can offer without seriously compromising quality.
Going lower than I-Mini (e.g. IQ2_XSS/IQ2_XS range) would severely degrade the model's performance, and I don't recommend it for this model. The expert weights and attention layers are already at very low precision in I-Mini, and pushing further would break the model's reasoning and coding capabilities.
If 13.35 GB is still too large for your use case, I'd suggest trying the I-Mini with --fit on (llama.cpp) so it can offload some expert weights to CPU. Hope that helps!
If it was possible could you quantize the model for more quants like iq2_xss and xs?
i have 12gb vram on my gpu
Hi! Good news — this is a MoE model, so 12GB VRAM is enough. You don't need lower quants.
The I-Compact (17 GB) quant in this repo can run on your 12GB VRAM setup in two ways:
llama.cpp: Just use --fit on (enabled by default), which automatically allocates the model layers across GPU and CPU based on your available VRAM. You can also use --cpu-moe or --n-cpu-moe N for finer control over MoE expert offloading. The beauty of MoE models is that expert weights are only loaded when selected by the router, so most experts can safely sit in CPU/RAM.
LM Studio: There's a "Force Model Expert Weights onto CPU" option in settings that offloads MoE expert weights to CPU while keeping attention layers on GPU — saving VRAM while maintaining performance.
Give it a try!
hey there yes i know of the cpu moe offloading im using a a llama.cpp eden fork and also
my concern is the speed i downloaded bf16 gguf and im trying to quantize it to iq2 xss range at least xd i have working qwen 3.6 35b already at that quant so
could u upload a imatrix please i would appreciate itthe 17gb is too much for my use case
Hi! I've uploaded a new I-Mini quant (Ornith-1.0-35B-MTP-APEX-I-Mini.gguf, 13.35 GB) — it's the most compact APEX tier I can offer without seriously compromising quality.
Going lower than I-Mini (e.g. IQ2_XSS/IQ2_XS range) would severely degrade the model's performance, and I don't recommend it for this model. The expert weights and attention layers are already at very low precision in I-Mini, and pushing further would break the model's reasoning and coding capabilities.
If 13.35 GB is still too large for your use case, I'd suggest trying the I-Mini with --fit on (llama.cpp) so it can offload some expert weights to CPU. Hope that helps!
alright thanks a lot i will try it
did it work, also hows the compacts performance