--- quantized_by: PollardWeights pipeline_tag: image-text-to-text base_model: kai-os/Carnice-V3 base_model_relation: quantized license: apache-2.0 language: - en tags: - pollard - gguf - llama.cpp - qwen35 - multimodal - vision - image-text-to-text - hermes-agent - tool-use - agent - conversational --- # Pollard quantizations of Carnice-V3-27b by kai-os > ### Pollard shrank this model: **47.6 GB (f16) → 12.08 GB** — **75% smaller, 3.9× down**, vs ~12.6 GB at NVFP4. > Full quant ladder below — pick the size that fits your machine. **Carnice on a Mac, not a 3090.** [Carnice-V3-27b](https://huggingface.co/kai-os/Carnice-V3) is a 27B agent model (Qwen3.8-27B + a Hermes-agent SFT distilled from Qwen3.8-Max) that punches far above its size on agent tasks — built to fit a **3090 / 24 GB**. Pollard sizes it to **your** box: the recommended build is **~12 GB and runs on a 16 GB Mac.** Built with **[Pollard Weights](https://github.com/WestWaters/pollard-weights)** on a runtime with native `qwen35` support. Original model: https://huggingface.co/kai-os/Carnice-V3 ## Model details | | | |---|---| | Parameter count | 27B (dense) | | Architecture | `qwen35` (65 layers, 5120 hidden) | | Context length | 262,144 (256K) | | Input support | **text + image + video** (multimodal) | | Specialty | Hermes-agent, tool-use — "beats models 10× the size" on agent tasks | | imatrix | yes — Qwen3.8-27B importance matrix (same base; transferable) | | Source | Q8_0 (near-lossless; ~99.9% of bf16) | ## Which file should I choose? - **~16 GB Mac / 12 GB VRAM** → **`IQ3_S`** (12 GB). Runs the 27B where a 3090 was the bar. **Recommended headline build.** - **~24 GB (a 3090)** → **`IQ4_XS`** (15.7 GB) — *smaller than Carnice's own build*, more fidelity. - **~32 GB** → **`Q6_K`** (19.6 GB) — near-lossless. ## Available files | file | quant | size | held-out PPL | M4 tok/s | |---|---|---|---|---| | [Carnice-V3-27b-Pollard-IQ3_S.gguf](https://huggingface.co/PollardWeights/Carnice-V3-27b-Pollard/blob/main/Carnice-V3-27b-Pollard-IQ3_S.gguf) | IQ3_S mix | 12.08 GB | 7.88 | ~6.9 | | [Carnice-V3-27b-Pollard-IQ4_XS.gguf](https://huggingface.co/PollardWeights/Carnice-V3-27b-Pollard/blob/main/Carnice-V3-27b-Pollard-IQ4_XS.gguf) | IQ4_XS mix | 15.72 GB | — | 24 GB tier | | [Carnice-V3-27b-Pollard-Q6_K.gguf](https://huggingface.co/PollardWeights/Carnice-V3-27b-Pollard/blob/main/Carnice-V3-27b-Pollard-Q6_K.gguf) | Q6_K mix | 19.63 GB | — | 32 GB tier | | [mmproj-Carnice-V3-BF16.gguf](https://huggingface.co/PollardWeights/Carnice-V3-27b-Pollard/blob/main/mmproj-Carnice-V3-BF16.gguf) | vision projector (f16, **not quantized**) | 0.93 GB | — | ship w/ any | *(PPL/tok-s measured only on the IQ3 tier — the larger tiers exceed the test Mac's 16 GB, which is the whole point: those are for 24 GB / 32 GB boxes.)* *(tok/s is a 16 GB Mac at IQ3 — bandwidth-bound with partial Metal offload; it's markedly faster on 24 GB+ with the whole model resident, and on any CUDA GPU.)* ## Prompt format Carnice uses ChatML (Qwen3.5), with tool-use and agent formatting: ``` <|im_start|>system {system_prompt}<|im_end|> <|im_start|>user {prompt}<|im_end|> <|im_start|>assistant ``` ## Multimodal (vision + video) Ship the **mmproj** alongside the model (it's kept at f16 — never quantize the projector) and run with llama.cpp's multimodal CLI/server: ```bash llama-mtmd-cli -m Carnice-V3-27b-Pollard-IQ3_S.gguf \ --mmproj mmproj-Carnice-V3-BF16.gguf --image photo.jpg -p "Describe this image." # or serve vision: llama-server -m Carnice-V3-27b-Pollard-IQ3_S.gguf --mmproj mmproj-Carnice-V3-BF16.gguf -ngl 99 ``` ## Download ```bash pip install -U "huggingface_hub[cli]" hf download PollardWeights/Carnice-V3-27b-Pollard \ --include "Carnice-V3-27b-Pollard-IQ3_S.gguf" "mmproj-Carnice-V3-BF16.gguf" --local-dir ./ ``` ## How to run (text) ```bash curl -LsSf https://llama.app/install.sh | sh llama-server -hf PollardWeights/Carnice-V3-27b-Pollard:IQ3_S # OpenAI API + web UI :8080 llama-cli -m Carnice-V3-27b-Pollard-IQ3_S.gguf -ngl 99 -st -p "Plan a 3-step task." ``` Needs a **recent llama.cpp with `qwen35` support**. Also runs in LM Studio, koboldcpp, Jan, etc. once they ship a new-enough llama.cpp. ## Notes - **License:** Apache-2.0, inherited from Carnice-V3 / Qwen3.8-27B. - **Requantized from Q8_0** (near-lossless source; an f16 source is marginally better, but for an IQ3/IQ4 target the difference is negligible). - imatrix is the **base Qwen3.8-27B** matrix (Carnice is a merged-LoRA fine-tune of the same base, so it transfers); imatrix-uncovered tensors are pinned to `q6_K`. - **Quantized, not fine-tuned** — same weights, sized to your machine. ## Credits - Base model: [kai-os/Carnice-V3](https://huggingface.co/kai-os/Carnice-V3) — Qwen3.8-27B + Hermes-agent SFT (thanks to NousResearch, LambdaAPI, per the author). - Foundation: [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B). - Quantization runtime: [llama.cpp](https://github.com/ggml-org/llama.cpp). - Method + tooling: [Pollard Weights](https://github.com/WestWaters/pollard-weights) — *measure first, no claim before a number.*