Instructions to use mradermacher/A3-Qwen3.5-2B-i1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mradermacher/A3-Qwen3.5-2B-i1-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("mradermacher/A3-Qwen3.5-2B-i1-GGUF", dtype="auto")

llama-cpp-python

How to use mradermacher/A3-Qwen3.5-2B-i1-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="mradermacher/A3-Qwen3.5-2B-i1-GGUF",
	filename="A3-Qwen3.5-2B.i1-IQ1_M.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use mradermacher/A3-Qwen3.5-2B-i1-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf mradermacher/A3-Qwen3.5-2B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf mradermacher/A3-Qwen3.5-2B-i1-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf mradermacher/A3-Qwen3.5-2B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf mradermacher/A3-Qwen3.5-2B-i1-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf mradermacher/A3-Qwen3.5-2B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf mradermacher/A3-Qwen3.5-2B-i1-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf mradermacher/A3-Qwen3.5-2B-i1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf mradermacher/A3-Qwen3.5-2B-i1-GGUF:Q4_K_M

Use Docker

docker model run hf.co/mradermacher/A3-Qwen3.5-2B-i1-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use mradermacher/A3-Qwen3.5-2B-i1-GGUF with Ollama:
```
ollama run hf.co/mradermacher/A3-Qwen3.5-2B-i1-GGUF:Q4_K_M
```

Unsloth Studio

How to use mradermacher/A3-Qwen3.5-2B-i1-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for mradermacher/A3-Qwen3.5-2B-i1-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for mradermacher/A3-Qwen3.5-2B-i1-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for mradermacher/A3-Qwen3.5-2B-i1-GGUF to start chatting

How to use mradermacher/A3-Qwen3.5-2B-i1-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf mradermacher/A3-Qwen3.5-2B-i1-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "mradermacher/A3-Qwen3.5-2B-i1-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use mradermacher/A3-Qwen3.5-2B-i1-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf mradermacher/A3-Qwen3.5-2B-i1-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default mradermacher/A3-Qwen3.5-2B-i1-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat new
Docker Model Runner
How to use mradermacher/A3-Qwen3.5-2B-i1-GGUF with Docker Model Runner:
```
docker model run hf.co/mradermacher/A3-Qwen3.5-2B-i1-GGUF:Q4_K_M
```

Lemonade

How to use mradermacher/A3-Qwen3.5-2B-i1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mradermacher/A3-Qwen3.5-2B-i1-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.A3-Qwen3.5-2B-i1-GGUF-Q4_K_M

List all available models

lemonade list

mradermacher commited on Apr 14

Commit

27e4f5b

verified ·

1 Parent(s): e4b39cd

uploaded from nico1

Browse files

Files changed (25) hide show

.gitattributes +24 -0
A3-Qwen3.5-2B.i1-IQ1_M.gguf +3 -0
A3-Qwen3.5-2B.i1-IQ1_S.gguf +3 -0
A3-Qwen3.5-2B.i1-IQ2_M.gguf +3 -0
A3-Qwen3.5-2B.i1-IQ2_S.gguf +3 -0
A3-Qwen3.5-2B.i1-IQ2_XS.gguf +3 -0
A3-Qwen3.5-2B.i1-IQ2_XXS.gguf +3 -0
A3-Qwen3.5-2B.i1-IQ3_M.gguf +3 -0
A3-Qwen3.5-2B.i1-IQ3_S.gguf +3 -0
A3-Qwen3.5-2B.i1-IQ3_XS.gguf +3 -0
A3-Qwen3.5-2B.i1-IQ3_XXS.gguf +3 -0
A3-Qwen3.5-2B.i1-IQ4_NL.gguf +3 -0
A3-Qwen3.5-2B.i1-IQ4_XS.gguf +3 -0
A3-Qwen3.5-2B.i1-Q2_K.gguf +3 -0
A3-Qwen3.5-2B.i1-Q2_K_S.gguf +3 -0
A3-Qwen3.5-2B.i1-Q3_K_L.gguf +3 -0
A3-Qwen3.5-2B.i1-Q3_K_M.gguf +3 -0
A3-Qwen3.5-2B.i1-Q3_K_S.gguf +3 -0
A3-Qwen3.5-2B.i1-Q4_0.gguf +3 -0
A3-Qwen3.5-2B.i1-Q4_1.gguf +3 -0
A3-Qwen3.5-2B.i1-Q4_K_M.gguf +3 -0
A3-Qwen3.5-2B.i1-Q4_K_S.gguf +3 -0
A3-Qwen3.5-2B.i1-Q5_K_M.gguf +3 -0
A3-Qwen3.5-2B.i1-Q5_K_S.gguf +3 -0
A3-Qwen3.5-2B.i1-Q6_K.gguf +3 -0

.gitattributes CHANGED Viewed

@@ -34,3 +34,27 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 A3-Qwen3.5-2B.imatrix.gguf filter=lfs diff=lfs merge=lfs -text

 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 A3-Qwen3.5-2B.imatrix.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-IQ1_S.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-IQ2_M.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-IQ2_S.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-IQ2_XS.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-IQ3_XS.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-IQ3_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-Q2_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+A3-Qwen3.5-2B.i1-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text

A3-Qwen3.5-2B.i1-IQ1_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d7758263968140ad66899172d908d36b4460339a12f91d9857225e3d1dacfec8
+size 915075616

A3-Qwen3.5-2B.i1-IQ1_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ac77d909f74a37a168e1585213c0e18c88cb9bb13a386d90c04a6fe51d94211a
+size 889832992

A3-Qwen3.5-2B.i1-IQ2_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:34b5afbf9022f8d7831bb3f4056d7e802276ab42eb61e6e174839ff2845a04ef
+size 1084270112

A3-Qwen3.5-2B.i1-IQ2_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d8b52011e5160afedfcce19c1af1c348d8ac50ff92833e79da342fc258e45ed7
+size 1050613280

A3-Qwen3.5-2B.i1-IQ2_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a91ce675889f978dac3877dbdcf8835a5d865c73f69f80da4f36392853c50b41
+size 991589920

A3-Qwen3.5-2B.i1-IQ2_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:787e056671e658420950b258a975851cc1162ca1c35f6abb1ae14998983fecdd
+size 957146656

A3-Qwen3.5-2B.i1-IQ3_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b44acd364cdde189d487a803dec00877e766308d4a2507dfff0f8d7c7a11a83c
+size 1277967904

A3-Qwen3.5-2B.i1-IQ3_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6a62571882788bb7b243edc614cfc5395b5b46ef20da523a7318cda70c939fc4
+size 1269612064

A3-Qwen3.5-2B.i1-IQ3_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0138f27cd9c24c69cceae5dd9648ddff06584fb2f0173afa612297853acc3763
+size 1245724192

A3-Qwen3.5-2B.i1-IQ3_XXS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8ed9f492ce610937d05f696f67be39cb0a21f301196ed8890d52d5b0272f9677
+size 1146275360

A3-Qwen3.5-2B.i1-IQ4_NL.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4a88b84e1bb7554907c07b3579ffc7469a0fc35cedf5392a0b5e4eff319442e1
+size 1517650464

A3-Qwen3.5-2B.i1-IQ4_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0ada09a6667e1e6dc6508464210e88b9dbd758ff5953b57060b9e57d88f12f52
+size 1466135072

A3-Qwen3.5-2B.i1-Q2_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b90f23afe3b045ad19d96394146d352af6e09b4c9891c0ee2884fc28dfa26df1
+size 1135413792

A3-Qwen3.5-2B.i1-Q2_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:147dacd6c2fa16d877ee327c97d3a83dac7bfeba66745e5c81da56d933ac9d4e
+size 1111034400

A3-Qwen3.5-2B.i1-Q3_K_L.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b601e7648a2fb252d6219675e3ffbe4533e096a0adcdd318d4dd047b96737114
+size 1383054880

A3-Qwen3.5-2B.i1-Q3_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c19455bb762d35e3b82212ea0f7e7d1969f869e3ec395b0595edf33ad4bae415
+size 1317781024

A3-Qwen3.5-2B.i1-Q3_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fa81a62ae9262bfeebf6b2d59fcee90101b6e68abe40910f5777ce220d39b6cc
+size 1238695456

A3-Qwen3.5-2B.i1-Q4_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7c4d2a294e412f92974fe68ce1a2650cd6f1cb5f4c4f950a87892e05c4b47341
+size 1490911776

A3-Qwen3.5-2B.i1-Q4_1.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:32f07bf175920457695802911ea90cd3f07cf33bdd427905b8f45cbf10486378
+size 1606132256

A3-Qwen3.5-2B.i1-Q4_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9501373296839bd9c314095799788ebcb61c1192164da0f7abdb9d0de025fa9d
+size 1560461856

A3-Qwen3.5-2B.i1-Q4_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f588cbfbbb2f42add535d570a90847f52d1cc1c040a66c8ab3b40f69a9bbb2e7
+size 1498120736

A3-Qwen3.5-2B.i1-Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c16ed4381fecdef1482a767c0111e8f82021cb5cb4156ccb13cafcd7243a8024
+size 1760756256

A3-Qwen3.5-2B.i1-Q5_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e6c1ad4c3e4def92e9a590537e156775f27b9d40806cefbcfd2da9e35df17e74
+size 1723712032

A3-Qwen3.5-2B.i1-Q6_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7606ac1eaba3d40be2fbe8c1f064b1a273c65f3b6a5be91d6857f9b2034dac21
+size 1973569056