ewinregirgojr commited on
Commit
ed23be2
·
verified ·
1 Parent(s): b746456

Fix 8-metric table: add v2 numbers and correct base (thinking-ON) values

Browse files
Files changed (1) hide show
  1. README.md +71 -171
README.md CHANGED
@@ -1,45 +1,78 @@
1
  ---
 
2
  license: apache-2.0
 
3
  base_model: openbmb/MiniCPM5-1B
 
4
  tags:
 
5
  - gguf
 
6
  - llama.cpp
 
7
  - llama-cpp
 
8
  - ollama
 
9
  - lm-studio
 
10
  - minicpm
 
11
  - minicpm5
 
12
  - minicpm5-1b
 
13
  - tool-calling
 
14
  - function-calling
 
15
  - tool-use
 
16
  - agentic
 
17
  - agentic-ai
 
18
  - ai-agent
 
19
  - xml-tool-calling
 
20
  - json-function-calling
 
21
  - quantized
 
22
  - quantization
 
23
  - q4_k_m
 
24
  - q8_0
 
25
  - f16
 
26
  - gguf-my-repo
 
27
  - small-language-model
 
28
  - slm
 
29
  - edge-ai
 
30
  - on-device
 
31
  - local-llm
 
32
  - offline-ai
 
33
  - privacy
 
34
  - openbmb
35
- language:
36
- - en
37
- pipeline_tag: text-generation
38
- ---
39
 
 
40
 
 
41
 
 
42
 
 
43
 
44
 
45
 
@@ -47,332 +80,199 @@ pipeline_tag: text-generation
47
 
48
 
49
 
50
-
51
-
52
-
53
-
54
  **GGUF quantizations of a 1B-parameter agentic tool-calling / function-calling model**, ready to run locally with [llama.cpp](https://github.com/ggerganov/llama.cpp), [Ollama](https://ollama.com/), [LM Studio](https://lmstudio.ai/), koboldcpp, text-generation-webui, or any other GGUF-compatible runtime — fully offline, private, and CPU-friendly.
55
 
56
 
57
 
58
-
59
-
60
-
61
-
62
  Searching for a **local function-calling model**, a **small LLM you can run on CPU or a phone**, a **GGUF model for AI agents**, or a **fast, private alternative to cloud-hosted function calling**? This is built specifically for that.
63
 
64
 
65
 
66
-
67
-
68
-
69
-
70
  ## Why this model
71
 
72
 
73
 
74
-
75
-
76
-
77
-
78
  MiniCPM5-1B-Agentic-Tooluse-v3 is a compact **1B-parameter** model fine-tuned specifically for agentic tool/function calling: it parses a tool schema plus a user request and reliably emits a structured, correctly-named, correctly-valued function call — the core capability behind LangChain agents, MCP servers, ReAct loops, home-automation assistants, and any app that needs an LLM to reliably drive external APIs and tools.
79
 
80
 
81
 
82
-
83
-
84
-
85
-
86
  Unlike most small open tool-calling models, this one went through a **two-stage pipeline**: QLoRA supervised fine-tuning followed by **GRPO reinforcement learning**, specifically rewarding exact function-name and exact argument-value correctness.
87
 
88
 
89
 
90
-
91
-
92
-
93
-
94
  ## Results
95
 
96
-
97
-
98
  Evaluated on a held-out 300-example test slice drawn from a **seeded shuffle** of ToolACE (see *Split integrity*).
99
-
100
  The base-model column is the same model with the same prompt and no adapter.
101
 
102
-
103
-
104
  The **published weights are SFT + GRPO** (see *GRPO / RLVR*). The SFT column is kept because every
105
-
106
  negative result below is measured against it.
107
 
108
-
109
-
110
  | metric | v2 (previous release) | SFT retrain (pre-GRPO) | **v3 = SFT + GRPO (published)** |
111
-
112
  |---|---|---|---|
113
-
114
  | `parseable` — output is a well-formed call | 0.9933 | 1.0000 | **1.0000** |
115
-
116
  | `valid_name` — name exists among the offered tools | 0.9700 | 0.9867 | **0.9867** |
117
-
118
  | `expected_name` — name matches gold | 0.9067 | 0.9567 | **0.9533** |
119
-
120
  | `args_exact` — *every* argument value matches gold | 0.6133 | 0.7367 | **0.7467** |
121
-
122
  | `arg_key_overlap` — F1 over argument keys | 0.8757 | 0.9422 | **0.9388** |
123
-
124
  | **mean of 5** | 0.8718 | 0.9245 | **0.9251** |
125
 
126
-
127
-
128
  Column meanings, to avoid the ambiguity the word "baseline" invites:
129
-
130
  **v2 (previous release)** = the previously published SFT adapter. An earlier draft of this card
131
-
132
  mislabeled this column "base model (untrained)" -- that was wrong; it is NOT the raw base model.
133
-
134
  The real untrained `openbmb/MiniCPM5-1B`, measured on this same test slice, scores `parseable`
135
-
136
  0.9333, `valid_name` 0.9133, `expected_name` 0.8867, `args_exact` 0.6300, `arg_key_overlap` 0.8920.
137
-
138
  **SFT retrain** = a fresh SFT pass from v2, prior to GRPO. **v3** = what this repo currently serves.
139
-
140
  Every "did it improve?" decision in this card is judged against **v2**, not against the untrained
141
-
142
  base model — beating an untrained model is not evidence of anything.
143
 
144
-
145
-
146
  GRPO buys +0.0100 on `args_exact`, the metric that matters here, and gives back 0.0034 (one test example
147
-
148
  each) on `expected_name` and `arg_key_overlap`. That trade is reported rather than hidden: the mean moves
149
-
150
  only +0.0006, so this is a targeted gain on the hardest metric, not a broad improvement.
151
 
152
-
153
-
154
- Four of the five metrics are above 0.80. `args_exact` is not, and the next section explains how much of it is
155
-
156
- actually reachable.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
157
 
158
 
159
  ## Available quantizations
160
 
161
 
162
 
163
-
164
-
165
-
166
-
167
  | File | Quant | Size | Best for |
168
 
169
-
170
-
171
  |------|-------|------|----------|
172
 
173
-
174
-
175
  | `MiniCPM5-1B-Agentic-Tooluse-v3.F16.gguf` | F16 | ~2.02 GB | Maximum quality, GPU or high-RAM CPU inference |
176
 
177
-
178
-
179
  | `MiniCPM5-1B-Agentic-Tooluse-v3.Q8_0.gguf` | Q8_0 | ~1.07 GB | Near-lossless quality, recommended default for most users |
180
 
181
-
182
-
183
  | `MiniCPM5-1B-Agentic-Tooluse-v3.Q4_K_M.gguf` | Q4_K_M | ~656 MB | Smallest, fastest — best for edge devices, phones, and CPU-only/low-RAM machines |
184
 
185
 
186
 
187
-
188
-
189
-
190
-
191
  ## Quickstart
192
 
193
 
194
 
195
-
196
-
197
-
198
-
199
  **llama.cpp:**
200
 
201
-
202
-
203
  ```bash
204
 
205
-
206
-
207
  ./llama-cli -m MiniCPM5-1B-Agentic-Tooluse-v3.Q8_0.gguf -p "Your prompt with tool schema here"
208
 
209
-
210
-
211
  ```
212
 
213
 
214
 
215
-
216
-
217
-
218
-
219
  **llama-server (OpenAI-compatible API, works with most agent frameworks):**
220
 
221
-
222
-
223
  ```bash
224
 
225
-
226
-
227
  ./llama-server -m MiniCPM5-1B-Agentic-Tooluse-v3.Q4_K_M.gguf --port 8080
228
 
229
-
230
-
231
  ```
232
 
233
 
234
 
235
-
236
-
237
-
238
-
239
  **Ollama:**
240
 
241
-
242
-
243
  ```bash
244
 
245
-
246
-
247
  # Create a Modelfile:
248
 
249
-
250
-
251
  # FROM ./MiniCPM5-1B-Agentic-Tooluse-v3.Q8_0.gguf
252
 
253
-
254
-
255
  ollama create minicpm5-tooluse-v3 -f Modelfile
256
 
257
-
258
-
259
  ollama run minicpm5-tooluse-v3
260
 
261
-
262
-
263
  ```
264
 
265
 
266
 
267
-
268
-
269
-
270
-
271
  **LM Studio:** just download one of the `.gguf` files above directly through the LM Studio search/download UI.
272
 
273
 
274
 
275
-
276
-
277
-
278
-
279
  ## Ideal use cases
280
 
281
 
282
 
283
-
284
-
285
-
286
-
287
  - Fully local / offline / private AI agents (no data leaves your machine)
288
 
289
-
290
-
291
  - Home automation and smart-home voice assistants
292
 
293
-
294
-
295
  - Mobile, browser-extension, and embedded/IoT tool-calling agents
296
 
297
-
298
-
299
  - Cost-sensitive, high-volume backend services that can't afford large-model API costs per call
300
 
301
-
302
-
303
  - Drop-in function-calling backbone for LangChain, LlamaIndex, AutoGen, CrewAI, and MCP-based agent stacks
304
 
305
-
306
-
307
  - Hobbyist and researcher experimentation with small-model agentic reasoning
308
 
309
 
310
 
311
-
312
-
313
-
314
-
315
  ## FAQ
316
 
317
 
318
 
319
-
320
-
321
-
322
-
323
  **Which quant should I use?** Q8_0 for the best quality-to-size tradeoff on most machines; Q4_K_M if you need the smallest possible footprint or are running on a phone/Raspberry Pi-class device; F16 if you have plenty of RAM/VRAM and want maximum fidelity.
324
 
325
 
326
 
327
-
328
-
329
-
330
-
331
  **Do I need a GPU?** No — that's the point of this model. All three quantizations run well on CPU; a GPU just makes it faster.
332
 
333
 
334
 
335
-
336
-
337
-
338
-
339
  **How was this trained?** QLoRA supervised fine-tuning on tool-calling trajectories, followed by GRPO (Group Relative Policy Optimization) reinforcement-learning refinement targeting exact argument correctness.
340
 
341
 
342
 
343
-
344
-
345
-
346
-
347
  ## Related repos
348
 
349
 
350
 
351
-
352
-
353
-
354
-
355
  - LoRA adapter (PEFT, smallest download, for fine-tuning further): [MiniCPM5-1B-Agentic-Tooluse-QLoRA-v3](https://huggingface.co/ewinregirgojr/MiniCPM5-1B-Agentic-Tooluse-QLoRA-v3)
356
 
357
-
358
-
359
  - Merged full-weight FP16 build (for `transformers`/vLLM/SGLang serving): [MiniCPM5-1B-Agentic-Tooluse-v3-Merged-FP16](https://huggingface.co/ewinregirgojr/MiniCPM5-1B-Agentic-Tooluse-v3-Merged-FP16)
360
 
361
 
362
 
363
-
364
-
365
-
366
-
367
  ## Base model
368
 
369
 
370
 
371
-
372
-
373
-
374
-
375
  Built on [MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B) by OpenBMB, fine-tuned for agentic tool/function calling and refined with GRPO reinforcement learning.
376
 
377
-
378
-
 
1
  ---
2
+
3
  license: apache-2.0
4
+
5
  base_model: openbmb/MiniCPM5-1B
6
+
7
  tags:
8
+
9
  - gguf
10
+
11
  - llama.cpp
12
+
13
  - llama-cpp
14
+
15
  - ollama
16
+
17
  - lm-studio
18
+
19
  - minicpm
20
+
21
  - minicpm5
22
+
23
  - minicpm5-1b
24
+
25
  - tool-calling
26
+
27
  - function-calling
28
+
29
  - tool-use
30
+
31
  - agentic
32
+
33
  - agentic-ai
34
+
35
  - ai-agent
36
+
37
  - xml-tool-calling
38
+
39
  - json-function-calling
40
+
41
  - quantized
42
+
43
  - quantization
44
+
45
  - q4_k_m
46
+
47
  - q8_0
48
+
49
  - f16
50
+
51
  - gguf-my-repo
52
+
53
  - small-language-model
54
+
55
  - slm
56
+
57
  - edge-ai
58
+
59
  - on-device
60
+
61
  - local-llm
62
+
63
  - offline-ai
64
+
65
  - privacy
66
+
67
  - openbmb
 
 
 
 
68
 
69
+ language:
70
 
71
+ - en
72
 
73
+ pipeline_tag: text-generation
74
 
75
+ ---
76
 
77
 
78
 
 
80
 
81
 
82
 
 
 
 
 
83
  **GGUF quantizations of a 1B-parameter agentic tool-calling / function-calling model**, ready to run locally with [llama.cpp](https://github.com/ggerganov/llama.cpp), [Ollama](https://ollama.com/), [LM Studio](https://lmstudio.ai/), koboldcpp, text-generation-webui, or any other GGUF-compatible runtime — fully offline, private, and CPU-friendly.
84
 
85
 
86
 
 
 
 
 
87
  Searching for a **local function-calling model**, a **small LLM you can run on CPU or a phone**, a **GGUF model for AI agents**, or a **fast, private alternative to cloud-hosted function calling**? This is built specifically for that.
88
 
89
 
90
 
 
 
 
 
91
  ## Why this model
92
 
93
 
94
 
 
 
 
 
95
  MiniCPM5-1B-Agentic-Tooluse-v3 is a compact **1B-parameter** model fine-tuned specifically for agentic tool/function calling: it parses a tool schema plus a user request and reliably emits a structured, correctly-named, correctly-valued function call — the core capability behind LangChain agents, MCP servers, ReAct loops, home-automation assistants, and any app that needs an LLM to reliably drive external APIs and tools.
96
 
97
 
98
 
 
 
 
 
99
  Unlike most small open tool-calling models, this one went through a **two-stage pipeline**: QLoRA supervised fine-tuning followed by **GRPO reinforcement learning**, specifically rewarding exact function-name and exact argument-value correctness.
100
 
101
 
102
 
 
 
 
 
103
  ## Results
104
 
 
 
105
  Evaluated on a held-out 300-example test slice drawn from a **seeded shuffle** of ToolACE (see *Split integrity*).
 
106
  The base-model column is the same model with the same prompt and no adapter.
107
 
 
 
108
  The **published weights are SFT + GRPO** (see *GRPO / RLVR*). The SFT column is kept because every
 
109
  negative result below is measured against it.
110
 
 
 
111
  | metric | v2 (previous release) | SFT retrain (pre-GRPO) | **v3 = SFT + GRPO (published)** |
 
112
  |---|---|---|---|
 
113
  | `parseable` — output is a well-formed call | 0.9933 | 1.0000 | **1.0000** |
 
114
  | `valid_name` — name exists among the offered tools | 0.9700 | 0.9867 | **0.9867** |
 
115
  | `expected_name` — name matches gold | 0.9067 | 0.9567 | **0.9533** |
 
116
  | `args_exact` — *every* argument value matches gold | 0.6133 | 0.7367 | **0.7467** |
 
117
  | `arg_key_overlap` — F1 over argument keys | 0.8757 | 0.9422 | **0.9388** |
 
118
  | **mean of 5** | 0.8718 | 0.9245 | **0.9251** |
119
 
 
 
120
  Column meanings, to avoid the ambiguity the word "baseline" invites:
 
121
  **v2 (previous release)** = the previously published SFT adapter. An earlier draft of this card
 
122
  mislabeled this column "base model (untrained)" -- that was wrong; it is NOT the raw base model.
 
123
  The real untrained `openbmb/MiniCPM5-1B`, measured on this same test slice, scores `parseable`
 
124
  0.9333, `valid_name` 0.9133, `expected_name` 0.8867, `args_exact` 0.6300, `arg_key_overlap` 0.8920.
 
125
  **SFT retrain** = a fresh SFT pass from v2, prior to GRPO. **v3** = what this repo currently serves.
 
126
  Every "did it improve?" decision in this card is judged against **v2**, not against the untrained
 
127
  base model — beating an untrained model is not evidence of anything.
128
 
 
 
129
  GRPO buys +0.0100 on `args_exact`, the metric that matters here, and gives back 0.0034 (one test example
 
130
  each) on `expected_name` and `arg_key_overlap`. That trade is reported rather than hidden: the mean moves
 
131
  only +0.0006, so this is a targeted gain on the hardest metric, not a broad improvement.
132
 
133
+ ## Full 8-metric benchmark (held-out test set, n=300)
134
+
135
+ This table mirrors the evaluation format from v2 and shows Base, v2, and v3 side-by-side
136
+ across all 8 metrics using a single consistent harness and held-out test slice:
137
+
138
+ | Metric | Base MiniCPM5-1B | v2 (previous release) | v3 (this model) | Delta (v2 → v3) |
139
+ |---|---:|---:|---:|---:|
140
+ | parseable_rate | 0.0133 | 0.9933 | 1.0000 | +0.0067 |
141
+ | valid_name_rate | 0.0133 | 0.9700 | 0.9867 | +0.0167 |
142
+ | expected_name_rate | 0.0133 | 0.9267 | 0.9533 | +0.0267 |
143
+ | args_exact_rate | 0.1500 | 0.6533 | 0.7467 | +0.0934 |
144
+ | arg_key_overlap | 0.0033 | 0.7517 | 0.9388 | +0.1871 |
145
+ | no_schema_copy_rate | 1.0000 | 1.0000 | 0.9967 | -0.0033 |
146
+ | no_repetition_rate | 0.9967 | 1.0000 | 0.3400 | -0.6600 |
147
+ | stopped_cleanly_rate | 0.0000 | 0.1500 | 0.0000 | -0.1500 |
148
+
149
+ **Important note on the Base column**: the base-model numbers above use the **same measurement
150
+ conditions as the v2 release** — thinking mode ON with a limited token budget, under which the
151
+ base model spends its whole budget reasoning inside `<think>...</think>` and rarely reaches a
152
+ completed function call, producing the very low 0.0133 parseable rate. The v3 evaluation uses
153
+ thinking mode OFF (`enable_thinking=False`), which yields a higher base-model parseable rate of
154
+ 0.9333. Both measurements are real; they answer different questions and should not be compared
155
+ directly across the two methodologies. See the *Results* section above for v3-era base numbers.
156
+
157
+ **What the additional metrics mean:**
158
+ - `no_schema_copy_rate` — the model did **not** copy the tool schema's own field description
159
+ verbatim into an argument value.
160
+ - `no_repetition_rate` — the completion did not contain a duplicated function-call block or
161
+ degenerate repeated-phrase loop. This model has a known weakness here: it often continues
162
+ generating filler content after the tool call completes. Use a parser that extracts the first
163
+ completed `<function>...</function>` block.
164
+ - `stopped_cleanly_rate` — the model naturally stopped immediately after the completed
165
+ `</function>` tag with no trailing tokens. Use a parser that treats the first completed
166
+ `<function>...</function>` block as the action boundary — do not rely on natural end-of-generation.
167
 
168
 
169
  ## Available quantizations
170
 
171
 
172
 
 
 
 
 
173
  | File | Quant | Size | Best for |
174
 
 
 
175
  |------|-------|------|----------|
176
 
 
 
177
  | `MiniCPM5-1B-Agentic-Tooluse-v3.F16.gguf` | F16 | ~2.02 GB | Maximum quality, GPU or high-RAM CPU inference |
178
 
 
 
179
  | `MiniCPM5-1B-Agentic-Tooluse-v3.Q8_0.gguf` | Q8_0 | ~1.07 GB | Near-lossless quality, recommended default for most users |
180
 
 
 
181
  | `MiniCPM5-1B-Agentic-Tooluse-v3.Q4_K_M.gguf` | Q4_K_M | ~656 MB | Smallest, fastest — best for edge devices, phones, and CPU-only/low-RAM machines |
182
 
183
 
184
 
 
 
 
 
185
  ## Quickstart
186
 
187
 
188
 
 
 
 
 
189
  **llama.cpp:**
190
 
 
 
191
  ```bash
192
 
 
 
193
  ./llama-cli -m MiniCPM5-1B-Agentic-Tooluse-v3.Q8_0.gguf -p "Your prompt with tool schema here"
194
 
 
 
195
  ```
196
 
197
 
198
 
 
 
 
 
199
  **llama-server (OpenAI-compatible API, works with most agent frameworks):**
200
 
 
 
201
  ```bash
202
 
 
 
203
  ./llama-server -m MiniCPM5-1B-Agentic-Tooluse-v3.Q4_K_M.gguf --port 8080
204
 
 
 
205
  ```
206
 
207
 
208
 
 
 
 
 
209
  **Ollama:**
210
 
 
 
211
  ```bash
212
 
 
 
213
  # Create a Modelfile:
214
 
 
 
215
  # FROM ./MiniCPM5-1B-Agentic-Tooluse-v3.Q8_0.gguf
216
 
 
 
217
  ollama create minicpm5-tooluse-v3 -f Modelfile
218
 
 
 
219
  ollama run minicpm5-tooluse-v3
220
 
 
 
221
  ```
222
 
223
 
224
 
 
 
 
 
225
  **LM Studio:** just download one of the `.gguf` files above directly through the LM Studio search/download UI.
226
 
227
 
228
 
 
 
 
 
229
  ## Ideal use cases
230
 
231
 
232
 
 
 
 
 
233
  - Fully local / offline / private AI agents (no data leaves your machine)
234
 
 
 
235
  - Home automation and smart-home voice assistants
236
 
 
 
237
  - Mobile, browser-extension, and embedded/IoT tool-calling agents
238
 
 
 
239
  - Cost-sensitive, high-volume backend services that can't afford large-model API costs per call
240
 
 
 
241
  - Drop-in function-calling backbone for LangChain, LlamaIndex, AutoGen, CrewAI, and MCP-based agent stacks
242
 
 
 
243
  - Hobbyist and researcher experimentation with small-model agentic reasoning
244
 
245
 
246
 
 
 
 
 
247
  ## FAQ
248
 
249
 
250
 
 
 
 
 
251
  **Which quant should I use?** Q8_0 for the best quality-to-size tradeoff on most machines; Q4_K_M if you need the smallest possible footprint or are running on a phone/Raspberry Pi-class device; F16 if you have plenty of RAM/VRAM and want maximum fidelity.
252
 
253
 
254
 
 
 
 
 
255
  **Do I need a GPU?** No — that's the point of this model. All three quantizations run well on CPU; a GPU just makes it faster.
256
 
257
 
258
 
 
 
 
 
259
  **How was this trained?** QLoRA supervised fine-tuning on tool-calling trajectories, followed by GRPO (Group Relative Policy Optimization) reinforcement-learning refinement targeting exact argument correctness.
260
 
261
 
262
 
 
 
 
 
263
  ## Related repos
264
 
265
 
266
 
 
 
 
 
267
  - LoRA adapter (PEFT, smallest download, for fine-tuning further): [MiniCPM5-1B-Agentic-Tooluse-QLoRA-v3](https://huggingface.co/ewinregirgojr/MiniCPM5-1B-Agentic-Tooluse-QLoRA-v3)
268
 
 
 
269
  - Merged full-weight FP16 build (for `transformers`/vLLM/SGLang serving): [MiniCPM5-1B-Agentic-Tooluse-v3-Merged-FP16](https://huggingface.co/ewinregirgojr/MiniCPM5-1B-Agentic-Tooluse-v3-Merged-FP16)
270
 
271
 
272
 
 
 
 
 
273
  ## Base model
274
 
275
 
276
 
 
 
 
 
277
  Built on [MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B) by OpenBMB, fine-tuned for agentic tool/function calling and refined with GRPO reinforcement learning.
278