sukhrobnurali/Qwen3-1.7B-xlam-toolcall

Qwen3-1.7B xLAM Tool-Calling (QLoRA)

QLoRA fine-tune of Qwen/Qwen3-1.7B specialised for single-turn, single-call function/tool calling, evaluated BFCL-style (AST matching, not string match) against the untuned base both in-distribution (held-out xLAM) and out-of-distribution (BFCL).

In-distribution: xLAM held-out (n=300)

metric	base	tuned	delta
full_acc	0.657	0.813	+0.157
name_acc	0.843	0.990	+0.147
parse_rate	0.850	0.993	+0.143
halluc_rate	0.000	0.000	+0.000

Out-of-distribution: BFCL v4 live_simple (n=258)

metric	base	tuned	delta
full_acc	0.484	0.609	+0.124
name_acc	0.717	0.942	+0.225
parse_rate	0.721	0.953	+0.233
halluc_rate	0.004	0.012	+0.008

• full_acc - full-call AST/exact match (name + all arguments correct)
• name_acc - function-name accuracy
• parse_rate - emitted a parseable <tool_call> JSON
• halluc_rate - called a function not in the provided tools (lower is better)

The gain transfers to BFCL live_simple (real-world queries never seen in training), so the model learned tool-calling skill, not just xLAM's surface style. Absolute BFCL numbers are lower (expected for the live set); the delta is the signal.

Evaluation methodology

AST matching, not string comparison. Function name normalised (dots->underscores); arguments compared with type-aware + string-normalised (case/punctuation/whitespace-insensitive) values; lists order-sensitive; numbers and numeric strings compared numerically.

• xLAM held-out: disjoint from training (carved before sampling, deduped by query). Single ground-truth, so a parameter on only one side is accepted when empty or equal to its schema default (optional-arg leniency).
• BFCL: uses BFCL's possible_answer (each parameter maps to a list of acceptable values; an empty-string entry marks an omittable parameter).
• Matcher is committed (eval.py, unit-tested); all generations are saved, so every number re-scores from disk.

Caveat (honest lower bound): xLAM full_acc understates true quality - some held-out golds are unanswerable from the query (specific API IDs) or stored as unevaluated Python-expression strings, which cap absolute accuracy for both models equally. The base-vs-tuned delta is unaffected.

Intended use

Single-turn function calling: given a user query and tool/function schemas, emit one <tool_call> with the correct function name and arguments, using Qwen3's native tool template.

Out of scope

• Multi-turn / agentic tool dialogues
• Executable / runtime tool categories
• Parallel or multiple calls in one turn (training restricted to exactly one call)
• The multimodal Qwen3.5 path

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import re, json

m = AutoModelForCausalLM.from_pretrained("sukhrobnurali/Qwen3-1.7B-xlam-toolcall", torch_dtype="auto", device_map="auto")
tok = AutoTokenizer.from_pretrained("sukhrobnurali/Qwen3-1.7B-xlam-toolcall")

tools = [{"type": "function", "function": {
    "name": "get_weather", "description": "Get current weather for a city.",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
msgs = [{"role": "user", "content": "What is the weather in Paris?"}]
prompt = tok.apply_chat_template(msgs, tools=tools, add_generation_prompt=True,
                                 enable_thinking=False, tokenize=False)
enc = tok(prompt, return_tensors="pt", add_special_tokens=False).to(m.device)
out = m.generate(**enc, max_new_tokens=256, do_sample=False)
gen = tok.decode(out[0][enc.input_ids.shape[1]:], skip_special_tokens=True)
print([json.loads(b.strip()) for b in re.findall(r"<tool_call>(.*?)</tool_call>", gen, re.DOTALL)])

Training

• 4-bit QLoRA via Unsloth; LoRA r=16, alpha=32, dropout=0.
• 1 epoch, ~6000 single-call xLAM examples, max_seq_len=1024, packing, lr=2e-4 cosine, adamw_8bit, bf16, seed=3407. ~8 min on one A100.
• Data formatted through Qwen3's native tool template (apply_chat_template(tools=...), enable_thinking=False) so train and inference prompts are byte-identical.

Data & efficiency notes

A ~6000-example single-call subset of xLAM (not the full 60k) - format specialisation converges fast.

Reproducibility

Fixed seed 3407; hyperparameters above; resolved library versions in requirements-lock.txt.

License & citation

Base model Apache-2.0. Training data: Salesforce/xlam-function-calling-60k (CC-BY-4.0); cite APIGen / xLAM.