abdelstark/llama-3.1-nemotron-nano-8b-xlam-tool-calling-fr-en-lora

Llama-3.1-Nemotron-Nano-8B — bilingual (en+fr) xlam tool-calling LoRA

Built with Llama.

A QLoRA adapter for nvidia/Llama-3.1-Nemotron-Nano-8B-v1 that turns free-form user requests, in English or French, plus a set of JSON tool schemas into exactly one schema-valid JSON tool call — no prose, no markdown fences, no explanations.

This is the v2 of llama-3.1-nemotron-nano-8b-xlam-tool-calling-lora: same base model, same hyperparameters, same pipeline, one changed variable — the training data adds a French paired variant of every selected row (abdelstark/sommelier-xlam-single-call-splits-fr), where only the query is translated and tool schemas and gold answers stay byte identical. Trained and evaluated end to end with sommelier; this repository contains the adapter weights, tokenizer metadata, and the machine-readable evaluation evidence for the exact run that produced them (nemotron-8b-fr-full-4).

Why this exists

Tool calling should work as well in French as in English, and that is a claim worth measuring rather than assuming. Measured on this task family (n=1000 en, n=879 fr, same prompts by digest, greedy decoding, conservative parser):

• The base model loses 4.2 points of full-call exact match on French input (0.663 vs 0.705).
• The English-only v1 adapter transfers surprisingly well, narrowing the gap to 2.3 points (fr 0.851).
• This adapter closes the gap to measurement noise: fr 0.873 vs en 0.870 (+0.3 points, French slightly ahead).

Evaluation

Base model vs. this adapter, per language slice, on the held-out test splits. Both evaluations used byte-identical prompts per slice, greedy decoding (temperature 0.0, max_new_tokens 512), and the same conservative parser (sommelier.parser.v1) that counts every parse failure as a metric failure. The comparison is only written when config, test-split, per-slice prompt-set, parser, and decoding digests all match.

English slice (n=1000)

Metric	Base	Adapter	Delta
valid_json_rate	0.9160	0.9970	+0.0810
function_name_accuracy	0.9110	0.9930	+0.0820
argument_exact_match	0.7070	0.8730	+0.1660
argument_f1	0.7569	0.9211	+0.1642
full_call_exact_match	0.7050	0.8700	+0.1650

French slice (n=879)

Metric	Base	Adapter	Delta
valid_json_rate	0.9044	0.9954	+0.0910
function_name_accuracy	0.8976	0.9898	+0.0922
argument_exact_match	0.6655	0.8760	+0.2105
argument_f1	0.7091	0.9208	+0.2117
full_call_exact_match	0.6633	0.8726	+0.2093

Relative to the v1 English-only adapter, the English slice sits 0.3 to 0.8 points lower (full-call 0.870 vs 0.874, argument F1 0.9211 vs 0.9291), within one standard error at n=1000, while French gains 2.2 points of full-call exact match. The full per-slice reports, the gated comparison with its language-gaps section, and the runtime evidence (L40S, 5 h 42 m training, peak 26,369 MiB, pinned package versions) are in reports/.

Training

QLoRA (NF4, bf16 compute), rank 16, alpha 32, dropout 0.05, all seven projection modules, 2 epochs, effective batch 16, cosine schedule with 3 percent warmup, max_sequence_length 4096, completion-only loss with a proven prompt boundary. Data: the 15,000-row English train split plus 13,113 French paired rows (the French set runs short where the gold contract rejects translation; the drop accounting is in the dataset card). The system prompt stays English for both languages: the query language is the only moving variable.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "nvidia/Llama-3.1-Nemotron-Nano-8B-v1"
adapter = "abdelstark/llama-3.1-nemotron-nano-8b-xlam-tool-calling-fr-en-lora"

tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter)

Prompt format: system message with the instruction and the canonical JSON of the available tools, then the user request (English or French); the model answers with the canonical JSON of one tool call. See the sommelier documentation for the exact template contract.

Limitations

• Single tool call per request; multi-call plans are out of scope.
• One run, one seed. The French test slice is machine-translated (reviewed on samples, not row by row) and excludes rows whose gold arguments embed English text, so it is slightly biased toward language-neutral arguments.
• Exact canonical-JSON scoring penalizes semantically equivalent forms; both models face the identical contract.
• Instruction-language effects are unmeasured: the system prompt is English for both slices by design.

License and attribution

Adapter weights: NVIDIA Open Model License, as a derivative of nvidia/Llama-3.1-Nemotron-Nano-8B-v1. Built with Llama: the base model derives from Llama 3.1 and this repository follows the Llama 3.1 Community License naming and notice requirements. Training data derives from Salesforce/xlam-function-calling-60k (CC BY 4.0).