README.md

---
language:
- en
- fr
license: other
license_name: nvidia-open-model-license
license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
library_name: peft
pipeline_tag: text-generation
base_model: nvidia/Llama-3.1-Nemotron-Nano-8B-v1
base_model_relation: adapter
datasets:
- abdelstark/sommelier-xlam-single-call-splits
- abdelstark/sommelier-xlam-single-call-splits-fr
- Salesforce/xlam-function-calling-60k
tags:
- lora
- qlora
- peft
- function-calling
- tool-calling
- nemotron
- llama-3.1
- built-with-llama
- multilingual
model-index:
- name: llama-3.1-nemotron-nano-8b-xlam-tool-calling-fr-en-lora
  results:
  - task:
      type: text-generation
      name: Single JSON tool-call generation (English slice)
    dataset:
      type: abdelstark/sommelier-xlam-single-call-splits
      name: sommelier-xlam-single-call-splits
      split: test
    metrics:
    - type: valid_json_rate
      name: Valid JSON rate
      value: 0.997
    - type: function_name_accuracy
      name: Function name accuracy
      value: 0.993
    - type: argument_exact_match
      name: Argument exact match
      value: 0.873
    - type: argument_f1
      name: Argument F1 (micro, flattened keys)
      value: 0.9211
    - type: full_call_exact_match
      name: Full-call exact match
      value: 0.87
  - task:
      type: text-generation
      name: Single JSON tool-call generation (French slice)
    dataset:
      type: abdelstark/sommelier-xlam-single-call-splits-fr
      name: sommelier-xlam-single-call-splits-fr
      split: test
    metrics:
    - type: valid_json_rate
      name: Valid JSON rate
      value: 0.9954
    - type: function_name_accuracy
      name: Function name accuracy
      value: 0.9898
    - type: argument_exact_match
      name: Argument exact match
      value: 0.8760
    - type: argument_f1
      name: Argument F1 (micro, flattened keys)
      value: 0.9208
    - type: full_call_exact_match
      name: Full-call exact match
      value: 0.8726
---

# Llama-3.1-Nemotron-Nano-8B — bilingual (en+fr) xlam tool-calling LoRA

**Built with Llama.**

A QLoRA adapter for
[nvidia/Llama-3.1-Nemotron-Nano-8B-v1](https://huggingface.co/nvidia/Llama-3.1-Nemotron-Nano-8B-v1)
that turns free-form user requests, in English or French, plus a set of
JSON tool schemas into **exactly one schema-valid JSON tool call** — no
prose, no markdown fences, no explanations.

This is the v2 of
[llama-3.1-nemotron-nano-8b-xlam-tool-calling-lora](https://huggingface.co/abdelstark/llama-3.1-nemotron-nano-8b-xlam-tool-calling-lora):
same base model, same hyperparameters, same pipeline, one changed
variable — the training data adds a French paired variant of every
selected row ([abdelstark/sommelier-xlam-single-call-splits-fr](https://huggingface.co/datasets/abdelstark/sommelier-xlam-single-call-splits-fr)),
where only the query is translated and tool schemas and gold answers stay
byte identical. Trained and evaluated end to end with
[sommelier](https://github.com/AbdelStark/sommelier); this repository
contains the adapter weights, tokenizer metadata, and the machine-readable
evaluation evidence for the exact run that produced them
(`nemotron-8b-fr-full-4`).

## Why this exists

Tool calling should work as well in French as in English, and that is a
claim worth measuring rather than assuming. Measured on this task family
(n=1000 en, n=879 fr, same prompts by digest, greedy decoding, conservative
parser):

- The base model loses 4.2 points of full-call exact match on French input
  (0.663 vs 0.705).
- The English-only v1 adapter transfers surprisingly well, narrowing the
  gap to 2.3 points (fr 0.851).
- This adapter closes the gap to measurement noise: **fr 0.873 vs en
  0.870** (+0.3 points, French slightly ahead).

## Evaluation

Base model vs. this adapter, per language slice, on the held-out test
splits. Both evaluations used byte-identical prompts per slice, greedy
decoding (temperature 0.0, `max_new_tokens` 512), and the same conservative
parser (`sommelier.parser.v1`) that counts every parse failure as a metric
failure. The comparison is only written when config, test-split, per-slice
prompt-set, parser, and decoding digests all match.

### English slice (n=1000)

| Metric | Base | Adapter | Delta |
|--------|------|---------|-------|
| valid_json_rate | 0.9160 | **0.9970** | +0.0810 |
| function_name_accuracy | 0.9110 | **0.9930** | +0.0820 |
| argument_exact_match | 0.7070 | **0.8730** | +0.1660 |
| argument_f1 | 0.7569 | **0.9211** | +0.1642 |
| full_call_exact_match | 0.7050 | **0.8700** | +0.1650 |

### French slice (n=879)

| Metric | Base | Adapter | Delta |
|--------|------|---------|-------|
| valid_json_rate | 0.9044 | **0.9954** | +0.0910 |
| function_name_accuracy | 0.8976 | **0.9898** | +0.0922 |
| argument_exact_match | 0.6655 | **0.8760** | +0.2105 |
| argument_f1 | 0.7091 | **0.9208** | +0.2117 |
| full_call_exact_match | 0.6633 | **0.8726** | +0.2093 |

Relative to the v1 English-only adapter, the English slice sits 0.3 to 0.8
points lower (full-call 0.870 vs 0.874, argument F1 0.9211 vs 0.9291), within
one standard error at n=1000, while French gains 2.2 points of full-call
exact match. The full per-slice reports, the gated comparison with its
language-gaps section, and the runtime evidence (L40S, 5 h 42 m training,
peak 26,369 MiB, pinned package versions) are in [`reports/`](./reports).

## Training

QLoRA (NF4, bf16 compute), rank 16, alpha 32, dropout 0.05, all seven
projection modules, 2 epochs, effective batch 16, cosine schedule with
3 percent warmup, `max_sequence_length` 4096, completion-only loss with a
proven prompt boundary. Data: the 15,000-row English train split plus
13,113 French paired rows (the French set runs short where the gold
contract rejects translation; the drop accounting is in the dataset card).
The system prompt stays English for both languages: the query language is
the only moving variable.

## Usage

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "nvidia/Llama-3.1-Nemotron-Nano-8B-v1"
adapter = "abdelstark/llama-3.1-nemotron-nano-8b-xlam-tool-calling-fr-en-lora"

tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter)
```

Prompt format: system message with the instruction and the canonical JSON
of the available tools, then the user request (English or French); the
model answers with the canonical JSON of one tool call. See the
[sommelier documentation](https://abdelstark.github.io/sommelier/) for the
exact template contract.

## Limitations

- Single tool call per request; multi-call plans are out of scope.
- One run, one seed. The French test slice is machine-translated (reviewed
  on samples, not row by row) and excludes rows whose gold arguments embed
  English text, so it is slightly biased toward language-neutral arguments.
- Exact canonical-JSON scoring penalizes semantically equivalent forms;
  both models face the identical contract.
- Instruction-language effects are unmeasured: the system prompt is English
  for both slices by design.

## License and attribution

Adapter weights: [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/),
as a derivative of nvidia/Llama-3.1-Nemotron-Nano-8B-v1. **Built with
Llama**: the base model derives from Llama 3.1 and this repository follows
the [Llama 3.1 Community License](https://www.llama.com/llama3_1/license/)
naming and notice requirements. Training data derives from
[Salesforce/xlam-function-calling-60k](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k)
(CC BY 4.0).