Add the bilingual en+fr adapter, tokenizer metadata, card, and run reports

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,199 @@
+---
+language:
+- en
+- fr
+license: other
+license_name: nvidia-open-model-license
+license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
+library_name: peft
+pipeline_tag: text-generation
+base_model: nvidia/Llama-3.1-Nemotron-Nano-8B-v1
+base_model_relation: adapter
+datasets:
+- abdelstark/sommelier-xlam-single-call-splits
+- abdelstark/sommelier-xlam-single-call-splits-fr
+- Salesforce/xlam-function-calling-60k
+tags:
+- lora
+- qlora
+- peft
+- function-calling
+- tool-calling
+- nemotron
+- llama-3.1
+- built-with-llama
+- multilingual
+model-index:
+- name: llama-3.1-nemotron-nano-8b-xlam-tool-calling-fr-en-lora
+  results:
+  - task:
+      type: text-generation
+      name: Single JSON tool-call generation (English slice)
+    dataset:
+      type: abdelstark/sommelier-xlam-single-call-splits
+      name: sommelier-xlam-single-call-splits
+      split: test
+    metrics:
+    - type: valid_json_rate
+      name: Valid JSON rate
+      value: 0.997
+    - type: function_name_accuracy
+      name: Function name accuracy
+      value: 0.993
+    - type: argument_exact_match
+      name: Argument exact match
+      value: 0.873
+    - type: argument_f1
+      name: Argument F1 (micro, flattened keys)
+      value: 0.9211
+    - type: full_call_exact_match
+      name: Full-call exact match
+      value: 0.87
+  - task:
+      type: text-generation
+      name: Single JSON tool-call generation (French slice)
+    dataset:
+      type: abdelstark/sommelier-xlam-single-call-splits-fr
+      name: sommelier-xlam-single-call-splits-fr
+      split: test
+    metrics:
+    - type: valid_json_rate
+      name: Valid JSON rate
+      value: 0.9954
+    - type: function_name_accuracy
+      name: Function name accuracy
+      value: 0.9898
+    - type: argument_exact_match
+      name: Argument exact match
+      value: 0.8760
+    - type: argument_f1
+      name: Argument F1 (micro, flattened keys)
+      value: 0.9208
+    - type: full_call_exact_match
+      name: Full-call exact match
+      value: 0.8726
+---
+
+# Llama-3.1-Nemotron-Nano-8B — bilingual (en+fr) xlam tool-calling LoRA
+
+**Built with Llama.**
+
+A QLoRA adapter for
+[nvidia/Llama-3.1-Nemotron-Nano-8B-v1](https://huggingface.co/nvidia/Llama-3.1-Nemotron-Nano-8B-v1)
+that turns free-form user requests, in English or French, plus a set of
+JSON tool schemas into **exactly one schema-valid JSON tool call** — no
+prose, no markdown fences, no explanations.
+
+This is the v2 of
+[llama-3.1-nemotron-nano-8b-xlam-tool-calling-lora](https://huggingface.co/abdelstark/llama-3.1-nemotron-nano-8b-xlam-tool-calling-lora):
+same base model, same hyperparameters, same pipeline, one changed
+variable — the training data adds a French paired variant of every
+selected row ([abdelstark/sommelier-xlam-single-call-splits-fr](https://huggingface.co/datasets/abdelstark/sommelier-xlam-single-call-splits-fr)),
+where only the query is translated and tool schemas and gold answers stay
+byte identical. Trained and evaluated end to end with
+[sommelier](https://github.com/AbdelStark/sommelier); this repository
+contains the adapter weights, tokenizer metadata, and the machine-readable
+evaluation evidence for the exact run that produced them
+(`nemotron-8b-fr-full-4`).
+
+## Why this exists
+
+Tool calling should work as well in French as in English, and that is a
+claim worth measuring rather than assuming. Measured on this task family
+(n=1000 en, n=879 fr, same prompts by digest, greedy decoding, conservative
+parser):
+
+- The base model loses 4.2 points of full-call exact match on French input
+  (0.663 vs 0.705).
+- The English-only v1 adapter transfers surprisingly well, narrowing the
+  gap to 2.3 points (fr 0.851).
+- This adapter closes the gap to measurement noise: **fr 0.873 vs en
+  0.870** (+0.3 points, French slightly ahead).
+
+## Evaluation
+
+Base model vs. this adapter, per language slice, on the held-out test
+splits. Both evaluations used byte-identical prompts per slice, greedy
+decoding (temperature 0.0, `max_new_tokens` 512), and the same conservative
+parser (`sommelier.parser.v1`) that counts every parse failure as a metric
+failure. The comparison is only written when config, test-split, per-slice
+prompt-set, parser, and decoding digests all match.
+
+### English slice (n=1000)
+
+| Metric | Base | Adapter | Delta |
+|--------|------|---------|-------|
+| valid_json_rate | 0.9160 | **0.9970** | +0.0810 |
+| function_name_accuracy | 0.9110 | **0.9930** | +0.0820 |
+| argument_exact_match | 0.7070 | **0.8730** | +0.1660 |
+| argument_f1 | 0.7569 | **0.9211** | +0.1642 |
+| full_call_exact_match | 0.7050 | **0.8700** | +0.1650 |
+
+### French slice (n=879)
+
+| Metric | Base | Adapter | Delta |
+|--------|------|---------|-------|
+| valid_json_rate | 0.9044 | **0.9954** | +0.0910 |
+| function_name_accuracy | 0.8976 | **0.9898** | +0.0922 |
+| argument_exact_match | 0.6655 | **0.8760** | +0.2105 |
+| argument_f1 | 0.7091 | **0.9208** | +0.2117 |
+| full_call_exact_match | 0.6633 | **0.8726** | +0.2093 |
+
+Relative to the v1 English-only adapter, the English slice sits 0.3 to 0.8
+points lower (full-call 0.870 vs 0.874, argument F1 0.9211 vs 0.9291), within
+one standard error at n=1000, while French gains 2.2 points of full-call
+exact match. The full per-slice reports, the gated comparison with its
+language-gaps section, and the runtime evidence (L40S, 5 h 42 m training,
+peak 26,369 MiB, pinned package versions) are in [`reports/`](./reports).
+
+## Training
+
+QLoRA (NF4, bf16 compute), rank 16, alpha 32, dropout 0.05, all seven
+projection modules, 2 epochs, effective batch 16, cosine schedule with
+3 percent warmup, `max_sequence_length` 4096, completion-only loss with a
+proven prompt boundary. Data: the 15,000-row English train split plus
+13,113 French paired rows (the French set runs short where the gold
+contract rejects translation; the drop accounting is in the dataset card).
+The system prompt stays English for both languages: the query language is
+the only moving variable.
+
+## Usage
+
+```python
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+base = "nvidia/Llama-3.1-Nemotron-Nano-8B-v1"
+adapter = "abdelstark/llama-3.1-nemotron-nano-8b-xlam-tool-calling-fr-en-lora"
+
+tokenizer = AutoTokenizer.from_pretrained(base)
+model = AutoModelForCausalLM.from_pretrained(base, dtype="auto", device_map="auto")
+model = PeftModel.from_pretrained(model, adapter)
+```
+
+Prompt format: system message with the instruction and the canonical JSON
+of the available tools, then the user request (English or French); the
+model answers with the canonical JSON of one tool call. See the
+[sommelier documentation](https://abdelstark.github.io/sommelier/) for the
+exact template contract.
+
+## Limitations
+
+- Single tool call per request; multi-call plans are out of scope.
+- One run, one seed. The French test slice is machine-translated (reviewed
+  on samples, not row by row) and excludes rows whose gold arguments embed
+  English text, so it is slightly biased toward language-neutral arguments.
+- Exact canonical-JSON scoring penalizes semantically equivalent forms;
+  both models face the identical contract.
+- Instruction-language effects are unmeasured: the system prompt is English
+  for both slices by design.
+
+## License and attribution
+
+Adapter weights: [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/),
+as a derivative of nvidia/Llama-3.1-Nemotron-Nano-8B-v1. **Built with
+Llama**: the base model derives from Llama 3.1 and this repository follows
+the [Llama 3.1 Community License](https://www.llama.com/llama3_1/license/)
+naming and notice requirements. Training data derives from
+[Salesforce/xlam-function-calling-60k](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k)
+(CC BY 4.0).

adapter_config.json

@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "nvidia/Llama-3.1-Nemotron-Nano-8B-v1",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "down_proj",
+    "gate_proj",
+    "v_proj",
+    "q_proj",
+    "up_proj",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5fe9eaa16fde9317e0e2bcb5801d683ad8dc87108bd1b8275e54776384c74d89
+size 167832240

chat_template.jinja

@@ -0,0 +1,17 @@
+{%- if messages[0]['role'] == 'system' -%}{%- set system_message = messages[0]['content'] | trim -%}{%- set messages = messages[1:] -%}{%- else -%}{%- set system_message = '' -%}{%- endif -%}{%- if tools is not none -%}{{- '<|begin_of_text|><|start_header_id|>system<|end_header_id|>' + '
+
+' + system_message -}} {{- '
+
+' if system_message else '' -}} {{- '<AVAILABLE_TOOLS>[' -}} {% for t in tools %}{{- (t.function if t.function is defined else t) | tojson() -}}{{- ', ' if not loop.last else '' -}}{%- endfor -%} {{- ']</AVAILABLE_TOOLS>' -}} {{- '<|eot_id|>' -}}{%- else -%}{{- '<|begin_of_text|><|start_header_id|>system<|end_header_id|>' + '
+
+' + system_message + '<|eot_id|>' -}}{%- endif -%}{%- for message in messages -%}{%- if (message['role'] in ['user', 'tool']) != (loop.index0 % 2 == 0) -%}{{- raise_exception('Conversation roles must alternate between user/tool and assistant') -}}{%- elif message['role'] == 'user' -%}{{- '<|start_header_id|>user<|end_header_id|>' + '
+
+' + message['content'] | trim + '<|eot_id|>' -}}{%- elif message['role'] == 'tool' -%}{%- set tool_response = '<TOOL_RESPONSE>[' + message['content'] | trim + ']</TOOL_RESPONSE>' -%}{{- '<|start_header_id|>user<|end_header_id|>' + '
+
+' + tool_response + '<|eot_id|>' -}}{%- elif message['role'] == 'assistant' and message.get('tool_calls') is not none -%}{%- set tool_calls = message['tool_calls'] -%}{{- '<|start_header_id|>assistant<|end_header_id|>' + '
+
+' + '<TOOLCALL>[' -}}{%- for tool_call in tool_calls -%}{{ '{' + '"name": "' + tool_call.function.name + '", "arguments": ' + tool_call.function.arguments | tojson + '}' }}{%- if not loop.last -%}{{ ', ' }}{%- else -%}{{ ']</TOOLCALL>' + '<|eot_id|>' }}{%- endif -%}{%- endfor -%}{%- elif message['role'] == 'assistant' -%}{{- '<|start_header_id|>assistant<|end_header_id|>' + '
+
+' + message['content'] | trim + '<|eot_id|>' -}}{%- endif -%}{%- endfor -%}{%- if add_generation_prompt -%}{{ '<|start_header_id|>assistant<|end_header_id|>' + '
+
+' }}{%- endif -%}

reports/comparison_report.json

@@ -0,0 +1,286 @@
+{
+  "adapter": {
+    "adapter_source": {
+      "kind": "local_directory",
+      "revision": null,
+      "source": "/__modal/volumes/vo-I7HI9wRvHXpMpsKs7Yc61o/artifacts/runs/nemotron-8b-fr-full-4/train/adapter"
+    },
+    "metrics": {
+      "argument_exact_match": {
+        "denominator": 1879,
+        "numerator": 1643,
+        "value": 0.8744012772751464
+      },
+      "argument_f1": {
+        "denominator": 10207,
+        "numerator": 9400,
+        "value": 0.9209366121289311
+      },
+      "full_call_exact_match": {
+        "denominator": 1879,
+        "numerator": 1637,
+        "value": 0.8712080894092602
+      },
+      "function_name_accuracy": {
+        "denominator": 1879,
+        "numerator": 1863,
+        "value": 0.9914848323576371
+      },
+      "valid_json_rate": {
+        "denominator": 1879,
+        "numerator": 1872,
+        "value": 0.9962746141564662
+      }
+    },
+    "run_id": "nemotron-8b-fr-full-4"
+  },
+  "base": {
+    "adapter_source": null,
+    "metrics": {
+      "argument_exact_match": {
+        "denominator": 1879,
+        "numerator": 1292,
+        "value": 0.6875997871208089
+      },
+      "argument_f1": {
+        "denominator": 9328,
+        "numerator": 6858,
+        "value": 0.7352058319039451
+      },
+      "full_call_exact_match": {
+        "denominator": 1879,
+        "numerator": 1288,
+        "value": 0.6854709952102181
+      },
+      "function_name_accuracy": {
+        "denominator": 1879,
+        "numerator": 1700,
+        "value": 0.9047365620010644
+      },
+      "valid_json_rate": {
+        "denominator": 1879,
+        "numerator": 1711,
+        "value": 0.9105907397551889
+      }
+    },
+    "run_id": "nemotron-8b-fr-full-4"
+  },
+  "created_at": "2026-07-06T09:06:44.927671+00:00",
+  "deltas": {
+    "argument_exact_match": 0.18680149015433745,
+    "argument_f1": 0.18573078022498601,
+    "full_call_exact_match": 0.18573709419904205,
+    "function_name_accuracy": 0.08674827035657262,
+    "valid_json_rate": 0.08568387440127734
+  },
+  "language_gaps": {
+    "adapter": {
+      "fr": {
+        "argument_exact_match": 0.002995449374288972,
+        "argument_f1": -0.0003166404540679846,
+        "full_call_exact_match": 0.0025824800910124734,
+        "function_name_accuracy": -0.0032389078498293866,
+        "valid_json_rate": -0.0015506257110352584
+      }
+    },
+    "base": {
+      "fr": {
+        "argument_exact_match": -0.04147098976109209,
+        "argument_f1": -0.04786359932831308,
+        "full_call_exact_match": -0.04174630261660972,
+        "function_name_accuracy": -0.013389078498293516,
+        "valid_json_rate": -0.01156313993174063
+      }
+    },
+    "reference": "en"
+  },
+  "run_id": "nemotron-8b-fr-full-4",
+  "runtime": {
+    "available": true,
+    "cost_source": "unavailable",
+    "hardware": {
+      "gpu": "L40S",
+      "source": "config"
+    },
+    "observed_cost_usd": null,
+    "peak_gpu_memory_mb": 26369,
+    "schema_version": "sommelier.runtime_metadata.v1",
+    "stages": {
+      "data": {
+        "elapsed_seconds": 8.057
+      },
+      "eval-adapter": {
+        "elapsed_seconds": 2835.133
+      },
+      "eval-base": {
+        "elapsed_seconds": 1805.156
+      },
+      "format": {
+        "elapsed_seconds": 30.606
+      },
+      "train": {
+        "elapsed_seconds": 20540.327
+      }
+    }
+  },
+  "schema_version": "sommelier.comparison_report.v2",
+  "shared": {
+    "config_sha256": "87a6c067167d801d85cbf3105d08b145c62955f4298de0559a0c44608e3fca89",
+    "decoding": {
+      "do_sample": false,
+      "max_new_tokens": 512,
+      "temperature": 0.0
+    },
+    "parser_version": "sommelier.parser.v1",
+    "split": "test",
+    "test_split_sha256": "11267f2e2e6293b6132a1a955b28a84caa12b15c980a2a39653d4e4ee33d80e9"
+  },
+  "slices": {
+    "en": {
+      "adapter": {
+        "metrics": {
+          "argument_exact_match": {
+            "denominator": 1000,
+            "numerator": 873,
+            "value": 0.873
+          },
+          "argument_f1": {
+            "denominator": 5550,
+            "numerator": 5112,
+            "value": 0.9210810810810811
+          },
+          "full_call_exact_match": {
+            "denominator": 1000,
+            "numerator": 870,
+            "value": 0.87
+          },
+          "function_name_accuracy": {
+            "denominator": 1000,
+            "numerator": 993,
+            "value": 0.993
+          },
+          "valid_json_rate": {
+            "denominator": 1000,
+            "numerator": 997,
+            "value": 0.997
+          }
+        }
+      },
+      "base": {
+        "metrics": {
+          "argument_exact_match": {
+            "denominator": 1000,
+            "numerator": 707,
+            "value": 0.707
+          },
+          "argument_f1": {
+            "denominator": 5097,
+            "numerator": 3858,
+            "value": 0.7569158328428487
+          },
+          "full_call_exact_match": {
+            "denominator": 1000,
+            "numerator": 705,
+            "value": 0.705
+          },
+          "function_name_accuracy": {
+            "denominator": 1000,
+            "numerator": 911,
+            "value": 0.911
+          },
+          "valid_json_rate": {
+            "denominator": 1000,
+            "numerator": 916,
+            "value": 0.916
+          }
+        }
+      },
+      "deltas": {
+        "argument_exact_match": 0.16600000000000004,
+        "argument_f1": 0.16416524823823242,
+        "full_call_exact_match": 0.16500000000000004,
+        "function_name_accuracy": 0.08199999999999996,
+        "valid_json_rate": 0.08099999999999996
+      },
+      "examples": 1000,
+      "generation_artifacts": {
+        "adapter": "runs/nemotron-8b-fr-full-4/eval/adapter/generations.en.jsonl",
+        "base": "runs/nemotron-8b-fr-full-4/eval/base/generations.en.jsonl"
+      },
+      "prompt_set_sha256": "a0da8fa28835a329dba5c5314ada3aff21f950939af2e1ae186155d3b494f39a"
+    },
+    "fr": {
+      "adapter": {
+        "metrics": {
+          "argument_exact_match": {
+            "denominator": 879,
+            "numerator": 770,
+            "value": 0.875995449374289
+          },
+          "argument_f1": {
+            "denominator": 4657,
+            "numerator": 4288,
+            "value": 0.9207644406270131
+          },
+          "full_call_exact_match": {
+            "denominator": 879,
+            "numerator": 767,
+            "value": 0.8725824800910125
+          },
+          "function_name_accuracy": {
+            "denominator": 879,
+            "numerator": 870,
+            "value": 0.9897610921501706
+          },
+          "valid_json_rate": {
+            "denominator": 879,
+            "numerator": 875,
+            "value": 0.9954493742889647
+          }
+        }
+      },
+      "base": {
+        "metrics": {
+          "argument_exact_match": {
+            "denominator": 879,
+            "numerator": 585,
+            "value": 0.6655290102389079
+          },
+          "argument_f1": {
+            "denominator": 4231,
+            "numerator": 3000,
+            "value": 0.7090522335145356
+          },
+          "full_call_exact_match": {
+            "denominator": 879,
+            "numerator": 583,
+            "value": 0.6632536973833902
+          },
+          "function_name_accuracy": {
+            "denominator": 879,
+            "numerator": 789,
+            "value": 0.8976109215017065
+          },
+          "valid_json_rate": {
+            "denominator": 879,
+            "numerator": 795,
+            "value": 0.9044368600682594
+          }
+        }
+      },
+      "deltas": {
+        "argument_exact_match": 0.2104664391353811,
+        "argument_f1": 0.2117122071124775,
+        "full_call_exact_match": 0.20932878270762223,
+        "function_name_accuracy": 0.09215017064846409,
+        "valid_json_rate": 0.09101251422070533
+      },
+      "examples": 879,
+      "generation_artifacts": {
+        "adapter": "runs/nemotron-8b-fr-full-4/eval/adapter/generations.fr.jsonl",
+        "base": "runs/nemotron-8b-fr-full-4/eval/base/generations.fr.jsonl"
+      },
+      "prompt_set_sha256": "b6111339b6dd6d6a911aeea4e0bf960bd3d78cd68ed14a1da05403aeacb76783"
+    }
+  }
+}

reports/comparison_report.md

@@ -0,0 +1,103 @@
+# Sommelier Comparison Report
+
+The JSON report (`comparison_report.json`) is authoritative for automation; this document is a human rendering.
+
+## Run Identity
+
+- Run ID: `nemotron-8b-fr-full-4`
+- Evidence class: full run
+- Created at: 2026-07-06T09:06:44.927671+00:00
+- Config digest: `87a6c067167d801d85cbf3105d08b145c62955f4298de0559a0c44608e3fca89`
+- Parser version: `sommelier.parser.v1`
+- Decoding: `{"do_sample": false, "max_new_tokens": 512, "temperature": 0.0}`
+- Adapter source: `/__modal/volumes/vo-I7HI9wRvHXpMpsKs7Yc61o/artifacts/runs/nemotron-8b-fr-full-4/train/adapter` (local_directory, revision None)
+
+## Split Summary
+
+- Split: test
+- Slices: `en`, `fr`
+- Examples evaluated: 1879 across all slices
+- Test split digest: `11267f2e2e6293b6132a1a955b28a84caa12b15c980a2a39653d4e4ee33d80e9`
+
+## Metrics, all slices
+
+| Metric | Base | Adapter | Delta |
+|--------|------|---------|-------|
+| argument_exact_match | 0.6876 (1292/1879) | 0.8744 (1643/1879) | +0.1868 |
+| argument_f1 | 0.7352 (6858/9328) | 0.9209 (9400/10207) | +0.1857 |
+| full_call_exact_match | 0.6855 (1288/1879) | 0.8712 (1637/1879) | +0.1857 |
+| function_name_accuracy | 0.9047 (1700/1879) | 0.9915 (1863/1879) | +0.0867 |
+| valid_json_rate | 0.9106 (1711/1879) | 0.9963 (1872/1879) | +0.0857 |
+
+## Metrics, slice `en`
+
+- Examples: 1000
+- Prompt set digest: `a0da8fa28835a329dba5c5314ada3aff21f950939af2e1ae186155d3b494f39a`
+
+| Metric | Base | Adapter | Delta |
+|--------|------|---------|-------|
+| argument_exact_match | 0.7070 (707/1000) | 0.8730 (873/1000) | +0.1660 |
+| argument_f1 | 0.7569 (3858/5097) | 0.9211 (5112/5550) | +0.1642 |
+| full_call_exact_match | 0.7050 (705/1000) | 0.8700 (870/1000) | +0.1650 |
+| function_name_accuracy | 0.9110 (911/1000) | 0.9930 (993/1000) | +0.0820 |
+| valid_json_rate | 0.9160 (916/1000) | 0.9970 (997/1000) | +0.0810 |
+
+## Metrics, slice `fr`
+
+- Examples: 879
+- Prompt set digest: `b6111339b6dd6d6a911aeea4e0bf960bd3d78cd68ed14a1da05403aeacb76783`
+
+| Metric | Base | Adapter | Delta |
+|--------|------|---------|-------|
+| argument_exact_match | 0.6655 (585/879) | 0.8760 (770/879) | +0.2105 |
+| argument_f1 | 0.7091 (3000/4231) | 0.9208 (4288/4657) | +0.2117 |
+| full_call_exact_match | 0.6633 (583/879) | 0.8726 (767/879) | +0.2093 |
+| function_name_accuracy | 0.8976 (789/879) | 0.9898 (870/879) | +0.0922 |
+| valid_json_rate | 0.9044 (795/879) | 0.9954 (875/879) | +0.0910 |
+
+## Language Gaps
+
+Each slice against the `en` reference slice (positive means the slice scores higher):
+
+### `fr` minus `en`
+
+| Metric | Base gap | Adapter gap |
+|--------|----------|-------------|
+| argument_exact_match | -0.0415 | +0.0030 |
+| argument_f1 | -0.0479 | -0.0003 |
+| full_call_exact_match | -0.0417 | +0.0026 |
+| function_name_accuracy | -0.0134 | -0.0032 |
+| valid_json_rate | -0.0116 | -0.0016 |
+
+
+## Runtime and Cost
+
+- Hardware: L40S (source: config)
+- Peak GPU memory: 26369 MiB
+- Observed cost: unavailable (source: unavailable)
+- data: 8.057 s elapsed
+- eval-adapter: 2835.133 s elapsed
+- eval-base: 1805.156 s elapsed
+- format: 30.606 s elapsed
+- train: 20540.327 s elapsed
+
+## Reproduction
+
+Using the resolved config stored in this run directory:
+
+```bash
+sommelier eval run --config config.resolved.yaml --model base --data formatted --out eval/base --run-id nemotron-8b-fr-full-4
+sommelier train run --config config.resolved.yaml --data formatted --out train/adapter --run-id nemotron-8b-fr-full-4
+sommelier eval run --config config.resolved.yaml --model adapter --adapter train/adapter --data formatted --out eval/adapter --run-id nemotron-8b-fr-full-4
+sommelier report compare --base eval/base --adapter eval/adapter --out report
+```
+
+Generation artifacts per slice: `en`: `runs/nemotron-8b-fr-full-4/eval/base/generations.en.jsonl` (base), `runs/nemotron-8b-fr-full-4/eval/adapter/generations.en.jsonl` (adapter); `fr`: `runs/nemotron-8b-fr-full-4/eval/base/generations.fr.jsonl` (base), `runs/nemotron-8b-fr-full-4/eval/adapter/generations.fr.jsonl` (adapter).
+
+## Limitations
+
+- Metrics measure schema-valid single tool calls on the configured held-out test split only; multi-call plans are out of scope.
+- Non-English slices are machine-translated variants of the English test rows, not natively authored requests, and share their gold answers by construction.
+- Argument comparisons are exact canonical-JSON matches; semantically equivalent but differently formatted values count as mismatches.
+- Results describe the recorded run (hardware, dependencies, dataset revision) and do not claim production readiness, broad reliability, or generalization beyond the evaluated split.
+- Parse failures count against every metric; raw generations are retained for audit.

reports/evaluation_report_adapter.json

@@ -0,0 +1,113 @@
+{
+  "adapter_source": {
+    "kind": "local_directory",
+    "revision": null,
+    "source": "/__modal/volumes/vo-I7HI9wRvHXpMpsKs7Yc61o/artifacts/runs/nemotron-8b-fr-full-4/train/adapter"
+  },
+  "config_sha256": "87a6c067167d801d85cbf3105d08b145c62955f4298de0559a0c44608e3fca89",
+  "created_at": "2026-07-06T09:06:43.409328+00:00",
+  "decoding": {
+    "do_sample": false,
+    "max_new_tokens": 512,
+    "temperature": 0.0
+  },
+  "metrics": {
+    "argument_exact_match": {
+      "denominator": 1879,
+      "numerator": 1643,
+      "value": 0.8744012772751464
+    },
+    "argument_f1": {
+      "denominator": 10207,
+      "numerator": 9400,
+      "value": 0.9209366121289311
+    },
+    "full_call_exact_match": {
+      "denominator": 1879,
+      "numerator": 1637,
+      "value": 0.8712080894092602
+    },
+    "function_name_accuracy": {
+      "denominator": 1879,
+      "numerator": 1863,
+      "value": 0.9914848323576371
+    },
+    "valid_json_rate": {
+      "denominator": 1879,
+      "numerator": 1872,
+      "value": 0.9962746141564662
+    }
+  },
+  "model_kind": "adapter",
+  "parser_version": "sommelier.parser.v1",
+  "run_id": "nemotron-8b-fr-full-4",
+  "schema_version": "sommelier.evaluation_report.v2",
+  "slices": {
+    "en": {
+      "examples": 1000,
+      "generation_artifact": "runs/nemotron-8b-fr-full-4/eval/adapter/generations.en.jsonl",
+      "metrics": {
+        "argument_exact_match": {
+          "denominator": 1000,
+          "numerator": 873,
+          "value": 0.873
+        },
+        "argument_f1": {
+          "denominator": 5550,
+          "numerator": 5112,
+          "value": 0.9210810810810811
+        },
+        "full_call_exact_match": {
+          "denominator": 1000,
+          "numerator": 870,
+          "value": 0.87
+        },
+        "function_name_accuracy": {
+          "denominator": 1000,
+          "numerator": 993,
+          "value": 0.993
+        },
+        "valid_json_rate": {
+          "denominator": 1000,
+          "numerator": 997,
+          "value": 0.997
+        }
+      },
+      "prompt_set_sha256": "a0da8fa28835a329dba5c5314ada3aff21f950939af2e1ae186155d3b494f39a"
+    },
+    "fr": {
+      "examples": 879,
+      "generation_artifact": "runs/nemotron-8b-fr-full-4/eval/adapter/generations.fr.jsonl",
+      "metrics": {
+        "argument_exact_match": {
+          "denominator": 879,
+          "numerator": 770,
+          "value": 0.875995449374289
+        },
+        "argument_f1": {
+          "denominator": 4657,
+          "numerator": 4288,
+          "value": 0.9207644406270131
+        },
+        "full_call_exact_match": {
+          "denominator": 879,
+          "numerator": 767,
+          "value": 0.8725824800910125
+        },
+        "function_name_accuracy": {
+          "denominator": 879,
+          "numerator": 870,
+          "value": 0.9897610921501706
+        },
+        "valid_json_rate": {
+          "denominator": 879,
+          "numerator": 875,
+          "value": 0.9954493742889647
+        }
+      },
+      "prompt_set_sha256": "b6111339b6dd6d6a911aeea4e0bf960bd3d78cd68ed14a1da05403aeacb76783"
+    }
+  },
+  "split": "test",
+  "test_split_sha256": "11267f2e2e6293b6132a1a955b28a84caa12b15c980a2a39653d4e4ee33d80e9"
+}

reports/evaluation_report_base.json

@@ -0,0 +1,109 @@
+{
+  "adapter_source": null,
+  "config_sha256": "87a6c067167d801d85cbf3105d08b145c62955f4298de0559a0c44608e3fca89",
+  "created_at": "2026-07-06T02:37:07.786757+00:00",
+  "decoding": {
+    "do_sample": false,
+    "max_new_tokens": 512,
+    "temperature": 0.0
+  },
+  "metrics": {
+    "argument_exact_match": {
+      "denominator": 1879,
+      "numerator": 1292,
+      "value": 0.6875997871208089
+    },
+    "argument_f1": {
+      "denominator": 9328,
+      "numerator": 6858,
+      "value": 0.7352058319039451
+    },
+    "full_call_exact_match": {
+      "denominator": 1879,
+      "numerator": 1288,
+      "value": 0.6854709952102181
+    },
+    "function_name_accuracy": {
+      "denominator": 1879,
+      "numerator": 1700,
+      "value": 0.9047365620010644
+    },
+    "valid_json_rate": {
+      "denominator": 1879,
+      "numerator": 1711,
+      "value": 0.9105907397551889
+    }
+  },
+  "model_kind": "base",
+  "parser_version": "sommelier.parser.v1",
+  "run_id": "nemotron-8b-fr-full-4",
+  "schema_version": "sommelier.evaluation_report.v2",
+  "slices": {
+    "en": {
+      "examples": 1000,
+      "generation_artifact": "runs/nemotron-8b-fr-full-4/eval/base/generations.en.jsonl",
+      "metrics": {
+        "argument_exact_match": {
+          "denominator": 1000,
+          "numerator": 707,
+          "value": 0.707
+        },
+        "argument_f1": {
+          "denominator": 5097,
+          "numerator": 3858,
+          "value": 0.7569158328428487
+        },
+        "full_call_exact_match": {
+          "denominator": 1000,
+          "numerator": 705,
+          "value": 0.705
+        },
+        "function_name_accuracy": {
+          "denominator": 1000,
+          "numerator": 911,
+          "value": 0.911
+        },
+        "valid_json_rate": {
+          "denominator": 1000,
+          "numerator": 916,
+          "value": 0.916
+        }
+      },
+      "prompt_set_sha256": "a0da8fa28835a329dba5c5314ada3aff21f950939af2e1ae186155d3b494f39a"
+    },
+    "fr": {
+      "examples": 879,
+      "generation_artifact": "runs/nemotron-8b-fr-full-4/eval/base/generations.fr.jsonl",
+      "metrics": {
+        "argument_exact_match": {
+          "denominator": 879,
+          "numerator": 585,
+          "value": 0.6655290102389079
+        },
+        "argument_f1": {
+          "denominator": 4231,
+          "numerator": 3000,
+          "value": 0.7090522335145356
+        },
+        "full_call_exact_match": {
+          "denominator": 879,
+          "numerator": 583,
+          "value": 0.6632536973833902
+        },
+        "function_name_accuracy": {
+          "denominator": 879,
+          "numerator": 789,
+          "value": 0.8976109215017065
+        },
+        "valid_json_rate": {
+          "denominator": 879,
+          "numerator": 795,
+          "value": 0.9044368600682594
+        }
+      },
+      "prompt_set_sha256": "b6111339b6dd6d6a911aeea4e0bf960bd3d78cd68ed14a1da05403aeacb76783"
+    }
+  },
+  "split": "test",
+  "test_split_sha256": "11267f2e2e6293b6132a1a955b28a84caa12b15c980a2a39653d4e4ee33d80e9"
+}

reports/runtime_metadata.json

@@ -0,0 +1,30 @@
+{
+  "cost_source": "unavailable",
+  "hardware": {
+    "gpu": "L40S",
+    "source": "config"
+  },
+  "observed_cost_usd": null,
+  "peak_gpu_memory_mb": 26369,
+  "schema_version": "sommelier.runtime_metadata.v1",
+  "stages": {
+    "compare": {
+      "elapsed_seconds": 1.121
+    },
+    "data": {
+      "elapsed_seconds": 8.057
+    },
+    "eval-adapter": {
+      "elapsed_seconds": 2835.133
+    },
+    "eval-base": {
+      "elapsed_seconds": 1805.156
+    },
+    "format": {
+      "elapsed_seconds": 30.606
+    },
+    "train": {
+      "elapsed_seconds": 20540.327
+    }
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b9e4e7fb171f92fd137b777cc2714bf87d11576700a1dcd7a399e7bbe39537b
+size 17209920

tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|begin_of_text|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 131072,
+  "pad_token": "<|eot_id|>",
+  "tokenizer_class": "TokenizersBackend"
+}