Add files using upload-large-folder tool

README.md

@@ -1,187 +1,10 @@
 ---
-language:
-- th
-library_name: transformers
+license: apache-2.0
 pipeline_tag: text-generation
-tags:
-- medical
-- rag
-- citation
-- question-answering
-- thai
+language: [th]
+tags: [sft]
 ---
 
-# ThaiLLM-30B-IQ (Medical)
+# IQ-ThaiLLM-30B-SFT
 
-## Model Description
-
-**ThaiLLM-30B-IQ (Medical)** is a Thai-language large language model specialized for **medical information query** and **citation-grounded question answering**.
-
-The model is fine-tuned from **ThaiLLM-30B** using supervised fine-tuning (SFT) with medical-domain data. It is designed for **retrieval-augmented generation (RAG)** workflows, where answers must be generated **only from provided medical contexts** and returned with explicit citations.
-
-> This model does not include a retriever and must be used with externally supplied medical documents.
-
----
-
-## Intended Use
-
-### Suitable for
-- Medical RAG systems (Thai)
-- Medical document question answering
-- Citation-grounded medical QA
-- Medical education and evaluation
-
-### Not suitable for
-- Medical diagnosis or treatment decisions
-- Patient-facing clinical advice
-- Emergency or critical-care use
-
----
-
-## Model Details
-
-- **Base model:** ThaiLLM-30B
-- **Domain:** Medical  
-- **Model type:** Decoder-only causal language model  
-- **Fine-tuning:** Supervised fine-tuning (LoRA)  
-- **Language:** Thai  
-- **Precision:** bfloat16  
-
----
-
-## Performance (Medical Information Query Setting)
-
-| Model | Response (BLEU) | Citations (Jaccard) |
-|------|-----------------|---------------------|
-| ThaiLLM-30B | 0.406 | 0.1786 |
-| **ThaiLLM-8B-SFT-IQ (Medical)** | **0.4331** | **0.5458** |
-
----
-
-## Training
-
-- **Framework:** LLaMA-Factory  
-- **Method:** LoRA-based supervised fine-tuning  
-
-| Hyperparameter | Value |
-|---------------|-------|
-| Learning rate | 2e-4 |
-| LoRA rank | 16 |
-| LoRA alpha | 16 |
-| Sequence length | 4096 |
-| Epochs | 3 |
-| Batch size | 4 |
-
-Training data consists of Thai medical question–answer pairs with context grounding and citation supervision.
-- **Training Script**
-- ```bash
-  llamafactory-cli train \
-    --stage sft \
-    --do_train True \
-    --model_name_or_path ${MODEL_PATH} \
-    --preprocessing_num_workers 16 \
-    --deepspeed ./examples/deepspeed/ds_z2_config.json \
-    --finetuning_type lora \
-    --template qwen3 \
-    --flash_attn auto \
-    --dataset thaillm-SFT \
-    --cutoff_len 4096 \
-    --learning_rate 2e-4 \
-    --num_train_epochs 3.0 \
-    --max_samples 100000 \
-    --per_device_train_batch_size 1 \
-    --gradient_accumulation_steps 4 \
-    --gradient_checkpointing True \
-    --gradient_checkpointing_kwargs '{"use_reentrant":false}' \
-    --lr_scheduler_type cosine \
-    --max_grad_norm 1.0 \
-    --logging_steps 5 \
-    --save_steps 100 \
-    --warmup_steps 0 \
-    --packing False \
-    --enable_thinking True \
-    --report_to tensorboard \
-    --quantization_bit 4 \
-    --output_dir "./checkpoints/ThaiLLM-30B-SFT-IQ" \
-    --overwrite_output_dir True \
-    --bf16 True \
-    --plot_loss True \
-    --trust_remote_code True \
-    --include_num_input_tokens_seen True \
-    --optim adamw_8bit \
-    --lora_rank 16 \
-    --lora_alpha 16 \
-    --upcast_layernorm True \
-    --ddp_timeout 180000000 \
-    --lora_dropout 0 \
-    --lora_target all \
-    --freeze_vision_tower True \
-    --freeze_multi_modal_projector True \
-    --use_unsloth False \
-    --seed 1234
-  ```
-
----
-
-## Usage
-
-The model is optimized for **strict JSON output** and **context-only medical answers**.
-
-### Expected Instruction Format
-
-```python
-import torch
-from transformers import AutoModelForCausalLM, AutoTokenizer
-
-model_name = "ThaiLLM-30B-SFT-IQ"
-
-tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
-model = AutoModelForCausalLM.from_pretrained(
-    model_name,
-    device_map="auto",
-    torch_dtype=torch.bfloat16,
-)
-
-system_prompt = """\
-Answer in JSON format with citations only.
-Use only the provided medical contexts to answer the question.
-Include the fact IDs that support your answer in the following format:
-{"answer": "<ANSWER_TEXT>", "citations": ["<FACT_ID1>", "<FACT_ID2>"]}
-
-If unknown, respond with:
-{"answer": "unknown", "citations": []}
-"""
-
-question = "การบาดเจ็บจากอุบัติเหตุที่ข้อต่อขากรรไกรสามารถทำให้อาการปวดร้าวไปที่ใดเมื่อเคี้ยวอาหาร?"
-
-facts = """\
-[1] การบาดเจ็บจากอุบัติเหตุที่ข้อต่อขากรรไกรสามารถทำให้ปวดร้าวไปที่หูเมื่อเคี้ยวอาหาร
-[2] ข้อต่อขากรรไกรอักเสบ (TMJ) สามารถทำให้ปวดบริเวณใกล้ติ่งหูและร้าวไปที่หูเมื่อเคี้ยวอาหาร
-"""
-
-prompt = f"""{system_prompt}
-
-Question:
-{question}
-
-Facts:
-{facts}
-"""
-
-inputs = tokenizer.apply_chat_template(
-    [{"role": "user", "content": prompt}],
-    add_generation_prompt=True,
-    return_tensors="pt",
-).to(model.device)
-
-with torch.inference_mode():
-    outputs = model.generate(
-        inputs,
-        max_new_tokens=256,
-        do_sample=False,
-        temperature=0.0,
-    )
-
-generated = outputs[0, inputs.shape[-1]:]
-print(tokenizer.decode(generated, skip_special_tokens=True))
-```
+Instruction-tuned version of **IQ-ThaiLLM-30B-SFT**.

added_tokens.json

@@ -0,0 +1,28 @@
+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.jinja

@@ -0,0 +1,7 @@
+{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% endif %}{% if system_message is defined %}{{ '<|im_start|>system
+' + system_message + '<|im_end|>
+' }}{% endif %}{% for message in loop_messages %}{% set content = message['content'] %}{% if message['role'] == 'user' %}{{ '<|im_start|>user
+' + content + '<|im_end|>
+<|im_start|>assistant
+' }}{% elif message['role'] == 'assistant' %}{{ content + '<|im_end|>
+' }}{% endif %}{% endfor %}

config.json

@@ -0,0 +1,38 @@
+{
+  "architectures": [
+    "Qwen3MoeForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "decoder_sparse_step": 1,
+  "dtype": "bfloat16",
+  "eos_token_id": 151643,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 6144,
+  "max_position_embeddings": 32768,
+  "max_window_layers": 48,
+  "mlp_only_layers": [],
+  "model_type": "qwen3_moe",
+  "moe_intermediate_size": 768,
+  "norm_topk_prob": true,
+  "num_attention_heads": 32,
+  "num_experts": 128,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 48,
+  "num_key_value_heads": 4,
+  "output_router_logits": false,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000.0,
+  "router_aux_loss_coef": 0.001,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.1",
+  "use_cache": false,
+  "use_sliding_window": false,
+  "vocab_size": 151936
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "bos_token_id": 151643,
+  "eos_token_id": 151643,
+  "max_new_tokens": 2048,
+  "transformers_version": "4.57.1"
+}

model-00001-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:878ac143bc76a48679ed0bbb64b2162a7fdab476b395428d6568a01df1b8607a
+size 1998123464

model-00002-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0de9a808696d407af2743b77b02a63e575709580c82137867dea19342dab6f4b
+size 1998147512

model-00003-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c0f63af47d016f4d1deac7091c41a73e18d35686ed223a4f88318f52b90618f
+size 1998680096

model-00004-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a21600a98b4e8bd3abb15a94e31a6a77721472b1750c9bab509cdd714b94a5d9
+size 1998147560

model-00005-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b06c4eb6beaaa35078842128642d20fb2bd5a3da6424e3db5a395a1d777b98a
+size 1998680048

model-00006-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3834f89b897f0e9ae65f615cebb32c1e95e2b1364627ee159b176bb748745ebf
+size 1998680096

model-00007-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de7be3ec28f54e94adc96478fabaaa8c7a133b47823baa19c5d33607be487a69
+size 1998147808

model-00008-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a91ad49bba213ea430690bc130da22f480fbf504e352d8f0a592d4ff47980c58
+size 1998680728

model-00009-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:62e3b9cb4e4606386dd77466685ebf98be87aa7dde5a0853dc1f362834901828
+size 1998148200

model-00010-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37867ca20a83c010d41148c1fcfc69b30e42dfebfa2d5b4135d788387ac5b6d0
+size 1998680664

model-00011-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b03d5f0b576318f6b9b56ff76439eead3d1292dd633d30caa365be0cc6ffc5c
+size 1998680728

model-00012-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:efa8e870735de2ecd5614f5e39f9e9d69a4180569eb7ee3ab4664489c5250ec7
+size 1998148144

model-00013-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bba62768af36bba2369b705895913bc2eed7e850fd3e20adb425746128a90da7
+size 1998680728

model-00014-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:450dbaec574d451c6727d5c1c1aa255cfd45324e4bec4cd491638af8443fa478
+size 1998148208

model-00015-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f45a7e64fc45cb99cab3d203e5ebd0990195c9e5ef4cd4e4cae7a26eb60440df
+size 1998680656

model-00016-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50ca99d9ef3a354095910f347364c99f3abec2c4db0929807b0ab8385ff6a43e
+size 1998680728

model-00017-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b44c93caf0c08d8b4c74bdf605d04f923d1dfae4f416480fd94f2993b7bf82e
+size 1998148144

model-00018-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef9820cc412a4d60f5d5c3aa74184a9cd0bb01fc6cce2653426d4164fd8a1282
+size 1998680728

model-00019-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:14af6339988aa9eca3ab63cdb0ea7b0869f5759fc8a96a5fddd11e03ea1657a4
+size 1998148208

model-00020-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b28cd2cc3edd368bd5c6436a123c49f78f56136a65c08514f479428418c31e2a
+size 1998680656

model-00021-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cb9711271d2523aa87fed6d6d325f212a6fe393624850d5cbded2297496c4c63
+size 1998680728

model-00022-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed1ec37171ce1962438053bb1fb60344485a1a52a11eb6d5982d96bad4cca120
+size 1998148144

model-00023-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:815d32bcc72a2e8c1df8705d5be46ad7b673ea6ea85fe79a14222352de75f655
+size 1998680728

model-00024-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad8496c51818ec1384cc5958c17f68f33f697a6f3fbd3fe9216dce279ce5db0e
+size 1998148216

model-00025-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:33ac59d7697055d8858179093f8795b19c5df243fecc0f513c18abee10d70a44
+size 1998680648

model-00026-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f3c3c1a260c54feff4f0a06e67af6c6fc252e8e72faf63b05476324a6ce976a0
+size 1998680728

model-00027-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d58670675b5eca6e040f275b363e8020ff7df5ce4aadfaa0bf8d6532c39fa148
+size 1998148144

model-00028-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2fa76bcf3cf6c4a9a7f5e02eda4bd680bd74abb9596dbfe71bd571f169548b7e
+size 1998680728

model-00029-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c2da35da40ae03ff111d48915814656eefd0d489d720d954fcb6d292b842b3c1
+size 1995010792

model-00030-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d07dfe2c8b215a40006eb2aa4fb669f17d52995cec0a6fcfdea4cb086ebdebf
+size 1998672224

model-00031-of-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02ea4a466e7001d3a07720de88e30a12257a358e2a45313269c4f0baa5203dbf
+size 1116241256

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,240 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "right",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}