Upload folder using huggingface_hub

README.md

@@ -0,0 +1,101 @@
+---
+language:
+- en
+license: apache-2.0
+base_model: HuggingFaceTB/SmolLM2-360M
+tags:
+- telecom
+- 3gpp
+- etsi
+- standards
+- domain-adaptation
+- causal-lm
+datasets:
+- nareshmodina/TeleSpec-Data
+metrics:
+- perplexity
+---
+
+# SmolLM-TS-360M
+
+A 360M parameter language model specialised in 3GPP and ETSI telecommunications standards, trained via full fine-tuning on [TeleSpec-Data](https://huggingface.co/datasets/nareshmodina/TeleSpec-Data).
+
+Part of the **SmolLM-TS** series — small language models adapted exclusively to telecommunications standards documents, with zero arXiv or web content in the training corpus.
+
+> **Looking for the instruction-tuned version?** See [nareshmodina/SmolLM-TS-360M-it](https://huggingface.co/nareshmodina/SmolLM-TS-360M-it)
+
+---
+
+## Model Details
+
+| | |
+|---|---|
+| **Base model** | HuggingFaceTB/SmolLM2-360M |
+| **Parameters** | 360M |
+| **Training** | Full fine-tuning on TeleSpec-Data |
+| **Pretraining data** | TeleSpec-Data (1.87B tokens) |
+| **Context length** | 4096 tokens |
+| **Hardware** | 3× NVIDIA L40S (48GB) |
+
+---
+
+## Training
+
+Full fine-tuning of all model weights on 457,160 packed 4096-token blocks (1.87B tokens) from 38,302 standards documents — 15,054 3GPP (Rel-8 to Rel-19) and 23,248 ETSI documents spanning 15 working groups (2000–2024). Zero arXiv or web content — 100% standards text.
+
+- Epochs: 2
+- Effective batch size: 128 — LR: 5e-5 (cosine with warmup)
+- Context length: 4096 tokens
+
+---
+
+## Usage
+
+This is a base model — it continues text rather than following instructions. For instruction following, use [SmolLM-TS-360M-it](https://huggingface.co/nareshmodina/SmolLM-TS-360M-it).
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+model_id  = "nareshmodina/SmolLM-TS-360M"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model     = AutoModelForCausalLM.from_pretrained(
+    model_id, dtype=torch.bfloat16, device_map="auto"
+)
+
+prompt  = "The RRC Connection Establishment procedure in LTE is"
+inputs  = tokenizer(prompt, return_tensors="pt").to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=100, do_sample=False)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+---
+
+## Limitations
+
+- **Base model only** — does not follow instructions, use SmolLM-TS-360M-it for Q&A
+- **Standards only** — strong 3GPP/ETSI knowledge, limited general telecom knowledge
+- **Not for production** — intended for research purposes only
+
+---
+
+## Links
+
+- 📦 Dataset: [nareshmodina/TeleSpec-Data](https://huggingface.co/datasets/nareshmodina/TeleSpec-Data)
+- 🤖 Instruct version: [nareshmodina/SmolLM-TS-360M-it](https://huggingface.co/nareshmodina/SmolLM-TS-360M-it)
+- 📊 Benchmark: [AliMaatouk/Tele-Eval](https://huggingface.co/datasets/AliMaatouk/Tele-Eval)
+- 🗂️ Collection: [nareshmodina/SmolLM-TS](https://huggingface.co/collections/nareshmodina/smollm-ts)
+
+---
+
+## Citation
+
+```bibtex
+@misc{modina2025smollmts,
+  author    = {Naresh Modina},
+  title     = {SmolLM-TS: Small Language Models for Telecommunications Standards},
+  year      = {2025},
+  publisher = {Hugging Face},
+  url       = {https://huggingface.co/nareshmodina/SmolLM-TS-360M}
+}
+```

config.json

@@ -0,0 +1,34 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "bfloat16",
+  "eos_token_id": 0,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 960,
+  "initializer_range": 0.02,
+  "intermediate_size": 2560,
+  "is_llama_config": true,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 15,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 5,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_interleaved": false,
+  "rope_parameters": {
+    "rope_theta": 100000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.3.0",
+  "use_cache": false,
+  "vocab_size": 49152
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": [
+    0
+  ],
+  "pad_token_id": 0,
+  "transformers_version": "5.3.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f3bb6f16e34a98c6d3d2f32359c9fc3d55046dd5c8f2cf26242eef5f4899e10b
+size 723674912

tokenizer_config.json

@@ -0,0 +1,33 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|endoftext|>",
+    "<|im_start|>",
+    "<|im_end|>",
+    "<repo_name>",
+    "<reponame>",
+    "<file_sep>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<jupyter_script>",
+    "<empty_output>"
+  ],
+  "is_local": false,
+  "model_max_length": 8192,
+  "pad_token": null,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9b4c80afdad7bcfa119b85bfa7d826f49a3b82074b95f924450ac43c0dc2ad6c
+size 5201