Duplicate from KSP-NMAI/Boris-1.3-75M

.gitattributes

@@ -0,0 +1,38 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+Boris-75M.png filter=lfs diff=lfs merge=lfs -text
+Boris-1.3-75M.png filter=lfs diff=lfs merge=lfs -text
+benchmarks.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,124 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+datasets:
+- HuggingFaceFW/fineweb-edu
+- mlfoundations/dclm-baseline-1.0-parquet
+tags:
+- boris
+- nmai
+- gpt2
+- 75M
+---
+ 
+![Boris](Boris-1.3-75M.png)
+ 
+# Boris-1.3-75M
+ 
+Boris-1.3-75M is a 75 million-parameter language model created by New Millennium
+Artificial Intelligence (NMAI). It extends the original Boris-75M base checkpoint
+with additional continued pretraining aimed at closing gaps found in Boris-75M's
+own benchmark results (see *Continued pretraining* below).
+ 
+This is a **base (pretrained) model**. It has not been instruction-tuned and does
+not follow instructions or hold a conversation — it continues text. For an
+instruction-following version, see
+[KSP-NMAI/Boris-1.3-75M-Instruct](https://huggingface.co/KSP-NMAI/Boris-1.3-75M-Instruct).
+ 
+## Usage
+ 
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+ 
+tok = AutoTokenizer.from_pretrained("KSP-NMAI/Boris-1.3-75M")
+model = AutoModelForCausalLM.from_pretrained("KSP-NMAI/Boris-1.3-75M")
+ 
+ids = tok("The ocean is", return_tensors="pt").input_ids
+out = model.generate(ids, max_new_tokens=40, do_sample=True, top_p=0.95)
+print(tok.decode(out[0], skip_special_tokens=True))
+```
+ 
+## Details
+ 
+| | |
+|---|---|
+| Architecture | GPT-2 (pre-LN, learned positional embeddings, tied embeddings) |
+| Layers / heads / d_model | 12 / 9 / 576 |
+| Context length | 1024 |
+| Vocab | 50304 (GPT-NeoX-20B BPE, padded) |
+| Tokenizer | `EleutherAI/gpt-neox-20b` |
+| Precision | trained in bf16 autocast with fp32 master weights |
+ 
+## Base model training
+ 
+The Boris-75M base checkpoint was trained on 1.55B tokens of FineWeb-Edu for
+14:49:08 on one RTX 3060.
+ 
+| | |
+|---|---|
+| Final loss | 3.6356 |
+| Final grad norm | 0.328 |
+| Final learning rate | 6.00e-05 |
+ 
+## Continued pretraining
+ 
+Boris-75M's own benchmark results showed a gap on HellaSwag/CommonsenseQA-style
+tasks consistent with FineWeb-Edu's educational-content skew. Boris-1.3-75M adds
+three sequential continued-pretraining passes on top of the base checkpoint,
+each with a re-warmed learning rate, extending total training by 2.4B tokens
+(~60% more than the original 1.55B-token pretraining run):
+ 
+| Pass | Data | Tokens | Wall-clock (RTX 3060) |
+|---|---|---|---|
+| 1 | DCLM-baseline | 1.5B | 14h 57m |
+| 2 | FineWeb-Edu | 0.3B | ~2.5h *(estimated)* |
+| 3 | FineWeb-Edu | 0.6B | ~5.0h *(estimated)* |
+ 
+| | |
+|---|---|
+| Final loss | *3.3302* |
+| Final grad norm | *3.3302* |
+| Final learning rate | *1.00e-05* |
+ 
+**Why this recipe:** DCLM alone improved fluency/coherence tasks (LAMBADA,
+WinoGrande) but noticeably cost ARC-Easy/ARC-Challenge performance. The two
+follow-up FineWeb-Edu passes were run specifically to test whether that cost was
+recoverable — it was: ARC-Easy and ARC-Challenge both ended above their original
+Boris-75M base values, while most of the DCLM-driven fluency gains held.
+ 
+| Task | Boris-75M | +DCLM | +FineWeb-Edu | Boris-1.3-75M |
+|---|---|---|---|---|
+| HellaSwag (acc_norm) | 27.20 | 27.14 | 27.27 | 27.57 |
+| PIQA (acc_norm) | 57.18 | 58.81 | 59.30 | 59.41 |
+| WinoGrande (acc) | 49.72 | 51.70 | 51.93 | 51.54 |
+| ARC-Easy (acc_norm) | 39.14 | 38.76 | 39.48 | 40.57 |
+| ARC-Challenge (acc_norm) | 23.04 | 21.84 | 22.78 | 23.46 |
+| LAMBADA (acc) | 15.21 | 19.27 | 19.17 | 18.16 |
+| **Mean-6** | **35.25** | **36.25** | **36.66** | **36.79** |
+ 
+![Benchmarks](benchmarks.png)
+ 
+## Limitations
+ 
+A base model of this size will produce text that is frequently inaccurate,
+inconsistent, or offensive. It has received no alignment or safety tuning and
+should not be used for factual reference or deployed without supervision.
+ 
+## Copyright & License
+ 
+*Copyright 2026 Joseph Jones*
+ 
+This project and all associated files (the "Work") are licensed under the Apache
+License, Version 2.0 (the "License"); you may not use this project except in
+compliance with the License. You may obtain a copy of the License at:
+ 
+    http://www.apache.org/licenses/LICENSE-2.0
+ 
+Unless required by applicable law or agreed to in writing, software distributed
+under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
+CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+ 

config.json

@@ -0,0 +1,35 @@
+{
+  "activation_function": "gelu",
+  "add_cross_attention": false,
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.0,
+  "bos_token_id": 0,
+  "dtype": "float32",
+  "embd_pdrop": 0.0,
+  "eos_token_id": 0,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 576,
+  "n_head": 9,
+  "n_inner": null,
+  "n_layer": 12,
+  "n_positions": 1024,
+  "pad_token_id": null,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.0,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.13.1",
+  "use_cache": true,
+  "vocab_size": 50304
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.13.1",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6fee1ba947d5603c7d9297dc69830d7f1db9b8afa4c2bffbbe7060d9f4b73ed5
+size 309741640

tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|padding|>",
+  "tokenizer_class": "GPTNeoXTokenizer",
+  "trim_offsets": true,
+  "unk_token": "<|endoftext|>"
+}