---
language:
- code
license: apache-2.0
library_name: transformers
tags:
- code
- python
- javascript
- cpp
- sql
- html
- code-generation
- codebharat
- llama
- PyTorch
- byte-level-bpe
pipeline_tag: text-generation
widget:
- text: "def quicksort(arr):"
  example_title: "Python QuickSort"
- text: "function debounce(func, wait) {"
  example_title: "JavaScript Debounce"
- text: "int binarySearch(const std::vector<int>& arr, int target) {"
  example_title: "C++ Binary Search"
---

# CodeBharat-100M

**CodeBharat-100M** is a 100.68M parameter decoder-only Transformer pretrained from scratch on code (Python, JavaScript, TypeScript, C++, SQL, HTML/CSS, and synthetic textbooks).

## Model Details
- **Architecture:** Decoder-Only Transformer (Llama/Qwen-style: RMSNorm, RoPE, SwiGLU, Grouped-Query Attention)
- **Parameters:** 100,679,424 (100.68M)
- **Vocabulary:** 49,152 tokens (Byte-level BPE)
- **Context Window:** 1,024 tokens
- **Training Device:** NVIDIA GeForce RTX 5050 GPU
- **Final Validation Loss:** 1.3891

## Quickstart Usage

### Native PyTorch / Tokenizers Usage

```python
import torch
from tokenizers import Tokenizer
from pathlib import Path

# Load tokenizer and model weights
tokenizer = Tokenizer.from_file("tokenizer.json")
weights = torch.load("pytorch_model.bin", map_location="cuda" if torch.cuda.is_available() else "cpu")
```

### Run Inference via CLI
```bash
python 100m-codebharat/scripts/11_generate.py --prompt "def binary_search(arr, target):"
```

## Model Card Info
- **Developed by:** CodeBharat Team
- **Model Type:** Causal Language Model for Code
- **License:** Apache 2.0