Upload folder using huggingface_hub

README.md

@@ -0,0 +1,74 @@
+---
+license: apache-2.0
+language:
+  - en
+  - ta
+  - hi
+pipeline_tag: text-generation
+tags:
+  - education
+  - academic
+  - concept-first
+  - india
+  - from-scratch
+---
+
+# AQ-1B — Academic Quotient v1 (Base)
+
+**AQ (Academic Quotient) — India's Concept-First Academic AI.**
+*Raising the Academic Quotient of every student.*
+
+AQ-1B is a **1.26B-parameter foundation model built completely from scratch** by Zyora Labs — proprietary architecture, own training code (pure PyTorch), own tokenizer, own data pipeline. No fine-tune of any existing model.
+
+It is trained **concept-first**: the model learns the *concepts* of mathematics, physics, chemistry, biology, engineering, history, geography, civics and economics — from foundations to advanced — rather than curriculum checklists.
+
+## Highlights
+
+- **From scratch, end to end** — architecture, tokenizer (32k byte-level BPE), training loop, and data pipeline all built in-house
+- **20B tokens** of knowledge-dense pretraining: encyclopedic text, real textbooks and course notes, scientific papers, and mathematical reasoning corpora
+- **Final quality anneal** — the last 1.5B tokens use only the highest-quality sources (textbooks, course material, scientific papers, encyclopedic facts) with learning rate annealed to zero
+- **Tamil + Hindi inclusive** — trained with native Tamil and Hindi text alongside English
+- **Progressive growth training** — grown and continually trained through 75M → 300M → 1.26B parameter stages, each stage inheriting the previous stage's knowledge
+
+## Architecture (proprietary, from scratch)
+
+| | |
+|---|---|
+| Parameters | 1.26B |
+| Layers | 48 |
+| Hidden size | 1536 |
+| Attention heads | 24 (grouped-query, 8 KV heads) |
+| Feed-forward | SwiGLU, 4096 |
+| Positional encoding | Rotary (RoPE) |
+| Normalization | RMSNorm |
+| Context length | 2048 |
+| Vocabulary | 32,000 (byte-level BPE, English + Tamil + Hindi) |
+| Embeddings | Tied |
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained("zyoralabs/AQ-academic-ai")
+model = AutoModelForCausalLM.from_pretrained("zyoralabs/AQ-academic-ai", trust_remote_code=True)
+
+ids = tok("Photosynthesis is the process", return_tensors="pt").input_ids
+out = model.generate(ids, max_new_tokens=60)
+print(tok.decode(out[0]))
+```
+
+## Intended use
+
+AQ-1B is a **base (pretrained) model** — the foundation of the AQ educator stack (instruct tuning, retrieval grounding, and the AQ Playground sit on top of it). As a raw base model it predicts text continuations; it is not yet instruction-tuned.
+
+## Team
+
+| Name | Role | Affiliation |
+|---|---|---|
+| **Vasanth** | Chief AI Researcher | Zyora Labs |
+| **Adithi Sreedhar** | Jr AI Engineer | AI & DS, Arunachala College of Engineering for Women |
+
+## About
+
+Built in India by [Zyora Labs](https://zyora.in). AQ v1 is the first release of the Academic Quotient model family.

config.json

@@ -0,0 +1,25 @@
+{
+  "architectures": [
+    "AQForCausalLM"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_aq.AQConfig",
+    "AutoModelForCausalLM": "modeling_aq.AQForCausalLM"
+  },
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "head_dim": 64,
+  "hidden_size": 1536,
+  "intermediate_size": 4096,
+  "max_position_embeddings": 2048,
+  "model_type": "aq",
+  "num_attention_heads": 24,
+  "num_hidden_layers": 48,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 10000.0,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.53.0",
+  "vocab_size": 32000
+}

configuration_aq.py

@@ -0,0 +1,48 @@
+"""AQ model configuration.
+
+AQ (Academic Quotient) — Zyora Labs' proprietary, from-scratch decoder
+architecture: RMSNorm, rotary position embeddings, grouped-query attention,
+SwiGLU feed-forward, tied embeddings.
+"""
+
+from transformers import PretrainedConfig
+
+
+class AQConfig(PretrainedConfig):
+    model_type = "aq"
+
+    def __init__(
+        self,
+        vocab_size: int = 32000,
+        hidden_size: int = 1536,
+        intermediate_size: int = 4096,
+        num_hidden_layers: int = 48,
+        num_attention_heads: int = 24,
+        num_key_value_heads: int = 8,
+        head_dim: int = 64,
+        max_position_embeddings: int = 2048,
+        rope_theta: float = 10000.0,
+        rms_norm_eps: float = 1e-5,
+        tie_word_embeddings: bool = True,
+        bos_token_id: int = 0,
+        eos_token_id: int = 0,
+        pad_token_id: int = 0,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.num_key_value_heads = num_key_value_heads
+        self.head_dim = head_dim
+        self.max_position_embeddings = max_position_embeddings
+        self.rope_theta = rope_theta
+        self.rms_norm_eps = rms_norm_eps
+        super().__init__(
+            tie_word_embeddings=tie_word_embeddings,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            pad_token_id=pad_token_id,
+            **kwargs,
+        )

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "pad_token_id": 0,
+  "transformers_version": "4.53.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfb954c76b28c488b9ac8980f0f9e06edd99ae5536cb727ea2ec87cb4e03e436
+size 2514566480

modeling_aq.py

@@ -0,0 +1,193 @@
+"""AQ modeling code — Zyora Labs' proprietary, from-scratch decoder architecture.
+
+AQ (Academic Quotient) is a concept-first academic language model built from
+scratch in pure PyTorch. Architecture: RMSNorm, rotary position embeddings
+(RoPE), grouped-query attention (GQA), SwiGLU feed-forward, tied embeddings.
+
+Module names intentionally mirror the original AQ training code, so trained
+checkpoints load 1:1 with no key remapping.
+"""
+
+from __future__ import annotations
+
+from typing import Optional
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import PreTrainedModel
+from transformers.generation import GenerationMixin
+from transformers.modeling_outputs import CausalLMOutput
+
+from configuration_aq import AQConfig
+
+
+class AQRMSNorm(nn.Module):
+    def __init__(self, dim: int, eps: float = 1e-5):
+        super().__init__()
+        self.eps = eps
+        self.weight = nn.Parameter(torch.ones(dim))
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        dtype = x.dtype
+        x = x.float()
+        x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
+        return (x.to(dtype)) * self.weight
+
+
+def build_rope_cache(seq_len: int, head_dim: int, theta: float, device):
+    inv_freq = 1.0 / (theta ** (torch.arange(0, head_dim, 2, device=device).float() / head_dim))
+    t = torch.arange(seq_len, device=device).float()
+    freqs = torch.outer(t, inv_freq)
+    emb = torch.cat((freqs, freqs), dim=-1)
+    return emb.cos(), emb.sin()
+
+
+def rotate_half(x: torch.Tensor) -> torch.Tensor:
+    x1, x2 = x.chunk(2, dim=-1)
+    return torch.cat((-x2, x1), dim=-1)
+
+
+def apply_rope(q, k, cos, sin):
+    cos = cos.unsqueeze(0).unsqueeze(0)
+    sin = sin.unsqueeze(0).unsqueeze(0)
+    q = (q * cos) + (rotate_half(q) * sin)
+    k = (k * cos) + (rotate_half(k) * sin)
+    return q, k
+
+
+class AQAttention(nn.Module):
+    def __init__(self, cfg: AQConfig):
+        super().__init__()
+        self.n_heads = cfg.num_attention_heads
+        self.n_kv = cfg.num_key_value_heads
+        self.head_dim = cfg.head_dim
+        self.n_rep = self.n_heads // self.n_kv
+
+        self.q_proj = nn.Linear(cfg.hidden_size, self.n_heads * self.head_dim, bias=False)
+        self.k_proj = nn.Linear(cfg.hidden_size, self.n_kv * self.head_dim, bias=False)
+        self.v_proj = nn.Linear(cfg.hidden_size, self.n_kv * self.head_dim, bias=False)
+        self.o_proj = nn.Linear(self.n_heads * self.head_dim, cfg.hidden_size, bias=False)
+
+    def forward(self, x, cos, sin, attn_mask: Optional[torch.Tensor] = None):
+        B, T, _ = x.shape
+        q = self.q_proj(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2)
+        k = self.k_proj(x).view(B, T, self.n_kv, self.head_dim).transpose(1, 2)
+        v = self.v_proj(x).view(B, T, self.n_kv, self.head_dim).transpose(1, 2)
+
+        q, k = apply_rope(q, k, cos, sin)
+
+        k = k.repeat_interleave(self.n_rep, dim=1)
+        v = v.repeat_interleave(self.n_rep, dim=1)
+
+        if attn_mask is not None:
+            out = F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask)
+        else:
+            out = F.scaled_dot_product_attention(q, k, v, is_causal=True)
+        out = out.transpose(1, 2).contiguous().view(B, T, -1)
+        return self.o_proj(out)
+
+
+class AQSwiGLU(nn.Module):
+    def __init__(self, cfg: AQConfig):
+        super().__init__()
+        self.gate_proj = nn.Linear(cfg.hidden_size, cfg.intermediate_size, bias=False)
+        self.up_proj = nn.Linear(cfg.hidden_size, cfg.intermediate_size, bias=False)
+        self.down_proj = nn.Linear(cfg.intermediate_size, cfg.hidden_size, bias=False)
+
+    def forward(self, x):
+        return self.down_proj(F.silu(self.gate_proj(x)) * self.up_proj(x))
+
+
+class AQBlock(nn.Module):
+    def __init__(self, cfg: AQConfig):
+        super().__init__()
+        self.attn_norm = AQRMSNorm(cfg.hidden_size, cfg.rms_norm_eps)
+        self.attn = AQAttention(cfg)
+        self.mlp_norm = AQRMSNorm(cfg.hidden_size, cfg.rms_norm_eps)
+        self.mlp = AQSwiGLU(cfg)
+
+    def forward(self, x, cos, sin, attn_mask=None):
+        x = x + self.attn(self.attn_norm(x), cos, sin, attn_mask)
+        x = x + self.mlp(self.mlp_norm(x))
+        return x
+
+
+class AQPreTrainedModel(PreTrainedModel):
+    config_class = AQConfig
+    base_model_prefix = "aq"
+    supports_gradient_checkpointing = False
+    _no_split_modules = ["AQBlock"]
+
+    def _init_weights(self, module):
+        if isinstance(module, nn.Linear):
+            nn.init.normal_(module.weight, mean=0.0, std=0.02)
+        elif isinstance(module, nn.Embedding):
+            nn.init.normal_(module.weight, mean=0.0, std=0.02)
+
+
+class AQForCausalLM(AQPreTrainedModel, GenerationMixin):
+    """AQ decoder language model with a causal LM head (tied embeddings)."""
+
+    _tied_weights_keys = ["lm_head.weight"]
+
+    def __init__(self, config: AQConfig):
+        super().__init__(config)
+        self.embed = nn.Embedding(config.vocab_size, config.hidden_size)
+        self.layers = nn.ModuleList([AQBlock(config) for _ in range(config.num_hidden_layers)])
+        self.norm = AQRMSNorm(config.hidden_size, config.rms_norm_eps)
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+        cos, sin = build_rope_cache(
+            config.max_position_embeddings, config.head_dim, config.rope_theta, "cpu")
+        self.register_buffer("rope_cos", cos, persistent=False)
+        self.register_buffer("rope_sin", sin, persistent=False)
+
+        self.post_init()  # weight init + embedding tying (config.tie_word_embeddings)
+
+    def get_input_embeddings(self):
+        return self.embed
+
+    def set_input_embeddings(self, value):
+        self.embed = value
+
+    def get_output_embeddings(self):
+        return self.lm_head
+
+    def set_output_embeddings(self, new_embeddings):
+        self.lm_head = new_embeddings
+
+    def forward(
+        self,
+        input_ids: torch.LongTensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        **kwargs,
+    ) -> CausalLMOutput:
+        B, T = input_ids.shape
+        cos = self.rope_cos[:T].to(input_ids.device)
+        sin = self.rope_sin[:T].to(input_ids.device)
+
+        # Combined causal + padding mask (only when padding is actually present).
+        attn_mask = None
+        if attention_mask is not None and not bool(attention_mask.all()):
+            causal = torch.tril(torch.ones(T, T, dtype=torch.bool, device=input_ids.device))
+            pad = attention_mask[:, None, None, :].to(torch.bool)  # (B,1,1,T)
+            attn_mask = causal[None, None, :, :] & pad
+
+        x = self.embed(input_ids)
+        for layer in self.layers:
+            x = layer(x, cos, sin, attn_mask)
+        x = self.norm(x)
+        logits = self.lm_head(x)
+
+        loss = None
+        if labels is not None:
+            shift_logits = logits[:, :-1, :].contiguous()
+            shift_labels = labels[:, 1:].contiguous()
+            loss = F.cross_entropy(
+                shift_logits.view(-1, shift_logits.size(-1)),
+                shift_labels.view(-1),
+                ignore_index=-100,
+            )
+        return CausalLMOutput(loss=loss, logits=logits)

special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|endoftext|>"
+}

tokenizer_config.json

@@ -0,0 +1,19 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 2048,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}