himalaya-ai/himalaya-gemma-4-e2b-it-gguf

Gemma 4 E2B-it Full SFT (Nepali & English)

This repository contains optimized GGUF quantized weights for himalaya-gemma-4-e2b-it. The base model was trained using Full-Parameter SFT over approximately 125,000 steps to achieve deep bilingual proficiency in both English and Nepali.

These GGUF builds are explicitly configured for high-performance local inference using [llama.cpp] and compatible frontends like LM Studio or Ollama.

💾 Provided Quantization Formats File Name Precision File Size Description himalaya_gemma_4_bf16.gguf BF16 ~9.31 GB Native Precision. himalaya_gemma_4_q8.gguf Q8_0 ~4.95 GB Near-Lossless. Standard 8-bit static quantization. himalaya_gemma_4_q4_k_m.gguf Q4_K_M ~3.43 GB Optimal Squeeze (Recommended). Mixed-precision 4-bit K-Quant. himalaya_gemma_4_q4_0.gguf Q4_0 ~3.36 GB

⚙️ Training Details

Unlike standard QLoRA fine-tuning, this model uses Full-Parameter SFT. This means every parameter in the model is trainable.

The training run stabilized beautifully over approximately 125,000 steps. To fit this comprehensive training on a single 1x A100 GPU with a tight memory budget, we utilized:

• 8-bit AdamW optimizer
• Gradient Checkpointing

📚 Datasets

The training data is a 50/50 mix of two high-quality datasets:

1. Nepali: himalaya-ai/nepali-sft-dataset
2. English: teknium/OpenHermes-2.5

🚀 How to Use for Benchmarking

You can load and test this model using the Hugging Face transformers library.

1. Install dependencies

First, make sure you have the required libraries installed:

pip install transformers accelerate torch

from transformers import AutoTokenizer, AutoModelForCausalLM import torch

Put the exact Hugging Face repository name here

model_id = "himalaya-ai/himalaya-gemma-4-e2b-it"

1. Load the tokenizer and model

print("Loading model for benchmarking...")
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.float16
)

2. Set up your prompt

user_prompt = "Write a short poem about the mountains in Nepal."

Apply the chat template

    {"role": "user", "content": user_prompt}
]
formatted_prompt = tokenizer.apply_chat_template(
    messages, 
    tokenize=False, 
    add_generation_prompt=True
)

3. Generate the response

print("Generating response...")
outputs = model.generate(
    **inputs, 
    max_new_tokens=256,
    do_sample=True,
    temperature=0.7
)

4. Print the result

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print("\n--- Output ---\n")
print(response)