--- language: - hi - en base_model: - bharatgenai/Param-1-2.9B-Instruct pipeline_tag: text-generation tags: - Ayurvedic ---
BharatGen
Paper License Blog
# AyurParam BharatGen introduces AyurParam, a domain-specialized large language model fine-tuned from Param-1-2.9B-Instruct on a high-quality Ayurveda dataset. It is designed to handle Ayurvedic queries, classical text interpretation, clinical guidance, and wellness knowledge. Ayurveda offers vast traditional medical wisdom, yet most language models lack domain-specific understanding. AyurParam bridges this gap by combining Param-1โ€™s bilingual strengths with a curated Ayurvedic knowledge base, enabling contextually rich and culturally grounded responses. ## ๐Ÿ— Model Architecture AyurParam inherits the architecture of Param-1-2.9B-Instruct: * Hidden size: 204 * Intermediate size: 7168 * Attention heads: 16 * Hidden layers: 32 * Key-value heads: 8 * Max position embeddings: 2048 * Activation: SiLU * Positional Embeddings: Rotary (RoPE, theta=10000) * Attention Mechanism: Grouped-query attention * Precision: bf16-mixed * Base model: Param-1-2.9B-Instruct ## ๐Ÿ“š Data Preparation AyurParamโ€™s training corpus was carefully crafted to ensure deep Ayurvedic knowledge, Sanskrit/English bilingual accessibility, and clinical relevance. Steps involved: 1. Source Gathering * 15k+ passages from classical Ayurvedic texts (digitized and curated). * 10k+ passages from AYUSH ministry guidelines, research papers, and clinical case discussions. 2. Question Generation * 5 curated Q&A pairs generated per passage using an open-source LLM + domain expert review. 3. Domain Taxonomy & Personas * Built an Ayurveda-specific taxonomy (Dosha, Dhatu, Mala, Srotas, Nidana, Chikitsa, etc.). * Defined multiple personas: student, vaidya (physician), researcher, policymaker, wellness coach. 4. Dataset Construction * 1.5M Q&A pairs grounded in taxonomy and personas. * 4M multi-turn conversation samples created. * Sanskrit terminology preserved with transliteration and explanations. ## ๐Ÿ‹๏ธ Training Setup * Base model: Param-1-2.9B-Instruct * Training framework: Hugging Face + TRL (SFT) + torchrun multi-node setup * Prompt template: Custom-designed for Ayurvedic inference * Scheduler: Linear with warmup * Epochs: 3 * Total training samples: ~8M * Test samples: ~800k * Base learning rate: 5e-6 * Minimum learning rate: 0 * Additional tokens: , , , * Vocab size: 256k + 4 * Global batch size: 1024 * Micro batch size: 4 * Gradient accumulation steps: 32 ## ๐Ÿš€ Inference Example ```python from transformers import AutoTokenizer, AutoModelForCausalLM import torch model_name = "bharatgenai/AyurParam" tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False) model = AutoModelForCausalLM.from_pretrained( model_name, trust_remote_code=True, torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.bfloat32, device_map="auto" ) # Example Ayurvedic query user_input = "What is the Samprapti (pathogenesis) of Amavata according to Ayurveda?" # Prompt styles # 1. Generic QA: ... # 2. Context-based QA: ... ... # 3. Multi-turn conversation (supports up to 5 turns): ... ... ... prompt = f" {user_input} " inputs = tokenizer(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): output = model.generate( **inputs, max_new_tokens=300, do_sample=True, top_k=50, top_p=0.95, temperature=0.6, eos_token_id=tokenizer.eos_token_id, use_cache=False ) print(tokenizer.decode(output[0], skip_special_tokens=True)) ``` ## ๐Ÿ“Š Benchmark Results: Ayur Param vs Baselines - [BhashaBench-Ayur benchmark](https://huggingface.co/datasets/bharatgenai/BhashaBench-Ayur) --- ## 1. Overall Performance ### Similar Range Models | Model | bba | bba_English | bba_Hindi | |-----------------------|-------|-------------|-----------| | Llama-3.2-1B-Instruct | 26.41 | 26.77 | 25.82 | | Qwen2.5-3B-Instruct | 32.68 | 35.22 | 28.46 | | granite-3.1-2b | 31.10 | 33.39 | 27.30 | | Llama-3.2-3B-Instruct | 33.20 | 35.31 | 29.67 | | gemma-2-2b-it | 28.40 | 29.38 | 26.79 | | **AyurParam** | **39.97** | **41.12** | **38.04** | ### Larger Models | Model | bba | bba_English | bba_Hindi | |-----------------------------------------|-------|-------------|-----------| | Indic-gemma-7B-Navarasa-2.0 | 35.13 | 37.12 | 31.83 | | Pangea-7B | 37.41 | 40.69 | 31.93 | | aya-23-8B | 31.97 | 33.84 | 28.87 | | gpt-oss-20b | 36.34 | 38.30 | 33.09 | | Llama-3.1-8B-Instruct | 34.76 | 36.86 | 31.26 | | gemma-2-27b-it | 37.99 | 40.45 | 33.89 | | Nemotron-4-Mini-Hindi-4B-Instruct | 33.54 | 33.38 | 33.82 | | **AyurParam** | **39.97** | **41.12** | **38.04** | --- ## 2. Question Difficulty ### Similar Range Models | Difficulty | Llama-3.2-1B | Qwen2.5-3B | granite-3.1-2b | Llama-3.2-3B | gemma-2-2b-it | **AyurParam** | |------------|--------------|------------|----------------|--------------|---------------|----------------| | **Easy** | 27.44 | 35.55 | 33.90 | 36.42 | 29.96 | **43.93** | | **Medium** | 25.23 | 29.57 | 28.06 | 29.66 | 26.83 | **35.95** | | **Hard** | 25.39 | 28.23 | 26.81 | 28.51 | 24.96 | **31.21** | ### Larger Models | Difficulty | Indic-gemma-7B | Pangea-7B | aya-23-8B | gpt-oss-20b | Llama-3.1-8B | gemma-2-27b-it | Nemotron-4-Mini-Hindi-4B | **AyurParam** | |------------|----------------|-----------|-----------|-------------|--------------|----------------|--------------------------|----------------| | **Easy** | 38.54 | 41.45 | 35.51 | 42.03 | 39.43 | 43.47 | 36.08 |**43.93** | | **Medium** | 31.72 | 32.94 | 28.29 | 30.27 | 29.36 | 31.90 | 30.80 |**35.95** | | **Hard** | 27.23 | 31.77 | 25.11 | 26.67 | 30.50 | 30.78 | 29.50 |**31.21** | --- ## 3. Question Type ### Similar Range Models | Type | Llama-3.2-1B | Qwen2.5-3B | granite-3.1-2b | Llama-3.2-3B | gemma-2-2b-it | **AyurParam** | |----------------------|--------------|------------|----------------|--------------|---------------|----------------| | Assertion/Reasoning | 59.26 | 51.85 | 33.33 | 40.74 | 33.33 | **44.44** | | Fill in the blanks | 26.97 | 29.21 | 21.35 | 34.83 | 32.02 | **29.78** | | MCQ | 26.34 | 32.70 | 31.22 | 33.17 | 28.33 | **40.12** | | Match the column | 26.83 | 29.27 | 29.27 | 29.27 | 36.59 | **24.39** | ### Larger Models | Type | Indic-gemma-7B | Pangea-7B | aya-23-8B | gpt-oss-20b | Llama-3.1-8B | gemma-2-27b-it | Nemotron-4-Mini-Hindi-4B | **AyurParam** | |----------------------|----------------|-----------|-----------|-------------|--------------|----------------|--------------------------|----------------| | Assertion/Reasoning | 59.26 | 62.96 | 18.52 | 25.93 | 29.63 | 55.56 | 37.04 | **44.44** | | Fill in the blanks | 35.39 | 24.16 | 30.90 | 32.02 | 26.97 | 35.96 | 30.34 | **29.78** | | MCQ | 35.10 | 37.53 | 32.05 | 36.39 | 34.83 | 37.98 | 33.60 | **40.12** | | Match the column | 31.71 | 34.15 | 17.07 | 46.34 | 46.34 | 39.02 | 24.39 | **24.39** | --- From the above results, **Ayur Param not only outperforms all similar-sized models** but also achieves **competitive or better performance than larger models** across multiple metrics. ## Contact For any questions or feedback, please contact: - Sravan Kumar (sravan.kumar@tihiitb.org) - Kundeshwar Pundalik (kundeshwar.pundalik@tihiitb.org)