---
language:
- en
- hi
tags:
- indic
- hindi
- हिंदी
- indian
- language
- english
license: apache-2.0
datasets:
  - mc4
  - c4
inference: false
---
*[To be released soon]*

# BHASHA-7B-8K-HI

A 7B foundation language model pre-trained on hindi text with 8k context size. Weights initialised from MPT-7B-8K model. Uses extended vocabulary with knowledge transfer within embedding space. 

## Model Description

| Hyperparameter | Value |
|----------------|-------|
|n_parameters | 6695735296 (6.69B) |
|n_layers | 32 |
| n_heads | 32 |
| d_model | 4096 |
| vocab size | 61772 |
| sequence length | 8192 |

*This model is still getting pre-trained. Updated weights along with more details will be available soon.*

[Follow us](https://www.linkedin.com/company/soketlabs) to get updates on the progress.