---
library_name: transformers
license: apache-2.0
language:
- mn
- en
tags:
- Mongolian
- QLora
- Llama3
- Instructed-model
- unsloth
pipeline_tag: text-generation
---

## Mongolian-Llama3.1

![ Alt Text](Mongolian_llama3_1.jpg)

### Model Description

To implement Mongolian-Llama3.1 in a Chat UI, you'll need to set up an interface that interacts with the Llama 3.1 model. Here’s a step-by-step guide to achieve this:
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/11QQdExUnojmWDmjpk_DCidAXLhgPvWd-?usp=sharing])

Mongolian-Llama3.1 is the second open source instruction-tuned language model for Mongolian & English users with various abilities such as roleplaying & tool-using built upon the quantized Meta-Llama-3.1-8B model.

Developed by: Dorjzodovsuren

License: Llama-3 License

Base Model: llama-3.1-8b-bnb-4bit

Model Size: 4.65B

Context length: 8K

## Bias, Risks, and Limitations

To combat fake news, current strategies rely heavily on synthetic and translated data. However, these approaches have inherent biases, risks, and limitations:

1. **Synthetic Data Bias**: Algorithms may inadvertently perpetuate biases present in training data.

2. **Translation Inaccuracy**: Translations can distort meaning or lose context, leading to misinformation.

3. **Cultural Nuances**: Synthetic and translated data may miss cultural intricacies, risking amplification of stereotypes.

4. **Algorithmic Limits**: Effectiveness is constrained by algorithm capabilities and training data quality.

5. **Dependency on Data**: Accuracy hinges on quality and representativeness of training data.

6. **Adversarial Attacks**: Malicious actors can exploit vulnerabilities to manipulate content.

7. **Different answer based on language**: Answer might be a bit different based on language.

### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
Due to hallucinations and pretraining datasets characteristics, some information might be misleading, and answer might be a bit different based on language. 

Please ask in <b>Mongolian</b> if possible.

### Disclaimer: 

> We are not responsible for any consequences resulting from the use of this model or the outputs it generates. Users are advised to employ the model's predictions with caution and to independently verify the information provided.

## How to Get Started with the Model

Use the code below to get started with the model.

```python
import torch
import gradio as gr
from threading import Thread
from unsloth import FastLanguageModel
from transformers import TextStreamer
from transformers import AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList, TextIteratorStreamer

max_seq_length = 2048
load_in_4bit = True
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "Dorjzodovsuren/Mongolian_Llama3-v1.1",
    max_seq_length = max_seq_length,
    dtype = None,
    load_in_4bit = load_in_4bit,
)

EOS_TOKEN = tokenizer.eos_token # Must add EOS_TOKEN

alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{}

### Input:
{}

### Response:
{}"""


# Enable native 2x faster inference
FastLanguageModel.for_inference(model)

# Create a text streamer
text_streamer = TextStreamer(tokenizer, skip_prompt=False,skip_special_tokens=True)

# Get the device based on GPU availability
device = 'cuda' if torch.cuda.is_available() else 'cpu'

# Move model into device
model = model.to(device)

class StopOnTokens(StoppingCriteria):
    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
        stop_ids = [29, 0]
        for stop_id in stop_ids:
            if input_ids[0][-1] == stop_id:
                return True
        return False

# Current implementation does not support conversation based on history.
# Highly recommend to experiment on various hyper parameters to compare qualities.
def predict(message, history):
    stop = StopOnTokens()
    messages = alpaca_prompt.format(
        message,
        "",
        "",
    )

    model_inputs = tokenizer([messages], return_tensors="pt").to(device)

    streamer = TextIteratorStreamer(tokenizer, timeout=10., skip_prompt=True, skip_special_tokens=True)
    generate_kwargs = dict(
        model_inputs,
        streamer=streamer,
        max_new_tokens=max_seq_length,
        temperature=0.7,
        top_p=0.9,
        top_k=50,
        do_sample=True,
        stopping_criteria=StoppingCriteriaList([stop])
        )
    t = Thread(target=model.generate, kwargs=generate_kwargs)
    t.start()

    partial_message = ""
    for new_token in streamer:
        if new_token != '<':
            partial_message += new_token
            yield partial_message

gr.ChatInterface(predict).launch(debug=True, share=True, show_api=True)
```