inference card added

README.md

@@ -0,0 +1,58 @@
+This is the 8-bit quantized model of the [ministral-8B](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410) by Mistral-AI.Please follow the following instruction to run the model on your device:
+
+There are multiple ways to infer the model. Firstly, let's install `llama.cpp` and use it for the inference:
+
+1. Install
+```
+git clone https://github.com/ggerganov/llama.cpp
+!mkdir llama.cpp/build && cd llama.cpp/build && cmake .. && cmake --build . --config Release
+```
+
+2. Inference
+```
+./llama.cpp/build/bin/llama-cli -m ./ministral-8b_Q8_0.gguf -cnv -p "You are a helpful assistant"
+```
+
+Here, you can interact with model from your terminal.
+
+
+**Alternatively**, we can use python binding of the `llama.cpp` to run the model on both CPU and GPU.
+1. Install
+```
+pip install --no-cache-dir llama-cpp-python==0.2.85 --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu122
+```
+
+2. Inference on CPU
+```
+from llama_cpp import Llama
+
+model_path = "./ministral-8b_Q8_0.gguf"
+llm = Llama(model_path=model_path, n_threads=8, verbose=False)
+
+prompt = "What should I do when my eyes are dry?"
+output = llm(
+        prompt=f"<|user|>\n{prompt}<|end|>\n<|assistant|>",
+        max_tokens=4096,
+        stop=["<|end|>"],
+        echo=False,  # Whether to echo the prompt
+)
+print(output)
+```
+
+3. Inference on GPU
+```
+from llama_cpp import Llama
+
+model_path = "./ministral-8b_Q8_0.gguf"
+llm = Llama(model_path=model_path, n_threads=8, n_gpu_layers=-1, verbose=False)
+
+prompt = "What should I do when my eyes are dry?"
+output = llm(
+        prompt=f"<|user|>\n{prompt}<|end|>\n<|assistant|>",
+        max_tokens=4096,
+        stop=["<|end|>"],
+        echo=False,  # Whether to echo the prompt
+)
+print(output)
+```
+