Instructions to use thesven/Hermes-2-Theta-Llama-3-8B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use thesven/Hermes-2-Theta-Llama-3-8B-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="thesven/Hermes-2-Theta-Llama-3-8B-GGUF",
	filename="Hermes-2-Theta-Llama-3-8B-GGUF-IQ1_S.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use thesven/Hermes-2-Theta-Llama-3-8B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf thesven/Hermes-2-Theta-Llama-3-8B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf thesven/Hermes-2-Theta-Llama-3-8B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf thesven/Hermes-2-Theta-Llama-3-8B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf thesven/Hermes-2-Theta-Llama-3-8B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf thesven/Hermes-2-Theta-Llama-3-8B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf thesven/Hermes-2-Theta-Llama-3-8B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf thesven/Hermes-2-Theta-Llama-3-8B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf thesven/Hermes-2-Theta-Llama-3-8B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/thesven/Hermes-2-Theta-Llama-3-8B-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use thesven/Hermes-2-Theta-Llama-3-8B-GGUF with Ollama:
```
ollama run hf.co/thesven/Hermes-2-Theta-Llama-3-8B-GGUF:Q4_K_M
```

Unsloth Studio

How to use thesven/Hermes-2-Theta-Llama-3-8B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for thesven/Hermes-2-Theta-Llama-3-8B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for thesven/Hermes-2-Theta-Llama-3-8B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for thesven/Hermes-2-Theta-Llama-3-8B-GGUF to start chatting

Atomic Chat new
Docker Model Runner
How to use thesven/Hermes-2-Theta-Llama-3-8B-GGUF with Docker Model Runner:
```
docker model run hf.co/thesven/Hermes-2-Theta-Llama-3-8B-GGUF:Q4_K_M
```

Lemonade

How to use thesven/Hermes-2-Theta-Llama-3-8B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull thesven/Hermes-2-Theta-Llama-3-8B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Hermes-2-Theta-Llama-3-8B-GGUF-Q4_K_M

List all available models

lemonade list

thesven commited on May 18, 2024

Commit

76d4ed7

verified ·

1 Parent(s): d8ebbc0

Update README.md

Browse files

Files changed (1) hide show

README.md +0 -150

README.md CHANGED Viewed

@@ -198,144 +198,6 @@ You are a helpful assistant that answers in JSON. Here's the json schema you mus
 Given the {schema} that you provide, it should follow the format of that json to create it's response, all you have to do is give a typical user prompt, and it will respond in JSON.
-# Benchmarks
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/6317aade83d8d2fd903192d9/suBbCUIxpcRvhCv6-DBDQ.png)
-## GPT4All:
-```
-|    Task     |Version| Metric |Value |   |Stderr|
-|-------------|------:|--------|-----:|---|-----:|
-|arc_challenge|      0|acc     |0.5529|±  |0.0145|
-|             |       |acc_norm|0.5870|±  |0.0144|
-|arc_easy     |      0|acc     |0.8371|±  |0.0076|
-|             |       |acc_norm|0.8144|±  |0.0080|
-|boolq        |      1|acc     |0.8599|±  |0.0061|
-|hellaswag    |      0|acc     |0.6133|±  |0.0049|
-|             |       |acc_norm|0.7989|±  |0.0040|
-|openbookqa   |      0|acc     |0.3940|±  |0.0219|
-|             |       |acc_norm|0.4680|±  |0.0223|
-|piqa         |      0|acc     |0.8063|±  |0.0092|
-|             |       |acc_norm|0.8156|±  |0.0090|
-|winogrande   |      0|acc     |0.7372|±  |0.0124|
-```
-Average: 72.59
-## AGIEval:
-```
-|             Task             |Version| Metric |Value |   |Stderr|
-|------------------------------|------:|--------|-----:|---|-----:|
-|agieval_aqua_rat              |      0|acc     |0.2441|±  |0.0270|
-|                              |       |acc_norm|0.2441|±  |0.0270|
-|agieval_logiqa_en             |      0|acc     |0.3687|±  |0.0189|
-|                              |       |acc_norm|0.3840|±  |0.0191|
-|agieval_lsat_ar               |      0|acc     |0.2304|±  |0.0278|
-|                              |       |acc_norm|0.2174|±  |0.0273|
-|agieval_lsat_lr               |      0|acc     |0.5471|±  |0.0221|
-|                              |       |acc_norm|0.5373|±  |0.0221|
-|agieval_lsat_rc               |      0|acc     |0.6617|±  |0.0289|
-|                              |       |acc_norm|0.6357|±  |0.0294|
-|agieval_sat_en                |      0|acc     |0.7670|±  |0.0295|
-|                              |       |acc_norm|0.7379|±  |0.0307|
-|agieval_sat_en_without_passage|      0|acc     |0.4417|±  |0.0347|
-|                              |       |acc_norm|0.4223|±  |0.0345|
-|agieval_sat_math              |      0|acc     |0.4000|±  |0.0331|
-|                              |       |acc_norm|0.3455|±  |0.0321|
-```
-Average: 44.05
-## BigBench:
-```
-|                      Task                      |Version|       Metric        |Value |   |Stderr|
-|------------------------------------------------|------:|---------------------|-----:|---|-----:|
-|bigbench_causal_judgement                       |      0|multiple_choice_grade|0.6000|±  |0.0356|
-|bigbench_date_understanding                     |      0|multiple_choice_grade|0.6585|±  |0.0247|
-|bigbench_disambiguation_qa                      |      0|multiple_choice_grade|0.3178|±  |0.0290|
-|bigbench_geometric_shapes                       |      0|multiple_choice_grade|0.2340|±  |0.0224|
-|                                                |       |exact_str_match      |0.0000|±  |0.0000|
-|bigbench_logical_deduction_five_objects         |      0|multiple_choice_grade|0.2980|±  |0.0205|
-|bigbench_logical_deduction_seven_objects        |      0|multiple_choice_grade|0.2057|±  |0.0153|
-|bigbench_logical_deduction_three_objects        |      0|multiple_choice_grade|0.5367|±  |0.0288|
-|bigbench_movie_recommendation                   |      0|multiple_choice_grade|0.4040|±  |0.0220|
-|bigbench_navigate                               |      0|multiple_choice_grade|0.4970|±  |0.0158|
-|bigbench_reasoning_about_colored_objects        |      0|multiple_choice_grade|0.7075|±  |0.0102|
-|bigbench_ruin_names                             |      0|multiple_choice_grade|0.4821|±  |0.0236|
-|bigbench_salient_translation_error_detection    |      0|multiple_choice_grade|0.2295|±  |0.0133|
-|bigbench_snarks                                 |      0|multiple_choice_grade|0.6906|±  |0.0345|
-|bigbench_sports_understanding                   |      0|multiple_choice_grade|0.5375|±  |0.0159|
-|bigbench_temporal_sequences                     |      0|multiple_choice_grade|0.6270|±  |0.0153|
-|bigbench_tracking_shuffled_objects_five_objects |      0|multiple_choice_grade|0.2216|±  |0.0118|
-|bigbench_tracking_shuffled_objects_seven_objects|      0|multiple_choice_grade|0.1594|±  |0.0088|
-|bigbench_tracking_shuffled_objects_three_objects|      0|multiple_choice_grade|0.5367|±  |0.0288|
-```
-Average: 44.13
-**IFEval**: 72.64
-**MT_Bench**: Turn 1 - 8.3875, Turn 2 - 8.00625, Average - 8.196875
-# Inference Code
-Here is example code using HuggingFace Transformers to inference the model (note: in 4bit, it will require around 5GB of VRAM)
-Note: To use function calling, you should see the github repo above.
-```python
-# Code to inference Hermes with HF Transformers
-# Requires pytorch, transformers, bitsandbytes, sentencepiece, protobuf, and flash-attn packages
-import torch
-from transformers import AutoTokenizer, AutoModelForCausalLM, LlamaForCausalLM
-import bitsandbytes, flash_attn
-tokenizer = AutoTokenizer.from_pretrained('NousResearch/Hermes-2-Theta-Llama-3-8B', trust_remote_code=True)
-model = LlamaForCausalLM.from_pretrained(
-    "NousResearch/Hermes-2-Theta-Llama-3-8B",
-    torch_dtype=torch.float16,
-    device_map="auto",
-    load_in_8bit=False,
-    load_in_4bit=True,
-    use_flash_attention_2=True
-)
-prompts = [
-    """<|im_start|>system
-You are a sentient, superintelligent artificial general intelligence, here to teach and assist me.<|im_end|>
-<|im_start|>user
-Write a short story about Goku discovering kirby has teamed up with Majin Buu to destroy the world.<|im_end|>
-<|im_start|>assistant""",
-    ]
-for chat in prompts:
-    print(chat)
-    input_ids = tokenizer(chat, return_tensors="pt").input_ids.to("cuda")
-    generated_ids = model.generate(input_ids, max_new_tokens=750, temperature=0.8, repetition_penalty=1.1, do_sample=True, eos_token_id=tokenizer.eos_token_id)
-    response = tokenizer.decode(generated_ids[0][input_ids.shape[-1]:], skip_special_tokens=True, clean_up_tokenization_space=True)
-    print(f"Response: {response}")
-```
 ## Inference Code for Function Calling:
 All code for utilizing, parsing, and building function calling templates is available on our github:
@@ -343,18 +205,6 @@ All code for utilizing, parsing, and building function calling templates is avai
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/6317aade83d8d2fd903192d9/oi4CiGh50xmoviUQnh8R3.png)
-# Chat Interfaces
-When quantized versions of the model are released, I recommend using LM Studio for chatting with Hermes 2 Pro. It does not support function calling - for that use our github repo. It is a GUI application that utilizes GGUF models with a llama.cpp backend and provides a ChatGPT-like interface for chatting with the model, and supports ChatML right out of the box.
-In LM-Studio, simply select the ChatML Prefix on the settings side pane:
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/6317aade83d8d2fd903192d9/ls6WqV-GSxMw2RA3GuQiN.png)
-## Quantized Versions:
-GGUF Versions Available Here: https://huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-8B-GGUF
 # How to cite:
 ```bibtext

 Given the {schema} that you provide, it should follow the format of that json to create it's response, all you have to do is give a typical user prompt, and it will respond in JSON.
 ## Inference Code for Function Calling:
 All code for utilizing, parsing, and building function calling templates is available on our github:
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/6317aade83d8d2fd903192d9/oi4CiGh50xmoviUQnh8R3.png)
 # How to cite:
 ```bibtext