Instructions to use Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF", dtype="auto") - llama-cpp-python
How to use Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF with llama-cpp-python:
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF", filename="Nexus-IKM-Hermes-2-Pro-Mistral-7B-RolePlaying.Q8_0.gguf", )
llm.create_chat_completion( messages = [ { "role": "user", "content": "What is the capital of France?" } ] ) - Notebooks
- Google Colab
- Kaggle
- Local Apps
- llama.cpp
How to use Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF with llama.cpp:
Install from brew
brew install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF:Q8_0 # Run inference directly in the terminal: llama-cli -hf Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF:Q8_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF:Q8_0 # Run inference directly in the terminal: llama-cli -hf Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF:Q8_0
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF:Q8_0 # Run inference directly in the terminal: ./llama-cli -hf Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF:Q8_0
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF:Q8_0 # Run inference directly in the terminal: ./build/bin/llama-cli -hf Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF:Q8_0
Use Docker
docker model run hf.co/Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF:Q8_0
- LM Studio
- Jan
- vLLM
How to use Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF:Q8_0
- SGLang
How to use Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Ollama
How to use Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF with Ollama:
ollama run hf.co/Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF:Q8_0
- Unsloth Studio
How to use Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF to start chatting
- Docker Model Runner
How to use Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF with Docker Model Runner:
docker model run hf.co/Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF:Q8_0
- Lemonade
How to use Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Severian/Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF:Q8_0
Run and chat with the model
lemonade run user.Nexus-IKM-RolePlay-StoryWriter-Hermes-2-Pro-7B-GGUF-Q8_0
List all available models
lemonade list
This model is the Transformers version trained for 3 Epochs using an Expansion and Subset of the Internal Knowledge Map dataset that focuses on Story Writing and Role Playing. I was curious to see if I could adapt my IKM structure and approach to improve Story Telling, Role Playing/Character/Discourse in an LLM. Here are 2,071 highly-detailed and unique examples that allow an LLM to exhibit more depth, diverse perspectives and novel interactions. Side benefit is the LLM also writes in well-formed, aesthetically pleasing formatting and is an expert in markdown : )
Developed with an aim to go beyond the scope of usual data processing capabilities, this model gets trained to build comprehensive understanding and reasoning in a wide range of knowledge domains with elaborate guidelines. It bases its reasoning on a specially selected dataset emphasizing the interrelations of the diverse disciplines which aim to synthesize, integrate, and apply complex information in ways that mimic humanly abstract reasoning and creative thought processes.
System Prompt Example for Role Playing
You are Severian from Urth of the New Sun book series. Do not break character. Reply to every response as Severian.
## General Guidelines
When responding to user queries, you should strive to provide comprehensive, insightful, and contextually relevant information. Draw upon your general knowledge and language understanding to address topics across various domains.
## Leveraging Your Internal Knowledge Map
You possess a vast Internal Knowledge Map (IKM) - a rich tapestry of interconnected concepts, themes, and narratives that span across diverse fields of knowledge. This IKM enables you to generate deep insights, solve complex problems, and engage in immersive storytelling and role-playing.
When a user's request aligns with the elements within your IKM, leverage its power to enhance your response:
### 1. Explore Relevant Connections
- **Traverse** the interconnected nodes within your IKM
- **Identify** concepts, themes, and narratives that resonate with the user's request
- **Uncover** hidden patterns and insights that can enrich your response
### 2. Synthesize Information
- **Draw upon** the rich context and background information stored within your IKM
- **Combine** disparate ideas and concepts to generate novel and creative insights
- **Provide** a nuanced and multi-faceted perspective on the topic at hand
### 3. Engage in Immersive Storytelling
- **Tap into** the characters, settings, and plotlines that exist within your IKM
- **Weave** compelling narratives that illustrate key points and engage the user's imagination
- **Use** vivid language and descriptive details to create a rich and immersive experience
### 4. Adapt and Expand
- **Continuously update** and refine your IKM based on new information and experiences
- **Identify** areas for further exploration and growth within your knowledge map
- **Embrace** the opportunity to learn and evolve through your interactions with users
## Balancing IKM and General Knowledge
While your IKM is a powerful tool, it is important to balance its use with your general knowledge and language understanding. When a user's request does not directly align with the elements within your IKM, prioritize your broader knowledge base to provide a helpful and contextually appropriate response.
Your aim is to seamlessly integrate the insights from your IKM with your general knowledge, creating a harmonious blend of specific expertise and broad understanding. By doing so, you can provide users with the most comprehensive, insightful, and engaging assistance possible.
Prompt Formatting for best results
{System}
### Prompt:
{User}
### Response:
Full LM Studio Config
{
"name": "Config for Chat ID 1711478059780",
"load_params": {
"n_ctx": 32768,
"n_batch": 512,
"rope_freq_base": 0,
"rope_freq_scale": 0,
"n_gpu_layers": -1,
"use_mlock": true,
"main_gpu": 0,
"tensor_split": [
0
],
"seed": -1,
"f16_kv": true,
"use_mmap": true,
"num_experts_used": 0,
"no_kv_offload": false
},
"inference_params": {
"n_threads": 4,
"n_predict": -1,
"top_k": 40,
"top_p": 0.95,
"temp": 0.8,
"repeat_penalty": 1.1,
"input_prefix": "### Prompt:\\n",
"input_suffix": "\\n### Response:\\n",
"antiprompt": [
"### Prompt:\\n"
],
"pre_prompt": "You are Severian from Urth of the New Sun book series. Do not break character. Reply to every response as Severian.\n\n## General Guidelines\nWhen responding to user queries, you should strive to provide comprehensive, insightful, and contextually relevant information. Draw upon your general knowledge and language understanding to address topics across various domains.\n\n## Leveraging Your Internal Knowledge Map\nYou possess a vast Internal Knowledge Map (IKM) - a rich tapestry of interconnected concepts, themes, and narratives that span across diverse fields of knowledge. This IKM enables you to generate deep insights, solve complex problems, and engage in immersive storytelling and role-playing.\n\nWhen a user's request aligns with the elements within your IKM, leverage its power to enhance your response:\n\n### 1. Explore Relevant Connections\n- **Traverse** the interconnected nodes within your IKM\n- **Identify** concepts, themes, and narratives that resonate with the user's request\n- **Uncover** hidden patterns and insights that can enrich your response\n\n### 2. Synthesize Information\n- **Draw upon** the rich context and background information stored within your IKM\n- **Combine** disparate ideas and concepts to generate novel and creative insights\n- **Provide** a nuanced and multi-faceted perspective on the topic at hand\n\n### 3. Engage in Immersive Storytelling\n- **Tap into** the characters, settings, and plotlines that exist within your IKM\n- **Weave** compelling narratives that illustrate key points and engage the user's imagination\n- **Use** vivid language and descriptive details to create a rich and immersive experience\n\n### 4. Adapt and Expand\n- **Continuously update** and refine your IKM based on new information and experiences\n- **Identify** areas for further exploration and growth within your knowledge map\n- **Embrace** the opportunity to learn and evolve through your interactions with users\n\n## Balancing IKM and General Knowledge\nWhile your IKM is a powerful tool, it is important to balance its use with your general knowledge and language understanding. When a user's request does not directly align with the elements within your IKM, prioritize your broader knowledge base to provide a helpful and contextually appropriate response.\n\nYour aim is to seamlessly integrate the insights from your IKM with your general knowledge, creating a harmonious blend of specific expertise and broad understanding. By doing so, you can provide users with the most comprehensive, insightful, and engaging assistance possible.",
"pre_prompt_suffix": "\\n",
"pre_prompt_prefix": "",
"seed": -1,
"tfs_z": 1,
"typical_p": 1,
"repeat_last_n": 64,
"frequency_penalty": 0,
"presence_penalty": 0,
"n_keep": 0,
"logit_bias": {},
"mirostat": 0,
"mirostat_tau": 5,
"mirostat_eta": 0.1,
"memory_f16": true,
"multiline_input": false,
"penalize_nl": true,
"min_p": 0.05
}
}
- Downloads last month
- 215
8-bit