Not-For-All-Audiences

nsfw

Instructions to use benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2")
model = AutoModelForCausalLM.from_pretrained("benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2

SGLang

How to use benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2 with Docker Model Runner:
```
docker model run hf.co/benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2
```

Configuration Parsing Warning:In config.json: "quantization_config.bits" must be an integer

Exllamav2 3.75bpw quantization of NoromaidxOpenGPT4-2 from NeverSleep, quantized with default calibration dataset. Included is measurement json file, so you can do your own quants.

This bpw is the perfect size for 24GB GPUs, and can fit 32k context. Make sure to enable 4-bit cache option or you'll run into OOM errors.

Notes: This model is one of the better mixtral derivatives for rp, and I recommend using it with the Alpaca preset in SillyTavern.

Original Card

Description

This repo contains fp16 files of NoromaidxOpenGPT4-2.

The model was created by merging Noromaid-8x7b-Instruct with Open_Gpt4_8x7B_v0.2 the exact same way Rombodawg done his merge.

The only difference between NoromaidxOpenGPT4-1 and NoromaidxOpenGPT4-2 is that the first iteration use Mixtral-8x7B as a base for the merge (f16), where the second use Open_Gpt4_8x7B_v0.2 as a base (bf16).

After further testing and usage, the two model was released, because they each have their own qualities.

You can download the imatrix file to do many other quant HERE.

Prompt template:

Alpaca

### Instruction:
{system prompt}

### Input:
{prompt}

### Response:
{output}

Mistral

[INST] {prompt} [/INST]

Merge Details

Merge Method

This model was merged using the TIES merge method using rombodawg/Open_Gpt4_8x7B_v0.2 as a base.

Models Merged

The following models were included in the merge:

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: mistralai/Mixtral-8x7B-Instruct-v0.1
    parameters:
      density: .5
      weight: 1
  - model: NeverSleep/Noromaid-v0.1-mixtral-8x7b-Instruct-v3
    parameters:
      density: .5
      weight: .7
merge_method: ties
base_model: rombodawg/Open_Gpt4_8x7B_v0.2
parameters:
  normalize: true
  int8_mask: true
dtype: bfloat16

Support

If you want to support us, you can here.

Downloads last month: 11

Model tree for benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2

NeverSleep/Noromaid-v0.1-mixtral-8x7b-Instruct-v3

mistralai/Mixtral-8x7B-Instruct-v0.1

rombodawg/Open_Gpt4_8x7B_v0.2

Merge model

this model

Paper for benk04/NoromaidxOpenGPT4-2-3.75bpw-h6-exl2

Resolving Interference When Merging Models

Paper • 2306.01708 • Published Jun 2, 2023 • 19