Instructions to use TheBloke/Phind-CodeLlama-34B-v2-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use TheBloke/Phind-CodeLlama-34B-v2-GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="TheBloke/Phind-CodeLlama-34B-v2-GPTQ")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("TheBloke/Phind-CodeLlama-34B-v2-GPTQ")
model = AutoModelForCausalLM.from_pretrained("TheBloke/Phind-CodeLlama-34B-v2-GPTQ")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use TheBloke/Phind-CodeLlama-34B-v2-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TheBloke/Phind-CodeLlama-34B-v2-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/Phind-CodeLlama-34B-v2-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/TheBloke/Phind-CodeLlama-34B-v2-GPTQ

SGLang

How to use TheBloke/Phind-CodeLlama-34B-v2-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "TheBloke/Phind-CodeLlama-34B-v2-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/Phind-CodeLlama-34B-v2-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "TheBloke/Phind-CodeLlama-34B-v2-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/Phind-CodeLlama-34B-v2-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use TheBloke/Phind-CodeLlama-34B-v2-GPTQ with Docker Model Runner:
```
docker model run hf.co/TheBloke/Phind-CodeLlama-34B-v2-GPTQ
```

HumanEval Results

by m9e - opened Sep 11, 2023

Discussion

m9e

Sep 11, 2023

Just finished testing this against HumanEval, using this config:

which I believe ~matches the Phind config.

on the :gptq-4bit-32g-actorder_True branch

results

{'pass@1': 0.725609756097561}

which I believe means my first test on humaneval the model got 119/164 correct, vs 121/164 for the full precision according to the model card, although given it's not fully deterministic can't say for sure that's the quantization.

Still, nice to see it come in extremely tight on humaneval.

m9e

Sep 12, 2023

as a followup, I extended the context window using these settings:

{
"_from_model_config": true,
"bos_token_id": 1,
"do_sample": true,
"pad_token_id": 2,
"eos_token_id": 2,
"max_new_tokens": 384,
"temperature": 0.1,
"top_p": 0.75,
"top_k": 40,
"max_seq_length": 16384,
"rope_freq_base": 1000000,
"compress_pos_emb": 4,
"gpu_split": "19,23",
"transformers_version": "4.33.1"
}

Fumbling around a bit from various sources on the proper way to do the longer sequence with the CodeLlama model, but it was able to digest a 1000-line/35k character file and analyze the code in a way that wasn't crazy in text-generation-webui with these settings, so I spawned them in my server version (which is just a fastapi wrapper around AutoModelForCausalLM.from_pretrained basically)

and re-tested

{'pass@1': 0.7134146341463414}

again hard to say where the variance is, and my impression is that the compress_pos_emb setting needed to extend the context properly has a small negative effect, so this wasn't surprising but it still seemed competent.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment