Instructions to use glaiveai/glaive-function-calling-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use glaiveai/glaive-function-calling-v1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="glaiveai/glaive-function-calling-v1", trust_remote_code=True)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("glaiveai/glaive-function-calling-v1", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("glaiveai/glaive-function-calling-v1", trust_remote_code=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use glaiveai/glaive-function-calling-v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "glaiveai/glaive-function-calling-v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "glaiveai/glaive-function-calling-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/glaiveai/glaive-function-calling-v1

SGLang

How to use glaiveai/glaive-function-calling-v1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "glaiveai/glaive-function-calling-v1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "glaiveai/glaive-function-calling-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "glaiveai/glaive-function-calling-v1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "glaiveai/glaive-function-calling-v1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use glaiveai/glaive-function-calling-v1 with Docker Model Runner:
```
docker model run hf.co/glaiveai/glaive-function-calling-v1
```

glaive-function-calling-v1 / README.md

sahil2801

Update README.md

d94dd60 almost 3 years ago

preview code

raw

history blame contribute delete

3.14 kB

	---
	license: cc-by-sa-4.0
	datasets:
	- glaiveai/glaive-function-calling
	---
	# glaive-function-calling-v1

	glaive-function-calling-v1 is a 2.7B parameter open source chat model trained on data generated from Glaive’s synthetic data generation platform, which has similar function calling abilities as gpt-3.5 and gpt 4.

	The model is capable of having multi-turn conversations and intelligently choosing when to execute a function (provided at the beginning of the conversation as a system prompt) based on the conversation. The model is trained on top of the https://huggingface.co/replit/replit-code-v1-3b model.


	## Usage:

	You can run the model in the following way-
	```
	from transformers import AutoModelForCausalLM , AutoTokenizer

	tokenizer = AutoTokenizer.from_pretrained("glaiveai/glaive-function-calling-v1", trust_remote_code=True)
	model = AutoModelForCausalLM.from_pretrained("glaiveai/glaive-function-calling-v1", trust_remote_code=True).half().cuda()

	inputs = tokenizer(prompt,return_tensors="pt").to(model.device)

	outputs = model.generate(**inputs,do_sample=True,temperature=0.1,top_p=0.95,max_new_tokens=100)

	print(tokenizer.decode(outputs[0],skip_special_tokens=True))
	```

	This model uses the following prompt format-

	```
	SYSTEM: You are an helpful assistant who has access to the following functions to help the user, you can use the functions if needed-
	{
	"name": "plan_holiday",
	"description": "Plan a holiday based on user's interests",
	"parameters": {
	"type": "object",
	"properties": {
	"destination": {
	"type": "string",
	"description": "The destination of the holiday",
	},
	"duration": {
	"type": "integer",
	"description": "The duration of the trip in holiday",
	},
	},
	"required": ["destination", "duration"],
	},
	}
	USER: I am thinking of having a 10 day long vacation in Greece, can you help me plan it?
	```
	Based on which the model outputs-
	```
	ASSISTANT: <functioncall> {"name": "plan_holiday", "arguments": '{
	"destination": "Greece",
	"duration": 10
	}'}
	```
	The model precedes all function invocations with `<functioncall>`.

	The response of the function call should be sent to the model as-

	```
	FUNCTION CALL: {"places_to_visit":["Athens","Santorini","Mykonos"]}
	```

	The model can do multi-turn conversation in the above format.


	We're working on providing an inference server which can act as a drop in replacement to the OpenAI API, you can follow [this](https://github.com/glaive-ai/function-calling-server) repo for the server.

	## Known Limitations:

	- While the model does well on function calling use-cases, it doesn't always generalize very well to other chat use-cases. This is intentional as our thesis at Glaive is to provide use-case specialised model that are only used for the given task.
	- The model may sometimes hallucinate functions, v2 of the model will be aimed to fix that with a bigger dataset.