Instructions to use humarin/chatgpt_paraphraser_on_T5_base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use humarin/chatgpt_paraphraser_on_T5_base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="humarin/chatgpt_paraphraser_on_T5_base")

# Load model directly
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("humarin/chatgpt_paraphraser_on_T5_base")
model = AutoModelForSeq2SeqLM.from_pretrained("humarin/chatgpt_paraphraser_on_T5_base")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use humarin/chatgpt_paraphraser_on_T5_base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "humarin/chatgpt_paraphraser_on_T5_base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "humarin/chatgpt_paraphraser_on_T5_base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/humarin/chatgpt_paraphraser_on_T5_base

SGLang

How to use humarin/chatgpt_paraphraser_on_T5_base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "humarin/chatgpt_paraphraser_on_T5_base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "humarin/chatgpt_paraphraser_on_T5_base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "humarin/chatgpt_paraphraser_on_T5_base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "humarin/chatgpt_paraphraser_on_T5_base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use humarin/chatgpt_paraphraser_on_T5_base with Docker Model Runner:
```
docker model run hf.co/humarin/chatgpt_paraphraser_on_T5_base
```

Please Provide the parameters exact value used for the input in the demo

by Rkoy - opened Apr 20, 2023

Discussion

Rkoy

Apr 20, 2023

•

edited Apr 20, 2023

Using the parameters provided in the model card i am generating a different sentence compared to the sentences generated using the demo (inference api). It would be usefull if you are able to provide the exact parameters used for the demo (inference api)

Input: The Confidential Information disclosed under this agreement will be kept by the Disclosing Party and destroyed by either party upon its request

Output obtained by running model using the paramters mentioned in the model card: max_length=128, num_beams=5, num_return_sequences=5, num_beam_groups=5, repetition_penalty=10.0, diversity_penalty=3.0, no_repeat_ngram_size=2, temperature=0.7 is
Confidential Information disclosed under this agreement will be kept by the Disclosing Party and destroyed by either party upon its request.

But the output for the same sentence when given in the demo (inference api) is Confidential Information disclosed under this agreement will be destroyed by either party upon its request, and the Disclosing Party agrees to hold onto it for as long as is necessary to destroy it.

And another things i noticed while going through the config.json is that there are no configurations related to the paraphrase task. there are configurations related to other tasks like translation, summarization etc. Could you let me know is this the reason the difference in the output obtained or is it some other reason and why there are no configurations related to paraphrase.

Rkoy changed discussion title from Provide the parameters exact value used for the input in the demo to Please Provide the parameters exact value used for the input in the demo Apr 20, 2023

madm1952

May 19, 2023

Second this! The Readme.MD file does not contain the range of the parameters. A detailed explanation is needed.

Thank you

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment