Instructions to use fbellame/mistral-7b-json-quizz-fine-tuned with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use fbellame/mistral-7b-json-quizz-fine-tuned with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="fbellame/mistral-7b-json-quizz-fine-tuned")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("fbellame/mistral-7b-json-quizz-fine-tuned")
model = AutoModelForMultimodalLM.from_pretrained("fbellame/mistral-7b-json-quizz-fine-tuned")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use fbellame/mistral-7b-json-quizz-fine-tuned with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "fbellame/mistral-7b-json-quizz-fine-tuned"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fbellame/mistral-7b-json-quizz-fine-tuned",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/fbellame/mistral-7b-json-quizz-fine-tuned

SGLang

How to use fbellame/mistral-7b-json-quizz-fine-tuned with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "fbellame/mistral-7b-json-quizz-fine-tuned" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fbellame/mistral-7b-json-quizz-fine-tuned",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "fbellame/mistral-7b-json-quizz-fine-tuned" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fbellame/mistral-7b-json-quizz-fine-tuned",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use fbellame/mistral-7b-json-quizz-fine-tuned with Docker Model Runner:
```
docker model run hf.co/fbellame/mistral-7b-json-quizz-fine-tuned
```

mistral-7b-json-quizz-fine-tuned / cfg.yaml

fbellame

Upload cfg.yaml

edec775 over 2 years ago

Raw

History Blame Contribute Delete

2.92 kB

	architecture:
	backbone_dtype: int4
	force_embedding_gradients: false
	gradient_checkpointing: true
	intermediate_dropout: 0.0
	pretrained: true
	pretrained_weights: ''
	augmentation:
	neftune_noise_alpha: 0.0
	random_parent_probability: 0.0
	skip_parent_probability: 0.0
	token_mask_probability: 0.0
	dataset:
	add_eos_token_to_answer: true
	add_eos_token_to_prompt: false
	add_eos_token_to_system: true
	answer_column: output
	chatbot_author: H2O.ai
	chatbot_name: h2oGPT
	data_sample: 1.0
	data_sample_choice:
	- Train
	- Validation
	limit_chained_samples: false
	mask_prompt_labels: true
	parent_id_column: None
	personalize: false
	prompt_column:
	- instruction
	system_column: None
	text_answer_separator: ''
	text_prompt_start: ''
	text_system_start: <\|system\|>
	train_dataframe: /mnt/ssd2/h2o-llmstudio/data/user/train_data.2/train_data.parquet
	validation_dataframe: /mnt/ssd2/h2o-llmstudio/data/user/train_data.2/validate_data.parquet
	validation_size: 0.01
	validation_strategy: custom
	environment:
	compile_model: false
	deepspeed_reduce_bucket_size: 1000000
	deepspeed_stage3_param_persistence_threshold: 1000000
	deepspeed_stage3_prefetch_bucket_size: 1000000
	find_unused_parameters: false
	gpus:
	- '0'
	huggingface_branch: main
	mixed_precision: true
	number_of_workers: 8
	seed: -1
	trust_remote_code: true
	use_deepspeed: false
	experiment_name: mistral_7b_json_quizz.fine_tuned
	llm_backbone: mistralai/Mistral-7B-Instruct-v0.2
	logging:
	logger: None
	neptune_project: ''
	output_directory: /mnt/ssd2/h2o-llmstudio/output/user/mistral_7b_json_quizz.fine_tuned/
	prediction:
	batch_size_inference: 0
	do_sample: false
	max_length_inference: 256
	metric: GPT
	metric_gpt_model: gpt-3.5-turbo-0613
	metric_gpt_template: general
	min_length_inference: 2
	num_beams: 1
	num_history: 4
	repetition_penalty: 1.0
	stop_tokens: ''
	temperature: 0.0
	top_k: 0
	top_p: 1.0
	problem_type: text_causal_language_modeling
	tokenizer:
	add_prefix_space: false
	add_prompt_answer_tokens: false
	max_length: 1024
	max_length_answer: 512
	max_length_prompt: 512
	padding_quantile: 1.0
	use_fast: true
	training:
	batch_size: 1
	differential_learning_rate: 1.0e-05
	differential_learning_rate_layers: []
	drop_last_batch: true
	epochs: 2
	evaluate_before_training: true
	evaluation_epochs: 1.0
	grad_accumulation: 1
	gradient_clip: 0.0
	learning_rate: 0.0001
	lora: true
	lora_alpha: 16
	lora_dropout: 0.05
	lora_r: 4
	lora_target_modules: ''
	loss_function: TokenAveragedCrossEntropy
	optimizer: AdamW
	save_best_checkpoint: false
	schedule: Cosine
	train_validation_data: false
	use_flash_attention_2: true
	warmup_epochs: 0.0
	weight_decay: 0.0