Instructions to use MILVLG/Imp-v1.5-4B-Phi3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use MILVLG/Imp-v1.5-4B-Phi3 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="MILVLG/Imp-v1.5-4B-Phi3", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("MILVLG/Imp-v1.5-4B-Phi3", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use MILVLG/Imp-v1.5-4B-Phi3 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "MILVLG/Imp-v1.5-4B-Phi3" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "MILVLG/Imp-v1.5-4B-Phi3", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/MILVLG/Imp-v1.5-4B-Phi3
- SGLang
How to use MILVLG/Imp-v1.5-4B-Phi3 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "MILVLG/Imp-v1.5-4B-Phi3" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "MILVLG/Imp-v1.5-4B-Phi3", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "MILVLG/Imp-v1.5-4B-Phi3" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "MILVLG/Imp-v1.5-4B-Phi3", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use MILVLG/Imp-v1.5-4B-Phi3 with Docker Model Runner:
docker model run hf.co/MILVLG/Imp-v1.5-4B-Phi3
ModuleNotFoundError
Hello, I use transformers 4.41.2 and when I try to run your HF sample code I have :
Cell In[50], line 8
model = AutoModelForCausalLM.from_pretrained(
File ~\Anaconda3\envs\py39\lib\site-packages\transformers\models\auto\auto_factory.py:550 in from_pretrained
model_class = get_class_from_dynamic_module(
File ~\Anaconda3\envs\py39\lib\site-packages\transformers\dynamic_module_utils.py:513 in get_class_from_dynamic_module
return get_class_in_module(class_name, final_module)
File ~\Anaconda3\envs\py39\lib\site-packages\transformers\dynamic_module_utils.py:211 in get_class_in_module
module_spec.loader.exec_module(module)
File :850 in exec_module
File :228 in _call_with_frames_removed
File ~.cache\huggingface\modules\transformers_modules\MILVLG\Imp-v1.5-4B-Phi3\67394b9e49f912e892814ef922e1c485e91bf9e4\modeling_imp.py:44
from .vision_encoder import VisionTower
ModuleNotFoundError: No module named 'transformers_modules.MILVLG.Imp-v1'
Thanks for your issue.
You can add '/' after "MILVLG/Imp-v1.5-4B-Phi3", then it should be work.
And we will update it.
After I add '/' to the model path, I get this error.
from transformers import pipeline
messages = [
{"role": "user", "content": "Who are you?"},
]
pipe = pipeline(
"text-generation",
model="MILVLG/Imp-v1.5-4B-Phi3/",
trust_remote_code=True,
revision="main"
)
pipe(messages)
---------------------------------------------------------------------------
HFValidationError Traceback (most recent call last)
[/usr/local/lib/python3.11/dist-packages/transformers/utils/hub.py](https://localhost:8080/#) in cached_files(path_or_repo_id, filenames, cache_dir, force_download, resume_download, proxies, token, revision, local_files_only, subfolder, repo_type, user_agent, _raise_exceptions_for_gated_repo, _raise_exceptions_for_missing_entries, _raise_exceptions_for_connection_errors, _commit_hash, **deprecated_kwargs)
423 # This is slightly better for only 1 file
--> 424 hf_hub_download(
425 path_or_repo_id,
9 frames
HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'MILVLG/Imp-v1.5-4B-Phi3/'. Use `repo_type` argument if needed.
During handling of the above exception, another exception occurred:
HFValidationError Traceback (most recent call last)
[/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_validators.py](https://localhost:8080/#) in validate_repo_id(repo_id)
152
153 if repo_id.count("/") > 1:
--> 154 raise HFValidationError(
155 "Repo id must be in the form 'repo_name' or 'namespace/repo_name':"
156 f" '{repo_id}'. Use `repo_type` argument if needed."
HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'MILVLG/Imp-v1.5-4B-Phi3/'. Use `repo_type` argument if needed
Hi,
Thanks for your great work. I also get the same errors as above after adding the '/'. Any updates pls?