Instructions to use qazimbhat1/Crystal-based-MLLM-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use qazimbhat1/Crystal-based-MLLM-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="qazimbhat1/Crystal-based-MLLM-7B", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("qazimbhat1/Crystal-based-MLLM-7B", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use qazimbhat1/Crystal-based-MLLM-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "qazimbhat1/Crystal-based-MLLM-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "qazimbhat1/Crystal-based-MLLM-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/qazimbhat1/Crystal-based-MLLM-7B

SGLang

How to use qazimbhat1/Crystal-based-MLLM-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "qazimbhat1/Crystal-based-MLLM-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "qazimbhat1/Crystal-based-MLLM-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "qazimbhat1/Crystal-based-MLLM-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "qazimbhat1/Crystal-based-MLLM-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use qazimbhat1/Crystal-based-MLLM-7B with Docker Model Runner:
```
docker model run hf.co/qazimbhat1/Crystal-based-MLLM-7B
```

Crystal-based-MLLM-7B / README.md

qazimbhat1

Update README.md

bfb2d4e verified about 2 years ago

preview code

Raw

History Blame

6.41 kB

	---
	license: apache-2.0
	language:
	- en
	pipeline_tag: text-generation
	library_name: transformers
	tags:
	- nlp
	- llm
	- mllm
	---

	# CrystalChat-7B-MLLM: a fully-reproducible vision language model based on CrystalChat-7B

	## Model Description

	CrystalChat-7B based multi-modal large language model (MLLM) mimics the training recipe used for Vicuna-7B based [LLaVa-v1.5](https://huggingface.co/docs/transformers/main/model_doc/llava). CrystalChat-7B-MLLM models are entirely transparent, having open-sourced all materials, including code, data, model checkpoint, intermediate results, and more at [TODO: Add paper link]().

	### About CrystalChat-7B-MLLM:
	* 7 billion parameter LLM
	* CLIP ViT-L/14-336px vision encoder
	* Languages: English
	* Models Released: CrystalChat-7B-MLLM
	* Trained in 2 stages
	* License: ?

	Crystal-based models were developed as a collaboration between [MBZUAI](https://mbzuai.ac.ae/institute-of-foundation-models/), [Petuum](https://www.petuum.com/), and [LLM360](https://www.llm360.ai/) TODO- check????.


	## Evaluation

	General Evaluation Metrics for MLLMs. MME serves as an extensive evaluative benchmark,
	aiming to assess perceptual and cognitive capability of MLLMs within 14 sub-tasks. Additionally, we also evaluate the performance of our models on text-oriented visual question answering tasks employing a diverse set of benchmark datasets including ScienceQA and TextVQA. Furthermore, we assess our models’ ability toward anti-hallucination through POPE.

	\| LLM Backbone \| MME-P \| MME-C \| POPE \| SciQA \| TextVQA \|
	\|-----------------------------------\|---------\|--------\|-------\|--------\|---------\|
	\| CrystalCoder-7B \| 1359.83 \| 238.92 \| 86.18 \| 64.15 \| 50.39 \|
	\| CrystalChat-7B \| 1456.53 \| 308.21 \| 86.96 \| 67.77 \| 57.84 \|
	\| Vicuna-7B \| 1481.12 \| 302.85 \| 87.17 \| 67.97 \| 56.49 \|

	Table 1: Comparison of different LLM backbones on visual language understanding benchmarks. All models are instruction-tuned on the general domain data (i.e. LLaVA)


	## Data and Training Details

	### Pretrain Data
	LLaVA Visual Instruct Pretrain LCS-558K is a filtered subset of the LAION, CC, and SBU datasets, featuring a more balanced distribution of concept coverage. The file includes multimodal synthesized conversations generated from image-caption pairs by incorporating randomly selected instructions such as "Describe this image." It is used for pretraining in LLaVA, with the raw CC-3M caption serving as the default answer.

	### Finetune Data

	The dataset chosen was created by LLaVA with academic-task-oriented VQA data mixture and data from ShareGPT. LLaVA Visual Instruct 150K is a dataset of GPT-generated multimodal instruction-following data. It is designed for visual instruction tuning and aims to develop large multimodal models with capabilities akin to GPT-4 in both vision and language.

	<!-- The full data sequence can be found [here](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K) -->

	\| Data \| Size \| Response formatting prompts \|
	\|---------------\|------\|--------------------------------------------------------------------------\|
	\| LLaVA [36] \| 158K \| – \|
	\| ShareGPT [46] \| 40K \| – \|
	\| VQAv2 [19] \| 83K \| Answer the question using a single word or phrase. \|
	\| GQA [21] \| 72K \| Answer the question using a single word or phrase. \|
	\| OKVQA [41] \| 9K \| Answer the question using a single word or phrase. \|
	\| OCRVQA [42] \| 80K \| Answer the question using a single word or phrase. \|
	\| A-OKVQA [45] \| 66K \| Answer with the option’s letter from the given choices directly. \|
	\| TextCaps [47] \| 22K \| Provide a one-sentence caption for the provided image. \|
	\| RefCOCO [24, 40] \| 48K \| Note: randomly choose between the two formats. Provide a short description for this region. \|
	\| VG [25] \| 86K \| Provide the bounding box coordinate of the region this sentence describes. \|
	\| Total \| 665K \| \|

	Table 2. Instruction-following Data Mixture of LLaVA-1.5.


	TODO: Check if we need to publish these 2
	## Stage 2 - Finetuning
	\| Checkpoints \| \|
	\| ----------- \| ----------- \|
	\| [CrystalChat](https://huggingface.co/qazimbhat1/my-model-repo3/tree/main) \|
	\| [CrystalCoder](https://huggingface.co/qazimbhat1/Crystal-based-MLLM-7B/tree/Crystal-coder-7B) \|

	## Stage 1 - Pretraining
	\| Checkpoints \| \|
	\| ----------- \| ----------- \|
	\| [CrystalChat](https://huggingface.co/qazimbhat1/Crystal-based-MLLM-7B/tree/Crystal-based-MLLM-7B-pretrain) \|
	\| [CrystalCoder](https://huggingface.co/qazimbhat1/Crystal-based-MLLM-7B/tree/Crystal-coder-7B-pretrain) \|


	[to find all branches: git branch -a]

	## Examples

	TODO: Add image as sample example
	<center><img src="k2_table_of_tables.png" alt="k2 big eval table"/></center>


	## Loading Crystal
	```python
	from transformers import AutoModelForCausalLM, AutoTokenizer

	tokenizer = AutoTokenizer.from_pretrained(
	"LLM360/CrystalChat-7B-MLLM",
	padding_side="right",
	trust_remote_code=True)

	model = AutoModelForCausalLM.from_pretrained(
	"LLM360/CrystalChat-7B-MLLM",
	trust_remote_code=True,
	torch_dtype=torch.float16,
	device_map='auto',
	low_cpu_mem_usage=True
	)
	```



	## LLM-360
	LLM-360 is an open research lab enabling community-owned AGI through open-source large model research and development.

	Crystal-based Models enables community-owned AGI by creating standards and tools to advance the bleeding edge of LLM capability and empower knowledge transfer, research, and development.

	We believe in a future where artificial general intelligence (AGI) is created by the community, for the community. Through an open ecosystem of equitable computational resources, high-quality data, and flowing technical knowledge, we can ensure ethical AGI development and universal access for all innovators.

	[Visit us](https://www.llm360.ai/)


	## Citation

	BibTeX:

	```bibtex
	@article{
	title={},
	author={},
	year={},
	}
	```