Instructions to use zai-org/cogvlm2-video-llama3-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use zai-org/cogvlm2-video-llama3-base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="zai-org/cogvlm2-video-llama3-base", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("zai-org/cogvlm2-video-llama3-base", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use zai-org/cogvlm2-video-llama3-base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "zai-org/cogvlm2-video-llama3-base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/cogvlm2-video-llama3-base",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/zai-org/cogvlm2-video-llama3-base

SGLang

How to use zai-org/cogvlm2-video-llama3-base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "zai-org/cogvlm2-video-llama3-base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/cogvlm2-video-llama3-base",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "zai-org/cogvlm2-video-llama3-base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zai-org/cogvlm2-video-llama3-base",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use zai-org/cogvlm2-video-llama3-base with Docker Model Runner:
```
docker model run hf.co/zai-org/cogvlm2-video-llama3-base
```

zR commited on Jul 7, 2024

Commit

2978dc0

1 Parent(s): 3187e63

init

Browse files

Files changed (2) hide show

README.md +39 -17
README_zh.md +38 -17

README.md CHANGED Viewed

@@ -13,20 +13,41 @@ tags:
 inference: false
 ---
-# CogVLM2-Video
 [中文版本README](README_zh.md)
-CogVLM2-Video achieves state-of-the-art performance on multiple video question answering tasks. The following diagram
-shows the performance of CogVLM2-Video on
 the [MVBench](https://github.com/OpenGVLab/Ask-Anything), [VideoChatGPT-Bench](https://github.com/mbzuai-oryx/Video-ChatGPT)
 and Zero-shot VideoQA datasets (MSVD-QA, MSRVTT-QA, ActivityNet-QA). Where VCG-* refers to the VideoChatGPTBench, ZS-*
 refers to Zero-Shot VideoQA datasets and MV-* refers to main categories in the MVBench.
 ![Quantitative Evaluation](https://github.com/THUDM/CogVLM2/tree/main/resources/cogvlm2_video_bench.jpeg)
-## Detailed performance
 Performance on VideoChatGPT-Bench and Zero-shot VideoQA dataset:
 | Models                | VCG-AVG  | VCG-CI   | VCG-DO   | VCG-CU   | VCG-TU   | VCG-CO   | ZS-AVG    |
@@ -41,15 +62,15 @@ Performance on VideoChatGPT-Bench and Zero-shot VideoQA dataset:
 Performance on MVBench dataset:
-| Model                 | AVG      | AA       | AC       | AL       | AP       | AS       | CO       | CI       | EN    | ER       | FA       | FP       | MA       | MC       | MD       | OE       | OI       | OS   | ST       | SC   | UA       |
-|-----------------------|----------|----------|----------|----------|----------|----------|----------|----------|-------|----------|----------|----------|----------|----------|----------|----------|----------|------|----------|------|----------|
-| IG-VLM GPT4V          | 43.7     | 72.0     | 39.0     | 40.5     | **63.5** | 55.5     | 52.0     | 11.0     | 31.0  | 59.0     | 46.5     | 47.5     | 22.5     | 12.0     | 12.0     | 18.5     | 59.0     | 29.5 | 83.5     | 45.0 | 73.5     |
-| ST-LLM                | 54.9     | 84.0     | 36.5     | 31.0     | 53.5     | 66.0     | 46.5     | 58.5     | 34.5  | 41.5     | 44.0     | 44.5     | 78.5     | 56.5     | 42.5     | 80.5     | 73.5     | 38.5 | 86.5     | 43.0 | 58.5     |
-| ShareGPT4Video        | 51.2     | 79.5     | 35.5     | 41.5     | 39.5     | 49.5     | 46.5     | 51.5     | 28.5  | 39.0     | 40.0     | 25.5     | 75.0     | 62.5     | 50.5     | 82.5     | 54.5     | 32.5 | 84.5     | 51.0 | 54.5     |
-| VideoGPT+             | 58.7     | 83.0     | 39.5     | 34.0     | 60.0     | **69.0** | 50.0     | 60.0     | 29.5  | 44.0     | 48.5     | 53.0     | 90.5     | 71.0     | 44.0     | **85.5** | 75.5     | 36.0 | 89.5     | 45.0 | 66.5     |
-| VideoChat2_HD_mistral | 62.3     | 79.5     | **60.0** | **87.5** | 50.0     | 68.5     | **93.5** | 71.5     | 36.5  | 45.0     | 49.5     | **87.0** | 40.0     | **76.0** | **92.0** | 53.0     | 62.0     | 45.5 | 36.0     | 44.0 | 69.5     |
-| PLLaVA-34B            | 58.1     | 82.0     | 40.5     | 49.5     | 53.0     | 67.5     | 66.5     | 59.0     | l39.5 | **63.5** | 47.0     | 50.0     | 70.0     | 43.0     | 37.5     | 68.5     | 67.5     | 36.5 | **91.0** | 51.5 | **79.0** |
-| CogVLM2-Video         | **62.3** | **85.5** | 41.5     | 31.5     | 65.5     | 79.5     | 58.5     | **77.0** | 28.5  | 42.5     | **54.0** | 57.0     | **91.5** | 73.0     | 48.0     | **91.0** | **78.0** | 36.0 | **91.5** | 47.0 | 68.5     |
 ## Evaluation details
@@ -77,10 +98,11 @@ our [github](https://github.com/THUDM/CogVLM2/tree/main/video_demo).
 ## License
-This model is released under the CogVLM2 [LICENSE](LICENSE). For models built with Meta Llama 3, please also adhere to
-the [LLAMA3_LICENSE](LLAMA3_LICENSE).
 ## Training details
 Pleaser refer to our technical report for training formula and hyperparameters.

 inference: false
 ---
+# CogVLM2-Video-Llama3-Base
 [中文版本README](README_zh.md)
+## Introduction
+CogVLM2-Video achieves state-of-the-art performance on multiple video question answering tasks. It can achieve video
+understanding within one minute. We provide two example videos to demonstrate CogVLM2-Video's video understanding and
+video temporal grounding capabilities.
+<table>
+  <tr>
+    <td>
+      <video width="100%" controls>
+        <source src="https://github.com/THUDM/CogVLM2/raw/main/resources/videos/lion.mp4" type="video/mp4">
+      </video>
+    </td>
+    <td>
+      <video width="100%" controls>
+        <source src="https://github.com/THUDM/CogVLM2/raw/main/resources/videos/basketball.mp4" type="video/mp4">
+      </video>
+    </td>
+  </tr>
+</table>
+## BenchMark
+The following diagram shows the performance of CogVLM2-Video on
 the [MVBench](https://github.com/OpenGVLab/Ask-Anything), [VideoChatGPT-Bench](https://github.com/mbzuai-oryx/Video-ChatGPT)
 and Zero-shot VideoQA datasets (MSVD-QA, MSRVTT-QA, ActivityNet-QA). Where VCG-* refers to the VideoChatGPTBench, ZS-*
 refers to Zero-Shot VideoQA datasets and MV-* refers to main categories in the MVBench.
 ![Quantitative Evaluation](https://github.com/THUDM/CogVLM2/tree/main/resources/cogvlm2_video_bench.jpeg)
 Performance on VideoChatGPT-Bench and Zero-shot VideoQA dataset:
 | Models                | VCG-AVG  | VCG-CI   | VCG-DO   | VCG-CU   | VCG-TU   | VCG-CO   | ZS-AVG    |
 Performance on MVBench dataset:
+| Models                | AVG      | AA       | AC       | AL       | AP       | AS       | CO       | CI       | EN       | ER       | FA       | FP       | MA       | MC       | MD       | OE       | OI       | OS       | ST       | SC       | UA       |
+|-----------------------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|
+| IG-VLM GPT4V          | 43.7     | 72.0     | 39.0     | 40.5     | 63.5     | 55.5     | 52.0     | 11.0     | 31.0     | 59.0     | 46.5     | 47.5     | 22.5     | 12.0     | 12.0     | 18.5     | 59.0     | 29.5     | 83.5     | 45.0     | 73.5     |
+| ST-LLM                | 54.9     | 84.0     | 36.5     | 31.0     | 53.5     | 66.0     | 46.5     | 58.5     | 34.5     | 41.5     | 44.0     | 44.5     | 78.5     | 56.5     | 42.5     | 80.5     | 73.5     | 38.5     | 86.5     | 43.0     | 58.5     |
+| ShareGPT4Video        | 51.2     | 79.5     | 35.5     | 41.5     | 39.5     | 49.5     | 46.5     | 51.5     | 28.5     | 39.0     | 40.0     | 25.5     | 75.0     | 62.5     | 50.5     | 82.5     | 54.5     | 32.5     | 84.5     | 51.0     | 54.5     |
+| VideoGPT+             | 58.7     | 83.0     | 39.5     | 34.0     | 60.0     | 69.0     | 50.0     | 60.0     | 29.5     | 44.0     | 48.5     | 53.0     | 90.5     | 71.0     | 44.0     | 85.5     | 75.5     | 36.0     | 89.5     | 45.0     | 66.5     |
+| VideoChat2_HD_mistral | **62.3** | 79.5     | **60.0** | **87.5** | 50.0     | 68.5     | **93.5** | 71.5     | 36.5     | 45.0     | 49.5     | **87.0** | 40.0     | **76.0** | **92.0** | 53.0     | 62.0     | **45.5** | 36.0     | 44.0     | 69.5     |
+| PLLaVA-34B            | 58.1     | 82.0     | 40.5     | 49.5     | 53.0     | 67.5     | 66.5     | 59.0     | **39.5** | **63.5** | 47.0     | 50.0     | 70.0     | 43.0     | 37.5     | 68.5     | 67.5     | 36.5     | 91.0     | 51.5     | **79.0** |
+| CogVLM2-Video         | **62.3** | **85.5** | 41.5     | 31.5     | **65.5** | **79.5** | 58.5     | **77.0** | 28.5     | 42.5     | **54.0** | 57.0     | **91.5** | 73.0     | 48.0     | **91.0** | **78.0** | 36.0     | **91.5** | **47.0** | 68.5     |
 ## Evaluation details
 ## License
+This model is released under the
+CogVLM2  [LICENSE](https://modelscope.cn/models/ZhipuAI/cogvlm2-video-llama3-base/file/view/master?fileName=LICENSE&status=0).
+For models built with Meta Llama 3, please also adhere to
+the [LLAMA3_LICENSE](https://modelscope.cn/models/ZhipuAI/cogvlm2-video-llama3-base/file/view/master?fileName=LLAMA3_LICENSE&status=0).
 ## Training details
 Pleaser refer to our technical report for training formula and hyperparameters.

README_zh.md CHANGED Viewed

@@ -1,15 +1,34 @@
-# CogVLM2-Video
-CogVLM2-Video 在多个视频问答任务上实现了最先进的性能。下图显示了 CogVLM2-Video
 在 [MVBench](https://github.com/OpenGVLab/Ask-Anything)、[VideoChatGPT-Bench](https://github.com/mbzuai-oryx/Video-ChatGPT)
 和 Zero-shot VideoQA 数据集 (MSVD-QA、MSRVTT-QA、ActivityNet-QA) 上的性能。
-![Quantitative Evaluation](https://github.com/THUDM/CogVLM2/tree/main/resources/cogvlm2_video_bench.jpeg)
 其中 VCG 指的是 VideoChatGPTBench，ZS 指的是零样本 VideoQA 数据集，MV-* 指的是 MVBench 中的主要类别。
-## 评估结论
 具体榜单测试数据如下：
 | Models                | VCG-AVG  | VCG-CI   | VCG-DO   | VCG-CU   | VCG-TU   | VCG-CO   | ZS-AVG    |
@@ -24,15 +43,15 @@ CogVLM2-Video 在多个视频问答任务上实现了最先进的性能。下图
 CogVLM2-Video 在 MVBench 数据集上的表现
-| Model                 | AVG      | AA       | AC       | AL       | AP       | AS       | CO       | CI       | EN    | ER       | FA       | FP       | MA       | MC       | MD       | OE       | OI       | OS   | ST       | SC   | UA       |
-|-----------------------|----------|----------|----------|----------|----------|----------|----------|----------|-------|----------|----------|----------|----------|----------|----------|----------|----------|------|----------|------|----------|
-| IG-VLM GPT4V          | 43.7     | 72.0     | 39.0     | 40.5     | **63.5** | 55.5     | 52.0     | 11.0     | 31.0  | 59.0     | 46.5     | 47.5     | 22.5     | 12.0     | 12.0     | 18.5     | 59.0     | 29.5 | 83.5     | 45.0 | 73.5     |
-| ST-LLM                | 54.9     | 84.0     | 36.5     | 31.0     | 53.5     | 66.0     | 46.5     | 58.5     | 34.5  | 41.5     | 44.0     | 44.5     | 78.5     | 56.5     | 42.5     | 80.5     | 73.5     | 38.5 | 86.5     | 43.0 | 58.5     |
-| ShareGPT4Video        | 51.2     | 79.5     | 35.5     | 41.5     | 39.5     | 49.5     | 46.5     | 51.5     | 28.5  | 39.0     | 40.0     | 25.5     | 75.0     | 62.5     | 50.5     | 82.5     | 54.5     | 32.5 | 84.5     | 51.0 | 54.5     |
-| VideoGPT+             | 58.7     | 83.0     | 39.5     | 34.0     | 60.0     | **69.0** | 50.0     | 60.0     | 29.5  | 44.0     | 48.5     | 53.0     | 90.5     | 71.0     | 44.0     | **85.5** | 75.5     | 36.0 | 89.5     | 45.0 | 66.5     |
-| VideoChat2_HD_mistral | 62.3     | 79.5     | **60.0** | **87.5** | 50.0     | 68.5     | **93.5** | 71.5     | 36.5  | 45.0     | 49.5     | **87.0** | 40.0     | **76.0** | **92.0** | 53.0     | 62.0     | 45.5 | 36.0     | 44.0 | 69.5     |
-| PLLaVA-34B            | 58.1     | 82.0     | 40.5     | 49.5     | 53.0     | 67.5     | 66.5     | 59.0     | l39.5 | **63.5** | 47.0     | 50.0     | 70.0     | 43.0     | 37.5     | 68.5     | 67.5     | 36.5 | **91.0** | 51.5 | **79.0** |
-| CogVLM2-Video         | **62.3** | **85.5** | 41.5     | 31.5     | 65.5     | 79.5     | 58.5     | **77.0** | 28.5  | 42.5     | **54.0** | 57.0     | **91.5** | 73.0     | 48.0     | **91.0** | **78.0** | 36.0 | **91.5** | 47.0 | 68.5     |
 ## 评估和复现
@@ -57,8 +76,10 @@ prompt = f"The input consists of a sequence of key frames from a video. Answer t
 ## 模型协议
-此模型根据 CogVLM2 [LICENSE](LICENSE) 发布。对于使用 Meta Llama 3 构建的模型，还请遵守
-[LLAMA3_LICENSE](LLAMA3_LICENSE)。
 ## 引用

+# CogVLM2-Video-Llama3-Base
+## 介绍
+CogVLM2-Video 在多个视频问答任务上实现了最先进的性能。能够实现一分钟内的视频理解。 我们提供了两个示例视频，分别展现了
+CogVLM2-Video 的 视频理解和时间序列定位能力。
+<table>
+  <tr>
+    <td>
+      <video width="100%" controls>
+        <source src="https://github.com/THUDM/CogVLM2/raw/main/resources/videos/lion.mp4" type="video/mp4">
+      </video>
+    </td>
+    <td>
+      <video width="100%" controls>
+        <source src="https://github.com/THUDM/CogVLM2/raw/main/resources/videos/basketball.mp4" type="video/mp4">
+      </video>
+    </td>
+  </tr>
+</table>
+## 榜单性能
+下图显示了 CogVLM2-Video
 在 [MVBench](https://github.com/OpenGVLab/Ask-Anything)、[VideoChatGPT-Bench](https://github.com/mbzuai-oryx/Video-ChatGPT)
 和 Zero-shot VideoQA 数据集 (MSVD-QA、MSRVTT-QA、ActivityNet-QA) 上的性能。
+![Quantitative Evaluation](https://raw.githubusercontent.com/THUDM/CogVLM2/main/resources/cogvlm2_video_bench.jpeg)
 其中 VCG 指的是 VideoChatGPTBench，ZS 指的是零样本 VideoQA 数据集，MV-* 指的是 MVBench 中的主要类别。
 具体榜单测试数据如下：
 | Models                | VCG-AVG  | VCG-CI   | VCG-DO   | VCG-CU   | VCG-TU   | VCG-CO   | ZS-AVG    |
 CogVLM2-Video 在 MVBench 数据集上的表现
+| Models                | AVG      | AA       | AC       | AL       | AP       | AS       | CO       | CI       | EN       | ER       | FA       | FP       | MA       | MC       | MD       | OE       | OI       | OS       | ST       | SC       | UA       |
+|-----------------------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|
+| IG-VLM GPT4V          | 43.7     | 72.0     | 39.0     | 40.5     | 63.5     | 55.5     | 52.0     | 11.0     | 31.0     | 59.0     | 46.5     | 47.5     | 22.5     | 12.0     | 12.0     | 18.5     | 59.0     | 29.5     | 83.5     | 45.0     | 73.5     |
+| ST-LLM                | 54.9     | 84.0     | 36.5     | 31.0     | 53.5     | 66.0     | 46.5     | 58.5     | 34.5     | 41.5     | 44.0     | 44.5     | 78.5     | 56.5     | 42.5     | 80.5     | 73.5     | 38.5     | 86.5     | 43.0     | 58.5     |
+| ShareGPT4Video        | 51.2     | 79.5     | 35.5     | 41.5     | 39.5     | 49.5     | 46.5     | 51.5     | 28.5     | 39.0     | 40.0     | 25.5     | 75.0     | 62.5     | 50.5     | 82.5     | 54.5     | 32.5     | 84.5     | 51.0     | 54.5     |
+| VideoGPT+             | 58.7     | 83.0     | 39.5     | 34.0     | 60.0     | 69.0     | 50.0     | 60.0     | 29.5     | 44.0     | 48.5     | 53.0     | 90.5     | 71.0     | 44.0     | 85.5     | 75.5     | 36.0     | 89.5     | 45.0     | 66.5     |
+| VideoChat2_HD_mistral | **62.3** | 79.5     | **60.0** | **87.5** | 50.0     | 68.5     | **93.5** | 71.5     | 36.5     | 45.0     | 49.5     | **87.0** | 40.0     | **76.0** | **92.0** | 53.0     | 62.0     | **45.5** | 36.0     | 44.0     | 69.5     |
+| PLLaVA-34B            | 58.1     | 82.0     | 40.5     | 49.5     | 53.0     | 67.5     | 66.5     | 59.0     | **39.5** | **63.5** | 47.0     | 50.0     | 70.0     | 43.0     | 37.5     | 68.5     | 67.5     | 36.5     | 91.0     | 51.5     | **79.0** |
+| CogVLM2-Video         | **62.3** | **85.5** | 41.5     | 31.5     | **65.5** | **79.5** | 58.5     | **77.0** | 28.5     | 42.5     | **54.0** | 57.0     | **91.5** | 73.0     | 48.0     | **91.0** | **78.0** | 36.0     | **91.5** | **47.0** | 68.5     |
 ## 评估和复现
 ## 模型协议
+此模型根据
+CogVLM2 [LICENSE](https://modelscope.cn/models/ZhipuAI/cogvlm2-video-llama3-base/file/view/master?fileName=LICENSE&status=0)
+发布。对于使用 Meta Llama 3 构建的模型，还请遵守
+[LLAMA3_LICENSE](https://modelscope.cn/models/ZhipuAI/cogvlm2-video-llama3-base/file/view/master?fileName=LLAMA3_LICENSE&status=0)。
 ## 引用