XReyRobert commited on
Commit
8040574
·
verified ·
1 Parent(s): ca73e3d

Clean model card for public release

Browse files
Files changed (1) hide show
  1. README.md +18 -16
README.md CHANGED
@@ -1,6 +1,7 @@
1
  ---
2
  base_model: Jackrong/Qwopus3.6-27B-v2
3
  library_name: transformers
 
4
  tags:
5
  - qwen3.6
6
  - qwopus
@@ -10,17 +11,16 @@ tags:
10
  - vllm
11
  - int4
12
  - quantized
13
- license: other
14
  ---
15
 
16
  ![Qwopus3.6-27B-v2 GPTQ-Pro v1 RTX 3090 benchmark](assets/qwopus-gptq-pro-v1-rtx3090-benchmark.png)
17
 
18
-
19
  # Qwopus3.6-27B-v2 GPTQ-Pro v1
20
 
21
- This is the first public-ready GPTQ-Pro 4-bit release of `Jackrong/Qwopus3.6-27B-v2`, built to make this excellent Qwopus/Qwen3.6 model practical to run in vLLM with GPTQ-Marlin kernels and long-context inference.
22
 
23
- The goal is simple: preserve as much of the original model's character and capability as possible while making it efficient enough for real local/homelab serving, including RTX 3090-class deployments.
24
 
25
  This is not a new fine-tune. It is a quantized derivative of the original Qwopus3.6-27B-v2 model.
26
 
@@ -57,7 +57,9 @@ Thanks to Jackrong for the original Qwopus3.6 model, and to groxaxo for GPTQ-Pro
57
  | FOEM beta | `0.2` |
58
  | Batch size | `1` |
59
 
60
- Preserved modules include vision, `lm_head`, embeddings, norms, and MTP modules. Post-publication validation showed the internal `ns256-v2` artifact preserves MTP metadata but does **not** include actual `mtp.*` tensors in `model.safetensors.index.json`, so this release should be treated as non-MTP for vLLM speculative decoding.
 
 
61
 
62
  Post-save compatibility patch:
63
 
@@ -66,7 +68,7 @@ Post-save compatibility patch:
66
 
67
  ## Intended serving setup
68
 
69
- This checkpoint is intended for local/homelab vLLM serving on RTX 3090-class hardware.
70
 
71
  Recommended vLLM options:
72
 
@@ -92,7 +94,7 @@ vllm serve XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1 \
92
 
93
  ## Reasoning / thinking mode
94
 
95
- This model preserves Qwen3-style reasoning behavior. The Hermes validation below was run with thinking enabled.
96
 
97
  ## MTP / speculative decoding status
98
 
@@ -108,16 +110,16 @@ That MTP-GPTQ artifact works and reaches good draft acceptance, but it was still
108
 
109
  ## RTX 3090 validation status
110
 
111
- This checkpoint was validated locally on an RTX 3090 24GB with vLLM, `max_model_len=131072`, `kv_cache_dtype=fp8_e5m2`, prefix caching enabled, and Hermes running with thinking enabled.
112
 
113
- Observed Hermes/vLLM workload metrics:
114
 
115
  | Metric | Observed value | Notes |
116
  |---|---:|---|
117
- | Requests observed | `15` | Hermes session calls |
118
  | vLLM request success count | `15/15` | No vLLM errors observed during the sample |
119
- | Average prompt size | `33,172` tokens | Real multi-turn Hermes workload |
120
- | Average output size | `322` tokens | Real Hermes responses |
121
  | Average time to first token | `5.70s` | Prometheus TTFT summary |
122
  | Average end-to-end request latency | `13.07s` | Includes prefill, decode, and serving overhead |
123
  | Average time per output token | `0.0230s/token` | vLLM TPOT summary |
@@ -127,7 +129,7 @@ Observed Hermes/vLLM workload metrics:
127
  | Live 60s generation throughput | about `19.1 generated tok/s` | Aggregate over full window, including prefill and idle mix |
128
  | Live 60s prefix-cache hit ratio | `78.9%` | Delta over the observed window |
129
 
130
- These are practical Hermes session metrics, not a synthetic benchmark. They are useful for RTX 3090-class homelab serving expectations, especially multi-turn long-context usage with prefix caching.
131
 
132
  ## Compatibility notes
133
 
@@ -138,7 +140,7 @@ This artifact was built and validated for text-only vLLM serving without specula
138
  - Experimental quantization.
139
  - MTP/speculative decoding is not supported by this published artifact because `mtp.*` tensors are missing.
140
  - Quality has not yet been benchmarked against the BF16 source model.
141
- - RTX 3090 metrics above are observed Hermes workload numbers, not a controlled benchmark suite.
142
  - Use at your own risk, especially for long-context or tool-calling workflows.
143
 
144
  ## References
@@ -147,6 +149,6 @@ This artifact was built and validated for text-only vLLM serving without specula
147
  - GPTQ-Pro tooling: [groxaxo/GPTQ-Pro](https://github.com/groxaxo/GPTQ-Pro)
148
  - Reference GPTQ-Pro recipe: [groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit](https://huggingface.co/groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit)
149
 
150
- ## Personal project notice
151
 
152
- This repository is a personal research project. The model, benchmarks, opinions, and documentation are my own and are not affiliated with, sponsored by, or endorsed by my employer or any organization I am associated with.
 
1
  ---
2
  base_model: Jackrong/Qwopus3.6-27B-v2
3
  library_name: transformers
4
+ pipeline_tag: text-generation
5
  tags:
6
  - qwen3.6
7
  - qwopus
 
11
  - vllm
12
  - int4
13
  - quantized
14
+ license: apache-2.0
15
  ---
16
 
17
  ![Qwopus3.6-27B-v2 GPTQ-Pro v1 RTX 3090 benchmark](assets/qwopus-gptq-pro-v1-rtx3090-benchmark.png)
18
 
 
19
  # Qwopus3.6-27B-v2 GPTQ-Pro v1
20
 
21
+ This is a GPTQ-Pro 4-bit quantization of `Jackrong/Qwopus3.6-27B-v2`, built to make this excellent Qwopus/Qwen3.6 model practical to run in vLLM with GPTQ-Marlin kernels and long-context inference.
22
 
23
+ The goal is simple: preserve as much of the original model's character and capability as possible while making it efficient enough for single-GPU RTX 3090-class vLLM deployments.
24
 
25
  This is not a new fine-tune. It is a quantized derivative of the original Qwopus3.6-27B-v2 model.
26
 
 
57
  | FOEM beta | `0.2` |
58
  | Batch size | `1` |
59
 
60
+ Preserved modules include vision, `lm_head`, embeddings, and norms.
61
+
62
+ Validation showed that this artifact preserves MTP-related configuration metadata, but does **not** include actual `mtp.*` tensors in `model.safetensors.index.json`, so this release should be treated as non-MTP for vLLM speculative decoding.
63
 
64
  Post-save compatibility patch:
65
 
 
68
 
69
  ## Intended serving setup
70
 
71
+ This checkpoint is intended for text-only vLLM serving on RTX 3090-class hardware.
72
 
73
  Recommended vLLM options:
74
 
 
94
 
95
  ## Reasoning / thinking mode
96
 
97
+ This model preserves Qwen3-style reasoning behavior. The validation workload below was run with thinking enabled.
98
 
99
  ## MTP / speculative decoding status
100
 
 
110
 
111
  ## RTX 3090 validation status
112
 
113
+ This checkpoint was validated on an RTX 3090 24GB with vLLM, `max_model_len=131072`, `kv_cache_dtype=fp8_e5m2`, prefix caching enabled, and thinking enabled.
114
 
115
+ Observed vLLM multi-turn agent workload metrics:
116
 
117
  | Metric | Observed value | Notes |
118
  |---|---:|---|
119
+ | Requests observed | `15` | Multi-turn agent session calls |
120
  | vLLM request success count | `15/15` | No vLLM errors observed during the sample |
121
+ | Average prompt size | `33,172` tokens | Real multi-turn workload |
122
+ | Average output size | `322` tokens | Real generated responses |
123
  | Average time to first token | `5.70s` | Prometheus TTFT summary |
124
  | Average end-to-end request latency | `13.07s` | Includes prefill, decode, and serving overhead |
125
  | Average time per output token | `0.0230s/token` | vLLM TPOT summary |
 
129
  | Live 60s generation throughput | about `19.1 generated tok/s` | Aggregate over full window, including prefill and idle mix |
130
  | Live 60s prefix-cache hit ratio | `78.9%` | Delta over the observed window |
131
 
132
+ These are practical multi-turn serving metrics, not a synthetic benchmark. They are useful for RTX 3090-class long-context serving expectations, especially multi-turn usage with prefix caching.
133
 
134
  ## Compatibility notes
135
 
 
140
  - Experimental quantization.
141
  - MTP/speculative decoding is not supported by this published artifact because `mtp.*` tensors are missing.
142
  - Quality has not yet been benchmarked against the BF16 source model.
143
+ - RTX 3090 metrics above are observed workload numbers, not a controlled benchmark suite.
144
  - Use at your own risk, especially for long-context or tool-calling workflows.
145
 
146
  ## References
 
149
  - GPTQ-Pro tooling: [groxaxo/GPTQ-Pro](https://github.com/groxaxo/GPTQ-Pro)
150
  - Reference GPTQ-Pro recipe: [groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit](https://huggingface.co/groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit)
151
 
152
+ ## Individual project notice
153
 
154
+ This repository is an individual research project. It is not affiliated with, sponsored by, or endorsed by any employer or organization.