Hugging Face's logo

JohnTdi
/
Qwen3-14B-INT4-Mixed-GPTQ

Text Generation
English
Chinese
custom
int4
gptq
quantized
amd
rdna4
rocm
Model card Files
xet
 Community
JohnTdi commited on
Commit
154a2c2
·
verified ·
1 Parent(s): 9ca0c3e

Upload folder using huggingface_hub

Browse files
Files changed (45) hide show
  1. README.md +87 -0
  2. embed.pt +3 -0
  3. final_norm.pt +3 -0
  4. layer_000.pt +3 -0
  5. layer_001.pt +3 -0
  6. layer_002.pt +3 -0
  7. layer_003.pt +3 -0
  8. layer_004.pt +3 -0
  9. layer_005.pt +3 -0
  10. layer_006.pt +3 -0
  11. layer_007.pt +3 -0
  12. layer_008.pt +3 -0
  13. layer_009.pt +3 -0
  14. layer_010.pt +3 -0
  15. layer_011.pt +3 -0
  16. layer_012.pt +3 -0
  17. layer_013.pt +3 -0
  18. layer_014.pt +3 -0
  19. layer_015.pt +3 -0
  20. layer_016.pt +3 -0
  21. layer_017.pt +3 -0
  22. layer_018.pt +3 -0
  23. layer_019.pt +3 -0
  24. layer_020.pt +3 -0
  25. layer_021.pt +3 -0
  26. layer_022.pt +3 -0
  27. layer_023.pt +3 -0
  28. layer_024.pt +3 -0
  29. layer_025.pt +3 -0
  30. layer_026.pt +3 -0
  31. layer_027.pt +3 -0
  32. layer_028.pt +3 -0
  33. layer_029.pt +3 -0
  34. layer_030.pt +3 -0
  35. layer_031.pt +3 -0
  36. layer_032.pt +3 -0
  37. layer_033.pt +3 -0
  38. layer_034.pt +3 -0
  39. layer_035.pt +3 -0
  40. layer_036.pt +3 -0
  41. layer_037.pt +3 -0
  42. layer_038.pt +3 -0
  43. layer_039.pt +3 -0
  44. lm_head.pt +3 -0
  45. meta.pt +3 -0
README.md ADDED
@@ -0,0 +1,87 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ language:
4
+ - en
5
+ - zh
6
+ base_model: Qwen/Qwen3-14B
7
+ tags:
8
+ - int4
9
+ - gptq
10
+ - quantized
11
+ - amd
12
+ - rdna4
13
+ - rocm
14
+ library_name: custom
15
+ pipeline_tag: text-generation
16
+ ---
17
+
18
+ # Qwen3-14B INT4 Mixed (GPTQ + Hadamard)
19
+
20
+ Mixed INT4/INT8 quantization of [Qwen/Qwen3-14B](https://huggingface.co/Qwen/Qwen3-14B) using our custom quantization pipeline.
21
+
22
+ **Best quality** — sensitive layers kept at INT8 based on Hessian-weighted sensitivity analysis.
23
+
24
+ ## Quality
25
+
26
+ | Metric | This Model | FP16 Reference | llama.cpp Q4_K_M |
27
+ |---|---|---|---|
28
+ | **Perplexity** (WikiText-2) | **7.692** | ~7.5 | 7.715 |
29
+ | **ARC-Challenge** (250) | **92.8%** | ~94% | 90.8% |
30
+ | **MMLU** (250, 14 subjects) | **75.6%** | ~78% | 72.8% |
31
+
32
+ ## Performance (AMD Radeon AI PRO R9700)
33
+
34
+ | Metric | Speed |
35
+ |---|---|
36
+ | Decode (ctx=128) | **61 t/s** |
37
+ | Prefill (pp512) | **2076 t/s** |
38
+ | VRAM | **9.9 GB** |
39
+
40
+ ## Quantization Details
41
+
42
+ - **Method**: INT4 asymmetric with Hadamard rotation + GPTQ calibration
43
+ - **Sensitive layers**: INT8 (top 20% by Hessian-weighted error)
44
+ - **Block size**: 32
45
+ - **Calibration**: 64 samples × 256 tokens from WikiText-2
46
+ - **KV cache**: FP8 E4M3 at inference time
47
+
48
+ ## File Format
49
+
50
+ Custom `.pt` format — requires [rdna4-quant](https://github.com/JohnTDI-cpu/rdna4-quant) engine.
51
+
52
+ ```
53
+ embed.pt — embedding weights (FP16)
54
+ layer_000.pt — layer 0 (quantized weights + scales + metadata)
55
+ ...
56
+ layer_039.pt — layer 39
57
+ final_norm.pt — final RMSNorm
58
+ lm_head.pt — LM head (FP16)
59
+ meta.pt — quantization metadata
60
+ ```
61
+
62
+ ## Usage
63
+
64
+ ```bash
65
+ git clone https://github.com/JohnTDI-cpu/rdna4-quant
66
+ cd rdna4-quant
67
+ pip install -r requirements.txt
68
+
69
+ # Download weights
70
+ huggingface-cli download JohnTdi/Qwen3-14B-INT4-Mixed-GPTQ --local-dir quantized_v4_gptq
71
+
72
+ # Run inference
73
+ python int4_engine_v5.py --quant-dir quantized_v4_gptq --chat
74
+
75
+ # Or start API server
76
+ python api_server.py --quant-dir quantized_v4_gptq
77
+ ```
78
+
79
+ ## Hardware Requirements
80
+
81
+ - AMD GPU with ROCm 6.x+ support (RDNA3/4, MI300X)
82
+ - ~10 GB VRAM
83
+ - ROCm 6.x or 7.x, PyTorch 2.6+
84
+
85
+ ## License
86
+
87
+ MIT — same as the inference engine. Base model (Qwen3-14B) has its own license from Alibaba.
embed.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:608e029694531b6530100a5638b704bd8dbde0c6149a21766f736115fd0106f8
3
+ size 1555826139
final_norm.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:525d38e9c8a70da928ae83a1952ddba3d3e04b1fd1990f405d4889d5601a9ae1
3
+ size 11838
layer_000.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:155b1a1a7a71df1c590f6d9a1cd4ff03417b375ae195378b7f25961cd17d0ca4
3
+ size 196143327
layer_001.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:092aa6a66da38154b06374527940bc9296573adf45d1be26bce1dfd6248c2610
3
+ size 196143327
layer_002.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e3fc1568d1d1d3555c718145ce0d88df3897b22f0d202ff72d7e1a91e1051c5c
3
+ size 196143327
layer_003.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d5363db09568d705b05e279b5da4e6b929d5fe87400f15ee60f81205b46de39a
3
+ size 196143327
layer_004.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:deb8e06dc05bdb5c4310b5f1ab6f722ffb8beb965191418675a33077ed49c568
3
+ size 279701472
layer_005.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0b9cccab76b5ac47a4b29dc3eb23490efea385b249fd77be67c9c6ad28e5c79d
3
+ size 279701472
layer_006.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:173622a376272246eaa5def09d039e5baa0311f319000cb3a94c8c92bf4b23cd
3
+ size 321480417
layer_007.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4a5685d85092c565bb9c37f7821096c41e91f866daec027508cb40bbfd6bac7b
3
+ size 279701472
layer_008.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:843328ae20bec44c0be55fbc2642eaf4a8d2b8269c258945d0c7df5723ccaa74
3
+ size 196143327
layer_009.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:078adf215fb6ca12fcc3147247e1f5e4b23136cabb71ce3c13d8b4d0bca721b4
3
+ size 196143327
layer_010.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8d9102233b8010a5e63000868627e3734431b81093658d35ef54e3235e84d60e
3
+ size 196143327
layer_011.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:43625735be165272c69fcbb2b9de6770dfa495832332da940cb795a438d779e2
3
+ size 196143327
layer_012.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b0b44dbbe6d75301dd4ef8706ec69c58db4d68dfc27177b9aeb5d85e97d313c1
3
+ size 196143327
layer_013.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:267f4b4883c985c0ff4b41ad09dfde684f0439af6defd251973cc2a545478a5b
3
+ size 196143327
layer_014.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3eae39504c96c12420a97f0b7a28a5ef8736fc6a3047979f5da83125d46b8b32
3
+ size 196143327
layer_015.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:85460822e2a7cde749b47295197afa052c73e023628b26b53aa6f19a051aa76d
3
+ size 196143327
layer_016.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d9c76e368ae633f1fa85e18b5cfac2eaf1b401a520a9ba11e48c83b895958eee
3
+ size 196143327
layer_017.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d3156998480767b25a5471d4b541546653168677729c72cdf309d40d742ee76f
3
+ size 196143327
layer_018.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c63c7dc0c56caddfbfcc4f58fdef90809e1c8d784309cb9f6e775187ed1bafc0
3
+ size 196143327
layer_019.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d1315152be4f2a2c180ca5886763ccdd0b65722f9562d78fd11541b4bcbf750a
3
+ size 196143327
layer_020.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:22d40c8b711f51923d8b6b5bc86ba01ecf76ec3bc47ae8ede12ba54dad1dc9bb
3
+ size 196143327
layer_021.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b501142b41c234bc8f53167461ca7bd1e2f09a45678e844db5ef1cfaf5c7e9e1
3
+ size 196143327
layer_022.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7659eab9ea47d8ef2207cacfcc86bac6e049765bfe075b161e9bb49387ea6734
3
+ size 196143327
layer_023.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f223e80cf1164f6abc2cc6bb472beb43b6cdba4d931221ff6e87cc2cdbe30861
3
+ size 196143327
layer_024.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f63b4ae46454676184e2d0cb3c43c051286b891cb0e33bc393d6a0825788bb8c
3
+ size 196143327
layer_025.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0e93804a3b9ee81add2c83367a911da4d990ab2c50afca82f154fe79dffc2857
3
+ size 196143327
layer_026.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b71d9eb09743c0755c2341c2e0b0fba92ca2ea96dc8fe21c94b898837780dd11
3
+ size 196143327
layer_027.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2480dc807cde8fdd7526b2bf8261bfcaf1be0126769d5a727335264358d449fc
3
+ size 196143327
layer_028.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:92a6136f9b1685a6b97e1488513d87d797e05911f846fcc418ae4e745c00ccff
3
+ size 196143327
layer_029.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2ae630daa77a10bfc151d56a675489893ae2b66fb11cad949b950c287c2d0577
3
+ size 196143327
layer_030.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6871a2b8488133da20856f268deb1a403517a2085bbf7e01e5ff31bbed7f8e0c
3
+ size 196143327
layer_031.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:81760e1cdcb3bd72cafb9a2a1c201b113e8da477af3a5c0c7ab85ad5c3671567
3
+ size 296904417
layer_032.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cba77e9da0041b739770fe2f816c72b420bd2f357bf8c071184fb732c0ac3952
3
+ size 296904417
layer_033.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c540154cf2b0e094dcaa6cdac9cd44ba6626d68dc54904b7fdda3399af327c5c
3
+ size 338683362
layer_034.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9362728a4bc95fc4d13777166c0c2cb8bc127d11469bace7e1aae0e9c0550a01
3
+ size 338683362
layer_035.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ba4b6a6ed6a29d726ff03c1748c36fa2dfc055448d3b17cf7e1fab9011575cac
3
+ size 338683362
layer_036.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f19a2b76220c00c16880dc87f8ed9c3dcf16be7a7a44de2b06acc497d44ac2d8
3
+ size 338683362
layer_037.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:763a20388003b41ac85fcc87e21cf9ba1021e9e3298164c0bcd7d5f9b0974695
3
+ size 338683362
layer_038.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3f81c0224252b1cbeed3a200b41dae7f7ce5817ae79119fcf3c5c44c3a0457d3
3
+ size 350971043
layer_039.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0fe62ff546e9e441dba400bd50205dd71fd8f3b1f9973bc9bf93e127da8ba3be
3
+ size 350971043
lm_head.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2e445da677f636a85baa6ad22cc6d8639675a0366c026fdb05443b52b68bd452
3
+ size 437577573
meta.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0846de84c9486df01de1325b6664391a256706adf3a679b814c538524f26bcb8
3
+ size 2907