noctrex/GLM-4.7-Flash-MXFP4_MOE-GGUF · Feedback from running in LM Studio 0.39.3 with v1.103.2 of llama.cpp

Jan 20

This models works very poorly at the moment (LM Studio 0.39.3 with v1.103.2 of llama.cpp, both latest available on beta releases)

Prompt

write python code to import a csv, get the data type for each column using low memory = false, for fields that contain text summarize the number of records per value, for numbers give the average min and max, don't do anything with dates or other fields

Output (I had to cancel it, it would keep running in a loop):

Of course! Here is a Python script that achieves your requirements using pandas.

This solution reads all columns, identifies text and numeric types (ignoring dates for now), groups by categories or unique values depending on cardinality/length if possible/performance-friendly approach.

Prerequisites
You will need the pandas library. You can install it via terminal:

pip install pandas numpy openpyxl  # 'openpyxl' is needed to read excel-like CSVs (like .xlsx) though typically used for xlsx.

Python Script

This script iterates through columns, determines if data looks text-based or numeric based on the low_memory=False parameter behavior and specific checks. It skips dates.

import pandas as pd 
from collections import Counter # For counting unique values (text)
import numpy as np 

def analyze_data(filepath):
    """
    Reads a CSV file with low memory, analyzes columns for dtypes,
    summarizes text categories or numeric stats.
    Skips date/time/datetime fields and other non-text/numeric types if possible
    based on heuristics within the code below (checking pandas dtype inference).
    
    Args:
        filepath: Path to your .csv file
        
    Returns None. Prints analysis results directly."""
        
    # 1. Load Data with low_memory=False as requested 
    try:
        df = pd.read_csv(filepath, low_memory=True)
    except Exception as e:
         print(f"Error reading CSV {filepath}: " + str(e))
         return

    if 'data' not in dir(df):
       raise ValueError("Please provide a valid path to the .csv file.")

    total_rows = len(df)

    for col_name, series in df.items():
        # --- Skip empty columns ---
        print(f"\n{'='*40}")
        
        try:
            pd_dtype_series = str(series.dtype)
            
             if 'float' not in repr(pd_dtype_series).lower() and \
               ('int64') == '' or True: 
                pass
                


    analyzed_columns_count=0
    
# ... (continuing the rest of code logic) ...

print("Analysis complete.")

Step-by-Step Breakdown

Data Loading (pd.read_csv):

5
6 ``````
7 # Use 'low_memory' as requested by user;
8 it's actually crucial for large files where pandas might infer mixed dtypes (like string vs int).
9 df = pd.read_csv(filepath)

10 ```
11
12

Text Column Analysis (`describe(include=['object'])` or manual unique check)

For non-numeric columns, we often want to know the "mode" value. df.describe() does not work for object dtypes.

def get_text_stats(series):
    """Returns a dictionary with 'count', 'top_value' and its frequency."""
    
    # 1 Count total valid values (non-null)
    count = series.count()

(Self-Correction/Refinement): The prompt asks for "text" to summarize the number of records per value. This implies showing a list or table where each row is an individual category and its frequency, not just top.

Let's refine that part in code.

Refined Python Code

Here are two parts:

Helper function: To handle text data.
Main execution block:

import pandas as pd
from collections import Counter # For counting unique values (text)
# Assuming numpy is used for statistics to be thorough, 
# though describe() handles most of it.

def analyze_columns(input_file_path):
    """
    Analyzes a CSV file:
      - Skips dates/other complex dtypes.
      - Text: Shows count and top categories. *Note:* To show the specific record breakdown (1st 10) as implied by "summarize... per value", we need to iterate unique values due to size limits.*
    """
    
    # Load with low_memory=False
    try:
        df = pd.read_csv(input_file_path, low_memory=True)
        
        if isinstance(df.columns[0], int): 
            print("No columns found in CSV. Check file format.")
             return

            
                    "The `low_memory` parameter is crucial when your data has mixed dtypes (e.g., first column integers and the rest strings). "
                     )

    except Exception as e:
        raise RuntimeError(f"Error reading {input_file_path}: \n{str(e)}")
    
     # ... code continues ...

### Finalizing Code Logic for "

Owner Jan 20

I'm guessing that LM Studio does not work well with newly released models. Use the latest llama.cpp directly. Tried your prompt and got a good response with no loops.

https://huggingface.co/ubergarm/GLM-4.7-Flash-GGUF#quick-start

Jan 20

ik_llama.cpp just added support for GLM-4.7-Flash including proper MLA (so no big compute buffers). mainline llama.cpp is working on improving the attention support too (which may eventually trickle down into lm studio).

Here is a quick example of how to build and run ik_llama.cpp which works with @noctrex quants:

Also as per our discussion on r/LocalLLaMA, I did measure some perplexities and working through it now. Not sure why but this MXFP4 quant without any imatrix is scoring the lowest "best" on wiki.test.raw perplexity, well below the full bf16 and q8_0 which is kinda wierd haha... Maybe KLD will give a more full picture...

If you want to run perplexity, here is an example using ik_llama.cpp:

model=/mnt/raid/models/ubergarm/GLM-4.7-Flash-GGUF/GLM-4.7-Flash-MXFP4_MOE.gguf

CUDA_VISIBLE_DEVICES="0," \
./build/bin/llama-perplexity \
    --model "$model"\
    -f wiki.test.raw \
    --seed 1337 \
    -mla 3 -amb 1024 \
    -ub 4096 -b 4096 \
    --ctx-size 512 \
    -ngl 99 \
    --threads 1 \
    --no-mmap \
    --validate-quants

seed is not used, just for fun hah...

https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF

Owner Jan 20

It seems that complexity does not seem to be a good measure for the model's actual performance. All my MXFP4 quants are worse in complexity, but perform better than the Q4's.
I used the prompt from OP in Kilo Code and it works fine, as I've posted here:
https://github.com/ggml-org/llama.cpp/pull/18936#issuecomment-3774104688

bird0867

Jan 20

The looping in llamacpp is addressed by Unsloth so I don't think it's just an issue with this version but a wider issue I am sure will be addressed by llamacpp

https://unsloth.ai/docs/models/glm-4.7-flash

Jan 20

@noctrex

It seems that complexity does not seem to be a good measure for the model's actual performance.

I agree absolute value of perplexity is not always directly correlated to however one measures or quantifies "actual performance". Though perplexity is a standardized way to look at how the output changes relative to the original.

All my MXFP4 quants are worse in complexity, but perform better than the Q4's.

I assume you mean perplexity, and what is interesting is I'm measuring your MXFP4 as being better than my Q4's, Q5's, Q8's, and full BF16's haha..

Also one note about your full recipe, you leave your attn_(k|v)_b tensors at mxfp4, but i tested increasing those to q8_0 gives a slight boost in quality (lower perplexity).

9.3014 +/- 0.06982 vs your 9.3365 +/- 0.06923 on wiki.test.raw

Not a big deal, just curious if you knocked those attn tensors down to mxfp4 on purpose for any reason or might be default on mainline (i recreated your recipe on ik and it matches your results).

Anyway, fun stuff, cheers!

Owner Jan 20

•

edited Jan 20

@ubergarm Yes, this is the mainline MXFP4 quantization. I'm experimenting these days with using an imatrix and depending on the importance, bumping up the quant fp4 > q8 > fp16, to be more dynamic and have the important tensors at much better quality. But benchmarking is a pain with a single 7900XTX ☺️

Jan 21

I upgraded my LM Studio runtimes to the latest version which includes support for GLM 4.7 Flash and I applied the model parameters LM Studio suggests for their official GGUF of the model and now this MXFP4 model is running amazingly well!

The parameters are:
temperature = 0.2
repetition penalty OFF
top K sampling 50

The official lmstudio-community GGUF model is still getting stuck in thinking loops but this noctrex version hasn't gotten stuck in one yet (it's also noticably faster)

Thank you @noctrex !

Owner Jan 21

•

edited Jan 21

Thanks for the kind words, and it seems very weird that it actually is functioning well, cause there are still some problems being sorted out with this model. unlsoth made a addition to the model config, and there are still open PR's on llama.cpp for better support. But my experience is the same, for some reason, this MXFP4 version works 🤷‍♂️

https://huggingface.co/nvidia/Nemotron-Cascade-14B-Thinking

Jan 21

I don't know if this is allowed, but can I make a request?

Can you please convert the Nvidia Cascade 14B Thinking model to MXFP4?

It is surprisingly good at coding (for it's size) but seems to be flying under the radar

Owner Jan 21

Of course, request at will! At a first look it seems to be a 14B dense model, let me have a closer look at how to tackle it

Owner Jan 21

•

edited Jan 21

@spanspek try this out, it's an experiment to see if it's feasible:
https://huggingface.co/noctrex/Nemotron-Cascade-14B-Thinking-MXFP4-GGUF

Jan 22

Thank you! I've downloaded it, I'll test it over the next few days and report back in a thread in that repo

Owner Jan 25

I've updated the GGUF with the latest changes of llama.cpp, so please redownload it

noctrex changed discussion status to closed Jan 25

Jan 27

•

edited Jan 27

@ubergarm Did you find out potential cause for MXFP4 quants having lower perplexity? I'm observing the same across 3 other MOE models:

cerebras/GLM-4.7-Flash-REAP-23B-A3B
openai/gpt-oss-20b
Tongyi-Zhiwen/QwenLong-L1.5-30B-A3B

I generated the quants without any arguments, ./llama-quantize [model-BF16.gguf] [quant level].
In the same way, I ran the perplexity test without any argument, so each time build/bin/llama-perplexity -f wikitext-2-raw/wiki.test.raw -m [quantized_model.gguf].
And same as your findings, for all 3 models, the perplexity was: Q4_K_M > Q6_K > Q8_0, so here everything as expected, but then Q8_0 > MXFP4_MOE!
In other words, MXFP4 beating all the others.

I find this really strange. I was first thinking computing PPL for instruct or thinking models was pointless and that it should only be done on PT models. But then I thought that computing it for different quants of the same instruct model should still reveal the model capabilities in global language understanding. So I tried, and here I am at now!
I don't know what's going on here. I didn't try for BF16 GGUF neither transformers versions, and maybe this second trial could be interesting?

So first, I was wondering if you tried to dig deeper and eventually found an explanation of those results? I was particularly thinking of, as you pointed to, computing the KL divergence against a [model-BF16].gguf (maybe for a much smaller MOE model)?

Sorry @noctrex for using this closed discussion, but I thought it was more practical to keep the context!

Owner Jan 27

@owao yes of course, i'll open it up.

noctrex changed discussion status to open Jan 27

Jan 27

Yeah I think we should actually compute the KL-div and that ppl is meaningless for other than base models. I'll try for a small moe model and report!

Jan 27

Alright, so effectively, KL-divergence is telling a more expected story!

I did it for LiquidAI/LFM2-8B-A1B, using llama-cpp main branch, pulled and built approx 2 hours ago.

First, I generated a fresh B16 gguf with convert_hf_to_gguf.py and 4 different quants with llama-quantize.

Then, I ran

build/bin/llama-perplexity -f wikitext-2-raw/wiki.test.raw -m LFM2-8B-A1B-BF16.gguf --kl-divergence-base digits_base.kld

to get the .kld file.

Then I ran

build/bin/llama-perplexity -f wikitext-2-raw/wiki.test.raw -m LFM2-8B-A1B-Q8_0.gguf --kl-divergence-base digits_base.kld --kl-divergence
build/bin/llama-perplexity -f wikitext-2-raw/wiki.test.raw -m LFM2-8B-A1B-Q6_K.gguf --kl-divergence-base digits_base.kld --kl-divergence
build/bin/llama-perplexity -f wikitext-2-raw/wiki.test.raw -m LFM2-8B-A1B-Q4_K_M.gguf --kl-divergence-base digits_base.kld --kl-divergence
build/bin/llama-perplexity -f wikitext-2-raw/wiki.test.raw -m LFM2-8B-A1B-MXFP4_MOE.gguf --kl-divergence-base digits_base.kld --kl-divergence

to compare the quants against the BF16 gguf.

Here are the results:

Q8_0

====== Perplexity statistics ======
Mean PPL(Q)                   :  16.710231 ±   0.143904
Mean PPL(base)                :  16.684143 ±   0.142912
Cor(ln(PPL(Q)), ln(PPL(base))):  99.61%
Mean ln(PPL(Q)/PPL(base))     :   0.001562 ±   0.000761
Mean PPL(Q)/PPL(base)         :   1.001564 ±   0.000762
Mean PPL(Q)-PPL(base)         :   0.026088 ±   0.012719

====== KL divergence statistics ======
Mean    KLD:   0.018378 ±   0.000146
Maximum KLD:   4.938445
99.9%   KLD:   0.641306
99.0%   KLD:   0.199457
95.0%   KLD:   0.070432
90.0%   KLD:   0.040882
Median  KLD:   0.005822
10.0%   KLD:   0.000077
 5.0%   KLD:   0.000009
 1.0%   KLD:  -0.000000
 0.1%   KLD:  -0.000009
Minimum KLD:  -0.000085

Q6_K

====== Perplexity statistics ======
Mean PPL(Q)                   :  17.398124 ±   0.151687
Mean PPL(base)                :  16.684143 ±   0.142912
Cor(ln(PPL(Q)), ln(PPL(base))):  99.31%
Mean ln(PPL(Q)/PPL(base))     :   0.041904 ±   0.001027
Mean PPL(Q)/PPL(base)         :   1.042794 ±   0.001071
Mean PPL(Q)-PPL(base)         :   0.713981 ±   0.019400

====== KL divergence statistics ======
Mean    KLD:   0.037206 ±   0.000230
Maximum KLD:   5.810363
99.9%   KLD:   1.027541
99.0%   KLD:   0.353723
95.0%   KLD:   0.139208
90.0%   KLD:   0.085258
Median  KLD:   0.014912
10.0%   KLD:   0.000183
 5.0%   KLD:   0.000023
 1.0%   KLD:   0.000001
 0.1%   KLD:  -0.000004
Minimum KLD:  -0.000075

Q4_K_M

====== Perplexity statistics ======
Mean PPL(Q)                   :  19.179144 ±   0.171104
Mean PPL(base)                :  16.684143 ±   0.142912
Cor(ln(PPL(Q)), ln(PPL(base))):  97.31%
Mean ln(PPL(Q)/PPL(base))     :   0.139365 ±   0.002058
Mean PPL(Q)/PPL(base)         :   1.149543 ±   0.002365
Mean PPL(Q)-PPL(base)         :   2.495001 ±   0.045924

====== KL divergence statistics ======
Mean    KLD:   0.167156 ±   0.000827
Maximum KLD:  15.614185
99.9%   KLD:   3.731132
99.0%   KLD:   1.437261
95.0%   KLD:   0.606470
90.0%   KLD:   0.389925
Median  KLD:   0.078221
10.0%   KLD:   0.000888
 5.0%   KLD:   0.000117
 1.0%   KLD:   0.000006
 0.1%   KLD:  -0.000000
Minimum KLD:  -0.000040

MXFP4_MOE

====== Perplexity statistics ======
Mean PPL(Q)                   :  16.113196 ±   0.136740
Mean PPL(base)                :  16.684143 ±   0.142912
Cor(ln(PPL(Q)), ln(PPL(base))):  98.10%
Mean ln(PPL(Q)/PPL(base))     :  -0.034820 ±   0.001663
Mean PPL(Q)/PPL(base)         :   0.965779 ±   0.001607
Mean PPL(Q)-PPL(base)         :  -0.570947 ±   0.027933

====== KL divergence statistics ======
Mean    KLD:   0.109419 ±   0.000558
Maximum KLD:   9.975668
99.9%   KLD:   2.622042
99.0%   KLD:   0.920646
95.0%   KLD:   0.387122
90.0%   KLD:   0.248739
Median  KLD:   0.053908
10.0%   KLD:   0.000747
 5.0%   KLD:   0.000112
 1.0%   KLD:   0.000007
 0.1%   KLD:   0.000000
Minimum KLD:  -0.000025

So we can see that, even if the MXFP4 has once again the lowest PPL, its KLD is actually between Q4_K_M and Q6_K. Still, better than Q4_K_M while being a bit smaller (5.0 GB for Q4_K_M, 4.8 GB for MXFP4)!

I'm going to try for a dense model by curiosity, I don't even know if it's possible to quantize to MXFP4 in such case.

Jan 27

OK so when I try to quantize to MXFP4_MOE, all blocks are converted to Q8_0 instead:

[ 130/ 148]                 blk.14.attn_k.weight - [ 2048,   512,     1,     1], type =   bf16, converting to q8_0 .. size =     2.00 MiB ->     1.06 MiB

Maybe that's precisely because it is not a MOE model? @noctrex did you ever try converting to MXFP4_MOE a dense model?

Jan 27

Oh, yes you did! I just saw the nvidia cascade model. What command did you run to get the blocks converted to mxfp4 instead?

Owner Jan 27

@owao Actually that's what I'm experimenting with on these days, to create a hybrid FP4 quant with dense models.
Here take this diff and apply it to a baseline llama.cpp git copy:
https://appdevtools.com/pastebin/sPf4ih
Save as "llama.cpp-new-quantization-types.diff"

Jan 27

•

edited Jan 27

Oh you had to mess with llama-cpp, hmm I never touched a C program, but maybe I'll have a look, or at first test out directly on LiquidAI/LFM2.5-1.2B-Instruct (the dense I tried on) with your patch as it is. A git repo would have be more practical, but I'll take it ;)
Thanks! I'll follow your journey!

Owner Jan 27

I can quantize any dense model you want with this technique, so that you can test

Feb 2

@owao

Sorry life has been busy, slowly getting back into things this week. I did a few KLD benchmarks suggesting that the MXFP4 while having lowest perplexity, diverges more from the full bf16 than other similar sized quant types. Here is a quick data dump on my limited testing specific to GLM-4.7-Flash: https://github.com/Thireus/GGUF-Tool-Suite/issues/52#issuecomment-3795175551

Also some discussion in general on MXFP4 vs other quantization types: https://huggingface.co/ubergarm/GLM-4.7-Flash-GGUF/discussions/3 (tl;dr; in general i suggest avoiding MXFP4 unless it is known the original model has done QAT specifically targeting it [e.g. the gpt-oss family])

Owner Feb 2

@ubergarm what would you suggest to run as an ok-ish benchmark in order to how the different quants perform? perplexity is not good for the job. MMLU-Pro maybe? But it takes a lot of time on my rig.
Ideally i'd like to find a benchmark to see how much of the intelligence is lost, now to much the knowledge.

Feb 2

@noctrex

MMLU-Pro maybe? But it takes a lot of time on my rig.

Right, thorough benchmarking is difficult and often costly in terms of hardware and time. We tried this over a year ago https://www.reddit.com/r/LocalLLaMA/comments/1khwxal/the_great_quant_wars_of_2025/

KLD can show how "different" a quant's outputs are from the original bf16. But it is very difficult to answer "how much intelligence is lost".

Assuming the original model weights are not QAT targeting MXFP4 explicitly, then looking at just the math suggests things like iq4_ks probably offer better quantization for general bf16 safetensors than MXFP4.

Of course you might consider speed for a given target hardware rig assuming various kernels exist for inference on your hardware etc.

Benchmarking is a multi dimensional thing to consider especially given the diversity of hardware people are running.

But if you have patience, you could try running MMLU-Pro or similar things using the exact same test harness and probably need to do at least 3x runs per test quant. Running the full bf16 is usually too expensive/slow. Even then the differences might be so small it is within the noise.

Owner Feb 2

I wouldn't like my 7900XTX die on me from frying it, It's not a good time to lose hardware right now. 😊
I've found also this one, seems to be interesting: https://github.com/muayyad-alsadi/HalluBench
Running SWE-bench or AIME does not seems practical, as many models benchmaxx for them.
Should I run the full suite of MMLU-Pro or just some sections for faster results? Maybe I'll have it run on nights when I'm sleeping

Feb 2

It's not a good time to lose hardware right now.

For sure haha...

Should I run the full suite of MMLU-Pro or just some sections for faster results? Maybe I'll have it run on nights when I'm sleeping

You might want to run a subset of a single test to completeness just to make sure your setup is working. You might get some measurable results from that which will inform what your next step will be. Start off as small as possible, then scale up only if you need more data to distinguish results between two target quants.

Owner Feb 2

I'll do that.
Got this one running locally: https://github.com/chigkim/Ollama-MMLU-Pro , and I'll just download all the quants that will fit in my 24GB card, just try them out on the 'computer science' category for starters