I'm experimenting with BNB 4bit

quantize to FP8, ideally W8A16, take like 3 minutes to do

Here, see if this works for you
https://huggingface.co/hoborific/G4-MeroMero-v2-31B-W8A16-FP8

Hey,

It's hard to say without knowing what you're trying to use the model with, but there's a lot of quantization options out there. There should be one that fits your purposes, potentially like the one hoborific provided above.

Thank you, both really. I do hope earnestly you will create a V2 in MOE form. while i fall back tot he dense version when re-rolling doesn't help, MOE becomes very beneficial. I have a 2 million token epic adventure I am converting to a Dataset to fine tune the MeroMero model, but unsloth is very difficult on Dual r9700 pro ai 32GB gpus. I am trying to integrate a 16k token system prompt into the moe over three epochs. Honestly, I have been attempting to experiment so much, i branched many directions the project has me feeling overwhelmed, but collaborating with ChatGPT and Google's(free) AI assistant I am making very slow progress, a rate of learning i can retain. I'm not smart, but I am trying.

Unfortunately it wont work. I had strong hope.

• Failed (The current device_map had weights offloaded to the disk, which needed to be re-saved. This is either because the weights are not in safetensors format, or because the model uses an internal weight format different than the one saved (i.e. most MoE models). Please provide an
  offload_folder for them in from_pretrained.)

I can't use GGUF for what I want to do. .safetensors is what i need to fit.

Why / what are you trying to do that prevents you from using GGUF?

I'm trying to heretic the model, to use at a LM Studio server for generating a dataset using a python script, then merge them and train the single massive dataset over a few days. but when i try to use a non heretic model, it goes into loops or repetitively outputs gibberish. I don't know why, I have only been able to generate 1900 Rows of data using a heretic model, where i could only get a random number using a non-heretic version, sometimes i'll get 12 or even 20 rows, and then the following four hours is just gibberish. Honestly, I don't fully understand it if I'm using the suggested temperature, top_k and all. But I am using AI to help me understand. My script was also helped made with AI and i sort of understand a little more as I go along. I am not smart to be honest.

import json
import os
import random
from pydantic import BaseModel, Field
from openai import OpenAI
from tqdm import tqdm

1. Connect directly to your running LM Studio local server instance

client = OpenAI(base_url="http://localhost:1234/v1", api_key="lm-studio")

2. Strict Object Schemas to enforce your explicit game-world training structure

class ChatMessage(BaseModel):
role: str = Field(description="Must strictly be 'user' or 'assistant'")
content: str = Field(description="The actual dialogue or internal thought block text matching the game rules")

class DatasetRow(BaseModel):
messages: list[ChatMessage] = Field(description="Array containing system, user, or assistant interactions")

--- MULTI-PROCESS CONFIGURATION ---

INPUT_FILE = "dataset.jsonl"

print("=========================================================")
print(" LM STUDIO MULTI-PROCESS DATASET GENERATOR ")
print("=========================================================")
print("To run 8 copies safely, assign a unique ID to each window.")
process_id = input("Enter a unique Process ID for this terminal (e.g., 1, 2, 3...): ").strip()

Splitting 5,000 total rows evenly across your 8 parallel instances (~625 rows each)

TARGET_ROWS_PER_PROCESS = 625
OUTPUT_FILE = f"dataset_expanded_part_{process_id}.jsonl"

------------------------------------

def load_seed_data(filepath):
"""Loads your original 1,900 human-verified rows cleanly into memory."""
rows = []
if os.path.exists(filepath):
with open(filepath, "r", encoding="utf-8") as f:
for line in f:
if line.strip():
try:
rows.append(json.loads(line.strip()))
except json.JSONDecodeError:
continue
return rows

def generate_new_row(seed_examples):
"""Pings the 70B/72B flagship model with source contexts to build unique scenarios."""
# Pull 2 completely random rows from your original seed data to serve as contextual baseline
samples = random.sample(seed_examples, min(2, len(seed_examples)))
samples_str = json.dumps(samples, indent=2)

prompt = f"""You are a data synthesis engine building an expansion dataset for LLM fine-tuning.

Analyze these real training data examples from the game world:

{samples_str}

Generate ONE brand new, highly realistic training row.
Requirements:

1. Come up with a completely different user scenario, item, request, location, or obstacle in the game world.

2. The assistant's content MUST begin with a hidden '\n...\n' tag evaluating rules/inventory before replying out loud.

3. Mimic the exact vocabulary, formatting style, and setting constraints of the examples.
   """

   try:
   # Request a JSON-enforced structure from LM Studio
   response = client.beta.chat.completions.parse(
   model="local-model", # LM Studio defaults directly to your active 70B/72B model
   messages=[
   {"role": "system", "content": "You are a synthetic data engine that outputs strict JSON formats."},
   {"role": "user", "content": prompt}
   ],
   response_format=DatasetRow, # Structured outputs enforcement via Pydantic
   temperature=0.90, # High creativity setting to ensure unique outputs over long loops
   top_p=0.95 # Prevents erratic or corrupted word choices
   )
   
   # FIX: Safely unpack the choices whether it arrives as a list or a single object attribute
   if isinstance(response.choices, list):
   parsed_data = response.choices[0].message.parsed
   else:
   parsed_data = response.choices[0].message.parsed
   
   return parsed_data.model_dump()
   except Exception as e:
   print(f"\n[Process {process_id}] Skipping row due to processing error: {e}")
   return None

def main():
seed_data = load_seed_data(INPUT_FILE)
if not seed_data:
print(f"Error: Could not find or parse seed data in '{INPUT_FILE}'. check your file path.")
return

print(f"\nSuccessfully loaded {len(seed_data)} base source examples.")

# Calculate current progress in case you need to restart a process mid-night
existing_output = load_seed_data(OUTPUT_FILE)
current_count = len(existing_output)

if current_count == 0:
    # Seed this individual scratchpad file with a few starting entries
    with open(OUTPUT_FILE, "w", encoding="utf-8") as f:
        # Grab a random subset of original data so files don't start identically
        initial_seeds = random.sample(seed_data, min(5, len(seed_data)))
        for row in initial_seeds:
            f.write(json.dumps(row) + "\n")
    current_count = len(initial_seeds)
    print(f"Initialized '{OUTPUT_FILE}'. Starting generation loop...")
else:
    print(f"Resuming generation. '{OUTPUT_FILE}' already contains {current_count} rows.")

# Calculate exact remaining generation targets for this window
needed_rows = TARGET_ROWS_PER_PROCESS - current_count
if needed_rows <= 0:
    print(f"Target of {TARGET_ROWS_PER_PROCESS} rows already achieved for Process {process_id}!")
    return

# 4. Main Automation Loop
# Using unbuffered appends (flush=True) so data is preserved even if a process is terminated
with open(OUTPUT_FILE, "a", encoding="utf-8") as f:
    for _ in tqdm(range(needed_rows), desc=f"Process {process_id} Progress"):
        new_row = generate_new_row(seed_data)
        if new_row:
            f.write(json.dumps(new_row) + "\n")
            f.flush() # Forces writing directly to storage immediately

print(f"\nSuccess! Process {process_id} complete. Data saved to '{OUTPUT_FILE}'.")

if name == "main":
main()