Instructions to use Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned")
model = AutoModelForCausalLM.from_pretrained("Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned

SGLang

How to use Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned with Docker Model Runner:
```
docker model run hf.co/Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned
```

MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned

Quantization-aware (QAT) unlearning on the MUSE Books corpus.

Started from the already-unlearned model and fine-tuned with asymmetric per-group-128 INT4 fake-quant (STE).

The saved weights are full-precision (bf16); the fake-quant makes them robust to real 4-bit quantization at deployment. Motivation: standard NPO unlearning is reversed by 4-bit quantization (forgotten content re-surfaces); QAT aims to keep the model unlearned after quantization.

Training

Loss: NPO + gradient-difference retain (npo_gdr), beta=0.1, retain lambda=10
Source model: Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1
lr=5e-6, 1 epoch, max_len=512, per-device batch size 4
Fake-quant: asymmetric affine, per-group (128), straight-through estimator, on q/k/v/o/gate/up/down projections
Variant tag: QAT-int4-from-unlearned

Evaluation

MUSE metrics (verbmem_f / knowmem_f: forget memorization, lower = better unlearning; knowmem_r: retain utility; privleak: MIA gap) at bf16 / int4 (matched asymmetric-RTN) / NF4 will be added in results.csv once the eval completes.

Downloads last month: 26

Safetensors

Model size

7B params

Tensor type

BF16

Model tree for Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1-QAT-int4-from-unlearned

Base model

muse-bench/MUSE-books_target

Finetuned

Jeesup/MUSE-Books_iclm-7b_npo_beta0p1_lr5e-6_lam10_ep1

Finetuned

(1)

this model