brendanddev/Pleias-RAG-1B-Q4_K_M-GGUF

Pleias-RAG-1B (GGUF, Q4_K_M)

This repository provides a GGUF quantized version of the original Pleias-RAG-1B model, compatible with llama.cpp and llama-cpp-python.

---

Base Model

• Model: Pleias-RAG-1B
• Author: PleIAs
• Original Repository: https://huggingface.co/PleIAs/Pleias-RAG-1B
• License: Apache 2.0

All credit for the model architecture, training, and data goes to the original authors.

---

What This Repository Contains

• Pleias-RAG-1B-Q4_K_M.gguf — Q4_K_M quantized GGUF version of the model for efficient inference with llama.cpp

---

Changes from Original

This is a derivative work of the original model with the following modifications:

• Converted to GGUF format
• Quantized to Q4_K_M
• Enabled compatibility with llama.cpp / llama-cpp-python
• Added support for the pleias-rag tokenizer pre-type

---

Tokenizer Support (Important)

The original model uses a tokenizer configuration (pleias-rag) that is not supported by default in llama.cpp.

To enable correct tokenization, the following changes were required:

llama-vocab.h

LLAMA_VOCAB_PRE_TYPE_PLEIAS_RAG = 50,

llama-vocab.cpp (tokenizer mapping)

} else if (
    tokenizer_pre == "pleias-rag") {
    pre_type = LLAMA_VOCAB_PRE_TYPE_PLEIAS_RAG;
} else if (

llama-vocab.cpp (switch case)

case LLAMA_VOCAB_PRE_TYPE_PLEIAS_RAG:

Without these changes, the model will not tokenize correctly and may produce incorrect outputs.

---

Usage

This model follows a structured RAG-style prompting format.

Example (Python with llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="./Pleias-RAG-1B-Q4_K_M.gguf",
    n_gpu_layers=-1,
    n_ctx=2048,
    verbose=False
)

prompt = """<|query_start|>What is a pointer?<|query_end|>
<|source_start|><|source_id_start|>1<|source_id_end|>In computer science, a pointer is a variable that stores a memory address.<|source_end|>
<|final_answer_start|>"""

result = llm(
    prompt,
    max_tokens=80,
    temperature=0.0,
    repeat_penalty=1.0,
    stop=[
        "English",
        "<|source_analysis_start|>",
        "<|query_start|>",
        "#END#"
    ]
)

print(result["choices"][0]["text"].strip())

---

Prompt Format

The model expects structured tokens:

<|query_start|>...<|query_end|>
<|source_start|>...<|source_end|>
<|final_answer_start|>

If incorrect tokens are used, the model may output intermediate reasoning steps instead of a final answer.

---

Notes

• This is not the original model.
• This repository only provides a quantized GGUF version for local inference.
• Model behavior and quality depend on the original Pleias-RAG-1B model.

---

Technical Notes

This model required extending llama.cpp to support a custom tokenizer pre-type (pleias-rag). This was identified by inspecting the model metadata and reverse engineering tokenizer behavior.

---

License

This repository distributes a derivative of the original model under the terms of the Apache 2.0 License.

You must comply with the original license: https://www.apache.org/licenses/LICENSE-2.0

---

Attribution

• Original model: PleIAs
• This repository: GGUF quantization and llama.cpp compatibility work