Instructions to use BSC-LT/salamandraTA-7B-instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use BSC-LT/salamandraTA-7B-instruct-GGUF with Transformers:

# Use a pipeline as a high-level helper
# Warning: Pipeline type "translation" is no longer supported in transformers v5.
# You must load the model directly (see below) or downgrade to v4.x with:
# 'pip install "transformers<5.0.0'
from transformers import pipeline

pipe = pipeline("translation", model="BSC-LT/salamandraTA-7B-instruct-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("BSC-LT/salamandraTA-7B-instruct-GGUF")
model = AutoModelForCausalLM.from_pretrained("BSC-LT/salamandraTA-7B-instruct-GGUF", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use BSC-LT/salamandraTA-7B-instruct-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf BSC-LT/salamandraTA-7B-instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf BSC-LT/salamandraTA-7B-instruct-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf BSC-LT/salamandraTA-7B-instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf BSC-LT/salamandraTA-7B-instruct-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf BSC-LT/salamandraTA-7B-instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf BSC-LT/salamandraTA-7B-instruct-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf BSC-LT/salamandraTA-7B-instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf BSC-LT/salamandraTA-7B-instruct-GGUF:Q4_K_M

Use Docker

docker model run hf.co/BSC-LT/salamandraTA-7B-instruct-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use BSC-LT/salamandraTA-7B-instruct-GGUF with Ollama:
```
ollama run hf.co/BSC-LT/salamandraTA-7B-instruct-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use BSC-LT/salamandraTA-7B-instruct-GGUF with Docker Model Runner:
```
docker model run hf.co/BSC-LT/salamandraTA-7B-instruct-GGUF:Q4_K_M
```

Lemonade

How to use BSC-LT/salamandraTA-7B-instruct-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull BSC-LT/salamandraTA-7B-instruct-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.salamandraTA-7B-instruct-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

SalamandraTA-7B-instruct-GGUF Model Card

This model is the GGUF-quantized version of SalamandraTA-7b-instruct (v3).

The main branch of this repository contains the GGUF quantization of the latest SalamandraTA-7b-instruct release (v3). The previous quantized version is preserved on the v1 branch.

The model weights are quantized from FP16 to Q4_K_M (4-bit weights with K-quant, medium) using the Llama.cpp framework. Inferencing with this model can be done using VLLM.

SalamandraTA-7b-instruct is a translation LLM that has been instruction-tuned from SalamandraTA-7b-base. The base model results from continually pre-training Salamandra-7b on monolingual and parallel data and has not been published, but is reserved for internal use. SalamandraTA-7b-instruct (v3) is proficient in 40 languages (+ 3 varieties), including both European and non-European languages such as Arabic, Japanese, Hindi, Korean, and Simplified Chinese, and is mainly trained to perform general translation tasks at the sentence, paragraph, and document levels. This version adds translation-related tasks such as terminology-aware machine translation, structured text (e.g., HTML, XML) translation, translation post-editing, and named entity recognition, while maintaining the strong translation performance of the previous version.

License Notice

This release includes data licensed under GPL-3, and is therefore distributed under the terms of the GPL-3 license.

DISCLAIMER: This version of Salamandra is tailored exclusively for translation and the mentioned translation-related tasks. It lacks chat capabilities and has not been trained with any chat instructions.

Available files

File	Quantization	Size (approx.)
`salamandraTA_7B_v3_q4_k_m.gguf`	Q4_K_M — 4-bit, best size/quality balance	~4.1 GB

Quantization impact

The Q4_K_M quantization shows a 1–2 BLEU difference compared to the full-precision (unquantized) model.

How to Use

The following example code works under Python 3.10.4, vllm==0.7.3, torch==2.5.1 and torchvision==0.20.1, though it should run on any current version of the libraries. This is an example of translation using the model:

from huggingface_hub import snapshot_download
from vllm import LLM, SamplingParams

model_dir = snapshot_download(repo_id="BSC-LT/salamandraTA-7B-instruct-GGUF", revision="main")
model_name = "salamandraTA_7B_v3_q4_k_m.gguf"

llm = LLM(model=model_dir + '/' + model_name, tokenizer=model_dir)

source = "Spanish"
target = "English"
sentence = "Ayer se fue, tomó sus cosas y se puso a navegar. Una camisa, un pantalón vaquero y una canción, dónde irá, dónde irá. Se despidió, y decidió batirse en duelo con el mar. Y recorrer el mundo en su velero. Y navegar, nai-na-na, navegar."

prompt = f"Translate the following text from {source} into {target}.\\n{source}: {sentence} \\n{target}:"
messages = [{'role': 'user', 'content': prompt}]

outputs = llm.chat(messages,
                   sampling_params=SamplingParams(
                       temperature=0.1,
                       stop_token_ids=[5],
                       max_tokens=200)
                   )[0].outputs

print(outputs[0].text)

Additional information

Author

Machine Translation Group, AI Institute, the Barcelona Supercomputing Center (ai_institute_mt@bsc.es).

Contact

For further information, please send an email to ai_institute_mt@bsc.es.

Copyright

Funding

This work is funded by the Ministerio para la Transformación Digital y de la Función Pública and Plan de Recuperación, Transformación y Resiliencia - Funded by EU – NextGenerationEU within the framework of the project Desarrollo Modelos ALIA.

This work has been promoted and financed by the Government of Catalonia through the Aina Project.

Acknowledgements

The success of this project has been made possible thanks to the invaluable contributions of our partners in the ILENIA Project: HiTZ, and CiTIUS. Their efforts have been instrumental in advancing our work, and we sincerely appreciate their help and support.

Disclaimer

Be aware that the model may contain biases or other unintended distortions. When third parties deploy systems or provide services based on this model, or use the model themselves, they bear the responsibility for mitigating any associated risks and ensuring compliance with applicable regulations, including those governing the use of Artificial Intelligence.

The Barcelona Supercomputing Center, as the owner and creator of the model, shall not be held liable for any outcomes resulting from third-party use.