--- license: other library_name: transformers pipeline_tag: text-generation tags: - soofi - mamba-2 - moe - text-generation - sovereign-ai - base - preview language: - en - de --- # Soofi-S-30B-A3B Overview > ⚠️ **Preview / internal checkpoint.** Weights and metadata may still change. ## Description **Soofi-S-30B-A3B** is a pre-trained base model that generates free-form text continuations for a given prompt. It was developed by a German research consortium. SOOFI (Sovereign Open Source Foundation Models) is designed to provide a secure, European open-source alternative to US and Chinese AI models for industrial use, featuring strong reasoning and AI agent capabilities. As a base model, Soofi-S-30B-A3B has **not** been instruction-tuned, aligned, or safety-tuned. It is intended as a foundation for further post-training (e.g. SFT, preference tuning, domain adaptation) and for research — not for direct deployment as an end-user assistant. For general assistant and instruction-following tasks, see the post-trained variant [Soofi-S-Instruct-Preview](https://huggingface.co/Soofi-Project/Soofi-S-Instruct-Preview). For explicit chain-of-thought reasoning, see the thinking variants [Soofi-S-Isar-Preview](https://huggingface.co/Soofi-Project/Soofi-S-Isar-Preview) and [Soofi-S-Rhine-Preview](https://huggingface.co/Soofi-Project/Soofi-S-Rhine-Preview). This model is for research and development only (Preview). ## License/Terms of Use Released under a custom license ("Other"). TODO: add the full license text / link — the official card references a License section that is not yet filled in. ### Deployment Geography Global (open release on the Hugging Face Hub). Development and training infrastructure are located in Europe (see Computational Load). ### Use Case Enterprise developers and researchers seeking a sovereign, European open-source base LLM as a starting point for fine-tuning, continued pre-training, domain adaptation, and LLM research (e.g. building custom assistant or agent models). English and German are the primary languages. ### Release Date Hugging Face Hub — Preview at . TODO: final release date (MM/DD/YYYY). ## Reference(s) - Project: - Related models: see the *Related models* section below. - TODO: link the technical report / paper once published. ## Model Architecture **Architecture Type:** Transformer-based hybrid Mixture-of-Experts (MoE) with Mamba-2 state-space (SSM) layers and attention layers.
**Network Architecture:** Custom Hybrid Mamba-2/MoE (Nemotron-style), designed from scratch — 23 Mamba-2/MoE layers + 6 attention layers; 128 routing experts + 1 shared expert per MoE layer; 6 experts activated per token.
**This model was developed from scratch** (no base model).
**Number of model parameters:** 3.0×10^10 total (30B), with ~3.5B active parameters during inference. ## Computational Load **Cumulative Compute:** TODO.
**Estimated Energy and Emissions for Model Training:** TODO. Training infrastructure is hosted entirely in Europe on T-Systems' Industrial AI Cloud (Deutsche Telekom) to ensure data sovereignty. ## Input **Input Type(s):** Text
**Input Format(s):** String
**Input Parameters:** One-Dimensional (1D)
**Other Properties Related to Input:** Plain-text prompts (completion-style). As a base model, it uses no chat template and expects no system prompt. Context length: see `config.json` (TODO: confirm maximum context). ## Output **Output Type(s):** Text
**Output Format(s):** String
**Output Parameters:** One-Dimensional (1D)
**Other Properties Related to Output:** Raw text continuation of the input prompt (next-token prediction). No explicit reasoning trace, no chat or tool-calling format, and no alignment for helpfulness or safety. ## Software Integration **Runtime Engine(s):**
* Hugging Face `transformers` (`trust_remote_code=True`)
* vLLM, llama.cpp/Ollama via quantized variants (TODO: link once available)
**Supported Hardware Microarchitecture Compatibility:**
* NVIDIA GPUs (Ampere and newer recommended)
**Preferred/Supported Operating System(s):**
* Linux
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. ## Model Version(s) * **Soofi-S-30B-A3B** — bf16 safetensors, unquantized (this repo).
* Post-trained variants: Instruct and thinking models (see *Related models*).
* Quantized derivatives (`…-GGUF`, `…-FP8`): TODO — link once available. ## Installation & Usage SOOFI-S ships with custom modeling code. You must load it using `trust_remote_code=True` with `transformers`. ```python from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "Soofi-Project/Soofi-S-30B-A3B" tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=True, torch_dtype="auto", device_map="auto" ) # Base model: plain text completion — no chat template, no system prompt. prompt = "AI sovereignty is the idea that" inputs = tok(prompt, return_tensors="pt").to(model.device) out = model.generate(**inputs, max_new_tokens=128) # sampling defaults come from generation_config.json print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) ``` # Training, Testing, and Evaluation Datasets ### Dataset Overview * **Total Size:** ~2.5×10^13 tokens (25 trillion).
* **Languages:** English, German (primary); French, Italian, Spanish (limited). English acts as the pivot language.
* **Knowledge Cutoff:** End of 2025.
* **Training Start:** April 2026. ## Training Dataset **Link:** TODO.
**Data Modality:** Text.
**Text Training Data Size:** More than 10 Trillion Tokens (~25T).
**Data Collection Method by dataset:** Hybrid (freely available, high-quality sources). TODO: refine.
**Labeling Method by dataset:** TODO.
**Properties:** Trained entirely from scratch on freely available, high-quality tokens. ## Testing Dataset **Link:** TODO.
**Properties:** TODO. ## Evaluation Dataset **Link:** TODO.
**Benchmark Score:** TODO — add key base-model benchmarks (e.g. few-shot reasoning, multilingual) once available.
**Properties:** TODO. ## Inference **Acceleration Engine:** `transformers`; vLLM / llama.cpp via quantized variants (TODO).
**Specific Test Hardware:** TODO. ## Ethical Considerations The SOOFI consortium believes Trustworthy AI is a shared responsibility and has established policies and practices to enable development for a wide array of AI applications. When downloaded or used, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. Note that Soofi-S-30B-A3B is an unaligned base model: it has undergone no instruction tuning or safety alignment, and downstream developers are responsible for appropriate post-training, evaluation, and guardrails before any deployment. For more detailed information, see the Model Card++ subcards below. Please report model quality, risk, security vulnerabilities, or concerns to contact@soofi.info. ### Bias Subcard | Field | Response | |:---|:---| | Participation considerations from adversely impacted groups in model design and testing | TODO | | Measures taken to mitigate against unwanted bias | TODO | | Bias Metric (if measured) | TODO | ### Explainability Subcard | Field | Response | |:---|:---| | Intended Task/Domain | Text completion; foundation (base) model for downstream fine-tuning and research | | Model Type | Hybrid Mixture-of-Experts (MoE) autoregressive language model | | Intended Users | Enterprise developers and researchers (fine-tuning and research, not end users) | | Output | Text (String) | | Describe how the model works | Generates text autoregressively; a router activates 6 of 128 experts per token across hybrid Mamba-2/MoE and attention layers | | Technical Limitations | Preview checkpoint; unaligned base model — raw completions may be unhelpful, repetitive, or unsafe without post-training; non-primary languages (FR/IT/ES) are limited; may produce inaccurate or outdated content (knowledge cutoff end of 2025) | | Verified to have met prescribed quality standards | TODO | | Performance Metrics | TODO (see Evaluation Dataset) | | Potential Known Risks and Mitigation | May generate incorrect, biased, or unsafe content; as an unaligned base model, apply post-training, use-case-specific testing, and guardrails before deployment | | Terms of Use/Licensing | Other (see License/Terms of Use) | ### Privacy Subcard | Field | Response | |:---|:---| | Generatable or reverse engineerable personal data? | TODO | | Personal data used to create this model? | TODO | | Was consent obtained for any personal data used? | TODO | | How often is dataset reviewed? | TODO | | Was data from user interactions with the AI model used to train the model? | No | | Is there provenance for all datasets used in training? | TODO | | Applicable Privacy Policy | TODO | ### Safety & Security Subcard | Field | Response | |:---|:---| | Model Application Field(s) | Foundation model for industrial use; basis for fine-tuned assistant and agent applications | | Describe the life critical impact (if present) | None intended. Not for use in life-critical or safety-critical decision-making without independent validation | | Use Case Restrictions | Abide by the applicable license agreement (see License/Terms of Use). Not intended for direct end-user deployment without further post-training and safety alignment | | Model and dataset restrictions | TODO | ## Related models - Instruction-tuned variant: [Soofi-Project/Soofi-S-Instruct-Preview](https://huggingface.co/Soofi-Project/Soofi-S-Instruct-Preview) - Reasoning variants: [Soofi-Project/Soofi-S-Isar-Preview](https://huggingface.co/Soofi-Project/Soofi-S-Isar-Preview) and [Soofi-Project/Soofi-S-Rhine-Preview](https://huggingface.co/Soofi-Project/Soofi-S-Rhine-Preview) - Quantized derivatives of the Instruct variant: [Soofi-Project/Soofi-S-Instruct-Preview-GGUF](https://huggingface.co/Soofi-Project/Soofi-S-Instruct-Preview-GGUF), [Soofi-Project/Soofi-S-Instruct-Preview-FP8](https://huggingface.co/Soofi-Project/Soofi-S-Instruct-Preview-FP8) - TODO: quantized derivatives of this base model, if released. ## Citation ```bibtex @misc{soofi_s_30b_a3b, title = {Soofi-S-30B-A3B}, author = {SOOFI Consortium}, year = {2026}, url = {https://huggingface.co/Soofi-Project/Soofi-S-30B-A3B} } ```