--- language: - en license: cc-by-nc-sa-4.0 library_name: transformers pipeline_tag: any-to-any tags: - molmo - molmo-audio - multimodal - any-to-any - audio-text-to-text - image-text-to-text - audio - speech - diarization - vllm - peft - pytorch - blaster - cc-by-nc-sa-4.0 base_model: allenai/Molmo-7B-D-0924 base_model_relation: finetune pretty_name: "Blaster" --- # Blaster ![Blaster_Image_Logo](https://cdn-uploads.huggingface.co/production/uploads/68f5a89ddee3f58d40bbfb1b/deuAByepJ39o14D57fqJf.png) **Blaster** is a multimodal companion model: it can look at images, listen to microphone speech, and reply in text (with optional spoken replies via a codec). It is a fine-tune of [AllenAI Molmo-7B-D](https://huggingface.co/allenai/Molmo-7B-D-0924) with continuous audio understanding (SLAP), speaker markers, and discrete speech codes for generation. This repository is the **merged serving checkpoint** — the weights you load for inference (vLLM / Transformers). Adapters and companion modules live in sibling repos under the same `molmo-audio-*` ids (marketing name: **Blaster**). ## What it can do | Capability | Notes | | --- | --- | | **See** | Answer questions about attached images | | **Hear** | Respond to spoken turns without an ASR transcript bridge — audio is encoded and injected as continuous features | | **Talk (text)** | Short, conversational replies for mic and typed chat | | **Optional voice-out** | When the model emits audio codebook tokens, a Spectral FSQ (or legacy DiT) decoder can synthesize speech | | **Who spoke** | Vocabulary includes ``…``; a sibling diarization head supports multi-speaker cues | It is designed for **desk / appliance-style** multimodal chat (image + mic + text), not as a pure ASR system or a general web agent by itself. ## Quick start ### Transformers (research / offline) ```python from transformers import AutoModelForCausalLM, AutoProcessor repo = "0x8badbeef/molmo-audio-serving-diar-d" model = AutoModelForCausalLM.from_pretrained( repo, trust_remote_code=True, device_map="auto" ) processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True) # Full multimodal mic+image paths use the project serving stack (below). ``` ### Recommended: project Docker stack Blaster’s production path uses a small gateway + engine + optional DiT/FSQ workers: ```bash export SERVING_MODEL_DIR=$PWD/molmo-audio-serving-diar-d export MOLMO_LORA_DIR=$PWD/molmo-audio-lora-diar-d/checkpoint-112000 # or your tip export DIAR_MODEL_DIR=$PWD/molmo-audio-slap-diar export SPECTRAL_FSQ_DIR=$PWD/molmo-audio-spectral-fsq-v1 export MOLMO_AUDIO_CODEC=spectral_fsq docker compose -f deploy/compose.yaml up --build -d ``` Sibling LoRA (unmerged, for continued training): [`molmo-audio-lora-diar-d`](https://huggingface.co/0x8badbeef/molmo-audio-lora-diar-d). ## Model family (Blaster) | Hub id | Role | | --- | --- | | **`molmo-audio-serving-diar-d`** (this repo) | Merged weights for serving | | [`molmo-audio-lora-diar-d`](https://huggingface.co/0x8badbeef/molmo-audio-lora-diar-d) | LoRA + `audio_modules.pt` checkpoints | | [`molmo-audio-slap-stage-a`](https://huggingface.co/0x8badbeef/molmo-audio-slap-stage-a) | SLAP speech encoder (understanding) | | [`molmo-audio-slap-diar`](https://huggingface.co/0x8badbeef/molmo-audio-slap-diar) | Diarization head | | [`molmo-audio-spectral-fsq-v1`](https://huggingface.co/0x8badbeef/molmo-audio-spectral-fsq-v1) | Speech codec (voice-out) | | [`molmo-audio-dit-dac-v3`](https://huggingface.co/0x8badbeef/molmo-audio-dit-dac-v3) | Legacy DiT audio generator | | [`molmo-audio-phase-h-restore`](https://huggingface.co/0x8badbeef/molmo-audio-phase-h-restore) | Bill of materials / restore bundle | ## What’s inside this repo | Files | Purpose | | --- | --- | | `model-*-of-*.safetensors` | Merged language + vision weights | | `config.json` | Molmo config + audio special-token ids | | `audio_modules.pt` | SLAP encoder + projector used at serve time | | `tokenizer*` / `special_tokens_map.json` | `` `` `` `` `` | | `token_manifest.json` | Gateway / router ids | | `modeling_molmo.py` (etc.) | Remote-code sources for Transformers | **Architecture sketch:** frozen Molmo-7B-D backbone + folded LoRA; mic audio → SLAP → projector → continuous inject at ``; optional discrete `` codes for synthesis. ## Intended use - Interactive multimodal chat with **images and/or microphone audio** - Research and non-commercial demos of speech-aware VLMs - Starting point for further LoRA continues (prefer the LoRA sibling tip) ## Out of scope / limitations - **Not** a dedicated ASR engine — comprehension is end-to-end into the LM, not a transcript-first pipeline - Spoken and typed behavior still show **class confusions** on hard probes (e.g. status vs identity, instruction following); see training notes in the project `README_TRAINING.md` - Voice-out quality depends on the paired codec (Spectral FSQ / DiT), not only this merge - Tool-use / web search (when enabled in the UI) can still over-trigger on some turns - Base Molmo vision/text strengths and limits still apply ## Evaluation (high level) - Attribution / diarization gates (oracle speaker markers): strong text; solid spoken - Mic hearing and conversational probes are tracked per LoRA tip in the sibling LoRA card and local `hearing_eval_*.json` / `broad_baseline_matrix.md` artifacts Numbers move with each Stage D continue; treat Hub “primary tip” docs as approximate until you pin a checkpoint id. ## License **[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)** (Attribution–NonCommercial–ShareAlike). - **Attribution** required - **NonCommercial** — no commercial use of these weights without a separate grant - **ShareAlike** — adaptations must use the same license Marketing name: **Blaster**. Technical Hub id remains `molmo-audio-serving-diar-d`. The backbone [allenai/Molmo-7B-D-0924](https://huggingface.co/allenai/Molmo-7B-D-0924) remains **Apache-2.0**. Some training mixes include ShareAlike sources (e.g. People’s Speech / Dolly); this weight license is intentionally **more restrictive** (adds NonCommercial). ## Citation / credit If you use Blaster, please credit: - **Blaster** (`0x8badbeef/molmo-audio-serving-diar-d`), CC BY-NC-SA 4.0 - Base model: AllenAI **Molmo-7B-D** - Upstream data licenses as documented in the project training README ## Links - Model page: https://huggingface.co/0x8badbeef/molmo-audio-serving-diar-d - Base: https://huggingface.co/allenai/Molmo-7B-D-0924 - License deed: https://creativecommons.org/licenses/by-nc-sa/4.0/