espnet/bagpiper-sft

Bagpiper SFT

Bagpiper is an 8B audio foundation model for open-ended audio understanding and generation through rich captions. It accepts text and audio and can return text or generate speech, music, environmental sounds, and mixtures of them.

This repository contains the paper-selected general-SFT checkpoint from Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions.

Files and format

• model.pt: native ESPnet weight-only checkpoint, stored as {"module": state_dict}.
• train_stage3_qwen3_base.yaml: checkpoint-compatible model configuration, named consistently with the pre-trained espnet/bagpiper release.
• inference_text.yaml: audio-understanding/text-output decoding.
• inference_audio.yaml: text-then-audio generation decoding.
• MANIFEST.json and SHA256SUMS: provenance and integrity records.

model.pt contains 1,382 tensors: the 1,381 BF16 tensors from training plus the deterministically reconstructed FP32 vocab_weight buffer required by the current strict ESPnet loader. It contains no optimizer or exact-resume state. It is not a Transformers from_pretrained directory and no vLLM compatibility is claimed.

Download

hf download espnet/bagpiper-sft --local-dir bagpiper-sft
sha256sum -c bagpiper-sft/SHA256SUMS

ESPnet runtime

The release was strict-loaded with:

• ESPnet PR #6503, commit 1992821bfb6d68c6636d1b75340b8b2b591bc015;
• Python 3.11.15, PyTorch 2.12.1+cu129, Transformers 5.5.4;
• FlashAttention-3 3.0.0 at source commit 77aacb68d194ba9af1010eda5eac3e7c0df8e6f6; and
• one NVIDIA H100 80GB HBM3.

git clone --branch titan-pt-weightonly-init https://github.com/jctian98/espnet.git
cd espnet
git checkout 1992821bfb6d68c6636d1b75340b8b2b591bc015
python -m pip install -e '.[speechlm]'
python -m pip install --no-build-isolation \
  'flash-attn-3 @ git+https://github.com/Dao-AILab/flash-attention.git@77aacb68d194ba9af1010eda5eac3e7c0df8e6f6#subdirectory=hopper'

Use espnet2/speechlm/bin/inference.py with train_stage3_qwen3_base.yaml, one of the supplied inference YAMLs, and model.pt. Training backend configuration belongs to the ESPnet recipe and is not duplicated in this model repository.

Paper-reported results

These values are from the paper's checkpoint and evaluation protocols; they were not rerun as part of the Hub upload.

Evaluation	Result
LibriSpeech test-clean WER	2.5
MMAU-Mini	74.5
MMAU	73.1
MMAR	57.0
AIR-Bench chat	6.57
AudioBench aggregate	70.39
General-model TTS WER	2.7

The paper reports audio-fingerprint overlap between the SFT pool and 12.8% of AudioBench and 3.2% of AIR-Bench evaluation clips. Interpret those results with this disclosure.

Provenance and validation

• Selected checkpoint: step_270000/global_step269985.
• Raw source: JinchuanTian/bagpiper_sft at revision b11d5a0c11ad488edd04e3734d4bdff764977f57.
• Raw SHA-256: e0d8fdd57c8cb8c938cd61ada43b20e469d9cea33ad8450e0971f80d4f9667b6.
• model.pt SHA-256: 50980756fad38bbefeb326ac0012e39f431ec5af8c0caaac8bd254f179f33746.
• All 1,381 source tensors matched the converted file exactly with torch.equal; the only added key was vocab_weight.
• The 1,382-entry state dict passed the native ESPnet strict inference loader.

Limitations and responsible use

Understanding can hallucinate or mistranscribe. Generated audio can contain artifacts, wrong words, unsafe content, or unintended resemblance to people and styles. Most reported evaluations are English and some rely on model judges. The model has no built-in watermark, factuality verifier, copyright filter, or speaker-consent check. Do not use it for impersonation, deceptive media, surveillance, biometric authentication, or high-stakes automated decisions.

Training-data availability and terms are documented at Bagpiper SFT Data. Third-party component terms are listed in THIRD_PARTY_NOTICES.md.

Citation

@inproceedings{anonymous2026bagpiper,
  title={Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions},
  author={Jinchuan Tian and Haoran Wang and Bo-Hao Su and Chien-yu Huang and
          Qingzheng Wang and Jiatong Shi and William Chen and Xun Gong and
          Siddhant Arora and Chin-Jou Li and Masao Someki and Takashi Maekaku and
          Keita Goto and Yusuke Shinohara and Jin Sakuma and
          Chao-Han Huck Yang and Shinji Watanabe},
  booktitle={Third Conference on Language Modeling},
  year={2026},
  url={https://openreview.net/forum?id=FuHs64E3X6}
}