Qwen3-ASR-1.7B-JA-Anime-Galgame-hf

This is a Transformers-native -hf layout conversion of jaykwok/Qwen3-ASR-1.7B-JA-Anime-Galgame, which was fine-tuned from the non--hf Qwen3-ASR checkpoint.

The repository also ships an optional CTC alignment head, ctc_aligner.pt, trained on this model's audio encoder. It is not part of the ASR model and is not loaded by AutoModelForMultimodalLM; see CTC Alignment Head below.

Difference From The Non--hf Repository

The source repository is intended for the qwen-asr wrapper / original Qwen3-ASR layout. This repository is intended for native Hugging Face Transformers loading.

The conversion keeps the fine-tuned weights unchanged and only rewrites the repository layout to match Qwen/Qwen3-ASR-1.7B-hf:

• config / processor / tokenizer files come from the official -hf template.
• safetensors keys are rewritten as:
  • thinker.audio_tower.* -> model.audio_tower.*
  • thinker.audio_tower.proj1.* -> model.multi_modal_projector.linear_1.*
  • thinker.audio_tower.proj2.* -> model.multi_modal_projector.linear_2.*
  • thinker.model.* -> model.language_model.*
• tensor count after conversion: 707.
• converted tensor bytes: 4076104960.

Requirements

Requires the stable transformers >= 5.13.0 release for native Qwen3-ASR support:

pip install "transformers>=5.13.0"

With uv:

uv pip install "transformers>=5.13.0"

Usage

from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "jaykwok/Qwen3-ASR-1.7B-JA-Anime-Galgame-hf"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

inputs = processor.apply_transcription_request(
    audio="path/to/audio.wav",
    language="Japanese",
).to(model.device, model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
text = processor.decode(generated_ids, return_format="transcription_only")[0]
print(text)

CTC Alignment Head (ctc_aligner.pt)

ASR gives you text and a segment window. It does not tell you when inside that window each character was spoken, so a subtitle writer has to spread the text across the window in proportion to character count and hope. This head replaces that guess with a measurement.

It is a small CTC classifier over the frozen audio encoder of this model. It brings no acoustic model of its own — that is the point. General-purpose Japanese forced aligners are not adapted to this domain, so pairing one with a domain-fine-tuned ASR makes the aligner the bottleneck. Sitting on the encoder that was already fine-tuned means the domain adaptation is paid for once, and only a 3.8 M-parameter head has to be learned.

It is encoder-specific. It is trained against the features this fine-tune's encoder produces and will not transfer to Qwen/Qwen3-ASR-1.7B-hf or to another fine-tune. That is why it ships here, next to the encoder it belongs to, rather than in the application that consumes it.

File	ctc_aligner.pt (14.7 MB, torch.save payload, weights_only=False)
Schema	asr_ctc_alignment_head_v1
Input	(B, T, 2048) encoder hidden states, 13 fps (76.9 ms per frame)
Output	(B, T*2, 2328) log-probabilities, 38.5 ms resolution
Parameters	3,840,280
Vocabulary	2,328 = 2,326 characters + blank (index 0) + <unk> (index 1)
Targets	Japanese characters, NFKC-folded, whitespace stripped

Two design choices differ from the obvious ones:

• Characters, not kana or phonemes. Kana needs g2p (pyopenjtalk), which adds a dependency and, worse, a reading-error source on kanji. Characters need neither, and the density works out better: the training corpus runs 4.67 chars/s against a 13 fps encoder, i.e. ~2.8 frames per character, where kana would be nearer 2.
• The encoder is upsampled before the classifier. It adds no information, but CTC cannot emit more tokens than it has frames, and timestamp resolution is bounded by frame duration — 76.9 ms natively. A ×2 transposed convolution is cheap and buys back both.

Two readings of the same tensor

The head is run once per audio chunk; the resulting log-probabilities are read two different ways.

1. Character timestamps. CTC forced alignment (Viterbi over the standard blank-interleaved target lattice) of the known transcript against the log-probs yields a start/end frame per character. This is what gives subtitle cues real in-segment timing and real word gaps to split lines at.
2. Blank runs. A stretch the head covers entirely with blank is a stretch with no character evidence in it — read straight off the argmax, with no tuned threshold and no free parameter beyond a minimum run length. These are the natural pause locations, useful for choosing where to cut long audio into chunks.

Reading (2) is a gate on cut points only. Deciding that a stretch is silence and therefore deleting the audio would make a false blank unrecoverable; using it to choose where to cut leaves every sample in the stream either way.

Loading it

The payload is self-contained: architecture hyper-parameters, vocabulary and weights all travel in the same file.

import torch
from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="jaykwok/Qwen3-ASR-1.7B-JA-Anime-Galgame-hf",
    filename="ctc_aligner.pt",
)
payload = torch.load(path, map_location="cpu", weights_only=False)

payload["schema"]      # "asr_ctc_alignment_head_v1"
payload["input_dim"]   # 2048
payload["hidden_dim"]  # 512
payload["upsample"]    # 2
payload["blocks"]      # 4
payload["vocab"]       # {"schema", "size", "blank_index", "unk_index", "chars"}
payload["state_dict"]

The module it belongs to:

from torch import nn


class ResidualConvBlock(nn.Module):
    """Dilated depthwise-separable conv, pre-norm, residual.

    Convolutional rather than attentional on purpose: alignment is a monotonic,
    local problem, and a conv stack cannot learn to reorder time the way
    self-attention can.
    """

    def __init__(self, channels, dilation):
        super().__init__()
        self.norm = nn.LayerNorm(channels)
        self.depthwise = nn.Conv1d(
            channels, channels, kernel_size=5,
            padding=2 * dilation, dilation=dilation, groups=channels,
        )
        self.pointwise = nn.Conv1d(channels, channels, kernel_size=1)
        self.activation = nn.GELU()
        self.dropout = nn.Dropout(0.0)

    def forward(self, x):
        y = self.norm(x).transpose(1, 2)
        y = self.pointwise(self.activation(self.depthwise(y)))
        return x + self.dropout(y.transpose(1, 2))


class CtcAlignmentHead(nn.Module):
    def __init__(self, vocab_size, input_dim=2048, hidden_dim=512,
                 upsample=2, blocks=4):
        super().__init__()
        self.upsample = upsample
        self.input_norm = nn.LayerNorm(input_dim)
        self.project = nn.Linear(input_dim, hidden_dim)
        self.expand = nn.ConvTranspose1d(
            hidden_dim, hidden_dim, kernel_size=upsample, stride=upsample
        ) if upsample > 1 else None
        self.blocks = nn.ModuleList(
            [ResidualConvBlock(hidden_dim, dilation=2**i) for i in range(blocks)]
        )
        self.output_norm = nn.LayerNorm(hidden_dim)
        self.classifier = nn.Linear(hidden_dim, vocab_size)

    def forward(self, features):
        """(B, T, input_dim) -> (B, T*upsample, vocab) log-probabilities."""
        x = self.project(self.input_norm(features))
        if self.expand is not None:
            x = self.expand(x.transpose(1, 2)).transpose(1, 2)
        for block in self.blocks:
            x = block(x)
        return nn.functional.log_softmax(
            self.classifier(self.output_norm(x)), dim=-1
        )


head = CtcAlignmentHead(
    vocab_size=payload["vocab"]["size"],
    input_dim=payload["input_dim"],
    hidden_dim=payload["hidden_dim"],
    upsample=payload["upsample"],
    blocks=payload["blocks"],
)
head.load_state_dict(payload["state_dict"])
head.eval()

Running it

Feed it the audio encoder's output for the same audio, then read the result.

import unicodedata

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "jaykwok/Qwen3-ASR-1.7B-JA-Anime-Galgame-hf"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, torch_dtype="auto")

inputs = processor.apply_transcription_request(
    audio="path/to/audio.wav", language="Japanese",
).to(model.device, model.dtype)

with torch.inference_mode():
    encoded = model.get_audio_features(
        input_features=inputs["input_features"],
        input_features_mask=inputs["input_features_mask"],
    )
    features = encoded.pooler_output.float().cpu()    # (frames, 2048) at 13 fps
    log_probs = head(features.unsqueeze(0))[0]        # (frames*2, 2328)

Note that pooler_output is the batch's valid frames concatenated, not a padded (B, T, 2048) tensor. With a batch of one it is exactly that clip's frames; for a real batch, slice it with the per-item frame count:

def audio_output_lengths(input_lengths):
    """Mel frames -> encoder frames, 13 per 100."""
    leave = input_lengths % 100
    feat = (leave - 1) // 2 + 1
    return ((feat - 1) // 2 + 1 - 1) // 2 + 1 + (input_lengths // 100) * 13

lengths = audio_output_lengths(inputs["input_features_mask"].sum(dim=1))

Character indices, for building CTC targets:

chars = payload["vocab"]["chars"]                    # tuple of single chars
lookup = {ch: i + 2 for i, ch in enumerate(chars)}   # 0 blank, 1 unk

def encode(text):
    folded = unicodedata.normalize("NFKC", text)
    folded = "".join(ch for ch in folded if not ch.isspace())
    return [lookup.get(ch, 1) for ch in folded]

Out-of-vocabulary characters map to <unk> rather than being dropped: they still consume audio, and dropping them would shift every later timestamp.

Frame f of the output starts at f * (1 / 13) / upsample seconds. Blank runs need no extra machinery:

blank = log_probs.argmax(dim=-1).eq(0)   # per output frame

For character timestamps, run a standard CTC forced alignment (Viterbi over the blank-interleaved target lattice, backtracked to per-character frame spans) against encode(transcript). torchaudio.functional.forced_align does this if you have a torchaudio build for your Python/CUDA combination; otherwise it is about a hundred lines to implement directly, and doing so removes the dependency entirely.

Training and validation

Train / val clips	29,371 / 629
Best validation CTC loss	1.0212 (epoch 9)
Corpus	galgame speech only
Geometry eval	600 composites / 1,200 cores / 28,301 characters
Containment rate	98.75%
Median per-character context shift	1.9 ms (p90 64.5 ms)
Median first-character context shift	5.4 ms

"Context shift" is the movement of a character's predicted span when the same core utterance is embedded in different surrounding audio — an internal-consistency measure, not absolute onset accuracy.

Known limits. Accuracy above is established on clean speech. On noisy real-world audio the alignment scores are not worse (−1.44 to −1.50 versus −1.77 on the clean composites), but absolute onset accuracy on that domain has not been blind-audited. Mixed clean+noisy training variants were tried and rejected: blank discrimination degraded dose-dependently while alignment geometry stayed unchanged, so the noisy data cost the gate reading and bought nothing.

Treat the head as a measured improvement over proportional timing, not as ground truth.

Conversion

This repository was produced with:

uv run python -m tools.asr.convert_qwen3_asr_to_hf `
  --source-model-dir models/jaykwok-Qwen3-ASR-1.7B-JA-Anime-Galgame `
  --output-dir agents/temp/20260630_123000_qwen3_asr_hf_conversion `
  --template-repo Qwen/Qwen3-ASR-1.7B-hf `
  --max-shard-size 768MB

Notes

This model is specialized for Japanese anime / galgame style speech. It should be evaluated on your own data before production use.