cyh2236/LRAT-Qwen3-Embedding-0.6B-clean-v1

LRAT Qwen3-Embedding-0.6B Clean-v1

This repository contains a complete Qwen3-Embedding-0.6B-compatible dense retriever checkpoint prepared for the XIR/LRAT competition. It is a full model checkpoint, not a LoRA adapter.

Model details

• Base model: Qwen/Qwen3-Embedding-0.6B
• Architecture: Qwen3 (595,776,512 parameters)
• Embedding dimension: 1024
• Pooling: last non-padding token / EOS
• Similarity: cosine similarity after L2 normalization
• Query maximum length used for training: 512
• Passage maximum length used for training: 512
• Tokenizer vocabulary and token-to-ID mapping: unchanged from the base model

The weights are stored in FP32 because full-parameter DDP training retained FP32 master parameters. Evaluation may load them with BF16 without changing the architecture or embedding dimension.

Query format

Queries must use this exact instruction:

Instruct: Given a web search query, retrieve relevant passages that answer the query
Query:{query}

Passages are encoded without an instruction. Apply last-token pooling and L2 normalization to both query and passage embeddings.

Training provenance

• Dataset: official Yuqi-Zhou/LRAT-Train
• Dataset revision: 26df12d39ec3eb65a30a61d86664931bb0fa4359
• Source training pairs: 96,504 official rows
• Traceable clean-v1 training input: 96,503 rows; the single empty-query row was excluded
• Group size: one sampled positive and nine negatives
• Objective: weighted multi-positive InfoNCE using the official reweight_rate
• Temperature: 0.02
• Epochs: 2
• Global contrastive batch: 256 queries / 2,560 passages
• Precision during forward/backward: BF16
• Seed: 2025

Only the official LRAT training pairs, official agent trajectories, the competition-allowed offline corpus, and the original Qwen checkpoint were used. No external retrieval dataset, external teacher, external API relabeling, or untraceable synthetic data was used.

Transformers example

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model_path = "your-namespace/LRAT-Qwen3-Embedding-0.6B-clean-v1"
tokenizer = AutoTokenizer.from_pretrained(model_path, padding_side="left")
model = AutoModel.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    attn_implementation="sdpa",
).cuda().eval()

task = "Given a web search query, retrieve relevant passages that answer the query"
texts = [
    f"Instruct: {task}\nQuery:What is LRAT?",
    "LRAT trains retrievers from agent interaction trajectories.",
]
batch = tokenizer(
    texts,
    padding=True,
    truncation=True,
    max_length=512,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    hidden = model(**batch).last_hidden_state
    embeddings = F.normalize(hidden[:, -1].float(), dim=-1)

score = embeddings[0] @ embeddings[1]

Validation

The packaged checkpoint was checked with AutoConfig, AutoTokenizer, strict AutoModel loading, and the LRAT Tevatron DenseModel loader. Parameters and sample embeddings were finite, and normalized embeddings had unit L2 norm. No competition leaderboard score is claimed in this model card.