Lychee-JevEmbed
Collection
3 items • Updated • 1
JevEmbed-KaLM-Embedding-V2.5
This KaLM-Embedding v2.5 checkpoint was fine-tuned with LoRA on all 1,601,157 training questions in JevEmbed-Data, then merged for standalone use. The adapter is in lora/.
The model produces 896-dimensional embeddings. JevEmbed applies the prompts and scoring in jevembed.yaml to make Choice, Score, and Noul decisions. For the tested setup, use transformers==4.51.0 and sentence-transformers==5.3.0; KaLM requires trust_remote_code=True.
Use with JevEmbed
Install JevEmbed, then run the CLI from this directory (model_name_or_path: .):
cd /path/to/JevEmbed-KaLM-Embedding-V2.5
python -m jevembed --config jevembed.yaml --input request.json
For the Python API, resolve the model directory explicitly so your working directory does not matter:
import json
from dataclasses import replace
from pathlib import Path
from jevembed import JevEmbed, ModelConfig
model_dir = Path("/path/to/JevEmbed-KaLM-Embedding-V2.5").resolve()
config = replace(
ModelConfig.load(model_dir / "jevembed.yaml"),
model_name_or_path=str(model_dir),
)
client = JevEmbed(config=config)
request = json.loads((model_dir / "request.json").read_text())
response = client.evaluate(request)
The configuration uses a 1,024-token limit with truncation, Choice/Score temperature 0.1, and Noul slope 10. For raw embeddings, load the merged weights with SentenceTransformer(model_dir, trust_remote_code=True).
Test-set performance
The base and released merged model were evaluated on the same 66,482 questions in the JevEmbed-Data test split. Both used BF16, identical prompts and scoring, and 1,024-token truncation. Accuracy uses hard labels; Noul MAE includes 22,363 hard and 1,641 soft targets.
| Metric | Base | After LoRA fine-tuning (merged) | Change |
|---|---|---|---|
| Overall hard-label accuracy (64,110) | 32.33% | 76.03% | +43.70 pp |
| Choice accuracy (17,487) | 28.61% | 71.05% | +42.44 pp |
| Score level accuracy (24,260) | 27.45% | 66.17% | +38.73 pp |
| Noul binary accuracy (22,363) | 40.52% | 90.61% | +50.08 pp |
| Score MAE (24,287; lower is better) | 0.9990 | 0.4943 | -0.5047 |
| Noul MAE (24,004; lower is better) | 0.5925 | 0.1365 | -0.4560 |
The merged weights also matched the LoRA adapter's embeddings within 3.6 × 10⁻⁷ maximum absolute difference on five FP32 inputs; see MERGE_MANIFEST.json.
Training
One epoch on 1,601,157 questions used four GPUs, per-GPU batch 64, accumulation 2 (effective batch 512), BF16, LoRA rank 64, alpha 32, dropout 0.05, Q/K/V targets, learning rate 2 × 10⁻⁴, and 10% warmup. Inputs were truncated at 1,024 tokens. The final adapter is step 3,128.
The training loss uses query–candidate cosine similarities:
- Choice and categorical Score: cross-entropy against the target distribution after softmax with temperature 0.1. Hard labels are one-hot; soft labels remain soft.
- Continuous Score: squared error between the target score and the probability-weighted expected level, normalized by the number of level intervals.
- Noul: binary cross-entropy with logits. The logit is 10 times one cosine similarity without criteria, or 10 times the true-minus-false cosine difference with criteria.
The training objective is the mean of per-question losses in each batch. The implementation is in JevEmbed/src/jevembed/training/objective.py.
The base revision is 753c6fe26abc20a32aeb162003aa03457d15db2f. The training dataset has mixed source licenses; consult its source report when choosing release terms.
Test-set examples
Choice: target tracking
This illustrative Choice case comes from the JevEmbed-Data test split (Open-Jev Doom tracking; ID openjev:vizdoom-basic-v1:190919:id:205:0:choice).
Save the exact request as request.json:
{
"model": "jevembed-kalm-embedding-v2.5",
"state": {
"action_window_tics": 4,
"ammo": 50.0,
"elapsed_tics": 14,
"health": 100.0,
"movement": {"forward_units_per_tic": 0.0, "lateral_units_per_tic": 0.0},
"nearest_visible_enemy": {
"distance_units": 388.67414,
"name": "Cacodemon",
"relative_bearing_deg": 8.894711
},
"player": {"heading_deg": 0.0, "x_units": -384.0, "y_units": 32.0},
"visible_enemy_count": 1
},
"questions": {
"decision": {
"type": "choice",
"instructions": "Which lateral movement does the momentum-aware tracking policy choose to align with the nearest visible target?",
"criteria": {
"Strafe left": null,
"Strafe right": null,
"Hold lateral movement": null
}
}
}
}
JevEmbed-KaLM-Embedding-V2.5 selects Strafe left, matching the test label. Its Choice probabilities are:
| Strafe left | Strafe right | Hold lateral movement |
|---|---|---|
| 0.8763 | 0.1096 | 0.0140 |
Score: lining up a Doom shot
Three Open-Jev Doom states from the JevEmbed-Data test split (group ID openjev:vizdoom-basic-v1:190919:id:55) show the player strafing as a visible Cacodemon moves toward the crosshair. The relative bearing narrows from 8.57° to 1.74°. Score levels 0–4 rate shot alignment, not game reward.
The final frame (tick 42; ID openjev:vizdoom-basic-v1:190919:id:55:7:score) is a runnable request. Save it as request.json:
{
"model": "jevembed-kalm-embedding-v2.5",
"state": {
"action_window_tics": 4,
"ammo": 50.0,
"elapsed_tics": 42,
"health": 100.0,
"movement": {
"forward_units_per_tic": 0.0,
"lateral_units_per_tic": 0.690659
},
"nearest_visible_enemy": {
"distance_units": 384.176958,
"name": "Cacodemon",
"relative_bearing_deg": 1.739094
},
"player": {
"heading_deg": 0.0,
"x_units": -384.0,
"y_units": 120.630661
},
"visible_enemy_count": 1
},
"questions": {
"decision": {
"type": "score",
"instructions": "Rate the current shot alignment using the supplied ordered suitability grades. These grades are an aiming heuristic, not expected game reward.",
"criteria": [
"Cannot usefully aim a shot: no live visible target, no ammunition, dead player, or absolute target bearing above 22 degrees.",
"Very poor alignment: live visible target and ammunition, player alive, absolute target bearing above 12 and at most 22 degrees.",
"Coarse alignment: live visible target and ammunition, player alive, absolute target bearing above 6 and at most 12 degrees.",
"Near alignment: live visible target and ammunition, player alive, absolute target bearing above 3 and at most 6 degrees.",
"Aligned for the script's shot: live visible target and ammunition, player alive, absolute target bearing at most 3 degrees."
]
}
}
}
JevEmbed-KaLM-Embedding-V2.5 predicts the test label at each moment:
| Game tick | Relative bearing | Test level | Model level | Model probability of that level |
|---|---|---|---|---|
| 26 | 8.57° | 2 (coarse) | 2 | 0.6924 |
| 34 | 3.05° | 3 (near) | 3 | 0.5907 |
| 42 | 1.74° | 4 (aligned) | 4 | 0.8995 |
Noul: a safe move into the snake's tail
This Open-Jev test case (ID openjev:snake-v1:677b4c40f40fd3be2ca642ed:collision_up) asks whether moving up causes a collision. The destination is the tail's current cell, but the snake does not eat food on this move, so the tail vacates it. The test label is no collision.
{
"model": "jevembed-kalm-embedding-v2.5",
"state": {
"coordinates": "x increases right; y increases down; walls are outside the board",
"direction": "right",
"food": [3, 2],
"game": "snake",
"height": 6,
"snake_head_first": [[1, 1], [0, 1], [0, 0], [1, 0]],
"width": 6
},
"questions": {
"decision": {
"type": "noul",
"instructions": "Will moving up cause an immediate wall or body collision? The tail vacates unless food is eaten."
}
}
}
JevEmbed-KaLM-Embedding-V2.5 returns 0.1866 for the Noul probability of a collision.
Citation
If you find JevEmbed-KaLM-Embedding-V2.5 useful, please consider citing the following papers:
@misc{zhao2025kalmembeddingv2,
title={KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model},
author={Xinping Zhao and Xinshuo Hu and Zifei Shan and Shouzheng Huang and Yao Zhou and Xin Zhang and Zetian Sun and Zhenyu Liu and Dongfang Li and Xinyuan Wei and Youcheng Pan and Yang Xiang and Meishan Zhang and Haofen Wang and Jun Yu and Baotian Hu and Min Zhang},
year={2025},
eprint={2506.20923},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2506.20923},
}
@misc{hu2025kalmembedding,
title={KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model},
author={Xinshuo Hu and Zifei Shan and Xinping Zhao and Zetian Sun and Zhenyu Liu and Dongfang Li and Shaolin Ye and Xinyuan Wei and Qian Chen and Baotian Hu and Haofen Wang and Jun Yu and Min Zhang},
year={2025},
eprint={2501.01028},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2501.01028},
}
- Downloads last month
- 142
Model tree for HIT-TMG/JevEmbed-KaLM-Embedding-V2.5
Base model
Qwen/Qwen2-0.5B