johnyy212/moe-girl-v2

SentenceTransformer based on BAAI/bge-small-zh-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-small-zh-v1.5 on the json dataset. It maps sentences & paragraphs to a 512-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-small-zh-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 512 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • json

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 512, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("johnyy212/moe-girl-v2")
# Run inference
sentences = [
    '熟女、A型、召唤能力者',
    '角色：文豪野犬:尾崎红叶\n本名：尾崎、红叶、Ozaki Kouyou\n别名：红叶大姐、红叶君\n声优：小清水亚美\n发色：红\n瞳色：红\n生日：1月10日\n星座：摩羯\n血型：A\n萌点：和服、遮眼发、超能力者、熟女',
    '文豪野犬:尾崎红叶',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 512]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.3878, -0.0819],
#         [ 0.3878,  1.0000,  0.0236],
#         [-0.0819,  0.0236,  1.0000]])

Training Details

Training Dataset

json

• Dataset: json
• Size: 56,574 training samples
• Columns: anchor, positive, and char_name
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	char_name
  type	string	string	string
  details	min: 4 tokens mean: 11.84 tokens max: 34 tokens	min: 12 tokens mean: 80.61 tokens max: 295 tokens	min: 3 tokens mean: 7.38 tokens max: 48 tokens

• Samples:

  anchor	positive	char_name
  井上和彦、早水理沙	角色：黑田 本名：黒田 别名：小黑、幼年时 声优：小野友树、小野友树配音角色、井上和彦、井上和彦配音角色、早水理沙、早水理沙配音角色 发色：黑 瞳色：灰蓝 萌点：现在、工程师、浴衣、绷带、遮眼发、烟斗、人偶师、大叔、毒舌、反差萌、贪财、心理创伤、青年时、幼驯染、搞事、天才、傲娇、腹黑、强气、双向暗恋、单马尾、天然疯、武士、蔷薇	黑田
  平田裕香	角色：寺门卷子 本名：寺门 巻子 声优：平田裕香 萌点：同级生	寺门卷子
  蝴蝶结、高跟鞋、麦克风、制服、伞、乙女	角色：瞳(LIVE A HERO) 本名：绫歌瞳 别名：仁美、爱抖露 声优：夏怜 发色：黑 瞳色：渐变 萌点：麦克风、乙女、歌手、偶像、暴力女、短发、丸子头、耳环、制服、伞、比基尼、外套半脱、手套、蝴蝶结、百褶裙、长筒袜、四分之三袜、高跟鞋、元气、光	瞳(LIVE A HERO)

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

json

• Dataset: json
• Size: 3,403 evaluation samples
• Columns: anchor, positive, and char_name
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	char_name
  type	string	string	string
  details	min: 9 tokens mean: 15.96 tokens max: 30 tokens	min: 30 tokens mean: 80.32 tokens max: 512 tokens	min: 3 tokens mean: 7.28 tokens max: 26 tokens

• Samples:

  anchor	positive	char_name
  小天使、发珠、料理达人、傲娇、长直	角色：云雀丘琉璃 本名：云雀丘瑠璃 别名：云雀、ヒバリ、本场琉璃 声优：白石晴香 发色：琉璃色黑 瞳色：绿 生日：7月9日 星座：巨蟹 血型：B 萌点：不幸、单恋、微傲娇、小天使、美少女、认真、人妻、料理达人、长直、吊眼、发珠、高跟鞋、绝对领域	云雀丘琉璃
  黑瞳、银发、特工、B型	角色：风间让二 本名：风间、譲二 别名：二叔 声优：渡哲也配音角色 发色：灰 瞳色：黑 年龄：64 血型：B 萌点：特工、弟弟、长者、双胞胎	风间让二
  红发、机郎、傲娇、蓝瞳	角色：爱因兹希 本名：爱因兹希 别名：全连结指挥体 声优：诹访部顺一 发色：红 瞳色：蓝 萌点：机郎、傲娇、基佬	爱因兹希

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 64
• learning_rate: 2e-05
• num_train_epochs: 5
• warmup_steps: 0.1
• fp16: True
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 8
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.1
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss
0.1131	100	2.4777	-
0.2262	200	1.2126	-
0.3394	300	0.9734	-
0.4525	400	0.8498	-
0.5656	500	0.8329	-
0.6787	600	0.7920	-
0.7919	700	0.7039	-
0.9050	800	0.7013	-
1.0	884	-	0.1582
1.0181	900	0.6741	-
1.1312	1000	0.6307	-
1.2443	1100	0.6224	-
1.3575	1200	0.6090	-
1.4706	1300	0.5913	-
1.5837	1400	0.5812	-
1.6968	1500	0.5711	-
1.8100	1600	0.5614	-
1.9231	1700	0.5973	-
2.0	1768	-	0.1351
2.0362	1800	0.5275	-
2.1493	1900	0.5139	-
2.2624	2000	0.4939	-
2.3756	2100	0.5008	-
2.4887	2200	0.5044	-
2.6018	2300	0.4932	-
2.7149	2400	0.5083	-
2.8281	2500	0.5151	-
2.9412	2600	0.4993	-
3.0	2652	-	0.1247
3.0543	2700	0.4770	-
3.1674	2800	0.4751	-
3.2805	2900	0.4676	-
3.3937	3000	0.4480	-
3.5068	3100	0.4579	-
3.6199	3200	0.4581	-
3.7330	3300	0.4524	-
3.8462	3400	0.4532	-
3.9593	3500	0.4493	-
4.0	3536	-	0.1188
4.0724	3600	0.4443	-
4.1855	3700	0.4296	-
4.2986	3800	0.4117	-
4.4118	3900	0.4397	-
4.5249	4000	0.4223	-
4.6380	4100	0.4448	-
4.7511	4200	0.4304	-
4.8643	4300	0.4494	-
4.9774	4400	0.4271	-
5.0	4420	-	0.1192

• The bold row denotes the saved checkpoint.

Training Time

• Training: 14.9 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.1
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.8.5
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}