--- tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:7278 - loss:MultipleNegativesRankingLoss base_model: sentence-transformers/LaBSE widget: - source_sentence: '- Н-да!' sentences: - Акана — хохавава. - А о-отэнчя тихэс устя тэ и гыя дрэ фэлда ни прэ конэстэ на дыкхи. - — Аи! - source_sentence: Чего в ней! sentences: - >- И гарави пэстыр пэскиро доґалыпэ ёв пролыджия времё жыко бельвель, то ушты и псири пиро прахо, то пасиндой. - Со дрэ лэстэ! - Кошэна ман прэ грэда — розмэклы. - source_sentence: Ладно. sentences: - Задынэ андрэ грэн. - Шукар. - Адай ёв тырдыя и порискирдя шэро, пучибныткэс подыкхья прэ Макаростэ. - source_sentence: >- Его старуха была громадная, рослая старуха, и ему нести ее было бы еще труднее. sentences: - >- И Макаро газдыяпэ, ракири пэскэ тэло ушт, со дажэ после мулыпэ на дэн манушэскэ тэ пасёл. - >- Лэскири пхури сыс дрэван бари, зоралы пхури, и лэскэ тэ лыджял ла явэлас бы инкэ пхарыдыр. - Ёв лыя тэ ракир одолэстыр, со на камэл тэ джял ко трапэзнико дро грая. - source_sentence: Владимир разорвал их, не читая. sentences: - >- Коли, хамэ, ту зорало тэ явэс тэ умарав тут могискиравас, а то и адава нашты. - Мандэ кэ ёв сыс баро уважэниё и ёв ман дрэван камья. - Владимиро розрискирдя лэн на гины. pipeline_tag: sentence-similarity library_name: sentence-transformers metrics: - src2trg_accuracy - trg2src_accuracy - mean_accuracy model-index: - name: SentenceTransformer based on sentence-transformers/LaBSE results: - task: type: translation name: Translation dataset: name: ruska roma validation type: ruska-roma-validation metrics: - type: src2trg_accuracy value: 0.9023485784919654 name: Src2Trg Accuracy - type: trg2src_accuracy value: 0.892459826946848 name: Trg2Src Accuracy - type: mean_accuracy value: 0.8974042027194067 name: Mean Accuracy license: mit --- # SentenceTransformer based on sentence-transformers/LaBSE This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [sentence-transformers/LaBSE](https://huggingface.co/sentence-transformers/LaBSE). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval. Model was fine-tuned Ruska Romani on Russian language pairs. Ruska Romani is the dialect of Romani language attributed to Ruska Roma, the largest subgroup of Romani people in Russia. Model was trained and evaluated on parallel pairs data data from the [Russian National Corpus](https://ruscorpora.ru/en/search?search=CgkyBwgFEgNybWw%253D). The data curation process is described in the article [The Parallel Corpus of Russian and Ruska Romani Languages](https://aclanthology.org/2024.fieldmatters-1.1/). Please refer to that paper in any publications where the model was used. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [sentence-transformers/LaBSE](https://huggingface.co/sentence-transformers/LaBSE) - **Maximum Sequence Length:** 256 tokens - **Output Dimensionality:** 768 dimensions - **Similarity Function:** Cosine Similarity - **Supported Modality:** Text ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'}) (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'cls', 'include_prompt': True}) (2): Dense({'in_features': 768, 'out_features': 768, 'bias': True, 'activation_function': 'torch.nn.modules.activation.Tanh', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'}) (3): Normalize({}) ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("midwestcyr/labse-ruskaromani") # Run inference sentences = [ 'Владимир разорвал их, не читая.', 'Владимиро розрискирдя лэн на гины.', 'Мандэ кэ ёв сыс баро уважэниё и ёв ман дрэван камья.', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities) # tensor([[1.0000, 0.8380, 0.0814], # [0.8380, 1.0000, 0.1824], # [0.0814, 0.1824, 1.0000]]) ``` ## Evaluation ### Metrics #### Translation * Dataset: `ruska-roma-validation` * Evaluated with [TranslationEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.sentence_transformer.evaluation.TranslationEvaluator) | Metric | Value | |:------------------|:-----------| | src2trg_accuracy | 0.9023 | | trg2src_accuracy | 0.8925 | | **mean_accuracy** | **0.8974** | ## Training Details ### Training Dataset #### Unnamed Dataset * Size: 7,278 training samples * Columns: sentence_0, sentence_1, and label * Approximate statistics based on the first 100 samples: | | sentence_0 | sentence_1 | label | |:---------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:--------------------------------------------------------------| | type | string | string | float | | modality | text | text | | | details | | | | * Samples: | sentence_0 | sentence_1 | label | |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------| | Не бойтесь, государь милостив, я буду просить его. Он нас не обидит. Мы все его дети. А как ему за вас будет заступиться, если вы станете бунтовать и разбойничать». | На дарэнте, тагари куч, мэ лава тэ мангав лэс — ёв амэн на помэкэла — амэ сарэ лэскирэ чявэ — а сыр лэскэ пал тумэндэ тэ затэрдёл, коли тумэ лэна тэ газдэн бунто и разбоё. | 1.0 | | Можно было видеть, что мужчина высок и стоит у весла, широко расставив ноги, вполоборота к кругленькой, маленькой женщине, прислонившейся грудью к другому веслу, саженях в полутора от первого. | Могискирдо сыс тэ роздыкхэс, со гаджё учё и тэрдо пашо вёсло, буґлэс росчеви ґэра, дро паш обрисибэ кэ крэнглинько джювлы, сави припасия колынэса кэ вавир вёсло, надур екх екхэстыр. | 1.0 | | Мазурка кончилась, хозяева просили гостей к ужину, но полковник Б. отказался, сказав, что ему надо завтра рано вставать, и простился с хозяевами. | Мазурка кончисалыя, хулая мангнэ гостен ко хабэ, нэ полковнико Б. отпхэндяпэ и пхэндя, со лэскэ трэбинэ атася злокоса тэ уштэс и простиндяпэ хуланца. | 1.0 | * Loss: [MultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters: ```json { "scale": 20.0, "similarity_fct": "cos_sim", "gather_across_devices": false, "directions": [ "query_to_doc" ], "partition_mode": "joint", "hardness_mode": null, "hardness_strength": 0.0 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `per_device_train_batch_size`: 16 - `fp16`: True - `per_device_eval_batch_size`: 16 - `multi_dataset_batch_sampler`: round_robin #### All Hyperparameters
Click to expand - `per_device_train_batch_size`: 16 - `num_train_epochs`: 3 - `max_steps`: -1 - `learning_rate`: 5e-05 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: None - `warmup_steps`: 0 - `optim`: adamw_torch_fused - `optim_args`: None - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `optim_target_modules`: None - `gradient_accumulation_steps`: 1 - `average_tokens_across_devices`: True - `max_grad_norm`: 1 - `label_smoothing_factor`: 0.0 - `bf16`: False - `fp16`: True - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `use_liger_kernel`: False - `liger_kernel_config`: None - `use_cache`: False - `neftune_noise_alpha`: None - `torch_empty_cache_steps`: None - `auto_find_batch_size`: False - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `include_num_input_tokens_seen`: no - `log_level`: passive - `log_level_replica`: warning - `disable_tqdm`: False - `project`: huggingface - `trackio_space_id`: None - `trackio_bucket_id`: None - `trackio_static_space_id`: None - `per_device_eval_batch_size`: 16 - `prediction_loss_only`: True - `eval_on_start`: False - `eval_do_concat_batches`: True - `eval_use_gather_object`: False - `eval_accumulation_steps`: None - `include_for_metrics`: [] - `batch_eval_metrics`: False - `save_only_model`: False - `save_on_each_node`: False - `enable_jit_checkpoint`: False - `push_to_hub`: False - `hub_private_repo`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_always_push`: False - `hub_revision`: None - `load_best_model_at_end`: False - `ignore_data_skip`: False - `restore_callback_states_from_checkpoint`: False - `full_determinism`: False - `seed`: 42 - `data_seed`: None - `use_cpu`: False - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `parallelism_config`: None - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `dataloader_prefetch_factor`: None - `remove_unused_columns`: True - `label_names`: None - `train_sampling_strategy`: random - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `ddp_static_graph`: None - `ddp_backend`: None - `ddp_timeout`: 1800 - `fsdp`: None - `fsdp_config`: None - `deepspeed`: None - `debug`: [] - `skip_memory_metrics`: True - `do_predict`: False - `resume_from_checkpoint`: None - `warmup_ratio`: None - `local_rank`: -1 - `prompts`: None - `batch_sampler`: batch_sampler - `multi_dataset_batch_sampler`: round_robin - `router_mapping`: {} - `learning_rate_mapping`: {}
### Training Logs | Epoch | Step | Training Loss | ruska-roma-validation_mean_accuracy | |:------:|:----:|:-------------:|:-----------------------------------:| | 1.0 | 455 | - | 0.8535 | | 1.0989 | 500 | 0.1608 | 0.8684 | | 2.0 | 910 | - | 0.8875 | | 2.1978 | 1000 | 0.0325 | 0.8956 | | 3.0 | 1365 | - | 0.8974 | ### Training Time - **Training**: 11.7 minutes ### Framework Versions - Python: 3.12.13 - Sentence Transformers: 5.6.0 - Transformers: 5.12.1 - PyTorch: 2.11.0+cu128 - Accelerate: 1.14.0 - Datasets: 4.0.0 - Tokenizers: 0.22.2 ## Citation ### BibTeX #### Ruska Romani and Russian Parallel Corpus ```bibtex @inproceedings{koncha-etal-2024-parallel, title = "The Parallel Corpus of {R}ussian and Ruska {R}omani Languages", author = "Koncha, Kirill and Kukanova, Abina and Tatiana, Kazakova and Rozovskaya, Gloria", editor = "Serikov, Oleg and Voloshina, Ekaterina and Postnikova, Anna and Muradoglu, Saliha and Le Ferrand, Eric and Klyachko, Elena and Vylomova, Ekaterina and Shavrina, Tatiana and Tyers, Francis", booktitle = "Proceedings of the Third Workshop on NLP Applications to Field Linguistics", month = aug, year = "2024", address = "Bangkok, Thailand", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2024.fieldmatters-1.1/", doi = "10.18653/v1/2024.fieldmatters-1.1", pages = "1--5" } ``` #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{oord2019representationlearningcontrastivepredictive, title={Representation Learning with Contrastive Predictive Coding}, author={Aaron van den Oord and Yazhe Li and Oriol Vinyals}, year={2019}, eprint={1807.03748}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/1807.03748}, } ```