FareedKhan/mixedbread-ai_deepset-mxbai-embed-de-large-v1_FareedKhan_prime_synthetic_data_2k_3_8

SentenceTransformer based on mixedbread-ai/deepset-mxbai-embed-de-large-v1

This is a sentence-transformers model finetuned from mixedbread-ai/deepset-mxbai-embed-de-large-v1 on the json dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: mixedbread-ai/deepset-mxbai-embed-de-large-v1
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 tokens
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("FareedKhan/mixedbread-ai_deepset-mxbai-embed-de-large-v1_FareedKhan_prime_synthetic_data_2k_3_8")
# Run inference
sentences = [
    '\nThe list you provided seems to be a mix of various chemical substances, some of which appear to be medications, others are chemical compounds, and a few could be substances from other fields (e.g., water treatment, food additives). To be more precise, it would be helpful to categorize them properly based on their common usage:\n\n### Medications and Drugs:\n- **Antibiotics**: Cefoxitin, Tobramycin, Amikacin\n- ** pain and inflammation relievers**: Benoxaprofen, Daptomycin, Ceftolozane, Salicylates (Benzydamine, Dexamethasone sodium phosphate)\n- **Intravenous fluids**: Magnesium trisilicate\n- **Antivirals**: Ribavirin, Inotersen\n- **Antibacterial agents**: Epirizole, Floctafenine, Flunixin\n- **Vaccines**: Vaborbactam, Brincidofovir, Adefovir\n- **Neuromodulators**: Cefatrizine, Bumadizone, Alminoprofen\n- **Cancer treatments**: Colistin, Nitrofurantoin, Sisomicin\n\n### Chemical Compounds:\n- **Salts and salts of acidity**: Fosfomycin, Azosemide, Mofebutazone\n- **Amino acids**: Phenylalanine, Nitrosalicylic',
    'Which drugs interact with the SERPINA1 gene/protein as carriers?',
    'Is there a regulatory function associated with the epidermal growth factor receptor or its interacting proteins in the control of genes or proteins that participate in the inactivation of fast sodium channels during Phase 1 of cardiac action potential propagation?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.3911
cosine_accuracy@3	0.4752
cosine_accuracy@5	0.495
cosine_accuracy@10	0.5545
cosine_precision@1	0.3911
cosine_precision@3	0.1584
cosine_precision@5	0.099
cosine_precision@10	0.0554
cosine_recall@1	0.3911
cosine_recall@3	0.4752
cosine_recall@5	0.495
cosine_recall@10	0.5545
cosine_ndcg@10	0.467
cosine_mrr@10	0.4398
cosine_map@100	0.4462

Training Details

Training Dataset

json

• Dataset: json
• Size: 1,814 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 3 tokens mean: 267.06 tokens max: 512 tokens	min: 15 tokens mean: 39.66 tokens max: 120 tokens

• Samples:

  positive	anchor
  Based on the provided information, it appears you are describing a complex biological system involving various molecules, drugs, diseases, and anatomical structures. Here's a breakdown: ### Key Entities 1. Molecules and Targets - Mentioned molecules include metabolites, phenols, and drugs, with specific functional groups related to their chemical properties. - Targets include enzymes (like acetyl-CoA carboxylase) and diseases causing various health conditions (e.g., Finnish type amyloidosis, lung cancer). 2. Functionality and Interactions - The molecules and drugs interact with various biological processes, pathways, and bodily systems.	Identify common genetic targets that interact with both N-(3,5-dibromo-4-hydroxyphenyl)benzamide and 1-Naphthylamine-5-sulfonic acid.
  The provided list appears to be a collection of gene symbols related to cancer. Gene symbols are used in genetics and molecular biology to identify genes. Each symbol is associated with a specific gene that plays a role in cellular functions, including cancer processes. When studying cancer, researchers often analyze these genes to understand their roles in tumor development, potential as targets for therapy, or as indicators for patient prognosis. For example, some genes listed are known oncogenes or tumor suppressor genes: - TP53: A tumor suppressor gene that when mutated can lead to uncontrolled cell growth. - P53, POLD1, PTEN: These are well-known tumor suppressors that help regulate cell division and DNA repair. - BRCA	Which anatomical structures lack expression of genes or proteins involved in the homogentisate degradation pathway?
  The gene in question appears to have a wide range of functions across various biological processes and body systems. It's involved in several key areas that regulate cellular responses, metabolic processes, and organ development. Here is a summary of its potential roles: 1. Cell Growth and Regulation: The gene contributes to growth control in cells, particularly in smooth muscle cells, and seems to influence cell proliferation, which is essential for tissue repair and development. 2. Nerve Function: It plays a role in functions like signal transduction, neurotrophin signaling, and regulation of neural activity, suggesting it’s involved in neural health and development. 3. Metabolic Processes: There is evidence linking	Identify genes or proteins that interact with angiotensin-converting enzyme 2 (ACE2) and are linked to a common phenotype or effect.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768
      ],
      "matryoshka_weights": [
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• learning_rate: 1e-05
• warmup_ratio: 0.1
• bf16: True
• tf32: False
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_map@100
0	0	-	0.3930
0.0441	10	1.18	-
0.0881	20	1.0507	-
0.1322	30	0.9049	-
0.1762	40	0.8999	-
0.2203	50	0.6519	-
0.2643	60	0.5479	-
0.3084	70	0.6493	-
0.3524	80	0.4706	-
0.3965	90	0.5459	-
0.4405	100	0.5692	-
0.4846	110	0.7834	-
0.5286	120	0.5341	-
0.5727	130	0.5343	-
0.6167	140	0.4865	-
0.6608	150	0.3942	-
0.7048	160	0.3578	-
0.7489	170	0.5158	-
0.7930	180	0.3426	-
0.8370	190	0.5789	-
0.8811	200	0.5271	-
0.9251	210	0.577	-
0.9692	220	0.5193	-
1.0	227	-	0.4354
1.0132	230	0.4598	-
1.0573	240	0.2735	-
1.1013	250	0.2919	-
1.1454	260	0.3206	-
1.1894	270	0.2851	-
1.2335	280	0.3899	-
1.2775	290	0.3279	-
1.3216	300	0.2155	-
1.3656	310	0.3471	-
1.4097	320	0.327	-
1.4537	330	0.229	-
1.4978	340	0.2902	-
1.5419	350	0.3216	-
1.5859	360	0.2902	-
1.6300	370	0.4527	-
1.6740	380	0.1583	-
1.7181	390	0.3144	-
1.7621	400	0.2573	-
1.8062	410	0.2309	-
1.8502	420	0.3475	-
1.8943	430	0.3082	-
1.9383	440	0.3176	-
1.9824	450	0.2104	-
2.0	454	-	0.4453
2.0264	460	0.2615	-
2.0705	470	0.1599	-
2.1145	480	0.1015	-
2.1586	490	0.2154	-
2.2026	500	0.1161	-
2.2467	510	0.2208	-
2.2907	520	0.2035	-
2.3348	530	0.1622	-
2.3789	540	0.1758	-
2.4229	550	0.2782	-
2.4670	560	0.303	-
2.5110	570	0.1787	-
2.5551	580	0.2221	-
2.5991	590	0.1686	-
2.6432	600	0.2522	-
2.6872	610	0.1334	-
2.7313	620	0.1102	-
2.7753	630	0.2499	-
2.8194	640	0.2648	-
2.8634	650	0.1859	-
2.9075	660	0.2385	-
2.9515	670	0.2283	-
2.9956	680	0.1126	-
3.0	681	-	0.4462

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.10
• Sentence Transformers: 3.1.1
• Transformers: 4.45.1
• PyTorch: 2.2.1+cu121
• Accelerate: 0.34.2
• Datasets: 3.0.1
• Tokenizers: 0.20.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}