--- license: apache-2.0 base_model: - meta-llama/Meta-Llama-3-8B-Instruct language: - fr tags: - BEL - retrieval - entity-retrieval - named-entity-disambiguation - entity-disambiguation - named-entity-linking - entity-linking - text2text-generation - biomedical - healthcare - synthetic-data - causal-lm - llm library_name: transformers finetuning_task: - text2text-generation - entity-linking metrics: - recall model-index: - name: syncabel-quaero-emea-8b results: - task: type: entity-linking dataset: type: structured_dataset # Required field name: bigbio/quaero config: quaero_emea_bigbio_kb metrics: - type: recall value: 0.790 --- # SynCABEL: Synthetic Contextualized Augmentation for Biomedical Entity Linking ## SynCABEL **SynCABEL** is a novel framework that addresses data scarcity in biomedical entity linking through **synthetic data generation**. The method, introduced in our [paper] ## SynCABEL (QUAERO-EMEA Edition) This is a **finetuned version of LLaMA-3-8B** trained on **QUAERO-EMEA** using **[SynthQUAERO](https://huggingface.co/datasets/AnonymousARR42/SynCABEL)** (our synthetic dataset generated via the SynCABEL framework). | | | |--------|---------| | **Base Model** | [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) | | **Training Data** | [QUAERO-EMEA](https://huggingface.co/datasets/bigbio/quaero) (real) + [SynthQUAERO](https://huggingface.co/datasets/AnonymousARR42/SynCABEL) (synthetic) | | **Fine-tuning** | [Supervised Fine-Tuning](https://huggingface.co/docs/trl/en/sft_trainer) | ## Training Data Composition The model is trained on a mix of **human-annotated** and **synthetic** data: ``` QUAERO-EMEA (human) : 7,159 mentions SynthQUAERO (synthetic) : 396,914 mentions ``` To ensure balanced learning, **human data is upsampled during training** so that each batch contains: ``` 50% human-annotated data 50% synthetic data ``` In other words, although SynthMM is larger, the model always sees a **1:1 ratio of human to synthetic examples**, preventing synthetic data from overwhelming human supervision. ## Usage ### Loading ```python import torch from transformers import AutoModelForCausalLM # Load the model (requires trust_remote_code for custom architecture) model = AutoModelForCausalLM.from_pretrained( "AnonymousARR42/SynCABEL_QUAERO_EMEA", trust_remote_code=True, device_map="auto" ) ``` ### Unconstrained Generation ```python # Let the model freely generate concept names sentences = [ "Le patient atteint de [Embolie pulmonaire massive]{Disorders} a présenté des signes de détresse respiratoire.", "Le patient reçoit régulièrement des [corticoïdes]{Chemicals & Drugs} pour soulager les symptômes cutanés." ] results = model.sample( sentences=sentences, constrained=False, num_beams=2, ) for i, beam_results in enumerate(results): print(f"Input: {sentences[i]}") mention = beam_results[0]["mention"] print(f"Mention: {mention}") for j, result in enumerate(beam_results): print( f"Beam {j+1}:\n" f"Predicted concept name:{result['pred_concept_name']}\n" f"Predicted code: {result['pred_concept_code']}\n" f"Beam score: {result['beam_score']:.3f}\n" ) ``` **Output:** ``` Input: Le patient atteint de [Embolie pulmonaire massive]{Disorders} a présenté des signes de détresse respiratoire. Mention: Embolie pulmonaire massive Beam 1: Predicted concept name:Embolie pulmonaire massive Predicted code: NO_CODE Beam score: 0.820 Beam 2: Predicted concept name:Grade 5 Pulmonary embolus Predicted code: NO_CODE Beam score: 0.709 Input: Le patient reçoit régulièrement des [corticoïdes]{Chemicals & Drugs} pour soulager les symptômes cutanés. Mention: corticoïdes Beam 1: Predicted concept name:Corticoïdes Predicted code: C0001617 Beam score: 0.941 Beam 2: Predicted concept name:Corticosteroid Predicted code: NO_CODE Beam score: 0.520 ``` ### Constrained Decoding (Recommended for Entity Linking) ```python # Constrained to valid biomedical concepts sentences = [ "Le patient atteint de [Embolie pulmonaire massive]{Disorders} a présenté des signes de détresse respiratoire.", "Le patient reçoit régulièrement des [corticoïdes]{Chemicals & Drugs} pour soulager les symptômes cutanés." ] results = model.sample( sentences=sentences, constrained=True, num_beams=2, ) for i, beam_results in enumerate(results): print(f"Input: {sentences[i]}") mention = beam_results[0]["mention"] print(f"Mention: {mention}") for j, result in enumerate(beam_results): print( f"Beam {j+1}:\n" f"Predicted concept name:{result['pred_concept_name']}\n" f"Predicted code: {result['pred_concept_code']}\n" f"Beam score: {result['beam_score']:.3f}\n" ) ``` **Output:** ``` Input: Le patient atteint de [Embolie pulmonaire massive]{Disorders} a présenté des signes de détresse respiratoire. Mention: Embolie pulmonaire massive Beam 1: Predicted concept name:Embolie pulmonaire massive aiguë Predicted code: C0340535 Beam score: 0.460 Beam 2: Predicted concept name:Embolie pulmonaire Predicted code: C0034065 Beam score: 0.267 Input: Le patient reçoit régulièrement des [corticoïdes]{Chemicals & Drugs} pour soulager les symptômes cutanés. Mention: corticoïdes Beam 1: Predicted concept name:Corticoïdes Predicted code: C0001617 Beam score: 0.941 Beam 2: Predicted concept name:Corticosteroid therapy Predicted code: C1313874 Beam score: 0.320 ``` ## Scores Entity linking performance (Recall@1) on biomedical benchmarks. The best results are shown in **bold**, the second-best results are underlined, and the "Average" column reports the mean score across the four benchmarks. | Model | MM-ST21PV
(english) | QUAERO-MEDLINE
(french) | QUAERO-EMEA
(french) | SPACCC
(spanish) | Avg. | | :--- | :---: | :---: | :---: | :---: | :---: | | SciSpacy | 53.8 | 40.5 | 37.1 | 13.2 | 36.2 | | SapBERT | 51.1 | 50.6 | 49.8 | 33.9 | 46.4 | | CODER-all | 56.6 | 58.7 | 58.1 | 43.7 | 54.3 | | SapBERT-all | 64.6 | 74.7 | 67.9 | 47.9 | 63.8 | | ArboEL | 74.5 | 70.9 | 62.8 | 49.0 | 64.2 | | mBART-large | 65.5 | 61.5 | 58.6 | 57.7 | 60.8 | | + Guided inference | 70.0 | 72.8 | 71.1 | 61.8 | 68.9 | | **+ SynCABEL (Our method)** | 71.5 | 77.1 | 75.3 | 64.0 | 72.0 | | Llama-3-8B | 69.0 | 66.4 | 65.5 | 59.9 | 65.2 | | + Guided inference | 74.4 | 77.5 | 72.9 | 64.2 | 72.3 | | **+ SynCABEL (Our method)** | **75.4** | **79.7** | **79.0** | **67.0** | **75.3** | Here, we provide the source repositories for the baselines: - [**SciSpacy**](https://github.com/allenai/scispacy) - [**SapBERT**](https://hf.co/cambridgeltl/SapBERT-from-PubMedBERT-fulltext) - [**SapBERT-all**](https://hf.co/cambridgeltl/SapBERT-UMLS-2020AB-all-lang-from-XLMR) - [**CODER-all**](https://hf.co/GanjinZero/coder_all) - [**ArboEL**](https://github.com/dhdhagar/arboEL) - [**mBART-large**](https://hf.co/facebook/mbart-large-50) - [**LLaMA-3-8B**](https://hf.co/meta-llama/Meta-Llama-3-8B-Instruct). ### Speed and Memory | Model | Model (GB) | Cand. (GB) | Speed (/s) | |--------------|------------|------------|------------| | SapBERT | 2.1 | 20.1 | **575.5** | | ArboEL | **1.2** | 7.1 | 38.9 | | mBART | 2.3 | **5.4** | 51.0 | | Llama-3-8B | 28.6 | **5.4** | 19.1 | *Measured on single H100 GPU, constrained decoding*