Putting DoctoBERT to Work: A Practical Guide

We released a pretrained medical encoder. Here is how to run it, and how to finetune it for the tasks you actually have.

📄 Paper | 💻 Code | 🌐 FineMed | 🩺 DoctoBERT

Introduction

Much of clinical NLP is about reading text, not generating it. You have a consultation note and you need the drugs, the pathologies, and the symptoms out of it. You have a pile of documents to group by specialty, or a patient record where a single passage matters. For all of this, encoders are often the right tool.

An encoder is a BERT-style model that reads the whole text at once and turns every token into a vector that captures its meaning, in the context of the text. A word like "patient" will have a different vector depending on whether it appears in "patient medical file" or "patient was treated". This token representation is what makes encoders effective on many tasks, such as entity recognition and classification tasks. The encoder needs to be fine-tuned to adjust those token representations by giving enough task-specific examples. And that is why we are going to walk you through in the blog.

Why DoctoBERT?

For an ad-hoc question or a quick exploration, a large language model is usually the simplest choice. For a specific task that you run at scale, a small encoder brings a lot of advantages: it's cheaper and faster to run, it's easier to evaluate on your own test data, and it's small enough to run on your own hardware. In healthcare especially, that last point matters as it means that patient records never leave your network. These advantages come from how encoders work. An encoder produces its output in a single forward pass, while an autoregressive model builds its answer one token at a time, making them more efficient in terms of latency and compute.

DoctoBERT brings that tool to French medical text. Its models are pretrained from scratch on FineMed, a medical corpus we curated from the open web. Sourcing from the web gives it scale and diversity a small hand-built corpus cannot match, across many writing registers and medical entities. That scale and diversity, plus a targeted data-curation pipeline, contribute to DoctoBERT's state-of-the-art results on the public DrBenchmark and a real-world clinical NER task. For the story of how data is curated, see this blog post.

The rest of this blog will walk you through three use-cases:

• Named entity recognition
• Classification
• Retrieval

We will show how to run the models and finetune them on these tasks ans see how the model performs.

Kick the tires

Before any finetuning, it is worth checking that the model actually speaks medical French. Here we run a couple of masked clinical sentences through a general model (CamemBERT) and a medical one (DoctoModernBERT-fr, one of the DoctoBERT models), and compare what each proposes for the blank:

from transformers import pipeline

# Two masked clinical sentences
sentences = [
    "Devant une douleur thoracique, il faut évoquer un {mask}.",
    "On débute une antibiothérapie par {mask}.",
]

# Each model has its own mask token, so we insert fill.tokenizer.mask_token.
names = ["camembert-base", "doctolib-lab/doctomodernbert-fr-base"]
fillers = {name: pipeline("fill-mask", model=name) for name in names}

for sentence in sentences:
    print(sentence.replace("{mask}", "[MASK]"))
    for name, fill in fillers.items():
        masked = sentence.format(mask=fill.tokenizer.mask_token)
        top = [p["token_str"].strip() for p in fill(masked)[:5]]
        print(f"  {name.split('/')[-1]}")
        print(f"    {', '.join(top)}")
    print()

Output:

> Devant une douleur thoracique, il faut évoquer un [MASK].
>   camembert-base
>     traitement, médecin, diagnostic, médicament, traumatisme
>   doctomodernbert-fr-base
>     pneumothorax, infarctus, SCA, IDM, angor
>
> On débute une antibiothérapie par [MASK].
>   camembert-base
>     un, injection, le, exemple, une
>   doctomodernbert-fr-base
>     amoxicilline, quinolones, fluoroquinolone, cyclines, Augmentin

The contrast is hard to miss. CamemBERT falls back on filler and even bare articles (traitement, un), while DoctoBERT proposes the terms a clinician would use: pneumothorax and infarctus for chest pain, amoxicilline and Augmentin for the antibiotic. The difference comes down to exposure. CamemBERT saw these terms only rarely in general-text pretraining, so it treats them as unlikely and reaches for common words instead. DoctoBERT was pretrained on clinical text, where they appear frequently, so it ranks the right one first.

You can try it on your own sentences, or explore it in your browser: open the fill-mask demo.

The family

DoctoBERT comes in two models, both Apache-2.0 and available on the Hub under doctolib-lab.

Which one should you pick? We generally recommend DoctoModernBERT-fr-base: ModernBERT's sequence packing and unpadding make batched inference especially fast. Switching to DoctoBERT-fr-base is a one-line change.

We released the data curation pipeline which led to these models in our previous blog post. The annotators used to build FineMed can be useful and are available on Hugging Face:

• The subdomain classifier labels a document with one of 15 medical subdomains (e.g., clinical guidelines, patient education, biomedical science).
• The educational-quality scorer scores a document's educational quality on a 0 to 5 scale.
• The medical-entity extractor extracts medical-term spans under an 8-class taxonomy.

The corpora themselves are also available:

• FineMed is the 19.2B-word annotated corpus, released unfiltered so you can filter it your own way.
• FineMed-rephrased is a 4.5B-word LLM-rephrased subset.

The finetuning cookbook

An encoder without finetuning only predicts masked words. To adapt it to your task, you need to add a task-specific head on top and then finetune the entire model using your labeled dataset. Below are three practical examples: NER, classification, and semantic similarity.

Install the dependencies:

pip install "transformers>=4.48" datasets evaluate seqeval \
    sentence-transformers scikit-learn

Named entity recognition

Named entity recognition (NER) labels spans of text with their type. In clinical text, it is one of the most common tasks: find the drugs, the diseases, and the anatomy, and tag each span. DoctoBERT reaches state-of-the-art NER results on both public benchmarks and a real-world proprietary task.

Here is an end-to-end example, finetuning on QUAERO, a widely used French biomedical NER benchmark, openly available on the Hub.

1. Load the necessary packages

import numpy as np
import evaluate
from datasets import load_dataset
from transformers import (
    AutoTokenizer,
    AutoModelForTokenClassification,
    DataCollatorForTokenClassification,
    TrainingArguments,
    Trainer,
)

2. Pick a model and load the data

We use one of our models, doctolib-lab/doctomodernbert-fr-base. The other models, such as doctolib-lab/doctobert-fr-base, work for this task too.

QUAERO ships pre-tokenized: each example is a list of words, and every word carries a tag in the BIO scheme (B- starts an entity, I- continues it, O is outside any entity). We take the EMEA subset (drug leaflets), and MEDLINE works the same way.

model_id = "doctolib-lab/doctomodernbert-fr-base"  # or "doctolib-lab/doctobert-fr-base"
ds = load_dataset("DrBenchmark/QUAERO", "emea")

# Read the label set. QUAERO has 21 BIO tags: "O" plus B-/I- pairs for 10 UMLS
# entity types (CHEM = drugs, DISO = disorders, ANAT = anatomy, and so on).
labels = ds["train"].features["ner_tags"].feature.names
id2label = dict(enumerate(labels))
label2id = {name: i for i, name in id2label.items()}

# One raw example: words, plus a tag id per word read back through `labels`.
print(ds["train"][7]["tokens"])
print([labels[i] for i in ds["train"][7]["ner_tags"]])

Output:

> ['Prialt', 'ne', 'doit', 'pas', 'être', 'utilisé', 'chez', 'l', '’', 'enfant', '.']
> ['B-CHEM', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-LIVB', 'O']

"Prialt" (a drug) is tagged B-CHEM, "enfant" (a living being) is B-LIVB, and everything else is O.

3. Tokenize and align labels

The model reads tokens, not words, so the word-level tags have to be realigned to those tokens. We keep each word's tag on its first token and ignore the rest. Labeling every token of a word instead is another common choice.

tokenizer = AutoTokenizer.from_pretrained(model_id)

def tokenize_and_align(batch):
    """Tokenize each word list and align its NER tags to the tokens,
    labeling the first token of a word and masking the rest with -100
    (the label id the loss ignores)."""
    tokenized = tokenizer(batch["tokens"], is_split_into_words=True, truncation=True)

    aligned_labels = []
    for i, word_tags in enumerate(batch["ner_tags"]):
        # word_ids maps each token back to its source word (None for the
        # special tokens [CLS] and [SEP]).
        word_ids = tokenized.word_ids(batch_index=i)

        label_ids = []
        previous_word_id = None
        for word_id in word_ids:
            if word_id is None:  # special token
                label_ids.append(-100)
            elif word_id != previous_word_id:  # first token of the word
                label_ids.append(word_tags[word_id])
            else:  # another token of the same word
                label_ids.append(-100)
            previous_word_id = word_id
        aligned_labels.append(label_ids)

    tokenized["labels"] = aligned_labels
    return tokenized

ds_tok = ds.map(
    tokenize_and_align, batched=True, remove_columns=ds["train"].column_names
)

# The map turns each word list into tokens and aligned labels. For
# example 7, the tokens and their labels:
print(tokenizer.convert_ids_to_tokens(ds_tok["train"][7]["input_ids"]))
print(ds_tok["train"][7]["labels"])

Output:

> ['[CLS]', '▁P', 'rial', 't', '▁ne', '▁doit', '▁pas', '▁être', '▁utilisé', '▁chez', '▁l', '▁’', '▁enfant', '▁.', '[SEP]']
> [-100, 9, -100, -100, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, -100]

"Prialt" splits into ▁P / rial / t, so only ▁P keeps tag 9 (B-CHEM) and the rest are -100.

4. Add a token-classification head on the encoder

AutoModelForTokenClassification loads the pretrained encoder and adds a fresh linear layer that predicts one tag per token.

model = AutoModelForTokenClassification.from_pretrained(
    # id2label / label2id store tag names on the model, not just integer ids
    model_id, num_labels=len(labels), id2label=id2label, label2id=label2id,
)

5. Score entity predictions with seqeval

We score with seqeval, which works at the level of whole entities rather than single tokens. A predicted entity counts as correct only when its span and type both match the gold label exactly.

seqeval = evaluate.load("seqeval")

def compute_metrics(p):
    preds = np.argmax(p.predictions, axis=-1)
    pred_tags, gold_tags = [], []
    for pred_row, gold_row in zip(preds, p.label_ids):
        # keep only the labelled positions, dropping the -100 tokens
        kept = [
            (id2label[pr], id2label[gd])
            for pr, gd in zip(pred_row, gold_row) if gd != -100
        ]
        pred_tags.append([pr for pr, gd in kept])
        gold_tags.append([gd for pr, gd in kept])
    r = seqeval.compute(predictions=pred_tags, references=gold_tags)
    return {
        "precision": r["overall_precision"],
        "recall": r["overall_recall"],
        "f1": r["overall_f1"],
    }

6. Train the model

The Trainer takes the model, tokenized data, collator, and metric, and runs the training loop.

output_dir = "doctomodernbert-ft-ner-quaero-emea"
args = TrainingArguments(
    output_dir=output_dir,
    num_train_epochs=10,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=32,  # eval needs no gradients, so it can be larger
    learning_rate=3e-5,
    weight_decay=0.01,  # L2 regularization
    warmup_ratio=0.1,  # warm the learning rate up over the first 10% of steps
    lr_scheduler_type="linear",  # then decay it linearly to zero
    bf16=True,  # mixed precision, faster on recent GPUs
    eval_strategy="epoch",  # evaluate once per epoch
    save_strategy="epoch",  # checkpoint once per epoch
    load_best_model_at_end=True,  # reload the best checkpoint when training finishes
    metric_for_best_model="f1",  # "best" = highest entity-level F1
    greater_is_better=True,
    logging_steps=50,
    seed=42,  # reproducible runs
)
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=ds_tok["train"],
    eval_dataset=ds_tok["validation"],
    # pad inputs and labels together per batch
    data_collator=DataCollatorForTokenClassification(tokenizer),
    compute_metrics=compute_metrics,
)
trainer.train()  # a few minutes on a single GPU

7. (Optional) Score on the held-out test split

print(trainer.evaluate(ds_tok["test"]))

8. Save the finetuned model locally

trainer.save_model(output_dir)

9. Load the finetuned model and run it on new text

from transformers import pipeline

# Wrap the saved model in a token-classification pipeline. aggregation_strategy
# merges the per-token predictions into whole entity spans.
ner = pipeline(
    "token-classification",
    model="doctomodernbert-ft-ner-quaero-emea",
    aggregation_strategy="simple",
)

text = "Le patient reçoit de l'amoxicilline pour traiter une pneumonie."
print(ner(text))  # list of {entity_group, word, score, start, end}

Text classification

Text classification sorts a whole document into categories. In clinical text, that might be a specialty, a diagnostic group, or a triage bucket.

Here is an end-to-end example, finetuning on MORFITT, where each biomedical abstract can carry several specialties at once, so the task is multi-label.

1. Load the necessary packages

import numpy as np
from sklearn.metrics import f1_score
from datasets import load_dataset
from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
    DataCollatorWithPadding,
    TrainingArguments,
    Trainer,
)

2. Pick a model and load the data

We use the same base model as before, doctolib-lab/doctomodernbert-fr-base. MORFITT labels each abstract with one or more of 12 specialties, encoded as a 12-dimensional one-hot vector: one slot per specialty, set to 1 when the abstract belongs to it.

model_id = "doctolib-lab/doctomodernbert-fr-base"  # or "doctolib-lab/doctobert-fr-base"
ds = load_dataset("DrBenchmark/MORFITT", "source")

# The 12 specialty names (specialities_one_hot is the matching 0/1 vector).
specialties = ds["train"].features["specialities"].feature.names
# ['microbiology', 'etiology', 'virology', 'physiology', 'immunology',
#  'parasitology', 'genetics', 'chemistry', 'veterinary', 'surgery',
#  'pharmacology', 'psychology']

# One raw example. An abstract can carry more than one specialty.
example = ds["train"][1]
print(example["abstract"][:76], "...")
print("specialities:", [specialties[i] for i in example["specialities"]])
print("one-hot     :", [int(x) for x in example["specialities_one_hot"]])

Output:

> La dermatite idiopathie du philtrum nasal (DANP) est une atteinte vasculaire ...
> specialities: ['veterinary', 'immunology']
> one-hot     : [0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0]

This abstract is both veterinary and immunology, so two of the twelve slots are 1.

3. Tokenize the abstracts and attach their labels

Each abstract is tokenized, and its one-hot specialty vector becomes the training target. We cast that vector to floats because multi-label training uses binary cross-entropy, which expects float targets.

tokenizer = AutoTokenizer.from_pretrained(model_id)

def preprocess(batch):
    """Tokenize the abstracts and attach their one-hot vectors as float labels."""
    tokenized = tokenizer(batch["abstract"], truncation=True)
    tokenized["labels"] = [
        [float(x) for x in one_hot] for one_hot in batch["specialities_one_hot"]
    ]
    return tokenized

ds_tok = ds.map(preprocess, batched=True, remove_columns=ds["train"].column_names)

4. Add a classification head on the encoder

AutoModelForSequenceClassification adds a classification head on top of the encoder. Setting problem_type="multi_label_classification" makes the head apply a sigmoid and decide each specialty independently, yes or no.

model = AutoModelForSequenceClassification.from_pretrained(
    model_id, num_labels=len(specialties),
    problem_type="multi_label_classification",
)

5. Score with micro-F1

We score with micro-F1, which pools the yes/no decisions across all specialties into a single precision, recall, and F1.

def compute_metrics(p):
    probs = 1 / (1 + np.exp(-p.predictions))  # sigmoid -> per-specialty probability
    preds = (probs > 0.5).astype(int)         # threshold each specialty on its own
    return {"micro_f1": f1_score(p.label_ids, preds, average="micro")}

6. Train the model

Same Trainer setup as the NER recipe, this time with a padding collator for whole-sequence inputs.

output_dir = "doctomodernbert-ft-cls-morfitt"
args = TrainingArguments(
    output_dir=output_dir,
    learning_rate=3e-5,
    per_device_train_batch_size=16,
    num_train_epochs=10,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="micro_f1",
)
trainer = Trainer(
    model=model,
    args=args,
    train_dataset=ds_tok["train"],
    eval_dataset=ds_tok["validation"],
    # pad each batch to its longest input
    data_collator=DataCollatorWithPadding(tokenizer),
    compute_metrics=compute_metrics,
)
trainer.train()  # a few minutes on a single GPU

7. (Optional) Score on the held-out test split

print(trainer.evaluate(ds_tok["test"]))

8. Save the finetuned model locally

trainer.save_model(output_dir)

9. Load the finetuned model and run it on new text

from transformers import pipeline

clf = pipeline(
    "text-classification",
    model="doctomodernbert-ft-cls-morfitt",
    top_k=None,  # return a score for every specialty, then keep those above 0.5
)

text = "Étude de la résistance bactérienne aux antibiotiques chez le porc."
print(clf(text))  # list of {label, score} per specialty

For a single-label task such as DiaMED (one diagnostic class per document), drop problem_type and give each example a single integer class id instead of a one-hot vector. The head then applies a softmax over the classes and predicts the top one, and you would score it with accuracy or macro-F1 instead of micro-F1.

Semantic similarity and retrieval

Semantic similarity compares texts rather than labeling them: find similar clinical cases, retrieve relevant passages, or build a search index. This needs sentence embeddings, one vector per sentence instead of one per token, which is what Sentence Transformers is built for.

Here is an end-to-end example, finetuning on CLISTER, a French clinical similarity corpus where each row is a sentence pair and a human score from 0 to 5.

1. Load the necessary packages

from datasets import load_dataset
from sentence_transformers import (
    SentenceTransformer,
    SentenceTransformerTrainer,
    SentenceTransformerTrainingArguments,
    losses,
    models,
)
from sentence_transformers.evaluation import EmbeddingSimilarityEvaluator

2. Build a sentence encoder from the base model

DoctoBERT produces one vector per token. To compare whole sentences, we add a mean-pooling layer that averages a sentence's token vectors into a single sentence vector.

model_id = "doctolib-lab/doctomodernbert-fr-base"  # or "doctolib-lab/doctobert-fr-base"

backbone = models.Transformer(model_id, max_seq_length=256)
pooling = models.Pooling(
    backbone.get_word_embedding_dimension(), pooling_mode="mean"
)
model = SentenceTransformer(modules=[backbone, pooling])

3. Load the data

Each CLISTER row is two clinical sentences and a human similarity score, from 0 (unrelated) to 5 (equivalent).

ds = load_dataset("DrBenchmark/CLISTER", "source")
example = ds["train"][0]
print(example["text_1"][:76], "...")
print(example["text_2"][:76], "...")
print("label:", example["label"])  # 0 (unrelated) to 5 (equivalent)

Output:

> L'UIV a objectivé un retard de sécrétion avec importante dilatation pyélo-cali ...
> L'UIV a montré une importante dilatation urétéro-pyélo-calicielle en amont d'un ...
> label: 3.0

Two radiology sentences that overlap heavily but not fully, rated 3.0 of 5.

4. Prepare the pairs

Sentence Transformers expects three columns named sentence1, sentence2, and score, with the score in a 0 to 1 range. We rescale the 0 to 5 label and rename the columns to match.

def prepare_pairs(split):
    """Rescale the 0-5 score to 0-1 and rename the columns."""
    split = split.map(lambda row: {"score": row["label"] / 5.0})
    split = split.rename_columns({"text_1": "sentence1", "text_2": "sentence2"})
    return split.select_columns(["sentence1", "sentence2", "score"])

train_ds, eval_ds = prepare_pairs(ds["train"]), prepare_pairs(ds["validation"])

5. Define the loss

CosineSimilarityLoss pulls a pair's embeddings together or apart so their cosine similarity matches the target score. For query-passage retrieval data, MultipleNegativesRankingLoss is usually the better choice.

loss = losses.CosineSimilarityLoss(model)

6. Track dev correlation during training

An evaluator reports the Spearman correlation between predicted and gold similarity on the dev pairs at each evaluation, so you can watch the model improve.

dev_evaluator = EmbeddingSimilarityEvaluator(
    eval_ds["sentence1"], eval_ds["sentence2"], eval_ds["score"], name="clister-dev",
)

7. Train the model

The SentenceTransformerTrainer takes the sentence encoder, the prepared pairs, the loss, and the evaluator, then runs the training loop.

output_dir = "doctomodernbert-ft-sts-clister"
args = SentenceTransformerTrainingArguments(
    output_dir=output_dir,
    num_train_epochs=4,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=32,
    learning_rate=2e-5,
    warmup_ratio=0.1,  # warm the learning rate up over the first 10% of steps
    weight_decay=0.01,  # L2 regularization
    bf16=True,  # mixed precision, faster on recent GPUs
    eval_strategy="epoch",  # run the evaluator once per epoch
    save_strategy="epoch",
    logging_steps=50,
    seed=42,  # reproducible runs
)
trainer = SentenceTransformerTrainer(
    model=model,
    args=args,
    train_dataset=train_ds,
    eval_dataset=eval_ds,
    loss=loss,
    evaluator=dev_evaluator,
)
trainer.train()

8. Save the finetuned sentence encoder locally

model.save_pretrained(output_dir)

9. Load the finetuned model and score a new pair

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("doctomodernbert-ft-sts-clister")

pair = [
    "Le scanner montre une masse pulmonaire suspecte.",
    "L'imagerie révèle une lésion suspecte au poumon.",
]
emb = model.encode(pair)
print(model.similarity(emb[0], emb[1]))  # cosine similarity, higher = more similar

For higher retrieval accuracy, ColBERT-style late interaction keeps one vector per token instead of one per sentence, and scores a pair by matching each token to its closest counterpart. PyLate is a good choice for training such models.

Running it in production

A few practical notes once you have a finetuned model.

Half precision and Flash Attention. On GPU, load the weights in bfloat16, and for DoctoModernBERT add Flash Attention 2. ModernBERT then packs sequences and skips padding on its own, which helps most on the long notes its 8192-token context is built for:

import torch

model = AutoModelForTokenClassification.from_pretrained(
    "your-finetuned-model",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",  # ModernBERT only (pip install flash-attn)
)

For repeated calls, wrapping the loaded model in torch.compile speeds it up further.

Batch your inputs. At 111M to 149M parameters, these models take large batches on a single GPU. The pipeline API batches for you:

from transformers import pipeline

ner = pipeline(
    "token-classification", model="your-finetuned-model", device=0, batch_size=64
)

No GPU? Export to ONNX. For CPU or edge serving, convert the model with Optimum and optionally quantize it to int8. This is usually the biggest win without a GPU:

from optimum.onnxruntime import ORTModelForTokenClassification

model = ORTModelForTokenClassification.from_pretrained(
    "your-finetuned-model", export=True,
)

Have a question, or a tip of your own for running the models on different systems? Open an issue on GitHub. We would love to hear what works or not on your setup.

For fun: making DoctoBERT talk

Encoders are pretrained with masked language modeling: hide some tokens, predict them back. That is really a denoising objective, and masked diffusion models generalize it. Instead of unmasking everything at once, they start from a fully masked sequence and reveal tokens step by step, keeping the most confident predictions at each step.

Using the dLLM library, we finetuned DoctoModernBERT on 500k medical question-answer pairs into a text-diffusion model, doctomodernbert-fr-large-diffusion-v0.1. It answers a question by filling a masked canvas block by block, and you can watch it think out loud:

Try it yourself: open the diffusion demo.

Wrapping up

DoctoBERT gives you a French medical encoder that is small, fast, open, and genuinely good at clinical NLP. Finetune it on your own labeled data to get a model specialized for your use case, whether that is NER, classification, or retrieval. The models, annotators, and datasets are on the Hub under doctolib-lab, and the code is on GitHub.

Next steps

We started DoctoBERT with French, but the web-data recipe behind it is not tied to any one language. Extending it to other languages, and to multilingual medical encoders, is a natural next step. If there is a language or a clinical task you would like us to prioritize, let us know in the GitHub issues.

Citation

@misc{doctobert2026,
  title         = {Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining},
  author        = {Huang, Bofeng and Sun, Jacques and Bouchacourt, Diane and Barascud, Nicolas and Fogel, Fajwel},
  year          = {2026},
  eprint        = {2606.22079},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL}
}