mehmetkeremturkcan/SAM3-Joint-Surgical

Surgical SAM 3.1

Mehmet Kerem Turkcan
AIDL Lab, Columbia University

Surgical SAM 3.1 detects and segments surgical instruments, anatomy, and tissue from text prompts. The detector uses one vocabulary across CholecInstanceSeg, CholecSeg8k, the Dresden Surgical Anatomy Dataset, and Endoscapes. Training uses the grouped V3 dataset, which assigns each procedure, video, or patient group to one split.

The released checkpoint contains the fine tuned SAM 3.1 detector and the original SAM 3.1 multiplex tracker parameters. Image detection and instance masks use the fine tuned detector. Video propagation uses the original tracker components after the detector initializes the objects.

The galleries visualize reference masks from the grouped dataset. Positive annotations are available for 25 text concepts.

Results

The grouped test split contains 4,134 physical images in 22 groups. COCO AP averages precision across ten IoU thresholds from 0.50 through 0.95 and across the 24 concepts with positive test instances. Each image is evaluated only for the concepts listed in its annotation scope.

Model	Box AP	Mask AP	Box AP50	Mask AP50	Mask AP75
Surgical SAM 3.1, joint_grouped_sam31_best_multiplex.pt	0.4654	0.3503	0.6258	0.4620	0.3730
Base SAM 3.1, sam3.1_multiplex.pt	0.0241	0.0180	0.0304	0.0276	0.0199

Both evaluations use the same prompts, images, annotation scopes, postprocessing, and COCO evaluator.

Category group	Concepts with test instances	Box macro AP	Mask macro AP
Instruments	8	0.8417	0.7664
Anatomy and tissue	16	0.2773	0.1422

Test results by concept

Concept	Test images	Test instances	Box AP	Mask AP
grasper	2,808	3,695	0.797	0.565
hook	2,232	2,232	0.901	0.871
irrigator	241	241	0.934	0.886
clipper	142	142	0.935	0.915
bipolar	215	215	0.878	0.777
scissors	51	51	0.729	0.672
snare	2	2	0.718	0.751
surgical instrument	3,992	6,711	0.841	0.694
abdominal wall	769	769	0.444	0.072
liver	971	971	0.270	0.019
gastrointestinal tract	380	380	0.152	0.004
fat	1,280	1,280	0.720	0.000
connective tissue	960	960	0.038	0.000
cystic duct	264	264	0.053	0.018
gallbladder	843	843	0.074	0.025
liver ligament	1,099	1,099	0.429	0.000
cystic plate	19	19	0.148	0.100
calot triangle	14	14	0.252	0.239
cystic artery	28	28	0.054	0.033
colon	58	58	0.183	0.160
pancreas	17	17	0.269	0.240
small intestine	4	4	0.000	0.000
spleen	2	2	0.551	0.602
stomach	141	141	0.801	0.763

Model selection

Checkpoint and loss selection use mask AP on 4,144 validation images. The held out test annotations do not determine the selected checkpoint. The selected model starts from epoch 7 and updates the segmentation head with the loss configuration identified by the validation search.

Validation candidate	Box AP	Mask AP	Mask AP50	Large mask AP
dice_polish_epoch1	0.5130	0.3387	0.4534	0.3648
dice_polish_epoch2	0.5130	0.3386	0.4535	0.3647
dice_epoch1	0.5133	0.3386	0.4532	0.3648
sam31_epoch7	0.5128	0.3373	0.4521	0.3623
head_epoch1	0.5132	0.3366	0.4526	0.3617

The selected candidate is dice_polish_epoch1, with validation box AP 0.5130 and validation mask AP 0.3387.

Evaluation protocol

Each record contains exhaustive_category_ids, which lists the concepts that are annotated for that image. The evaluator issues only those text prompts and retains predictions for those categories.

The grouped split merges records that resolve to the same physical frame before assignment. Its audit found zero overlap between every pair of splits for group identifiers, physical frame identifiers, normalized frame identifiers, and SHA 256 image digests. Generated Endoscapes tool masks occur only in training.

Quick start

Download the image model builder patch:

hf download mehmetkeremturkcan/SAM3-Joint-Surgical \
  patches/sam31_image_training.patch \
  --revision main \
  --local-dir surgical_sam31

Install the SAM 3 source used for this run:

git clone https://github.com/facebookresearch/sam3.git
cd sam3
git checkout 46957e47805eaa273f4aa7bbbd25a88bca9108ce
git apply ../surgical_sam31/patches/sam31_image_training.patch
pip install -e .

Download the configuration and checkpoint:

hf download mehmetkeremturkcan/SAM3-Joint-Surgical \
  config.json \
  joint_grouped_sam31_best_multiplex.pt \
  --revision main \
  --local-dir checkpoints

Run image detection and segmentation:

import torch
from PIL import Image

from sam3.model.sam3_image_processor import Sam3Processor
from sam3.model_builder import build_sam31_image_model

model = build_sam31_image_model(
    checkpoint_path="checkpoints/joint_grouped_sam31_best_multiplex.pt",
    load_from_HF=False,
    device="cuda",
    eval_mode=True,
    enable_segmentation=True,
)
processor = Sam3Processor(model, device="cuda", confidence_threshold=0.5)
image = Image.open("frame.png").convert("RGB")

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = processor.set_image(image)
    output = processor.set_text_prompt(
        state=state,
        prompt="surgical instrument",
    )

boxes = output["boxes"]
masks = output["masks"]
scores = output["scores"]

The complete example is available in scripts/inference_sam31.py.

Checkpoint

File	Detector tensors	Tracker tensors	Size	SHA 256
joint_grouped_sam31_best_multiplex.pt	1,166	457	3,502,785,103 bytes	7cebc871ab6efbc4b06f4b82fcaf364c9187de30245c1553957109326a487da0

The checkpoint loads with weights_only=True. The unsafe serialization global check reports zero globals. The tracker tensors retain the values from facebook/sam3.1; the detector tensors contain the validation selected surgical model.

Training

Setting	Value
Base checkpoint	facebook/sam3.1, sam3.1_multiplex.pt
Dataset	SAM3 Joint Surgical Dataset, grouped V3 split
Physical training images	33,094
Effective training records per epoch	52,747
Input resolution	1008 by 1008
Full detector training	8 epochs
Segmentation search	two segmentation head loss configurations, with lower rate refinement when the time budget allowed
Hardware	8 NVIDIA A100 80 GB GPUs
Precision	BF16 automatic mixed precision
Optimizer	AdamW
Initial transformer learning rate	8 × 10−5
Initial vision backbone learning rate	2.5 × 10−5
Initial language backbone learning rate	5 × 10−6
Seed	123

The full detector run trains the SAM 3.1 image grounding path while retaining the interactive and propagation neck parameters from the base checkpoint. The refinement search updates only the segmentation head, which isolates mask quality from detector changes. Exact configurations and machine readable metrics are included in this repository.

Dataset

The associated SAM3 Joint Surgical Dataset contains 41,372 physical images with 175,674 annotations across 29 schema concepts. Positive annotations are available for 25 concepts. Train, validation, and test contain 33,094, 4,144, and 4,134 images, respectively.

Intended use

This repository supports research in surgical computer vision. The model has not been validated for clinical decision making or patient care.

Author

Mehmet Kerem Turkcan, AIDL Lab, Columbia University.

Citation

@misc{turkcan2026surgicalsam31,
  title  = {Surgical SAM 3.1},
  author = {Turkcan, Mehmet Kerem},
  year   = {2026},
  url    = {https://huggingface.co/mehmetkeremturkcan/SAM3-Joint-Surgical}
}

Licenses

The checkpoint follows the SAM License in LICENSE. Dataset records retain their source licenses. See the dataset repository for source attribution and redistribution terms.