README.md

metadata

title: MAOAM GLaMM Object & Material Selection
emoji: 🎯
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 5.46.0
app_file: app.py
pinned: false
python_version: '3.12'
short_description: Click/text object & material selection (GLaMM + SAM ViT-H).
preload_from_hub:
  - MBZUAI/GLaMM-GranD-Pretrained
  - openai/clip-vit-large-patch14-336
  - jpark677/maoam_ckpts

MAOAM: Unified Object & Material Selection with VLMs (GLaMM backend)

Faithful port of the authors' GLaMM/demo.py from adobe-research/obj-and-mat-selection, adapted for ZeroGPU. Select objects or materials in an image with star clicks, a text prompt, or both. Output is a cyan selection-mask overlay plus the raw binary mask, and a downloadable zip of all artifacts.

Selection types

• Material: click — drop 1-5 star points; <COLOR> is auto-filled from the first star's auto-contrast color.
• Material: text — describe a material (e.g. shiny chrome metal). No stars.
• Material: click + text — combine a star cue with a material description.
• Object: text — a RefCOCO-style object expression (e.g. the man in a red shirt).

Regions with the same base material but different colors count as different materials; lighting/shadow variation of the same material counts as the same.

Backend

GLaMM = LLaVA-Llama (GranD pretrained) as the vision-language model + SAM ViT-H as the grounding / mask encoder. Segmentation is a single teacher-forced forward pass that emits a [SEG] token whose hidden state is decoded by SAM's mask decoder (no autoregressive generation).

Weights:

• Base VLM: MBZUAI/GLaMM-GranD-Pretrained
• MAOAM fine-tuned checkpoint: jpark677/maoam_ckpts (glamm/mp_rank_00_model_states.pt)
• SAM ViT-H: sam_vit_h_4b8939.pth (fetched at runtime; set SAM_REPO to use an HF mirror)
• Vision tower / CLIP processor: openai/clip-vit-large-patch14-336

Citation

@inproceedings{park2026maoam,
  title     = {MAOAM: Unified Object and Material Selection with Vision-Language Models},
  author    = {Park, Jaden and Deschaintre, Valentin and Kuen, Jason and
               Liu, Kangning and Georgiev, Iliyan and Singh, Krishna Kumar and
               Lee, Yong Jae and Fischer, Michael},
  booktitle = {ACM SIGGRAPH 2026 Conference Papers},
  year      = {2026},
  publisher = {ACM},
  doi       = {10.1145/3799902.3811186},
}

Project page: https://jadenpark0.github.io/project_pages/maoam/

License: Adobe Research.