Spaces:
Sleeping
Sleeping
File size: 2,560 Bytes
c2cf032 a17007e c2cf032 a17007e c2cf032 a17007e c2cf032 a17007e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | ---
title: MAOAM GLaMM Object & Material Selection
emoji: 🎯
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 5.46.0
app_file: app.py
pinned: false
python_version: "3.12"
short_description: Click/text object & material selection (GLaMM + SAM ViT-H).
preload_from_hub:
- MBZUAI/GLaMM-GranD-Pretrained
- openai/clip-vit-large-patch14-336
- jpark677/maoam_ckpts
---
# MAOAM: Unified Object & Material Selection with VLMs (GLaMM backend)
Faithful port of the authors' `GLaMM/demo.py` from
[adobe-research/obj-and-mat-selection](https://github.com/adobe-research/obj-and-mat-selection),
adapted for ZeroGPU. Select **objects** or **materials** in an image with star
clicks, a text prompt, or both. Output is a cyan selection-mask overlay plus the
raw binary mask, and a downloadable zip of all artifacts.
## Selection types
- **Material: click** — drop 1-5 star points; `<COLOR>` is auto-filled from the
first star's auto-contrast color.
- **Material: text** — describe a material (e.g. *shiny chrome metal*). No stars.
- **Material: click + text** — combine a star cue with a material description.
- **Object: text** — a RefCOCO-style object expression (e.g. *the man in a red shirt*).
Regions with the same base material but different colors count as different
materials; lighting/shadow variation of the same material counts as the same.
## Backend
**GLaMM** = LLaVA-Llama (GranD pretrained) as the vision-language model +
**SAM ViT-H** as the grounding / mask encoder. Segmentation is a single
teacher-forced forward pass that emits a `[SEG]` token whose hidden state is
decoded by SAM's mask decoder (no autoregressive generation).
Weights:
- Base VLM: `MBZUAI/GLaMM-GranD-Pretrained`
- MAOAM fine-tuned checkpoint: `jpark677/maoam_ckpts` (`glamm/mp_rank_00_model_states.pt`)
- SAM ViT-H: `sam_vit_h_4b8939.pth` (fetched at runtime; set `SAM_REPO` to use an HF mirror)
- Vision tower / CLIP processor: `openai/clip-vit-large-patch14-336`
## Citation
```bibtex
@inproceedings{park2026maoam,
title = {MAOAM: Unified Object and Material Selection with Vision-Language Models},
author = {Park, Jaden and Deschaintre, Valentin and Kuen, Jason and
Liu, Kangning and Georgiev, Iliyan and Singh, Krishna Kumar and
Lee, Yong Jae and Fischer, Michael},
booktitle = {ACM SIGGRAPH 2026 Conference Papers},
year = {2026},
publisher = {ACM},
doi = {10.1145/3799902.3811186},
}
```
Project page: https://jadenpark0.github.io/project_pages/maoam/
License: **Adobe Research**.
|