Expand model card with setup, inference, evaluation and training instructions

README.md

@@ -13,48 +13,81 @@ tags:
 
 # CADReasoner-CM (cross-modality variant)
 
-CADReasoner-CM is the **cross-modality** variant of [CADReasoner](https://huggingface.co/kulibinai/cadreasoner),
-a vision–language model for **iterative CAD reverse engineering**. It generates a **runnable CadQuery program**
-and refines it over multiple iterations using geometric feedback from the discrepancy between the target shape
-and the current prediction.
+CADReasoner-CM is a variant of [CADReasoner](https://huggingface.co/kulibinai/cadreasoner), a
+vision–language model for **iterative CAD reverse engineering**. It generates a **runnable CadQuery
+program** and refines it over several iterations using geometric feedback from the discrepancy
+between the target shape and the current prediction.
 
-This checkpoint conditions on **both the point cloud and multi-view renders** of the target shape. Combining the
-two modalities improves geometric alignment and recovers fine details that either modality alone tends to miss.
+This checkpoint conditions on **both the point cloud and multi-view renders** of the target shape. Combining the two modalities improves geometric alignment and recovers fine details that either modality alone tends to miss.
 
 **Accepted to CVPR 2026 Findings Track**
 
 **Paper:** https://arxiv.org/abs/2603.29847
-**Code:** https://github.com/zhemdi/CADReasoner
+**Code:** https://github.com/zhemdi/CADReasoner (see the `pc_cm/` directory)
 **HF paper page:** https://huggingface.co/papers/2603.29847
 
 ## Variants
 
-| Model | Input modality |
-|---|---|
-| [`kulibinai/cadreasoner`](https://huggingface.co/kulibinai/cadreasoner) | multi-view renders |
-| [`kulibinai/cadreasoner-pc`](https://huggingface.co/kulibinai/cadreasoner-pc) | point cloud |
-| [`kulibinai/cadreasoner-cm`](https://huggingface.co/kulibinai/cadreasoner-cm) | point cloud + multi-view renders (cross-modality) |
+| Model | Input modality | Flags |
+|---|---|---|
+| [`kulibinai/cadreasoner`](https://huggingface.co/kulibinai/cadreasoner) | multi-view renders | — (root scripts) |
+| [`kulibinai/cadreasoner-pc`](https://huggingface.co/kulibinai/cadreasoner-pc) | point cloud | `--use_pc true --use_img false` |
+| [`kulibinai/cadreasoner-cm`](https://huggingface.co/kulibinai/cadreasoner-cm) | point cloud + multi-view renders | `--use_pc true --use_img true` |
 
-## Usage
+## Architecture
 
-Use the inference and evaluation scripts from the repository:
+This checkpoint uses the `Cadrille` architecture: a Qwen2-VL-2B backbone plus a Fourier point-cloud
+encoder. The point features are projected to the hidden size and **scattered into a run of pad
+tokens prepended to the prompt**, so the model cannot be loaded with a bare
+`Qwen2VLForConditionalGeneration.from_pretrained` — `config.json` declares
+`architectures: ["Cadrille"]`, and the class lives in `pc_cm/cadrille.py` in the repository.
+
+The point cloud is built from the **discrepancy** between the target mesh and the current
+prediction, not from the target alone: both surfaces are sampled, points whose distance to the
+opposite shape exceeds a percentile threshold are kept, and farthest point sampling reduces them.
+Each direction contributes `n_points` points — `n_points` from GT→pred and `n_points` from pred→GT
+— so the prompt reserves `2 * n_points` pad tokens and each point carries 6 features (position +
+displacement vector). On the first iteration, where no prediction exists yet, the bounding-box
+centre is used in place of the predicted mesh.
+
+`--n_points` must match the value used in training: **128**.
+
+## Setup
+
+```bash
+git clone https://github.com/zhemdi/CADReasoner.git
+cd CADReasoner
+```
+
+Build the environment from the provided `Dockerfile`, then add the two packages the `pc_cm/`
+scripts need on top of it:
+
+```bash
+pip install opencv-python rtree
+```
+
+`opencv-python` is used by the image augmentations, and `rtree` backs trimesh's closest-point
+queries — without it the code silently falls back to a KD-tree over mesh vertices, which is less
+accurate on coarse meshes.
+
+The scripts load the model with `attn_implementation="flash_attention_2"` and `torch.bfloat16`, so
+an Ampere-or-newer GPU with `flash-attn` installed is required. Inference shards samples across all
+visible GPUs, one process per GPU, and needs at least one.
+
+## Inference
 
 ```bash
-python3 test.py \
+python3 pc_cm/test.py \
     --dataset <test_dataset> \
     --checkpoint kulibinai/cadreasoner-cm \
     --use_pc true --use_img true \
     --n_points 128 \
-    --n_iters <n_iters> \
-    --outdir <outdir>
+    --n_iters 3 \
+    --n_samples 4 \
+    --outdir preds_cm
 ```
 
-The checkpoint uses the custom `Cadrille` architecture (Qwen2-VL plus a Fourier point-cloud
-encoder), so it must be loaded through `cadrille.py` from the repository rather than through a
-bare `Qwen2VLForConditionalGeneration.from_pretrained`. `--n_points` must match the value used
-in training (**128**).
-
-`<test_dataset>` can be one of:
+`<test_dataset>` is either a local directory of `.stl` files or a Hugging Face dataset repo id:
 
 * `maksimko123/deepcad_test_mesh`
 * `maksimko123/fusion360_test_mesh`
@@ -63,6 +96,80 @@ in training (**128**).
 * `kulibinai/fusion360_test_scan`
 * `kulibinai/mcb_test_scan`
 
+Each iteration writes a CadQuery program and its compiled mesh per candidate:
+
+```text
+preds_cm/<dataset_name>/<iter>/<shape_stem>/<chamfer_distance>.py
+preds_cm/<dataset_name>/<iter>/<shape_stem>/<chamfer_distance>.stl
+```
+
+Files are renamed to their chamfer distance so the next iteration can pick the best candidates to
+refine. `--n_samples` controls how many candidates are sampled per shape (the first is greedy, the
+rest use temperature 1.2).
+
+## Evaluation
+
+```bash
+python3 pc_cm/evaluate.py \
+    --dataset <test_dataset> \
+    --pred_dir preds_cm/<dataset_name>
+```
+
+For each refinement iteration this reports median chamfer distance (scaled by 1000), mean IoU, and
+the invalidity ratio — the fraction of shapes with no valid prediction. Each shape is scored with
+the best candidate seen up to that iteration, so the numbers are cumulative across iterations.
+
+## Training
+
+The curriculum runs over dataset groups `0, 1, 2` (see `data/README.md` for preparing the split).
+Group 0 starts from `Qwen/Qwen2-VL-2B-Instruct`; each later group starts from the previous group's
+final checkpoint.
+
+```bash
+torchrun --nproc_per_node <n_gpus> pc_cm/train.py \
+    --dataset_dir <train_dataset_dir> \
+    --use_pc true --use_img true \
+    --n_points 128
+```
+
+Checkpoints and logs are written to `runs/<experiment>/<experiment>_<timestamp>/`, with per-group
+weights under `model/<group>/final`. Pass `--run_dir` and `--groups` to resume an interrupted run,
+and `--skip_generate_code` / `--skip_generate_meshes` to reuse refinement samples already on disk.
+
+## Loading the model directly
+
+If you are integrating the model rather than using the scripts:
+
+```python
+import torch
+from transformers import AutoProcessor
+from cadrille import Cadrille  # from pc_cm/
+
+processor = AutoProcessor.from_pretrained(
+    "kulibinai/cadreasoner-cm",
+    resized_width=14 * 17 * 2,
+    resized_height=14 * 17 * 4,
+    padding_side="left",
+    use_fast=True,
+)
+model = Cadrille.from_pretrained(
+    "kulibinai/cadreasoner-cm",
+    torch_dtype=torch.bfloat16,
+    attn_implementation="flash_attention_2",
+).cuda().eval()
+```
+
+`forward` and `generate` additionally require `point_clouds` (a `(batch, 2 * n_points, 6)` tensor),
+`is_pc` and `is_img` — they are not optional, and the pad-token prefix must already be present in
+the prompt. See `generate_predictions_process` in `pc_cm/test.py` for a complete, working example.
+
+## Limitations
+
+The image-conditioned [`kulibinai/cadreasoner`](https://huggingface.co/kulibinai/cadreasoner) gives
+the best results in our implementation; the geometry-conditioned variants are released for
+reproducibility and for settings where rendered views are unavailable. The point-cloud modality also
+proved unstable under RL fine-tuning, so it is not being developed further at present.
+
 ## Citation
 
 ```bibtex