# Codex Guide: Open3DVQA on Jetson Orin NX

## Goal

Deploy the distilled Open3DVQA Qwen3-VL 4B model on a Jetson Orin NX
running JetPack 5, using Ollama when possible.

## Required Files

Keep these files in the same directory:

```text
student_4b_merged-Q4_K_M.gguf
mmproj-student_4b_merged-f16.gguf
Modelfile.ollama
```

The model is multimodal. The Q4_K_M file contains the language model, while
the F16 mmproj file contains the vision encoder and projector. Both are
required for image question answering.

Expected SHA256 checksums:

```text
6e8afbae9f375151a22d79725e61a979a5d48f49c1afe310f0df14e5930f4ee8  student_4b_merged-Q4_K_M.gguf
80611b31a5061473b676e507736e8486a1ffa3e832b7ecc7481abc188964e8e8  mmproj-student_4b_merged-f16.gguf
```

## Jetson Environment Check

Run:

```bash
cat /etc/nv_tegra_release
uname -m
free -h
df -h
nvidia-smi 2>/dev/null || tegrastats
```

Expected architecture:

```text
aarch64
```

Do not upgrade JetPack or CUDA as part of model deployment unless explicitly
requested. JetPack 5 compatibility is more important than using the newest
runtime.

## Install Ollama

The official installer detects Jetson and selects the JetPack package:

```bash
curl -fsSL https://ollama.com/install.sh | sh
```

Verify:

```bash
ollama --version
```

If systemd is unavailable, run the server manually:

```bash
OLLAMA_HOST=127.0.0.1:11434 ollama serve
```

For remote access on a trusted LAN:

```bash
OLLAMA_HOST=0.0.0.0:11434 ollama serve
```

Do not expose port 11434 directly to the public internet.

## Download from Hugging Face

Repository:

```text
dasvad/open3dvqa-qwen3vl-4b-distill-q4-k-m-gguf
```

Using `hf`:

```bash
python3 -m pip install -U huggingface_hub
hf download dasvad/open3dvqa-qwen3vl-4b-distill-q4-k-m-gguf \
  student_4b_merged-Q4_K_M.gguf \
  mmproj-student_4b_merged-f16.gguf \
  Modelfile.ollama \
  --local-dir ./open3dvqa-qwen3vl
```

Verify the files:

```bash
cd open3dvqa-qwen3vl
sha256sum student_4b_merged-Q4_K_M.gguf
sha256sum mmproj-student_4b_merged-f16.gguf
```

## Create the Ollama Model

The Modelfile must contain:

```text
FROM ./student_4b_merged-Q4_K_M.gguf
ADAPTER ./mmproj-student_4b_merged-f16.gguf
PARAMETER temperature 0
PARAMETER num_ctx 4096
```

Create and inspect the model:

```bash
ollama create open3dvqa-qwen3vl:4b-q4km -f Modelfile.ollama
ollama list
ollama show open3dvqa-qwen3vl:4b-q4km --modelfile
```

The imported model should be about 3.3 GB.

## Image Inference

Use `/api/chat`, not `/api/generate`. The latter returned an empty answer
during local validation even though it processed the image.

```python
import base64
import json
import urllib.request

image_path = "test.jpg"
payload = {
    "model": "open3dvqa-qwen3vl:4b-q4km",
    "messages": [
        {
            "role": "user",
            "content": "Describe the image briefly.",
            "images": [
                base64.b64encode(open(image_path, "rb").read()).decode()
            ],
        }
    ],
    "stream": False,
    "options": {
        "temperature": 0,
        "num_predict": 32,
        "num_ctx": 4096,
    },
}

opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
request = urllib.request.Request(
    "http://127.0.0.1:11434/api/chat",
    data=json.dumps(payload).encode(),
    headers={"Content-Type": "application/json"},
)
print(opener.open(request, timeout=600).read().decode())
```

If proxy variables are configured, bypass them for local Ollama requests:

```bash
env -u http_proxy -u https_proxy -u HTTP_PROXY -u HTTPS_PROXY \
  NO_PROXY=127.0.0.1,localhost no_proxy=127.0.0.1,localhost \
  python3 infer.py
```

## GPU Verification

Inspect Ollama logs while loading the model.

Desired behavior:

```text
clip_ctx: CLIP using CUDA backend
```

Undesired CPU fallback:

```text
--no-mmproj-offload
clip_ctx: CLIP using CPU backend
```

If CPU fallback occurs:

1. Stop other GPU workloads.
2. Restart Ollama.
3. Load the model again.
4. Keep `num_ctx=4096` for project-comparable testing.

Do not terminate processes owned by other users.

## Known Local Validation

The uploaded files were validated on an RTX 5090 with Ollama 0.30.8.

Test question:

```text
Does the blue and white apartment building appear over the cluster of cars in foreground?
A.Yes.
B.No.
```

Result:

```text
A.Yes
```

The same GGUF pair was also validated with `llama-mtmd-cli`.

## Fallback

If the JetPack 5 Ollama build cannot load Qwen3-VL, use a JetPack-compatible
llama.cpp build and load:

```bash
llama-mtmd-cli \
  -m student_4b_merged-Q4_K_M.gguf \
  --mmproj mmproj-student_4b_merged-f16.gguf \
  --image test.jpg \
  -p "Describe the image." \
  --image-min-tokens 1024 \
  -c 4096 \
  -n 32 \
  --temp 0
```

## Rules for Future Codex Sessions

1. Read this file and `README_JETSON_ORIN_NX_DEPLOY.md` before changing the
   deployment.
2. Preserve both GGUF files.
3. Use `/api/chat` for image input.
4. Keep the normal evaluation context at 4096.
5. Verify that the vision projector is on GPU before performance testing.
6. Never kill another user's GPU process.
7. Record end-to-end latency, decode token/s, sample/s, and peak device memory.