XXXXyu/Cosmos3-Nano-Policy-DROID-Marlin-W8A16

Base model | Runtime | Benchmark details | Cosmos 3 report

Run NVIDIA's 16B Cosmos3 DROID robot policy on a single 24GB RTX 4090. This repository provides a self-contained, packed W8A16 deployment of NVIDIA Cosmos3-Nano-Policy-DROID, served with Cosmos Lite and validated end to end in RoboLab.

At A Glance

Peak VRAM	Request latency	Banana success rate
21.42 GB	4,110 ms median	86% (43/50)

Measured on an RTX 4090 at batch size 1 with the common comparison sampler: four UniPC denoise steps and guidance 3.0. Request latency covers policy-server preprocessing, model generation, and response handling for one 32-action request; it excludes RoboLab and IsaacSim environment work.

Three BananaInBowlTask episodes selected from 10 rollout runs are shown above. W8 and mixed-precision demos use the three fastest successful episodes; the W4 demo uses its two fastest successes and one failure. Videos retain the original playback rate and full episode duration. Each final frame is held for three seconds while fading to black before the next episode fades in. All runs use the same guidance 3.0 / four-step sampler as the metrics. See the full quantization comparison and the five-model rollout view below.

General deployment default. Use guidance 3.0 and two denoise steps for approximately 2.4-second requests, 90% Banana SR, and the strongest tested cross-task result. Four steps remain the conservative rollback sampler.

Model Overview

Cosmos 3 is a family of omnimodal world models for Physical AI. The 16B Cosmos3-Nano model uses a Mixture-of-Transformers (MoT) architecture with two complementary transformer towers: an autoregressive transformer for discrete tokens and a diffusion transformer for continuous modalities. Continuous outputs, including robot actions, are generated through iterative denoising.

Cosmos3-Nano-Policy-DROID adapts this model to robot control: given a language instruction, visual observations, and robot state from the DROID embodiment, it predicts an action trajectory for manipulation. This repository changes the inference representation of the upstream policy; it does not retrain or distill the model.

Property	Value
Base checkpoint	`nvidia/Cosmos3-Nano-Policy-DROID`
Base model developer	NVIDIA
Quantized derivative	Cosmos Lite community release
Architecture	16B Mixture-of-Transformers world action model
Policy embodiment	DROID
Quantization	W8A16 weight-only; BF16 activations
Runtime	Cosmos Lite
Tested target	NVIDIA RTX 4090 24GB, Linux x86-64

Training And Evaluation Data

This derivative performs no additional policy training. Refer to the upstream model card for the base model's training description and data governance statements.

W4 and mixed-precision variants use activation statistics from 128 frames in 128 distinct successful episodes of the official nvidia/Cosmos3-DROID training split. These samples calibrate weight scales only; they do not update model parameters and are not specific to the Banana task. Full W8 does not require calibration data. Closed-loop quality was evaluated in RoboLab, with Banana as the 50-rollout primary task and three additional task sets used to check transfer of the recommended configuration.

Policy Interface

The validated RoboLab integration provides task text, robot state, and a 640x540 RGB composition made from three camera views. The runtime maps the image to the model's 736x544 inference bucket.

The released checkpoint and validated integration use a 32-step, 8-dimensional DROID action chunk. The chunk length is a property of this trained checkpoint and its deployment contract, not an immutable architectural limit of Cosmos 3. Changing it is outside the validated configuration.

This model is intended for robotics research, simulator evaluation, and controlled deployment development. Validate the observation/action contract, task success, and independent safety controls before using it on a real robot.

Benchmark Results

Quantization Comparison

This comparison fixes all four models to batch size 1, four UniPC denoise steps, guidance 3.0, and the same paired 50 initial states. Latency is median policy-server request time and excludes simulator/environment execution.

Quantized model	Peak VRAM (GB)	Request median (ms)	Banana SR (50 rollouts)
W8A16	21.42	4,110	86%
W4A16	14.67	4,248	52%
W4A16-AttnW8	16.21	4,153	84%
W4A16-GenW8	18.03	4,104	90%

Bold values are the best result in each metric column. The underlined W8A16 model is the general deployment default because it retained the strongest cross-task result; the mixed variants require task-specific validation.

Sampling Comparison

Denoise steps	Guidance	Request median (ms)	Banana SR (50 rollouts)
4	3.0	4,110	86%
2	3.0	2,403	90%
4	1.0	2,431	64%
2	1.0	1,565	80%

Bold values are the best metric in the table. The underlined row is the deployment recommendation. Guidance 1.0 reduced isolated request latency but increased failures and total rollout time, so it is not retained.

Across all variants, the general deployment recommendation remains W8A16 with two denoise steps and guidance 3.0. It achieved 90% (45/50) on Banana and 83% (25/30) across three additional RoboLab task sets. W4A16 provides the smallest memory footprint but did not pass the rollout quality gate.

These are protocol-specific point estimates and are not claims of improvement over BF16. See the full benchmark for request construction, latency statistics, replay metrics, repeated rollout counts, software versions, and rollback settings.

Rollout Comparison

All panels use the upstream default of guidance 3.0 and four denoise steps. Quantized panels play their selected episodes at the original rate and hold each final frame for three seconds while fading to black. Success and failure labels report the actual selected outcomes. The BF16 panel uses the only Banana rollout published with NVIDIA's base checkpoint. The upstream quickstart invokes the same default sampler, but the MP4 has no separate run manifest, so it remains a one-episode visual reference rather than a same-protocol BF16 benchmark. The displayed SR values come from the complete paired 50-rollout quantized evaluations, not these selected videos.

Open the comparison video

Quick Start

Install the Hugging Face CLI, download this self-contained bundle, and set up the locked policy runtime:

python -m pip install -U "huggingface_hub[cli]"
hf download "XXXXyu/Cosmos3-Nano-Policy-DROID-Marlin-W8A16" --local-dir /data/cosmos3_droid_quant

git clone https://github.com/xxxxyu/cosmos-lite.git
cd cosmos-lite
CUDA_VISIBLE_DEVICES=0 examples/robolab_quant/pipeline.sh setup

Validate every recorded SHA256 and packed tensor before GPU loading:

BUNDLE_DIR=/data/cosmos3_droid_quant \
STRATEGY=full_w8 \
examples/robolab_quant/pipeline.sh validate

Start the OpenPI-compatible policy server:

BUNDLE_DIR=/data/cosmos3_droid_quant \
POLICY_GPU=0 GUIDANCE=3.0 NUM_STEPS=2 \
examples/robolab_quant/pipeline.sh serve

For simulator setup and rollout commands, follow the RoboLab pipeline guide. The policy server and IsaacSim should use separate physical GPUs.

Quantization And Bundle Details

Field	Value
Strategy	`full_w8`
Packed linear modules	504
W4 / W8 modules	0 / 504
Bundle size	20.44 GB
Activation quantization	None; activations remain BF16
Weight kernel/layout	vLLM Marlin WNA16

Full W8 does not require calibration data. Every selected linear layer is packed to W8 using weight statistics only.

The bundle includes packed linear weights, residual weights, configuration, tokenizer, Wan VAE, runtime metadata, source revisions, file sizes, and SHA256 hashes. Serving does not require the original BF16 checkpoint or calibration data.

Format Note

This release uses vLLM's Marlin WNA16 packed layout and kernel. It is not a backend-neutral GPTQ or AWQ checkpoint and should be loaded through Cosmos Lite's bundle loader. All four variants are weight-only W4A16/W8A16 or fixed mixed-precision combinations; activation quantization is not used.

The server binds to 127.0.0.1:8000 by default and has no built-in TLS or authentication. Do not expose it directly to an untrusted network.

Provenance

Base policy: nvidia/Cosmos3-Nano-Policy-DROID at 6706d7680581c255ff61e0f3bb49d90eac55c79e
Qwen tokenizer/runtime assets: Qwen/Qwen3-VL-8B-Instruct at 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b
Wan VAE: Wan-AI/Wan2.2-TI2V-5B at 921dbaf3f1674a56f47e83fb80a34bac8a8f203e
Runtime: Cosmos Lite

Limitations And Safety

Results may not transfer to a new task, embodiment, camera layout, action contract, checkpoint, quantization strategy, sampler, or GPU architecture.
Open-loop action parity does not establish closed-loop task quality.
This model is not real-robot safety certified. Real deployments require an independent E-stop, watchdog, workspace/joint/action limits, rate limits, stale-command rejection, and operator supervision.
This is a community quantized derivative, not an official NVIDIA release.