---
library_name: pytorch
tags:
- robotics
- imitation-learning
- action-chunking-transformer
- mujoco
- franka
license: other
license_name: proprietary
license_link: LICENSE
inference: false
---

# PACT-128D with jointly finetuned encoder — v1010 fumehood, seed 3103

Exact final **60,000-update** checkpoint from the v1010/v1010c three-seed
comparison in [prox_learning](https://github.com/Jdvakil/prox_learning).
This repository contains **seed 3103 only**. Checkpoint selection follows the
original final-update evaluation; these are not validation-best checkpoints.

## Included files

- `policy_last.ckpt`: unchanged native PyTorch state dictionary, including the visual backbone.
- `dataset_stats.pkl` and equivalent `normalization.npz`: original training-only normalization.
- `model_config.json`: architecture, inputs, sensor order, action semantics, and execution settings.
- `prox_encoder.pt` and `checkpoint_pairs.json`: corresponding jointly finetuned encoder and hash-checked pairing.
- `runtime/`: original model definitions and third-party license notices.
- `inference.py`: portable inference adapter; no local training directories or simulator imports required.
- `example_observation.npz`: one retained simulation observation and its historical first action.
- `checksums.json`, `source_provenance.json`, `validation.json`: integrity and loading evidence.

## Download and run

Authenticate with an account that has access while the repository is private:

```bash
hf auth login
hf download Lundii/pact-128d-finetune-v1010c-seed3103 --local-dir ./pact-128d-finetune-v1010c-seed3103
cd pact-128d-finetune-v1010c-seed3103
python -m pip install -r requirements.txt
python inference.py --device cuda
```

The CUDA command verifies the package hashes, loads the entire model strictly,
and checks its first action against the retained evaluation action. CPU loading
is available for inspection, but the original experiment and action-equivalence
validation use CUDA. CPU outputs have measurable numerical differences; the CPU
example reports them without enforcing CUDA-reference equality. The validated
runtime versions are recorded in `source_provenance.json`.

```python
import numpy as np
from inference import RobotPolicy

policy = RobotPolicy('.', device='cuda')
policy.reset()  # once at the beginning of every episode
with np.load('example_observation.npz', allow_pickle=False) as example:
    action = policy.step(
        qpos=example['qpos'],           # 7 arm joints + 2 gripper joints
        wrist_rgb=example['wrist_rgb'], # HWC RGB
        proximity=example.get('proximity'),
    )
print(action['arm'], action['gripper'])
```

For an actual rollout, call `step` once for **every consecutive control step**
with the new observations. The adapter queries the policy each step and uses
100-action chunks with temporal aggregation over up to 100 overlapping chunks,
weighted by `exp(-0.01 * age)`. It returns seven absolute joint targets and a
gripper command: 0=open, 255=close (threshold 127.5). It does not step a simulator.
The reference evaluation uses 900 actions with no early stop on success.

Wrist images are resized to 240×320 with OpenCV area interpolation, scaled to
[0,1], and ImageNet-normalized inside the inference adapter. Supply RGB, not BGR.

PACT requires the included `prox_encoder.pt` paired with `policy_last.ckpt`. It uses the **128-D CLS readout**, not the older frozen 32-D embedding. The sensor order in `model_config.json` is the training order: link5 front before back. Do not replace it with the default alphabetical/back-before-front order.

Provide four 8×8 depth subframes per sensor per control step, in metres. The adapter minimum-pools these subframes, maintains eight consecutive control frames, left-pads the first frame, and uses the original encoder preprocessing. Returns beyond 20 cm or below 5 mm follow the original invalid-depth rules. The four repeated subframes inside the encoder are a compatibility representation, not additional physical measurements.

## Training and evaluation scope

Training uses 240 demonstrations and 40 validation demonstrations from the
v1010 four-object fumehood pick-and-place distribution. The policy has a ResNet18
visual backbone, width 512, seven encoder and seven decoder layers, feed-forward
width 3200, eight heads, state dimension 9, and action dimension 8.

Recorded seed-3103 evaluation on 50 source-task scenarios:

| Metric | Result |
|---|---:|
| Placement success | 27/50 (54%) |
| Collision-free placement | 24/50 (48%) |
| Collision-free rollout | 32/50 (64%) |

These are simulation results for one trained checkpoint, not hardware validation
or a claim of generalization to a different task or sensor layout. The original
simulation environment and task assets are maintained in the project repository
and are not part of this inference package. Download using a fixed Hub commit
revision when reproducing an experiment.

Model weights and project-specific code retain the project's existing
proprietary licensing. Vendored ACT/DETR files retain their included licenses.