README.md

metadata

library_name: pytorch
tags:
  - robotics
  - imitation-learning
  - action-chunking-transformer
  - mujoco
  - franka
license: other
license_name: proprietary
license_link: LICENSE
inference: false

PACT-128D with jointly finetuned encoder — v1010 fumehood, seed 3103

Exact final 60,000-update checkpoint from the v1010/v1010c three-seed comparison in prox_learning. This repository contains seed 3103 only. Checkpoint selection follows the original final-update evaluation; these are not validation-best checkpoints.

Included files

• policy_last.ckpt: unchanged native PyTorch state dictionary, including the visual backbone.
• dataset_stats.pkl and equivalent normalization.npz: original training-only normalization.
• model_config.json: architecture, inputs, sensor order, action semantics, and execution settings.
• prox_encoder.pt and checkpoint_pairs.json: corresponding jointly finetuned encoder and hash-checked pairing.
• runtime/: original model definitions and third-party license notices.
• inference.py: portable inference adapter; no local training directories or simulator imports required.
• example_observation.npz: one retained simulation observation and its historical first action.
• checksums.json, source_provenance.json, validation.json: integrity and loading evidence.

Download and run

Authenticate with an account that has access while the repository is private:

hf auth login
hf download Lundii/pact-128d-finetune-v1010c-seed3103 --local-dir ./pact-128d-finetune-v1010c-seed3103
cd pact-128d-finetune-v1010c-seed3103
python -m pip install -r requirements.txt
python inference.py --device cuda

The CUDA command verifies the package hashes, loads the entire model strictly, and checks its first action against the retained evaluation action. CPU loading is available for inspection, but the original experiment and action-equivalence validation use CUDA. CPU outputs have measurable numerical differences; the CPU example reports them without enforcing CUDA-reference equality. The validated runtime versions are recorded in source_provenance.json.

import numpy as np
from inference import RobotPolicy

policy = RobotPolicy('.', device='cuda')
policy.reset()  # once at the beginning of every episode
with np.load('example_observation.npz', allow_pickle=False) as example:
    action = policy.step(
        qpos=example['qpos'],           # 7 arm joints + 2 gripper joints
        wrist_rgb=example['wrist_rgb'], # HWC RGB
        proximity=example.get('proximity'),
    )
print(action['arm'], action['gripper'])

For an actual rollout, call step once for every consecutive control step with the new observations. The adapter queries the policy each step and uses 100-action chunks with temporal aggregation over up to 100 overlapping chunks, weighted by exp(-0.01 * age). It returns seven absolute joint targets and a gripper command: 0=open, 255=close (threshold 127.5). It does not step a simulator. The reference evaluation uses 900 actions with no early stop on success.

Wrist images are resized to 240×320 with OpenCV area interpolation, scaled to [0,1], and ImageNet-normalized inside the inference adapter. Supply RGB, not BGR.

PACT requires the included prox_encoder.pt paired with policy_last.ckpt. It uses the 128-D CLS readout, not the older frozen 32-D embedding. The sensor order in model_config.json is the training order: link5 front before back. Do not replace it with the default alphabetical/back-before-front order.

Provide four 8×8 depth subframes per sensor per control step, in metres. The adapter minimum-pools these subframes, maintains eight consecutive control frames, left-pads the first frame, and uses the original encoder preprocessing. Returns beyond 20 cm or below 5 mm follow the original invalid-depth rules. The four repeated subframes inside the encoder are a compatibility representation, not additional physical measurements.

Training and evaluation scope

Training uses 240 demonstrations and 40 validation demonstrations from the v1010 four-object fumehood pick-and-place distribution. The policy has a ResNet18 visual backbone, width 512, seven encoder and seven decoder layers, feed-forward width 3200, eight heads, state dimension 9, and action dimension 8.

Recorded seed-3103 evaluation on 50 source-task scenarios:

Metric	Result
Placement success	27/50 (54%)
Collision-free placement	24/50 (48%)
Collision-free rollout	32/50 (64%)

These are simulation results for one trained checkpoint, not hardware validation or a claim of generalization to a different task or sensor layout. The original simulation environment and task assets are maintained in the project repository and are not part of this inference package. Download using a fixed Hub commit revision when reproducing an experiment.

Model weights and project-specific code retain the project's existing proprietary licensing. Vendored ACT/DETR files retain their included licenses.