Add files using upload-large-folder tool

README.md

@@ -0,0 +1,98 @@
+---
+license: apache-2.0
+base_model: allenai/Molmo2-4B
+pipeline_tag: image-text-to-text
+tags:
+  - molmo2
+  - adacodec
+  - video
+  - long-video
+  - stage1
+  - stage2
+---
+
+# Molmo2 Codec V6 — Stage-2 step 100
+
+This is the evaluated V6 AdaCodec-on-Molmo2 research checkpoint. It packages
+the Stage-2 model, its matching Stage-1 P-tokenizer, the fixed motion-cost
+calibration, the exact saved training config, source provenance, and a paired
+six-benchmark evaluation summary.
+
+This is **not** a standalone `transformers` checkpoint. It must be loaded with
+the Molmo2 Codec code and its `lmms-eval` adapter.
+
+## Checkpoints
+
+| File | Purpose | SHA256 |
+|---|---|---|
+| `weights/stage2_step100_consolidated.pt` | Consolidated Molmo2 + trained Stage-2 parameters | `141467de9a28a53d6ebb87ef8bcf1dbf191880d39b64dd0bebc3728baaf79677` |
+| `weights/stage1_ptokenizer_step20000.pt` | V6+HEX Stage-1 P-tokenizer | `10fc7d0460199581d9a7f1677c79a6a519adc55184c4ba14cd44f50a4bf5eec9` |
+
+## Training contract
+
+- Base: Molmo2-4B-SFT lineage.
+- Stage-1: V6 P-tokenizer, RGB HEX motion search, step 20,000.
+- Stage-2 checkpoint: step 100 from the formal 32,768-token run.
+- Training visual ceiling: 16,384 tokens; text ceiling: 15,872 tokens.
+- Candidate timeline: full-span sampling, 2 FPS, at most 2,048 source frames.
+- Mixture: 40% long codec caption, 10% normal codec video, 37.5% native
+  video, and 12.5% native image replay.
+- Stage-2 trains the LLM, P-ViT, and P-connector. The I-frame visual path stays
+  on the native Molmo2 representation.
+- The durable Stage-2 GOP cache uses the explicit `fast_gray2` motion-search
+  contract and its matching gamma artifact. The packaged Stage-1 weights were
+  trained with RGB HEX; this run intentionally adapts the P branch in Stage-2.
+
+The complete serialized configuration is in
+`config/stage2_step100_config.yaml`.
+
+## Paired evaluation
+
+Each row uses the same frozen 200 examples for Dense and Codec. Evaluation
+uses full-span 2 FPS sampling, a 2,048-frame candidate timeline, 8,192 visual
+tokens, and a 16,384-token sequence. The checked machine-readable result is
+`eval/paired_summary_n200.json`.
+
+| Benchmark | Dense | Codec | Codec − Dense |
+|---|---:|---:|---:|
+| Video-MME | 70.0 | 69.0 | -1.0 |
+| LVBench | 48.0 | 49.5 | +1.5 |
+| MVBench | 75.5 | 68.5 | -7.0 |
+| TempCompass | 69.0 | 68.5 | -0.5 |
+| PerceptionTest | 82.0 | 83.0 | +1.0 |
+| NExT-QA | 84.0 | 83.5 | -0.5 |
+
+The checkpoint is close to Dense on five of six paired subsets, but the
+MVBench regression is material and remains an open limitation. These are
+bounded `n=200` diagnostics, not full benchmark submissions.
+
+## Code and loading
+
+- Main code: <https://github.com/weikaih04/molmo2-codec>, branch `adacodec`.
+- Main release commits: `69f20d99`, `63874827`, `ead01f2e`.
+- `lmms-eval` adapter commit: `6fda918` in the local Molmo2 evaluation fork.
+- Example launcher: `examples/molmo2_codec/run_smoke.sh` in that fork.
+
+The adapter expects paths to both packaged weight files and the gamma artifact.
+It invokes the production Molmo2 preprocessing, timestamp, position, packing,
+GOP-selection, and codec inference code instead of reimplementing them.
+
+## Provenance
+
+`provenance/SOURCE_SNAPSHOT.txt` records the exact source state captured by the
+formal training job, including the pre-commit working-tree digest and the
+post-requeue decode fix. `provenance/SOURCE_FILE_SHA256SUMS` records hashes of
+the evaluation-critical source files. The three release commits above organize
+that working tree and add the final regression tests; they should not be
+mistaken for a claim that the model was retrained after code cleanup.
+
+## Limitations
+
+- Research checkpoint after only 100 Stage-2 optimizer steps.
+- Requires the custom code path; it is not directly loadable through
+  `AutoModel.from_pretrained`.
+- The evaluation sample size is 200 per benchmark.
+- Motion cache and gamma artifacts are contract-bound. Do not silently mix
+  RGB HEX and `fast_gray2` cache namespaces.
+- The upstream base-model license and usage restrictions continue to apply.
+

SHA256SUMS

@@ -0,0 +1,7 @@
+141467de9a28a53d6ebb87ef8bcf1dbf191880d39b64dd0bebc3728baaf79677  weights/stage2_step100_consolidated.pt
+10fc7d0460199581d9a7f1677c79a6a519adc55184c4ba14cd44f50a4bf5eec9  weights/stage1_ptokenizer_step20000.pt
+467a3ee238d0a4d8c0ef6ffeeecb2c614813b57b9485c2850bdc7d58135af6f1  config/gamma_fast_gray2_2048.json
+6465ce2f7496d18c54a8980aa3d8bd3b72f19879ca74748837e5a5428f90cb23  config/stage2_step100_config.yaml
+2be473c31f78268729d831797f00dec6850627ba68086d8e693d36ed9fcbdfcc  eval/paired_summary_n200.json
+4653e29349ca1e3187c248f6713cd4e8196bfa115e1efa98dafbae681d4dbaf1  provenance/SOURCE_SNAPSHOT.txt
+a41e043fd83bfcffb4ac046d835bbac9b2a0d4a4625f8d0c83b58dbe8221aa1f  provenance/SOURCE_FILE_SHA256SUMS

config/gamma_fast_gray2_2048.json

@@ -0,0 +1,16 @@
+{
+  "base_fps": 2.0,
+  "cache_misses": 0,
+  "calibration_scope": "long10_20min_fast_motion_205",
+  "gamma": 964955.6875,
+  "max_frames": 2048,
+  "max_p": 16,
+  "motion_fingerprint": "2db9b823",
+  "motion_mode": "fast_gray2",
+  "num_cost_sequences": 212,
+  "num_videos": 212,
+  "sample_digest": "5c074a916966c538",
+  "sample_seed": 20260803,
+  "schema_version": "v6-ab",
+  "target_median_p": 8
+}

config/stage2_step100_config.yaml

@@ -0,0 +1,1618 @@
+run_name: multitask_train
+model:
+  _name: model/video_olmo
+  data_formatter:
+    prompt_templates: uber_model_v2
+    system_prompt: demo_or_style_v2
+    default_inference_len: 65
+    select_answer: best
+    debug: false
+    eval_system_prompt_mapping: null
+    p_choice_content_in_mc: 1.0
+    template_video_mc_questions: true
+    points_decimal_places: 1
+    use_seperate_non_pointing_qa_style: false
+    timestamp_mode: 50-percent-seconds
+    output_timestamp_mode: seconds
+    p_multi_point_all_image: 0.5
+    max_images: 5
+    use_seperate_count_without_pointing_style: false
+    sample_random_initial_point: true
+  llm:
+    d_model: 2560
+    n_heads: 32
+    n_kv_heads: 8
+    head_dim: 128
+    qkv_bias: false
+    clip_qkv: null
+    n_layers: 36
+    mlp_ratio: 4
+    mlp_hidden_size: 19456
+    activation_type: swiglu
+    block_type: sequential
+    rope: true
+    rope_full_precision: true
+    rope_theta: 5000000.0
+    rope_type: default
+    rope_factor: null
+    rope_high_freq_factor: null
+    rope_low_freq_factor: null
+    rope_original_max_position_embeddings: null
+    rope_attention_factor: null
+    rope_beta_fast: null
+    rope_beta_slow: null
+    rope_mscale: null
+    rope_mscale_all_dim: null
+    rope_truncate: null
+    attention_type: sdpa
+    full_attention_layers: null
+    sliding_attention_rope_scaling: false
+    float32_attention: true
+    attention_dropout: 0.0
+    attention_layer_norm: true
+    attention_layer_norm_type: qwen3
+    residual_dropout: 0.1
+    response_residual_dropout: 0.0
+    layer_norm_type: rms
+    layer_norm_with_affine: true
+    layer_norm_eps: 1.0e-06
+    attention_layer_norm_with_affine: true
+    max_sequence_length: 32768
+    max_position_embeddings: null
+    include_bias: false
+    bias_for_layer_norm: null
+    norm_after: false
+    moe_num_experts: 8
+    moe_top_k: 2
+    moe_mlp_impl: sparse
+    moe_log_expert_assignment: false
+    moe_shared_expert: false
+    moe_lbl_in_fp32: false
+    moe_interleave: false
+    moe_loss_weight: 0.1
+    moe_zloss_weight: null
+    moe_dropless: true
+    moe_capacity_factor: 1.25
+    embedding_dropout: 0.0
+    scale_logits: false
+    vocab_size: 151936
+    additional_vocab_size: 128
+    weight_tying: true
+    embedding_size: 151936
+    use_position_ids: true
+    tokenizer:
+      identifier: Qwen/Qwen3-4B-Instruct-2507
+      tokenizer_dir: null
+    init_path: /weka/oe-training-default/mm-olmo/pretrained_llms/qwen3-4b-instruct.pt
+    can_predict_extra_tokens: false
+    init_incremental: null
+    new_embedding_init_range: 0.02
+    initializer_range: 0.02
+    normalize_input_embeds: false
+    activation_checkpoint: whole_layer
+    compile: null
+    fix_pad_tokenizer: false
+    init_std: 0.02
+    init_fn: normal
+    init_cutoff_factor: null
+  vision_backbone:
+    vit:
+      image_model_type: siglip
+      image_default_input_size:
+      - 378
+      - 378
+      image_patch_size: 14
+      image_pos_patch_size: 14
+      image_emb_dim: 1152
+      image_num_heads: 16
+      image_num_key_value_heads: 16
+      image_num_layers: 27
+      image_head_dim: 72
+      image_mlp_dim: 4304
+      image_mlp_activations: gelu_pytorch_tanh
+      image_dropout_rate: 0.0
+      image_num_pos: 729
+      image_norm_eps: 1.0e-06
+      attention_dropout: 0.0
+      residual_dropout: 0.0
+      initializer_range: 0.02
+      float32_attention: true
+      attention_type: sdpa
+      sdpa_backend: all
+      activation_checkpointing: true
+      init_path: /weka/oe-training-default/mm-olmo/pretrained_image_encoders/siglip2-so400m-14-384.pt
+      resize_mode: siglip
+      pad_value: 0.0
+      normalize: siglip
+    image_pooling_2d: attention_meanq
+    pooling_attention_mask: true
+    image_projector: mlp
+    image_padding_embed: null
+    vit_layers:
+    - -3
+    - -9
+    skip_unused_layers: true
+    use_deepstack: false
+    share_connector: false
+    image_feature_dropout: 0.0
+    connector_activation_checkpointing: true
+    compile_vit: null
+    pool_size_embeds: null
+    compile_connector: null
+    normalize_on_gpu: true
+  mm_preprocessor:
+    video:
+      max_frames: 101
+      frame_sample_mode: uniform_last_frame
+      candidate_sampling_fps:
+      - 0.25
+      - 0.5
+      - 1.0
+      - 2.0
+      cache_videos: true
+      loading_method: decord_with_av_fallback
+      max_fps:
+      - 2.0
+      time_sampling: true
+      time_mode: per-frame-compact
+      subtitle_mode: frame_1
+      pooling_w: 3
+      pooling_h: 3
+      use_frame_special_tokens: true
+      per_frame_special_token: true
+      max_subtitle_tokens: null
+    image:
+      crop_mode: overlap-and-resize-c2
+      use_col_tokens: true
+      max_crops: 8
+      high_res_max_crops: 24
+      p_high_res: 0.0
+      pooling_w: 2
+      pooling_h: 2
+      overlap_margins:
+      - 4
+      - 4
+      max_images: null
+      max_multi_image_crops: 8
+      multi_image_pooling_w: 2
+      multi_image_pooling_h: 2
+      use_single_crop_col_tokens: false
+      use_single_crop_start_token: true
+    grounding:
+      coordinate_scale: '1000'
+      sort_order: xy
+      sort_by_object_id: true
+      sort_tracking: true
+      image_sep: ;
+      seconds_decimal_places: 1
+      output_timestamp_mode: seconds_decimal
+      sort_by_image_id: false
+    message_format: molmo2
+    loss_token_weighting: root_subsegments_root_tokens
+  bi_directional_attn: image_tokens
+  flex_attention: true
+  shared_low_high_embedding: true
+  keep_pointing_bias: 0.0
+  debug: null
+  cp_enabled: false
+  apply_cp_to_vision_backbone: false
+parallelism:
+  data_parallel_replicate_degree: 1
+  enable_compiled_autograd: false
+  data_parallel_shard_degree: -1
+  fsdp_reshard_after_forward: default
+  context_parallel_config:
+    degree: 1
+    attention_type: ulysses
+    load_balancer: ulysses
+    head_stride: 1
+  tensor_parallel_config:
+    degree: 1
+    enable_async: false
+  data_parallel_config:
+    name: fsdp
+    param_dtype: null
+    reduce_dtype: float32
+    num_replicas: null
+    shard_degree: null
+    wrapping_strategy: full
+    prefetch_factor: 0
+  context_parallel_rotate_method: allgather
+seed: 6198
+epoch: null
+dry_run: false
+ft_llm: true
+ft_vit: false
+ft_connector: false
+ft_p_vit: true
+ft_p_connector: true
+ft_embedding: lm_head
+optimizer:
+  name: adamw
+  default_group_config:
+    learning_rate: 0.0001
+    weight_decay: 0.0
+    betas:
+    - 0.9
+    - 0.95
+    eps: 1.0e-06
+  optimizer_group_configs:
+    llm:
+      learning_rate: 1.0e-05
+      weight_decay: 0.0
+      betas:
+      - 0.9
+      - 0.999
+      eps: 1.0e-08
+    connector:
+      learning_rate: 5.0e-06
+      weight_decay: 0.0
+      betas:
+      - 0.9
+      - 0.95
+      eps: 1.0e-06
+    vit:
+      learning_rate: 5.0e-06
+      weight_decay: 0.0
+      betas:
+      - 0.9
+      - 0.95
+      eps: 1.0e-06
+    frame_selector:
+      learning_rate: 0.0001
+      weight_decay: 0.0
+      betas:
+      - 0.9
+      - 0.95
+      eps: 1.0e-06
+    p_vit:
+      learning_rate: 1.0e-06
+      weight_decay: 0.0
+      betas:
+      - 0.9
+      - 0.95
+      eps: 1.0e-06
+    p_connector:
+      learning_rate: 5.0e-06
+      weight_decay: 0.0
+      betas:
+      - 0.9
+      - 0.95
+      eps: 1.0e-06
+  metrics_log_interval: -1
+scheduler:
+  connector:
+    _name: scheduler/CosWithWarmup
+    units: steps
+    grad_clip_warmup_steps: null
+    grad_clip_warmup_factor: null
+    warmup_min_lr: 0.0
+    warmup_steps: 200
+    alpha_f: 0.1
+    t_max: null
+  vit:
+    _name: scheduler/CosWithWarmup
+    units: steps
+    grad_clip_warmup_steps: null
+    grad_clip_warmup_factor: null
+    warmup_min_lr: 0.0
+    warmup_steps: 200
+    alpha_f: 0.1
+    t_max: null
+  llm:
+    _name: scheduler/CosWithWarmup
+    units: steps
+    grad_clip_warmup_steps: null
+    grad_clip_warmup_factor: null
+    warmup_min_lr: 0.0
+    warmup_steps: 4500
+    alpha_f: 0.1
+    t_max: null
+  frame_selector:
+    _name: scheduler/CosWithWarmup
+    units: steps
+    grad_clip_warmup_steps: null
+    grad_clip_warmup_factor: null
+    warmup_min_lr: 0.0
+    warmup_steps: 200
+    alpha_f: 0.1
+    t_max: null
+  temporal_token_scorer:
+    _name: scheduler/CosWithWarmup
+    units: steps
+    grad_clip_warmup_steps: null
+    grad_clip_warmup_factor: null
+    warmup_min_lr: 0.0
+    warmup_steps: 200
+    alpha_f: 0.1
+    t_max: null
+  p_vit:
+    _name: scheduler/CosWithWarmup
+    units: steps
+    grad_clip_warmup_steps: null
+    grad_clip_warmup_factor: null
+    warmup_min_lr: 0.0
+    warmup_steps: 200
+    alpha_f: 0.1
+    t_max: null
+  p_connector:
+    _name: scheduler/CosWithWarmup
+    units: steps
+    grad_clip_warmup_steps: null
+    grad_clip_warmup_factor: null
+    warmup_min_lr: 0.0
+    warmup_steps: 200
+    alpha_f: 0.1
+    t_max: null
+data:
+  dataset: null
+  mixture: null
+  multi_level_mixture:
+  - rate: 0.4
+    datasets:
+    - dataset:
+        _name: dataset/LLaVAOV2Gt10min
+        jsonl_path: ''
+        max_examples: 0
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    name: long_codec_caption
+  - rate: 0.1
+    datasets:
+    - dataset:
+        _name: dataset/TVQA
+        flat: false
+        max_per_video: null
+        with_subtitle: true
+      sampling_rate: null
+      root_size_factor: 60000.0
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/Paxion
+        flat: false
+        max_per_video: null
+        use_extracted_clips: true
+      sampling_rate: null
+      root_size_factor: 60000.0
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/LLaVAVideo178K
+        answer_type: multi_choice
+        flat: false
+        max_per_video: null
+        id_source: null
+        cap_source: lv
+        cap_kw: merged_caption
+        subset: academic
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/LLaVAVideo178K
+        answer_type: open_ended
+        flat: false
+        max_per_video: null
+        id_source: null
+        cap_source: lv
+        cap_kw: merged_caption
+        subset: academic
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/PerceptionTest
+        flat: false
+        max_per_video: null
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/NeXTQA
+        task: multiple-choice
+        flat: false
+        max_per_video: null
+        difficulty: all
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/NewsVideoQA
+        flat: false
+        filter_empty_answers: true
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/How2QA
+        flatten: false
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/SUTDTrafficQA
+        flat: false
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/SocialIQ2
+        flat: false
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/SportsQA
+        max_per_video: null
+        flat: false
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/CinepileHf
+        with_subtitle: true
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/CLEVRER
+        answer_type: all
+        include_multiple_correct: false
+        flat: false
+        max_per_video: 10
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/FunQA
+        answer_type: all
+        flat: false
+        max_per_video: 10
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/STAR
+        answer_type: all
+        flat: false
+        max_per_video: 10
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/IntentQA
+        answer_type: all
+        flat: false
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/VideoLocalizedNarratives
+        flat: false
+        answer_type: open_ended
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/RoadTextVQA
+        flat: false
+        answer_type: open_ended
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/Countix
+        answer_format: oe
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/CameraBenchTrain
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/MotionBenchCaption
+        flat: false
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/MomentsInTime
+        flat: false
+        max_per_video: null
+        qa_format: true
+      sampling_rate: null
+      root_size_factor: 60000.0
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/Kinetics710
+        flat: false
+        max_per_video: null
+        qa_format: true
+      sampling_rate: null
+      root_size_factor: 60000.0
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/CharadesSTA
+        flat: false
+        task: all
+        qa_format: true
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/COIN
+        flat: false
+        task: all
+        max_per_video: null
+        qa_format: true
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/Youcook2
+        flat: false
+        task: all
+        max_per_video: null
+        qa_format: true
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/ActivityNet
+        flat: false
+        max_per_video: null
+        task: all
+        qa_format: true
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/Ego4dCachedClips
+        task: all
+        max_per_video: null
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/EpicKitchens
+        flat: false
+        max_per_video: null
+        use_extracted_clips: true
+        qa_format: true
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/VideoLocalizedNarrativesCaptionHf
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/QVHighlights
+        minimum: 0.0
+        max_detected_windows: 5
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/VixMoSynCaptionsQA
+        version: v2
+        exclude_counting: false
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    - dataset:
+        _name: dataset/vixmo_syn_video_capqa_with_sub
+        answer_type: all
+        version: v2
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: codec
+    name: normal_codec_video
+  - rate: 0.375
+    datasets:
+    - dataset:
+        _name: dataset/TVQA
+        flat: false
+        max_per_video: null
+        with_subtitle: true
+      sampling_rate: null
+      root_size_factor: 60000.0
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/Paxion
+        flat: false
+        max_per_video: null
+        use_extracted_clips: true
+      sampling_rate: null
+      root_size_factor: 60000.0
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/LLaVAVideo178K
+        answer_type: multi_choice
+        flat: false
+        max_per_video: null
+        id_source: null
+        cap_source: lv
+        cap_kw: merged_caption
+        subset: academic
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/LLaVAVideo178K
+        answer_type: open_ended
+        flat: false
+        max_per_video: null
+        id_source: null
+        cap_source: lv
+        cap_kw: merged_caption
+        subset: academic
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/PerceptionTest
+        flat: false
+        max_per_video: null
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/NeXTQA
+        task: multiple-choice
+        flat: false
+        max_per_video: null
+        difficulty: all
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/NewsVideoQA
+        flat: false
+        filter_empty_answers: true
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/How2QA
+        flatten: false
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/SUTDTrafficQA
+        flat: false
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/SocialIQ2
+        flat: false
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/SportsQA
+        max_per_video: null
+        flat: false
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/CinepileHf
+        with_subtitle: true
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/CLEVRER
+        answer_type: all
+        include_multiple_correct: false
+        flat: false
+        max_per_video: 10
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/FunQA
+        answer_type: all
+        flat: false
+        max_per_video: 10
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/STAR
+        answer_type: all
+        flat: false
+        max_per_video: 10
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/IntentQA
+        answer_type: all
+        flat: false
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/VideoLocalizedNarratives
+        flat: false
+        answer_type: open_ended
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/RoadTextVQA
+        flat: false
+        answer_type: open_ended
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/Countix
+        answer_format: oe
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/CameraBenchTrain
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/MotionBenchCaption
+        flat: false
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/MomentsInTime
+        flat: false
+        max_per_video: null
+        qa_format: true
+      sampling_rate: null
+      root_size_factor: 60000.0
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/Kinetics710
+        flat: false
+        max_per_video: null
+        qa_format: true
+      sampling_rate: null
+      root_size_factor: 60000.0
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/CharadesSTA
+        flat: false
+        task: all
+        qa_format: true
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/COIN
+        flat: false
+        task: all
+        max_per_video: null
+        qa_format: true
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/Youcook2
+        flat: false
+        task: all
+        max_per_video: null
+        qa_format: true
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/ActivityNet
+        flat: false
+        max_per_video: null
+        task: all
+        qa_format: true
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/Ego4dCachedClips
+        task: all
+        max_per_video: null
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/EpicKitchens
+        flat: false
+        max_per_video: null
+        use_extracted_clips: true
+        qa_format: true
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/VideoLocalizedNarrativesCaptionHf
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/QVHighlights
+        minimum: 0.0
+        max_detected_windows: 5
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/VixMoSynCaptionsQA
+        version: v2
+        exclude_counting: false
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    - dataset:
+        _name: dataset/vixmo_syn_video_capqa_with_sub
+        answer_type: all
+        version: v2
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: native
+    name: native_video
+  - rate: 0.125
+    datasets:
+    - dataset:
+        _name: dataset/Molmo2ReplayImage
+        root: /fsx/home/weikai.huang/molmo2_codec_data/molmo2_replay_500k
+        index_path: null
+        validation_examples: 1024
+      sampling_rate: null
+      root_size_factor: null
+      message_weight: null
+      override_p_high_res: null
+      video_backend: null
+    name: replay_500k
+  split: train
+  seed: 50189
+  pad: to_max
+  sequence_length: 32768
+  max_text_seq_len: 15872
+  shuffle: true
+  start_index: 0
+  packing:
+    buffer_size: 4
+    mode: dynamic_solver
+    text_weight: 1.0
+    image_weight: 30.0
+    shortcut_max_len_images: false
+    track_packing_state: true
+  enable_variable_sized_token_pooling: true
+  override_max_crops: 8
+  num_workers: 2
+  drop_last: true
+  pin_memory: true
+  prefetch_factor: 1
+  persistent_workers: true
+  timeout: 0
+restore_dataloader: true
+fast_forward_batches: null
+evaluators:
+- label: llava_video_oe_academic
+  data:
+    dataset:
+      _name: dataset/LLaVAVideo178K
+      answer_type: open_ended
+      flat: false
+      max_per_video: null
+      id_source: null
+      cap_source: lv
+      cap_kw: merged_caption
+      subset: academic
+    mixture: null
+    multi_level_mixture: null
+    split: validation
+    seed: 691203
+    pad: to_max
+    sequence_length: 32768
+    max_text_seq_len: null
+    shuffle: true
+    start_index: 0
+    packing: null
+    enable_variable_sized_token_pooling: true
+    override_max_crops: null
+    num_workers: 2
+    drop_last: true
+    pin_memory: true
+    prefetch_factor: 1
+    persistent_workers: true
+    timeout: 0
+  device_batch_size: 2
+  subset_num_batches: null
+  max_examples: 2048
+  console_log_interval: 20
+  response_logits_only: false
+- label: pixmo_ask_model_anything
+  data:
+    dataset:
+      _name: dataset/PixMoAskModelAnything
+      prefix_how_many: true
+      keep_in_memory: false
+      flat: false
+      skip_counting: false
+      sample: null
+    mixture: null
+    multi_level_mixture: null
+    split: validation
+    seed: 691203
+    pad: to_max
+    sequence_length: 32768
+    max_text_seq_len: null
+    shuffle: true
+    start_index: 0
+    packing: null
+    enable_variable_sized_token_pooling: true
+    override_max_crops: null
+    num_workers: 2
+    drop_last: true
+    pin_memory: true
+    prefetch_factor: 1
+    persistent_workers: true
+    timeout: 0
+  device_batch_size: 2
+  subset_num_batches: null
+  max_examples: 2048
+  console_log_interval: 20
+  response_logits_only: false
+- label: pixmo_cap
+  data:
+    dataset:
+      _name: dataset/PixMoCap
+      mode: captions
+      prefix_how_many: true
+      keep_in_memory: false
+      flatten: false
+    mixture: null
+    multi_level_mixture: null
+    split: validation
+    seed: 691203
+    pad: to_max
+    sequence_length: 32768
+    max_text_seq_len: null
+    shuffle: true
+    start_index: 0
+    packing: null
+    enable_variable_sized_token_pooling: true
+    override_max_crops: null
+    num_workers: 2
+    drop_last: true
+    pin_memory: true
+    prefetch_factor: 1
+    persistent_workers: true
+    timeout: 0
+  device_batch_size: 2
+  subset_num_batches: null
+  max_examples: 2048
+  console_log_interval: 20
+  response_logits_only: false
+eval_interval: -1
+inf_evaluators:
+- label: chart_qa
+  data:
+    dataset:
+      _name: dataset/ChartQa
+      parts: both
+      weighted: false
+      use_exp: false
+    mixture: null
+    multi_level_mixture: null
+    split: validation
+    seed: 691203
+    pad: null
+    sequence_length: null
+    max_text_seq_len: 128
+    shuffle: true
+    start_index: 0
+    packing: null
+    enable_variable_sized_token_pooling: true
+    override_max_crops: null
+    num_workers: 2
+    drop_last: true
+    pin_memory: true
+    prefetch_factor: 1
+    persistent_workers: true
+    timeout: 0
+  use_default_evaluator: true
+  save_predictions: true
+  evaluators:
+  - _name: evaluator/SavePredictions
+    output_dir: null
+    save_json: true
+    save_tokens: false
+    log_examples: true
+    table: true
+    output_file: null
+  max_new_tokens: null
+  device_batch_size: 2
+  sampling:
+    temperature: 0.0
+    top_p: 1.0
+    top_k: null
+    ngram_size: null
+    repetition_penalty: null
+    frequency_penalty: null
+  subset_num_batches: null
+  max_examples: 1280
+  console_log_interval: 20
+  include_image: false
+- label: info_qa
+  data:
+    dataset:
+      _name: dataset/InfoQa
+    mixture: null
+    multi_level_mixture: null
+    split: validation
+    seed: 691203
+    pad: null
+    sequence_length: null
+    max_text_seq_len: 128
+    shuffle: true
+    start_index: 0
+    packing: null
+    enable_variable_sized_token_pooling: true
+    override_max_crops: null
+    num_workers: 2
+    drop_last: true
+    pin_memory: true
+    prefetch_factor: 1
+    persistent_workers: true
+    timeout: 0
+  use_default_evaluator: true
+  save_predictions: true
+  evaluators:
+  - _name: evaluator/SavePredictions
+    output_dir: null
+    save_json: true
+    save_tokens: false
+    log_examples: true
+    table: true
+    output_file: null
+  max_new_tokens: null
+  device_batch_size: 2
+  sampling:
+    temperature: 0.0
+    top_p: 1.0
+    top_k: null
+    ngram_size: null
+    repetition_penalty: null
+    frequency_penalty: null
+  subset_num_batches: null
+  max_examples: 1280
+  console_log_interval: 20
+  include_image: false
+- label: coco_2014_vqa_multi
+  data:
+    dataset:
+      _name: dataset/Vqa2
+      multi_question: false
+      sample: null
+    mixture: null
+    multi_level_mixture: null
+    split: validation
+    seed: 691203
+    pad: null
+    sequence_length: null
+    max_text_seq_len: 128
+    shuffle: true
+    start_index: 0
+    packing: null
+    enable_variable_sized_token_pooling: true
+    override_max_crops: null
+    num_workers: 2
+    drop_last: true
+    pin_memory: true
+    prefetch_factor: 1
+    persistent_workers: true
+    timeout: 0
+  use_default_evaluator: true
+  save_predictions: true
+  evaluators:
+  - _name: evaluator/SavePredictions
+    output_dir: null
+    save_json: true
+    save_tokens: false
+    log_examples: true
+    table: true
+    output_file: null
+  max_new_tokens: null
+  device_batch_size: 2
+  sampling:
+    temperature: 0.0
+    top_p: 1.0
+    top_k: null
+    ngram_size: null
+    repetition_penalty: null
+    frequency_penalty: null
+  subset_num_batches: null
+  max_examples: 1280
+  console_log_interval: 20
+  include_image: false
+- label: pixmo_clocks
+  data:
+    dataset:
+      _name: dataset/PixMoClocks
+      aug: true
+    mixture: null
+    multi_level_mixture: null
+    split: validation
+    seed: 691203
+    pad: null
+    sequence_length: null
+    max_text_seq_len: 128
+    shuffle: true
+    start_index: 0
+    packing: null
+    enable_variable_sized_token_pooling: true
+    override_max_crops: null
+    num_workers: 2
+    drop_last: true
+    pin_memory: true
+    prefetch_factor: 1
+    persistent_workers: true
+    timeout: 0
+  use_default_evaluator: true
+  save_predictions: true
+  evaluators:
+  - _name: evaluator/SavePredictions
+    output_dir: null
+    save_json: true
+    save_tokens: false
+    log_examples: true
+    table: true
+    output_file: null
+  max_new_tokens: null
+  device_batch_size: 2
+  sampling:
+    temperature: 0.0
+    top_p: 1.0
+    top_k: null
+    ngram_size: null
+    repetition_penalty: null
+    frequency_penalty: null
+  subset_num_batches: null
+  max_examples: 1280
+  console_log_interval: 20
+  include_image: false
+- label: pointing_eval_v2
+  data:
+    dataset:
+      _name: dataset/PixMoPointsEval
+      keep_in_memory: false
+      legacy: false
+    mixture: null
+    multi_level_mixture: null
+    split: test
+    seed: 691203
+    pad: null
+    sequence_length: null
+    max_text_seq_len: 128
+    shuffle: true
+    start_index: 0
+    packing: null
+    enable_variable_sized_token_pooling: true
+    override_max_crops: null
+    num_workers: 2
+    drop_last: true
+    pin_memory: true
+    prefetch_factor: 1
+    persistent_workers: true
+    timeout: 0
+  use_default_evaluator: true
+  save_predictions: true
+  evaluators:
+  - _name: evaluator/SavePredictions
+    output_dir: null
+    save_json: true
+    save_tokens: false
+    log_examples: true
+    table: true
+    output_file: null
+  max_new_tokens: null
+  device_batch_size: 2
+  sampling:
+    temperature: 0.0
+    top_p: 1.0
+    top_k: null
+    ngram_size: null
+    repetition_penalty: null
+    frequency_penalty: null
+  subset_num_batches: null
+  max_examples: 1280
+  console_log_interval: 20
+  include_image: false
+- label: muir_bench
+  data:
+    dataset:
+      _name: dataset/MuirBench
+      format: multiple_choice
+      legacy: false
+      keep_in_memory: false
+    mixture: null
+    multi_level_mixture: null
+    split: test
+    seed: 691203
+    pad: null
+    sequence_length: null
+    max_text_seq_len: 128
+    shuffle: true
+    start_index: 0
+    packing: null
+    enable_variable_sized_token_pooling: true
+    override_max_crops: null
+    num_workers: 2
+    drop_last: true
+    pin_memory: true
+    prefetch_factor: 1
+    persistent_workers: true
+    timeout: 0
+  use_default_evaluator: true
+  save_predictions: true
+  evaluators:
+  - _name: evaluator/SavePredictions
+    output_dir: null
+    save_json: true
+    save_tokens: false
+    log_examples: true
+    table: true
+    output_file: null
+  max_new_tokens: null
+  device_batch_size: 2
+  sampling:
+    temperature: 0.0
+    top_p: 1.0
+    top_k: null
+    ngram_size: null
+    repetition_penalty: null
+    frequency_penalty: null
+  subset_num_batches: null
+  max_examples: 1280
+  console_log_interval: 20
+  include_image: false
+- label: mvbench
+  data:
+    dataset:
+      _name: dataset/MVBench
+      difficulty: all
+      sample: null
+    mixture: null
+    multi_level_mixture: null
+    split: validation
+    seed: 691203
+    pad: null
+    sequence_length: null
+    max_text_seq_len: 128
+    shuffle: true
+    start_index: 0
+    packing: null
+    enable_variable_sized_token_pooling: true
+    override_max_crops: null
+    num_workers: 2
+    drop_last: true
+    pin_memory: true
+    prefetch_factor: 1
+    persistent_workers: true
+    timeout: 0
+  use_default_evaluator: true
+  save_predictions: true
+  evaluators:
+  - _name: evaluator/SavePredictions
+    output_dir: null
+    save_json: true
+    save_tokens: false
+    log_examples: true
+    table: true
+    output_file: null
+  max_new_tokens: null
+  device_batch_size: 2
+  sampling:
+    temperature: 0.0
+    top_p: 1.0
+    top_k: null
+    ngram_size: null
+    repetition_penalty: null
+    frequency_penalty: null
+  subset_num_batches: null
+  max_examples: 1280
+  console_log_interval: 20
+  include_image: false
+- label: vixmo_points_count_clip_63s
+  data:
+    dataset:
+      _name: dataset/VixMoPoints
+      subset: all
+      capability: count
+      mode:
+      - point_count
+      include_unsure: false
+      include_zero: true
+      include_nonzero_unanswerable: false
+      flat: true
+      min_points: -1
+      max_points: 60
+      point_sort_by: xy
+      use_2fps_video: true
+      max_seconds: -1
+      max_raw_duration: -1
+      fake_timestamp_fps: null
+      load_clip_times_from_metadata: false
+      fake_fps_candidates: null
+      multi_message_short_clips: false
+      p_multi_turn: 0.5
+      oversample: false
+    mixture: null
+    multi_level_mixture: null
+    split: val
+    seed: 691203
+    pad: null
+    sequence_length: null
+    max_text_seq_len: 128
+    shuffle: true
+    start_index: 0
+    packing: null
+    enable_variable_sized_token_pooling: true
+    override_max_crops: null
+    num_workers: 2
+    drop_last: true
+    pin_memory: true
+    prefetch_factor: 1
+    persistent_workers: true
+    timeout: 0
+  use_default_evaluator: true
+  save_predictions: true
+  evaluators:
+  - _name: evaluator/SavePredictions
+    output_dir: null
+    save_json: true
+    save_tokens: false
+    log_examples: true
+    table: true
+    output_file: null
+  max_new_tokens: null
+  device_batch_size: 2
+  sampling:
+    temperature: 0.0
+    top_p: 1.0
+    top_k: null
+    ngram_size: null
+    repetition_penalty: null
+    frequency_penalty: null
+  subset_num_batches: null
+  max_examples: 1280
+  console_log_interval: 20
+  include_image: false
+inf_eval_interval: -1
+save_inloop_predictions: true
+eval_on_last_step: true
+eval_on_load: false
+eval_on: []
+save_folder: /fsx/home/weikai.huang/molmo2_codec/codec_stage2_8k_hex_validation
+reset_parameters: true
+checkpointer_config:
+  save_thread_count: null
+  load_thread_count: null
+  pre_download: false
+  work_dir: null
+  throttle_uploads: false
+canceled_check_interval: 50
+save_interval: 100
+save_at: 10
+save_final_optim: true
+save_num_checkpoints_to_keep: 6
+save_final_unsharded_checkpoint: false
+save_interval_ephemeral: null
+save_overwrite: true
+load_path: null
+reset_optimizer_state: false
+reset_trainer_state: false
+initial_model_checkpoint: /fsx/home/weikai.huang/molmo2_codec/checkpoints/molmo2_4b_sft_dcp/step30000
+allow_resume: true
+max_duration: 45000
+global_train_batch_size: 128
+device_train_microbatch_size: 1
+max_grad_norm: 1.0
+multi_component_grad_norm: true
+batch_divisor: global_batch
+max_grad_norm_ratio: null
+precision: amp_bf16
+wandb: null
+beaker_log_interval: 50
+speed_monitor:
+  window_size: 20
+  gpu_flops_available: 989000000000000.0
+  flops_per_token: 21801071616
+console_log_interval: 1
+enable_timing_logs: false
+gen1_gc_interval: 1
+compile: null
+activation_checkpointing: true
+fsdp:
+  fsdp2: true
+  replicate_parameters: false
+  precision: pure
+  reshard_after_forward: true
+  use_orig_params: true
+  wrapping_strategy: null
+  sharding_strategy: FULL_SHARD
+  hybrid_sharding_num_model_replicas: null
+response_logits_only: true
+saliency_score_loss_wt: null
+frame_score_loss_wt: null
+frame_score_loss_type: mse
+frame_score_loss_target: 0.7
+time_limit: null
+extra_steps_after_cancel: 0
+python_profiling: false
+torch_profiling: false
+stop_at: 1000
+stop_after: null
+default_loss_fn:
+  fused_loss: false
+  compile: false
+  z_loss: 0.0001
+  float32_logits: true
+runtime_data:
+  args: launch_scripts/train_codec_sft.py /fsx/home/weikai.huang/molmo2_codec/checkpoints/molmo2_4b_sft_dcp/step30000
+    codec-mixed-final --seq_len 32768 --data.override_max_crops=8 --device_batch_size
+    1 --num_workers 2 --prefetch_factor 1 --max_duration=45000 --stop_at=1000 --save_at=10
+    --save_interval=100 --save_num_checkpoints_to_keep=6 --save_folder=/fsx/home/weikai.huang/molmo2_codec/codec_stage2_8k_hex_validation
+    --ft_p_vit=true --ft_p_connector=true --wandb=null
+  hostname: ip-10-1-122-30
+  date: 08/05/2026, 19:56
+  world_size: 8
+  resuming_from: null
+  beaker_experiment_id: null
+  beaker_experiment_url: null
+  wandb_id: null
+  wandb_url: null
+inf_eval_config:
+  distributed: true
+  offload_model: false
+  offload_optim: false

eval/paired_summary_n200.json

@@ -0,0 +1,108 @@
+{
+  "evaluation_contract": {
+    "sample_size_per_benchmark": 200,
+    "sequence_length": 16384,
+    "timeline_sampling": "full_span_2fps",
+    "visual_token_budget": 8192
+  },
+  "results": [
+    {
+      "benchmark": "lvbench",
+      "codec": {
+        "result_path": "/fsx/home/weikai.huang/molmo2_codec/evals/multibench_step100_fullspan_n200/lvbench/codec/codec_stage2_8k_hex_validation__step100_consolidated.pt/20260807_041030_results.json",
+        "sample_count": 200,
+        "score": 49.5
+      },
+      "delta": 1.5,
+      "dense": {
+        "result_path": "/fsx/home/weikai.huang/molmo2_codec/evals/multibench_step100_fullspan_n200/lvbench/dense/codec_stage2_8k_hex_validation__step100_consolidated.pt/20260807_035924_results.json",
+        "sample_count": 200,
+        "score": 48.0
+      },
+      "manifest": "/fsx/home/weikai.huang/adacodec_vlm/eval_manifests/lvbench_n200.json",
+      "paired_ids_verified": true
+    },
+    {
+      "benchmark": "mvbench",
+      "codec": {
+        "result_path": "/fsx/home/weikai.huang/molmo2_codec/evals/multibench_step100_fullspan_n200/mvbench/codec/codec_stage2_8k_hex_validation__step100_consolidated.pt/20260807_043534_results.json",
+        "sample_count": 200,
+        "score": 68.5
+      },
+      "delta": -7.0,
+      "dense": {
+        "result_path": "/fsx/home/weikai.huang/molmo2_codec/evals/multibench_step100_fullspan_n200/mvbench/dense/codec_stage2_8k_hex_validation__step100_consolidated.pt/20260807_043132_results.json",
+        "sample_count": 200,
+        "score": 75.5
+      },
+      "manifest": "/fsx/home/weikai.huang/adacodec_vlm/eval_manifests/mvbench_n200.json",
+      "paired_ids_verified": true
+    },
+    {
+      "benchmark": "nextqa",
+      "codec": {
+        "result_path": "/fsx/home/weikai.huang/molmo2_codec/evals/multibench_step100_fullspan_n200/nextqa/codec/codec_stage2_8k_hex_validation__step100_consolidated.pt/20260807_054907_results.json",
+        "sample_count": 200,
+        "score": 83.5
+      },
+      "delta": -0.5,
+      "dense": {
+        "result_path": "/fsx/home/weikai.huang/molmo2_codec/evals/multibench_step100_fullspan_n200/nextqa/dense/codec_stage2_8k_hex_validation__step100_consolidated.pt/20260807_054534_results.json",
+        "sample_count": 200,
+        "score": 84.0
+      },
+      "manifest": "/fsx/home/weikai.huang/adacodec_vlm/eval_manifests/nextqa_n200.json",
+      "paired_ids_verified": true
+    },
+    {
+      "benchmark": "perceptiontest",
+      "codec": {
+        "result_path": "/fsx/home/weikai.huang/molmo2_codec/evals/multibench_step100_fullspan_n200/perceptiontest/codec/codec_stage2_8k_hex_validation__step100_consolidated.pt/20260807_045116_results.json",
+        "sample_count": 200,
+        "score": 83.0
+      },
+      "delta": 1.0,
+      "dense": {
+        "result_path": "/fsx/home/weikai.huang/molmo2_codec/evals/multibench_step100_fullspan_n200/perceptiontest/dense/codec_stage2_8k_hex_validation__step100_consolidated.pt/20260807_044634_results.json",
+        "sample_count": 200,
+        "score": 82.0
+      },
+      "manifest": "/fsx/home/weikai.huang/adacodec_vlm/eval_manifests/perceptiontest_n200.json",
+      "paired_ids_verified": true
+    },
+    {
+      "benchmark": "tempcompass",
+      "codec": {
+        "result_path": "/fsx/home/weikai.huang/molmo2_codec/evals/multibench_step100_fullspan_n200/tempcompass/codec/codec_stage2_8k_hex_validation__step100_consolidated.pt/20260807_044308_results.json",
+        "sample_count": 200,
+        "score": 68.5
+      },
+      "delta": -0.5,
+      "dense": {
+        "result_path": "/fsx/home/weikai.huang/molmo2_codec/evals/multibench_step100_fullspan_n200/tempcompass/dense/codec_stage2_8k_hex_validation__step100_consolidated.pt/20260807_044022_results.json",
+        "sample_count": 200,
+        "score": 69.0
+      },
+      "manifest": "/fsx/home/weikai.huang/adacodec_vlm/eval_manifests/tempcompass_n200.json",
+      "paired_ids_verified": true
+    },
+    {
+      "benchmark": "videomme",
+      "codec": {
+        "result_path": "/fsx/home/weikai.huang/molmo2_codec/evals/multibench_step100_fullspan_n200/videomme/codec/codec_stage2_8k_hex_validation__step100_consolidated.pt/20260807_035721_results.json",
+        "sample_count": 200,
+        "score": 69.0
+      },
+      "delta": -1.0,
+      "dense": {
+        "result_path": "/fsx/home/weikai.huang/molmo2_codec/evals/multibench_step100_fullspan_n200/videomme/dense/codec_stage2_8k_hex_validation__step100_consolidated.pt/20260807_034942_results.json",
+        "sample_count": 200,
+        "score": 70.0
+      },
+      "manifest": "/fsx/home/weikai.huang/adacodec_vlm/eval_manifests/videomme_n200.json",
+      "paired_ids_verified": true
+    }
+  ],
+  "schema_version": 1,
+  "status": "complete"
+}

provenance/SOURCE_FILE_SHA256SUMS

@@ -0,0 +1,7 @@
+d14af0933f43d8792f520ac13264ad222a51ba6f1471d2b9ba7b5b313a1d1d4d  mm_olmo/codec/run_budget8k_probe_7way.sh
+33e86cf61f3bbc1dd89958fe0ef7fc1812efbffec61ec9c512adf3a07c1f54e9  mm_olmo/codec/codec_eval.py
+96d9baf59ef692dac7201713df948b953a961f54cc46972ac5e835d87ddfa337  mm_olmo/codec/codec_inference.py
+5c68bd2ec561072622d2c8112786839c4a18dfae110b549eb5629ded7c03987d  mm_olmo/codec/summarize_budget8k_formal.py
+e4b8b9b1d9809745fc365ba3902baad8aa1ec9e9c0c8989284dfe3b33a3c83eb  mm_olmo/olmo/models/molmo2/molmo2.py
+e1d3a71f8000fac3bd08d67252c66608ab1e102c46ff48ab5803990de25dbb34  mm_olmo/olmo/models/molmo2/example_preprocessor.py
+09a74b6f051cc8c43b0626e3329577a1a627760593ea56bf560312e91607e0b5  mm_olmo/olmo/data/video_loader.py

provenance/SOURCE_SNAPSHOT.txt

@@ -0,0 +1,15 @@
+Formal Stage-2 job: 794
+Snapshot created: 2026-08-05 UTC
+Snapshot decode fix: requeue 1, MKV direct sparse decode after reproduced step-47 straggler
+Decode-fix commit: d9cdab3f
+Repository HEAD: c04549779c46a3e23e6149386a9019ff1c359d33
+Tracked working-tree diff SHA256: fd139fc3550820aac005b19c98763c82ee108b94dc6bbc55642e3d79562dcbf4
+
+Evaluation-critical file SHA256:
+d14af0933f43d8792f520ac13264ad222a51ba6f1471d2b9ba7b5b313a1d1d4d  codec/run_budget8k_probe_7way.sh
+33e86cf61f3bbc1dd89958fe0ef7fc1812efbffec61ec9c512adf3a07c1f54e9  codec/codec_eval.py
+96d9baf59ef692dac7201713df948b953a961f54cc46972ac5e835d87ddfa337  codec/codec_inference.py
+5c68bd2ec561072622d2c8112786839c4a18dfae110b549eb5629ded7c03987d  codec/summarize_budget8k_formal.py
+e4b8b9b1d9809745fc365ba3902baad8aa1ec9e9c0c8989284dfe3b33a3c83eb  olmo/models/molmo2/molmo2.py
+e1d3a71f8000fac3bd08d67252c66608ab1e102c46ff48ab5803990de25dbb34  olmo/models/molmo2/example_preprocessor.py
+09a74b6f051cc8c43b0626e3329577a1a627760593ea56bf560312e91607e0b5  olmo/data/video_loader.py

weights/stage1_ptokenizer_step20000.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:10fc7d0460199581d9a7f1677c79a6a519adc55184c4ba14cd44f50a4bf5eec9
+size 2206186523

weights/stage2_step100_consolidated.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:141467de9a28a53d6ebb87ef8bcf1dbf191880d39b64dd0bebc3728baaf79677
+size 19609165128