Fix resize backend, low-precision loading and hierarchical predictor; add data pipeline

Video processor
- Disable torchvision antialiasing in resize(). The reference pipeline uses
cv2.resize(..., cv2.INTER_LINEAR), which does not antialias. The two agreed
only when upscaling: downscaling a 1080p frame to 384 they differed by up to
150/255 per pixel, mean 33/255, silently feeding the model a kind of image it
never saw in training. The port now matches the reference to 1/255, the 8-bit
quantisation floor, at 240p, 480p, 720p and 1080p.
- Drop the VideosKwargs subclass used as valid_kwargs. On transformers 5 an
empty TypedDict subclass loses the field defaults and preprocess() raised
StrictDataclassFieldValidationError on every call.

Modeling
- Build RoPE frequencies in float32 and cast back to the input dtype. Building
them in x.dtype promotes q and k to float32 under bf16/fp16 weights, and the
fused attention kernels reject the mismatch against v; loading with
dtype=torch.bfloat16 raised a RuntimeError. float32 results are unchanged and
still bit-exact against the reference.
- Feed the predictor the concatenated hierarchical features when
n_output_distillation > 1. On ViT-g and ViT-G the predictor input projection
expects hidden_size * 4 channels; it was receiving the last hidden state, so
any forward without skip_predictor=True failed with a shape error.
- Resolve the attention kernel through both the transformers 4.x mapping API
and the 5.x AttentionInterface.get_interface.
- Implement output_hidden_states and output_attentions, previously accepted and
silently ignored.
- Add out_layers, returning per-level normalised features, matching the
out_layers recipe of the reference frozen-evaluation probes.
- Accept (B, C, T, H, W), (B, T, C, H, W) and (B, T, H, W, C); the channel axis
is matched against config.in_chans.
- Wire up gradient checkpointing for encoder and predictor. It was declared as
supported but never implemented.
- Register AutoModelForVideoClassification and AutoVideoProcessor in auto_map.

Data pipeline
- Add video_io.py: clip_indices() and decode_frames(), transcribed from
src/datasets/video_dataset.py. The reference places clip frames with
np.linspace over fpc*frame_step, so the effective stride is fpc*fstp/(fpc-1);
range(0, fpc*fstp, fstp) selects a different frame 12 times out of 16.
decode_frames returns RGB and decodes sequentially rather than seeking with
CAP_PROP_POS_FRAMES, which snaps to keyframes on several codecs.
- Add dense_clip_indices, temporal_coverage, aggregate_predictions and
clip_scores_to_frame_scores. The reference partitioned sampling covers 98.7%
of a ten-second video but 14.2% of a two-minute one, with a 386-frame blind
gap; dense grids cover the timeline in full. aggregate_predictions averages
softmax outputs, matching the reference evaluation loop.

Tests
- 77 tests, green on transformers 4.57.1 and 5.14.1: functional suite, parity
against the reference implementation for both distillation regimes,
end-to-end preprocessing parity, downstream-pipeline properties, and frame
sampling and decoder contract tests.
- conftest.py fails any session in which every test was skipped.

Model card
- Rewritten: fixed the repository id in the usage snippet, documented the
per-layer normalisations and what n_output_distillation implies for them,
added measured precision and robustness figures for this checkpoint, and
narrowed the "Not verified" section to what is genuinely not verified.

README.md

@@ -215,6 +215,41 @@ explicitly; it also decodes sequentially rather than seeking with `CAP_PROP_POS_
 to the nearest keyframe on several codecs and quietly returns a neighbouring frame. `decord`, the
 backend the reference pipeline uses, is preferred when installed.
 
+### Long videos and multiple clips
+
+The reference evaluation splits a video into `num_segments` partitions, takes one clip and
+`num_views_per_segment` spatial crops from each, and averages the resulting softmax
+distributions into a single prediction per video. That is variance reduction for classifying a
+short video, and it does not transfer to a long one: with the reference settings for this
+checkpoint, eight segments cover 98.7% of a ten-second video but only 14.2% of a two-minute one,
+leaving a blind gap of 386 frames — thirteen seconds during which nothing is observed.
+
+`clip_indices` reproduces that partitioned sampling. When you need a score per position in time
+rather than one prediction per video, use `dense_clip_indices` instead, and map the results back
+onto frames:
+
+```python
+from video_io import dense_clip_indices, temporal_coverage, clip_scores_to_frame_scores
+
+clips = dense_clip_indices(video_len, frames_per_clip=16, frame_step=4)   # stride=window
+print(temporal_coverage(clips, video_len))     # {'covered_fraction': 1.0, 'max_gap': 0, ...}
+
+frame_scores = clip_scores_to_frame_scores(clips, scores, video_len, reduce="max")
+```
+
+`stride` controls overlap and therefore temporal resolution. `temporal_coverage` reports what a
+set of clips actually looks at, which is worth checking before trusting any per-frame metric.
+
+For video-level predictions, `aggregate_predictions` reproduces the reference combination —
+the mean of the softmax outputs, not of the logits. The two are different estimators and can rank
+classes differently; on a two-view example they give `[0.52, 0.19, 0.29]` against
+`[0.77, 0.10, 0.13]`.
+
+Note that dense extraction has a storage cost. A 16-frame clip at 384 is 4608 tokens, so caching
+token-level features for a two-minute video is about 1.8 GB at this hidden size. Pooling to one
+vector per clip is four orders of magnitude smaller, but forecloses any probe that consumes the
+token sequence.
+
 ## Validation
 
 All tests use float32 and `torch.no_grad()`.
@@ -284,13 +319,13 @@ reach. Both reproduction scripts are shipped here; point `VJEPA2_REPO` at a loca
 
 ### 4. Functional test suite
 
-77 tests ship with the repository, green on both `transformers==4.57.1` and
+88 tests ship with the repository, green on both `transformers==4.57.1` and
 `transformers==5.14.1`.
 
 | File | Tests | Needs |
 | --- | --- | --- |
 | `tests/test_vjepa21.py` | 30 | nothing |
-| `tests/test_frame_sampling.py` | 21 | ffmpeg |
+| `tests/test_frame_sampling.py` | 32 | ffmpeg |
 | `tests/test_parity_official.py` | 9 | `VJEPA2_REPO` |
 | `tests/test_end_to_end_pipeline.py` | 9 | `VJEPA2_REPO`, optionally `VJEPA21_CKPT` |
 | `tests/test_thesis_robustness.py` | 8 | optionally `VJEPA21_CKPT` |
@@ -302,6 +337,10 @@ checkpointing parity, low-precision weight loading in bf16 and fp16, the video p
 and normalization constants, and a full `save_pretrained` / `from_pretrained` round-trip through the
 Auto classes. It needs neither weights nor the reference repository.
 
+`tests/conftest.py` fails any session in which every test was skipped. A suite that silently
+disables itself — usually a missing `VJEPA2_REPO`, or a relative path resolved against the wrong
+working directory — prints `9 skipped` in the same colour as `9 passed`.
+
 ### 5. Properties relevant to downstream pipelines
 
 `tests/test_thesis_robustness.py` checks assumptions an experimental pipeline tends to make
@@ -359,9 +398,9 @@ Stated explicitly so the scope of the validation above is not overread:
 - **Your own data pipeline.** `video_io.py` reproduces the reference sampler and decoder and is
   tested against them, but if you write your own, only the contract tests in
   `tests/test_frame_sampling.py` stand between you and a silent mismatch.
-- **Multi-clip aggregation.** The reference evaluations average predictions over several clips and
-  several spatial crops per clip. `clip_indices` supports `num_clips`, but no aggregation policy is
-  implemented or validated here.
+- **Aggregation policy.** `aggregate_predictions` reproduces the reference combination and
+  `dense_clip_indices` covers the whole timeline, but which policy suits a given task — mean over
+  clips, max, top-k — is a modelling decision that is neither made nor evaluated here.
 - **Multiple mask pairs in the predictor.** Only a single `(context_mask, target_mask)` pair is
   supported.
 - **Reduced precision downstream.** bf16 and fp16 deviations are measured above, but their effect

tests/test_frame_sampling.py

@@ -363,3 +363,115 @@ def test_user_pipeline_end_to_end(indexed_video):
     got = [index_of(f) for f in frames]
     print(f"\n[pipeline] expected {expected.tolist()}\n[pipeline] got      {got}")
     assert got == expected.tolist()
+
+
+# --- 4. multi-clip: coverage and aggregation --------------------------------
+
+import sys as _sys  # noqa: E402
+
+_sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+
+def _video_io():
+    try:
+        import video_io
+    except ImportError:
+        pytest.skip("video_io.py not importable; run from the repository root")
+    return video_io
+
+
+def test_partitioned_sampling_leaves_long_videos_mostly_unseen():
+    """The reference sampler was designed for short single-label clips.
+
+    On a 10-second video eight segments see almost everything. On a two-minute
+    surveillance video the same settings see 14% of it, and leave a blind gap of
+    386 frames — thirteen seconds during which an event is not observed at all.
+    """
+    io = _video_io()
+    short = io.temporal_coverage(io.clip_indices(300, 16, 4, num_clips=8), 300)
+    long = io.temporal_coverage(io.clip_indices(3600, 16, 4, num_clips=8), 3600)
+
+    print(f"\n[coverage] 10 s video : {short['covered_fraction']*100:.1f}% seen, "
+          f"max gap {short['max_gap']} frames")
+    print(f"[coverage] 2 min video: {long['covered_fraction']*100:.1f}% seen, "
+          f"max gap {long['max_gap']} frames")
+
+    assert short["covered_fraction"] > 0.95
+    assert long["covered_fraction"] < 0.20
+    assert long["max_gap"] > 300
+
+
+@pytest.mark.parametrize("video_len", [300, 3600, 9000])
+def test_dense_grid_covers_everything(video_len):
+    """A sliding grid leaves no gap, which is what frame-level scoring needs."""
+    io = _video_io()
+    clips = io.dense_clip_indices(video_len, 16, 4)
+    metrics = io.temporal_coverage(clips, video_len)
+    print(f"\n[dense] {video_len} frames -> {len(clips)} clips, "
+          f"{metrics['covered_fraction']*100:.1f}% covered")
+    assert metrics["covered_fraction"] == 1.0
+    assert metrics["max_gap"] == 0
+    assert all(c.max() < video_len for c in clips)
+
+
+def test_dense_grid_stride_controls_overlap():
+    io = _video_io()
+    contiguous = io.dense_clip_indices(3600, 16, 4)
+    overlapping = io.dense_clip_indices(3600, 16, 4, stride=32)
+    assert len(overlapping) > len(contiguous)
+    assert io.temporal_coverage(overlapping, 3600)["covered_fraction"] == 1.0
+
+
+def test_aggregation_averages_probabilities_not_logits():
+    """The reference averages softmax outputs. Averaging logits is a different
+    estimator and can rank classes differently."""
+    io = _video_io()
+    views = [np.array([[6.0, 0.0, 0.0]]), np.array([[0.0, 2.0, 2.4]])]
+
+    probabilities = io.aggregate_predictions(views)
+    assert np.allclose(probabilities.sum(axis=-1), 1.0)
+
+    logit_mean = np.mean(views, axis=0)[0]
+    logit_mean = np.exp(logit_mean - logit_mean.max())
+    logit_mean /= logit_mean.sum()
+
+    print(f"\n[aggregate] probability mean {np.round(probabilities[0], 4)}")
+    print(f"[aggregate] logit mean       {np.round(logit_mean, 4)}")
+    assert not np.allclose(probabilities[0], logit_mean, atol=1e-3)
+
+
+def test_aggregation_is_order_independent():
+    io = _video_io()
+    views = [np.random.randn(2, 5) for _ in range(4)]
+    a = io.aggregate_predictions(views)
+    b = io.aggregate_predictions(views[::-1])
+    assert np.allclose(a, b)
+
+
+@pytest.mark.parametrize("reduce", ["max", "mean", "first"])
+def test_clip_scores_reach_every_frame(reduce):
+    """Frame-level AUC and AP need a score for every frame, including those no
+    clip covered."""
+    io = _video_io()
+    video_len = 3600
+    clips = io.dense_clip_indices(video_len, 16, 4)
+    scores = np.linspace(0, 1, len(clips))
+
+    frame_scores = io.clip_scores_to_frame_scores(clips, scores, video_len, reduce=reduce)
+    assert frame_scores.shape == (video_len,)
+    assert np.isfinite(frame_scores).all()
+    assert frame_scores.min() >= scores.min() - 1e-9
+    assert frame_scores.max() <= scores.max() + 1e-9
+
+
+def test_max_reduction_propagates_a_single_high_clip():
+    io = _video_io()
+    video_len = 1000
+    clips = io.dense_clip_indices(video_len, 16, 4)
+    scores = np.zeros(len(clips))
+    scores[3] = 1.0
+    frame_scores = io.clip_scores_to_frame_scores(clips, scores, video_len, reduce="max")
+    flagged = int((frame_scores > 0.5).sum())
+    print(f"\n[scores] one clip at 1.0 flags {flagged} frames")
+    assert flagged >= 64
+    assert flagged < video_len

video_io.py

@@ -219,4 +219,173 @@ def _check_complete(frames: dict, wanted: list[int], path: str) -> None:
         )
 
 
-__all__ = ["clip_indices", "frame_step_for_fps", "decode_frames", "available_backends"]
+__all__ = [
+    "clip_indices",
+    "dense_clip_indices",
+    "frame_step_for_fps",
+    "decode_frames",
+    "available_backends",
+    "aggregate_predictions",
+    "temporal_coverage",
+    "clip_scores_to_frame_scores",
+]
+
+
+# --- multi-clip: aggregation, coverage, dense grids --------------------------
+
+
+def aggregate_predictions(logits_per_view: Sequence[np.ndarray]) -> np.ndarray:
+    """Combine the predictions of several clips or spatial views of one video.
+
+    Matches the reference evaluation loop, which averages **softmax
+    probabilities**, not logits:
+    `sum(F.softmax(o, dim=1) for o in outputs) / len(outputs)`
+    (`evals/video_classification_frozen/eval.py`).
+
+    Averaging logits instead is a different estimator — it is a geometric rather
+    than an arithmetic mean over probabilities — and it lets one confident view
+    dominate. The two agree only when the views agree.
+
+    Args:
+        logits_per_view: One `(batch, num_classes)` array per clip or view.
+
+    Returns:
+        `(batch, num_classes)` probabilities that sum to one.
+    """
+    if not logits_per_view:
+        raise ValueError("logits_per_view is empty")
+    probabilities = []
+    for logits in logits_per_view:
+        logits = np.asarray(logits, dtype=np.float64)
+        shifted = logits - logits.max(axis=-1, keepdims=True)
+        exponentiated = np.exp(shifted)
+        probabilities.append(exponentiated / exponentiated.sum(axis=-1, keepdims=True))
+    return np.mean(probabilities, axis=0)
+
+
+def temporal_coverage(clips: Sequence[np.ndarray], video_len: int) -> dict:
+    """How much of a video a set of clips actually looks at.
+
+    The partitioned sampling used by the reference evaluation was designed for
+    short single-label videos. On a long one it leaves most of the timeline
+    unseen, and an event shorter than `max_gap` can fall entirely between two
+    clips without any of it being observed.
+
+    Returns:
+        `covered_fraction`, `max_gap` and `num_gaps`, in frames.
+    """
+    seen = np.zeros(video_len, dtype=bool)
+    for clip in clips:
+        clip = np.asarray(clip)
+        seen[int(clip.min()) : int(clip.max()) + 1] = True
+
+    gaps, run = [], 0
+    for visible in seen:
+        if visible:
+            if run:
+                gaps.append(run)
+            run = 0
+        else:
+            run += 1
+    if run:
+        gaps.append(run)
+
+    return {
+        "covered_fraction": float(seen.mean()),
+        "max_gap": int(max(gaps)) if gaps else 0,
+        "num_gaps": len(gaps),
+    }
+
+
+def dense_clip_indices(
+    video_len: int,
+    frames_per_clip: int,
+    frame_step: int,
+    stride: int | None = None,
+) -> list[np.ndarray]:
+    """A sliding grid of clips covering the whole video.
+
+    Use this instead of `clip_indices` when you need a score per position in
+    time rather than one prediction per video — temporal anomaly detection,
+    action localisation, anything scored frame by frame. `clip_indices`
+    partitions the video and samples one clip per partition, which is the right
+    thing for classifying a short video and the wrong thing here.
+
+    Args:
+        stride: Frames between the start of consecutive clips. Defaults to the
+            clip window `frames_per_clip * frame_step`, giving contiguous
+            non-overlapping clips. A smaller value overlaps them, which raises
+            temporal resolution at proportional cost.
+
+    Returns:
+        Clips in temporal order, the last one clamped to the end of the video.
+    """
+    window = frames_per_clip * frame_step
+    stride = window if stride is None else stride
+    if stride <= 0:
+        raise ValueError(f"stride must be positive, got {stride}")
+
+    starts = list(range(0, max(video_len - window, 0) + 1, stride))
+    if not starts:
+        starts = [0]
+    if starts[-1] + window < video_len:
+        starts.append(video_len - window)
+
+    clips = []
+    for start in starts:
+        idx = np.linspace(start, start + window, num=frames_per_clip)
+        clips.append(np.clip(idx, 0, video_len - 1).astype(np.int64))
+    return clips
+
+
+def clip_scores_to_frame_scores(
+    clips: Sequence[np.ndarray],
+    scores: Sequence[float],
+    video_len: int,
+    reduce: str = "max",
+) -> np.ndarray:
+    """Spread clip-level scores back over frames, for frame-level metrics.
+
+    Frame-level AUC on UCF-Crime and average precision on XD-Violence are
+    computed per frame, so a clip score has to be assigned to the frames the
+    clip covers. Overlapping clips give a frame several scores; `reduce` picks
+    between them. Frames covered by no clip keep the score of the nearest
+    covered frame, so the output is dense.
+
+    Args:
+        reduce: "max" (an anomaly anywhere in the window marks the window),
+            "mean" (smoother, blunter) or "first".
+    """
+    if len(clips) != len(scores):
+        raise ValueError(f"{len(clips)} clips but {len(scores)} scores")
+
+    accumulated = np.zeros(video_len, dtype=np.float64)
+    counts = np.zeros(video_len, dtype=np.int64)
+    assigned = np.zeros(video_len, dtype=bool)
+
+    for clip, score in zip(clips, scores):
+        clip = np.asarray(clip)
+        lo, hi = int(clip.min()), int(clip.max()) + 1
+        if reduce == "max":
+            accumulated[lo:hi] = np.where(
+                assigned[lo:hi], np.maximum(accumulated[lo:hi], score), score
+            )
+        elif reduce == "mean":
+            accumulated[lo:hi] += score
+        elif reduce == "first":
+            accumulated[lo:hi] = np.where(assigned[lo:hi], accumulated[lo:hi], score)
+        else:
+            raise ValueError(f"unknown reduce {reduce!r}")
+        counts[lo:hi] += 1
+        assigned[lo:hi] = True
+
+    if reduce == "mean":
+        accumulated[counts > 0] /= counts[counts > 0]
+
+    if not assigned.all():
+        covered = np.flatnonzero(assigned)
+        if covered.size == 0:
+            raise ValueError("no frame was covered by any clip")
+        nearest = covered[np.abs(np.subtract.outer(np.arange(video_len), covered)).argmin(axis=1)]
+        accumulated = np.where(assigned, accumulated, accumulated[nearest])
+    return accumulated