texture_probe: read cached latents, normalise for exposure, and a correction

Run against a real 3x243f chain, which found two defects and reversed a
finding.

Cached latents never loaded. torch.load has to import
comfy.nested_tensor to rebuild the object, and without the ComfyUI root
on sys.path store._get_latent caught the ModuleNotFoundError and the
probe printed "none cached for this hop" -- reporting a path problem as
an absent latent, which is the difference between a render to redo and
a one-line fix. enable_latent_reads() appends the root and imports
nothing; the module pulls in only torch when pickle reaches for it.

Band energy was not exposure-normalised. The docstring claimed a
band-pass "does not care about the local mean", which is true of an
offset and false of a scale -- a 5% brighter picture measures ~5% more
band energy with texture unchanged. This chain's luma rose 4.7%, so
whole-frame mid read x1.084 when the texture part was n1.035. Both
columns are printed now.

The correction: the ratchet is a staircase, not a ramp. Within each
hop, head mid last/first is 0.996 / 1.003 / 1.018 -- flat -- and each
join steps x1.042, twice, identically. The earlier "continuous climb"
came from binning someone else's master without knowing where their
joins were, which is an under-determined reading I should have flagged
as one. Re-read with boundaries in hand, their bins are a staircase
too. The hand-off is the cause, not the carrier, which means it can be
removed rather than damped.

And the measurement this was all for: latent sigma FALLS 1.2% across
the chain while the pixel mid band climbs 8%; the high-band fraction
rises 1.6%, monotone. pin_renorm=on would scale this pin up x1.012 on
a latent whose high band is already too hot -- the shipped lever pushes
the wrong way. Phase 2a is now evidenced rather than assumed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015MGjcAV8bDy93qJfmLi9kw

CLAUDE.md

@@ -363,7 +363,8 @@ hop N-1 ends. Hop N opens by holding what it was handed.
 - `ref_image_size="max"` (2048 short-edge) is slower per step than `"match"` — an explicit tradeoff, not a default to silently upgrade.
 - Fewer than 3 wired reference stills triggers a warning log; unconnected `Load Image` nodes do not count as wired.
 - **Never judge texture drift with mean `|Laplacian|`, or with any whole-frame scalar.** Measured on two 44 s H3 chains whose faces visibly come apart, it reads **0.961 and 0.973** -- one of them says the clip improved. It is an area average (a face is ~6% of a portrait frame and loses to the background sixteen to one) and it pools every spatial frequency, so band-to-band movement cancels. On those clips global contrast FELL while mid-band energy ROSE. Use `tools/texture_probe.py`: three Gaussian-difference bands (fine <1px / **mid 1-2.5px, the one that moves** / coarse 2.5-6px), a subject box against a background control, and the within-hop slope. `--video` runs it on any file, including other people's rigs. See DEVLOG 26.
-- **The texture ratchet is not injected at the join.** The climb is continuous through each hop with no step at the seams, so the boundary is the ratchet *pawl*, not the cause: `prev_imgs = imgs[-tail_n:]` hands forward the hop's tail, which is its most degraded stretch. It is also independent of exposure -- on the measured clips luma held at 92 -> 90 with anchoring on while mid-band climbed 17-35%. Do not expect `tone_compensate` to touch it, and do not tune it on a 2-hop chain; the effect only separates at 3+.
+- **The texture ratchet is injected AT the join, and nowhere else.** Measured off the hop cache on a 3x243f chain, head box, mid band: within each hop last/first is `0.996 / 1.003 / 1.018` -- flat -- and each join steps `x1.042`, twice, identically. It is a staircase, not a ramp. So the hand-off is the cause and not merely the carrier, which is the good news: a hand-off-side lever can remove this rather than damp it. `prev_imgs = imgs[-tail_n:]` and the pin latent are conditioning-only copies (`.clone()`, and `_condition_pin_latent` rebuilds rather than mutates), so they can be corrected without touching a single visible pixel.
+- **Texture drift is independent of exposure, but the metric is not.** A band-pass removes the local mean, so it ignores a brightness *offset* -- it is not scale-free, and a 5% brighter picture measures ~5% more band energy with texture unchanged. `texture_probe` prints both (`x` raw, `n` exposure-normalised); where they disagree the difference is the scene brightening. On the 3-hop chain whole-frame mid read `x1.084` raw but `n1.035` normalised. Do not expect `tone_compensate` to touch the ratchet, and do not tune on a 2-hop chain; the effect only separates at 3+.
 - Soundtrack is official H3, not a pack dialect: `(S1)` + `<d>…</d>` for lines, `overall_soundscape` for ambience/physical (or `N/A` for requested silence). Do not invent beat-keywords. “No speech” / “no dialogue” remains negation/gibberish.
 
 ## The engineering log

docs/DEVLOG.md

@@ -1137,3 +1137,77 @@ re-encodes decoded pixels, which is the decode/re-encode round trip the reporter
 measured at 1.530, "much worse", on their own rig. A chain that quietly fell
 back has both levers dead and the worse hand-off. The log says which pin ran;
 it is worth reading before trusting any A/B.
+
+### Correction, 2026-09-01: it is a staircase, not a ramp
+
+The section above says the climb is continuous with no step at the joins. That
+was wrong, and it was wrong in the way that matters most -- it is the claim
+that decides where a correction belongs.
+
+It came from binning the reporter's master at 60 frames **without knowing where
+their joins were**. A step function sampled that way, with content noise on
+top, reads as a ramp if you want it to. The inference was under-determined and
+I did not say so.
+
+The hop cache settles it, because there the boundaries are known. A 3 x 243f
+chain, 736x1280, overlap 22, Motion-Context pin, `pin_renorm off`, head box,
+mid band, with the regenerated overlap frames excluded:
+
+    hop 1   0.00985 0.00976 0.00963 0.00992 0.00981    last/first 0.996
+    hop 2   0.01050 0.01025 0.01029 0.01033 0.01053    last/first 1.003
+    hop 3   0.01058 0.01025 0.01044 0.01058 0.01076    last/first 1.018
+
+    join 1 -> 2   tail 0.01006 -> body start 0.01048   x1.042
+    join 2 -> 3   tail 0.01068 -> body start 0.01113   x1.042
+
+Flat inside every hop. **The same +4.2% at both joins.** Those two frames are
+adjacent in scene time -- hop N+1's frame 22 continues from hop N's last -- so
+it is a genuine discontinuity and not a gap the scene moved through.
+
+Re-reading the reporter's bins with this in hand, theirs is a staircase too:
+1.57 1.55 1.56 1.56 1.57 | 1.60 1.68 1.66 1.61 1.64 | 1.71 1.71 1.72 1.77 |
+1.85 1.85 1.93 -- four plateaus at ~1.56, ~1.64, ~1.73, ~1.88, stepping +5%,
++6%, +9%, on a chain they told us was four hops. Both rigs agree. I had the
+right data and read it wrong.
+
+This is better news than the original reading. "Self-conditioning drift inside
+the generation" could only ever be damped; a step injected at the hand-off can
+be removed at the hand-off, and the hand-off copies are conditioning-only.
+
+### And the latent measurement, which was the point
+
+From the same cache, per hop: component [0] sigma `1.0414 -> 1.0376 -> 1.0289`,
+its high band `0.3794 -> 0.3811 -> 0.3809`.
+
+Sigma **falls 1.2%** while the pixel mid band climbs 8%. The high-band
+*fraction* -- hi/sigma -- goes `0.3643 -> 0.3673 -> 0.3702`, up 1.6% and
+monotone. So the latent does carry the tilt, and total sigma does not see it.
+
+`pin_renorm=on` would have multiplied this pin by `1.0414/1.0289 = x1.012`,
+scaling every band up uniformly, on a latent whose high band was already 1.6%
+too hot. **On this chain the shipped lever pushes the wrong way.** That is not
+a small correction to it; it is the wrong statistic, and Phase 2a's band-matched
+rescale is now evidenced rather than assumed.
+
+One caveat kept in view: 1.6% in the latent against 8% in pixels. The VAE
+decode is nonlinear, so the two are not expected to be proportional, but the
+gap is large enough that the lever's gain will have to be fitted against
+measured output rather than derived from the latent ratio.
+
+### Two probe defects the real data exposed
+
+**Cached latents did not load at all.** `torch.load` has to import
+`comfy.nested_tensor` to rebuild the object; without the ComfyUI root on
+`sys.path`, `store._get_latent` caught the ModuleNotFoundError and the probe
+printed "none cached for this hop" -- reporting a path problem as an absent
+latent. `hopcache.enable_latent_reads()` appends the root and nothing else;
+the module imports only torch when pickle reaches for it, so it is safe to run
+beside a queued render.
+
+**Band energy was not exposure-normalised.** The probe's own docstring claimed
+band-pass output "does not care about the local mean", which is true of an
+offset and false of a scale: brighten a frame 5% and every band grows with it.
+The 3-hop chain's luma rose 4.7%, so whole-frame mid read `x1.084` when the
+texture part was `n1.035`. Both columns are printed now. The head box was
+unaffected either way -- the brightening was in the background -- which is
+exactly the kind of thing a single whole-frame number cannot tell you.

tools/hopcache.py

@@ -24,6 +24,26 @@ import types
 _COMFY_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(
     os.path.dirname(os.path.abspath(__file__)))))
 DEFAULT_ROOT = os.path.join(_COMFY_ROOT, "temp", "h3_ref_chain_hops")
+COMFY_ROOT = _COMFY_ROOT
+
+
+def enable_latent_reads():
+    """Put the ComfyUI root on sys.path so a cached latent will unpickle.
+
+    A hop's `.latent.pt` holds a `comfy.nested_tensor.NestedTensor`, and
+    `torch.load` has to import that class to rebuild it. Without this the load
+    raises ModuleNotFoundError, `store._get_latent` swallows it and prints
+    "cached latent unreadable ... falling back to the pixel pin", and a probe
+    reports "no latent cached" for every hop -- which reads as "the render did
+    not store one" rather than "this process cannot open it". The two need
+    telling apart: one is a render to redo, the other is a path entry.
+
+    Only a path entry. Nothing is imported here, and `comfy/nested_tensor.py`
+    imports nothing but torch when pickle does reach for it -- no CUDA, no
+    model management. Safe to call while a render is queued.
+    """
+    if _COMFY_ROOT not in sys.path:
+        sys.path.append(_COMFY_ROOT)
 
 
 def load_store():

tools/texture_probe.py

@@ -100,6 +100,13 @@ def bands(gray, torch):
         "fine": (gray - b1).abs().mean(dim=(1, 2)),
         "mid": (b1 - b2).abs().mean(dim=(1, 2)),
         "coarse": (b2 - b3).abs().mean(dim=(1, 2)),
+        # This box's own mean level, carried so the row can normalise by it.
+        # A band-pass removes the LOCAL mean, which is why it ignores a
+        # brightness offset -- but it is not scale-free: brighten a picture by
+        # 5% and every band amplitude grows about 5% with it, texture
+        # unchanged. On a chain whose exposure also drifts, the raw ratio and
+        # the normalised one answer different questions and both are wanted.
+        "_luma": gray.mean(dim=(1, 2)),
     }
 
 
@@ -183,13 +190,19 @@ def measure(gray, torch, boxes):
 
 def _row(label, per_band, ref, slopes):
     bits = []
+    lum = float(per_band["_luma"].mean())
     for b in BANDS:
         v = float(per_band[b].mean())
         if ref is None:
-            bits.append(f"{b} {v:.4f}        ")
+            bits.append(f"{b} {v:.4f}              ")
         else:
             r = v / ref[b] if ref[b] else float("nan")
-            bits.append(f"{b} {v:.4f} x{r:.3f}")
+            # n = the same ratio with exposure divided out. Where n and x
+            # disagree, the difference is the scene getting brighter, not
+            # rougher -- and only n is the texture ratchet.
+            rn = ((v / lum) / (ref[b] / ref["_luma"])
+                  if ref[b] and lum and ref["_luma"] else float("nan"))
+            bits.append(f"{b} {v:.4f} x{r:.3f} n{rn:.3f}")
     tail = ""
     if slopes is not None:
         tail = "   slope/100f " + " ".join(
@@ -283,13 +296,15 @@ def run_video(args, torch, stride):
 
     for name in ("head", "bg", "whole"):
         vals = per[name]
-        ref = {b: float(vals[b][edges[0]:edges[1]].mean()) for b in BANDS}
+        ref = {b: float(vals[b][edges[0]:edges[1]].mean())
+               for b in tuple(BANDS) + ("_luma",)}
         print(f"  [{name}]")
         for i in range(segs):
             a, z = edges[i], edges[i + 1]
             if z - a < 2:
                 continue
-            print(_row(f"seg {i + 1}", {b: vals[b][a:z] for b in BANDS},
+            print(_row(f"seg {i + 1}",
+                       {b: vals[b][a:z] for b in tuple(BANDS) + ("_luma",)},
                        None if i == 0 else ref, None))
         print()
 
@@ -306,6 +321,9 @@ def run_video(args, torch, stride):
 
 
 def run_cache(args, torch, stride):
+    # Before select(): the store reads latents during get(), and an
+    # unimportable NestedTensor is reported as an absent one.
+    hopcache.enable_latent_reads()
     store, hops, report = hopcache.select(args.root, args.chain)
     if report:
         print(report)
@@ -332,7 +350,8 @@ def run_cache(args, torch, stride):
         g = luma(imgs, torch)
         per = measure(g, torch, boxes)
         if ref is None:
-            ref = {k: {b: float(per[k][b].mean()) for b in BANDS} for k in per}
+            ref = {k: {b: float(per[k][b].mean()) for b in tuple(BANDS) + ("_luma",)}
+               for k in per}
         print(f"hop {hop} ({key[:8]}): {imgs.shape[0]} sampled frames of "
               f"{w}x{h}   luma {float(g.mean()):.4f}")
         for name in ("head", "bg", "whole"):