pin_renorm: a band-matched mode, because sigma is a no-op here
Browse filesThe ratchet drifts the high-band FRACTION of the pin latent. A fraction
is scale-invariant, and a uniform rescale is all `pin_renorm=on` ever
did. Driven end to end with a 12.74% band drift planted in hop 2:
`sigma` applied x0.9651 and left the drift at +12.74%; `band` took it
to -0.04%. There is no gain or anchor choice that rescues a
scale-invariant statistic from a scale -- the lever was mis-specified,
not mis-tuned. Kept as `sigma` because pre-0.5 workflows saved "on".
pin_renorm is now ["off", "sigma", "band"]; "on" maps to "sigma", and
the combo keeps its widget slot.
Two defects caught on the way, both of the kind that look like success:
match_band first used `k = target * sigma / hi_sigma`, but scaling the
high band changes the sigma it is a fraction of, so the target moves as
you apply it -- it landed 5% short, in the right direction. It now
solves the orthogonal fixed point and refines against the measured
statistic.
The test fixture used torch.randn, whose high-band fraction is 0.966 --
against its ceiling, where the clamp does the correcting and every
assertion measures the clamp. Real latents sit at 0.3643; the fixture
now lands there and an assertion holds it in that regime.
Also: pin_cond was keyed into hop 1, which has no pin, so flipping a
lever re-rendered a byte-identical hop. A third of every A/B, on runs
already too slow to repeat.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015MGjcAV8bDy93qJfmLi9kw
- CLAUDE.md +1 -0
- docs/DEVLOG.md +75 -0
- h3_ref_chain.py +110 -52
- latents.py +97 -0
- tools/check_texture.py +65 -0
|
@@ -364,6 +364,7 @@ hop N-1 ends. Hop N opens by holding what it was handed.
|
|
| 364 |
- Fewer than 3 wired reference stills triggers a warning log; unconnected `Load Image` nodes do not count as wired.
|
| 365 |
- **Never judge texture drift with mean `|Laplacian|`, or with any whole-frame scalar.** Measured on two 44 s H3 chains whose faces visibly come apart, it reads **0.961 and 0.973** -- one of them says the clip improved. It is an area average (a face is ~6% of a portrait frame and loses to the background sixteen to one) and it pools every spatial frequency, so band-to-band movement cancels. On those clips global contrast FELL while mid-band energy ROSE. Use `tools/texture_probe.py`: three Gaussian-difference bands (fine <1px / **mid 1-2.5px, the one that moves** / coarse 2.5-6px), a subject box against a background control, and the within-hop slope. `--video` runs it on any file, including other people's rigs. See DEVLOG 26.
|
| 366 |
- **The texture ratchet is injected AT the join, and nowhere else.** Measured off the hop cache on a 3x243f chain, head box, mid band: within each hop last/first is `0.996 / 1.003 / 1.018` -- flat -- and each join steps `x1.042`, twice, identically. It is a staircase, not a ramp. So the hand-off is the cause and not merely the carrier, which is the good news: a hand-off-side lever can remove this rather than damp it. `prev_imgs = imgs[-tail_n:]` and the pin latent are conditioning-only copies (`.clone()`, and `_condition_pin_latent` rebuilds rather than mutates), so they can be corrected without touching a single visible pixel.
|
|
|
|
| 367 |
- **Texture drift is independent of exposure, but the metric is not.** A band-pass removes the local mean, so it ignores a brightness *offset* -- it is not scale-free, and a 5% brighter picture measures ~5% more band energy with texture unchanged. `texture_probe` prints both (`x` raw, `n` exposure-normalised); where they disagree the difference is the scene brightening. On the 3-hop chain whole-frame mid read `x1.084` raw but `n1.035` normalised. Do not expect `tone_compensate` to touch the ratchet, and do not tune on a 2-hop chain; the effect only separates at 3+.
|
| 368 |
- Soundtrack is official H3, not a pack dialect: `(S1)` + `<d>…</d>` for lines, `overall_soundscape` for ambience/physical (or `N/A` for requested silence). Do not invent beat-keywords. “No speech” / “no dialogue” remains negation/gibberish.
|
| 369 |
|
|
|
|
| 364 |
- Fewer than 3 wired reference stills triggers a warning log; unconnected `Load Image` nodes do not count as wired.
|
| 365 |
- **Never judge texture drift with mean `|Laplacian|`, or with any whole-frame scalar.** Measured on two 44 s H3 chains whose faces visibly come apart, it reads **0.961 and 0.973** -- one of them says the clip improved. It is an area average (a face is ~6% of a portrait frame and loses to the background sixteen to one) and it pools every spatial frequency, so band-to-band movement cancels. On those clips global contrast FELL while mid-band energy ROSE. Use `tools/texture_probe.py`: three Gaussian-difference bands (fine <1px / **mid 1-2.5px, the one that moves** / coarse 2.5-6px), a subject box against a background control, and the within-hop slope. `--video` runs it on any file, including other people's rigs. See DEVLOG 26.
|
| 366 |
- **The texture ratchet is injected AT the join, and nowhere else.** Measured off the hop cache on a 3x243f chain, head box, mid band: within each hop last/first is `0.996 / 1.003 / 1.018` -- flat -- and each join steps `x1.042`, twice, identically. It is a staircase, not a ramp. So the hand-off is the cause and not merely the carrier, which is the good news: a hand-off-side lever can remove this rather than damp it. `prev_imgs = imgs[-tail_n:]` and the pin latent are conditioning-only copies (`.clone()`, and `_condition_pin_latent` rebuilds rather than mutates), so they can be corrected without touching a single visible pixel.
|
| 367 |
+
- **`pin_renorm=sigma` cannot fix the ratchet, and no tuning will rescue it.** The drifting statistic is the high-band *fraction* -- a ratio -- and a ratio is invariant under any uniform rescale, which is all a sigma match does. Measured end to end through the node: a 12.74% band drift stayed at **+12.74%** under `sigma` and went to **-0.04%** under `band`. Total sigma is not merely a weak proxy here; on the real chain it FELL 1.2% while the fraction rose, so it does not even point the right way. `sigma` is kept only because pre-0.5 workflows saved it as `on`.
|
| 368 |
- **Texture drift is independent of exposure, but the metric is not.** A band-pass removes the local mean, so it ignores a brightness *offset* -- it is not scale-free, and a 5% brighter picture measures ~5% more band energy with texture unchanged. `texture_probe` prints both (`x` raw, `n` exposure-normalised); where they disagree the difference is the scene brightening. On the 3-hop chain whole-frame mid read `x1.084` raw but `n1.035` normalised. Do not expect `tone_compensate` to touch the ratchet, and do not tune on a 2-hop chain; the effect only separates at 3+.
|
| 369 |
- Soundtrack is official H3, not a pack dialect: `(S1)` + `<d>…</d>` for lines, `overall_soundscape` for ambience/physical (or `N/A` for requested silence). Do not invent beat-keywords. “No speech” / “no dialogue” remains negation/gibberish.
|
| 370 |
|
|
@@ -1211,3 +1211,78 @@ The 3-hop chain's luma rose 4.7%, so whole-frame mid read `x1.084` when the
|
|
| 1211 |
texture part was `n1.035`. Both columns are printed now. The head box was
|
| 1212 |
unaffected either way -- the brightening was in the background -- which is
|
| 1213 |
exactly the kind of thing a single whole-frame number cannot tell you.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1211 |
texture part was `n1.035`. Both columns are printed now. The head box was
|
| 1212 |
unaffected either way -- the brightening was in the background -- which is
|
| 1213 |
exactly the kind of thing a single whole-frame number cannot tell you.
|
| 1214 |
+
|
| 1215 |
+
## 27. The band lever, and why the old one could never have worked (2026-09-01)
|
| 1216 |
+
|
| 1217 |
+
Built after §26's correction, on the finding that the ratchet is a step at the
|
| 1218 |
+
join: +4.2% mid-band, twice, identically, on a 3-hop chain. Two identical steps
|
| 1219 |
+
is already a model -- constant multiplicative step per join, geometric in hop
|
| 1220 |
+
count. It predicts hop 3 at 1.042^2 = 1.086 against 1.079 measured. So the
|
| 1221 |
+
shape did not need a 4-5 hop run to pin down, which matters: those runs are
|
| 1222 |
+
expensive enough that the user does not do them.
|
| 1223 |
+
|
| 1224 |
+
`pin_renorm` is now `["off", "sigma", "band"]`. `"on"` maps to `"sigma"`, so
|
| 1225 |
+
pre-0.5 workflows keep their behaviour, and the combo keeps its widget slot --
|
| 1226 |
+
adding options is safe, adding widgets is not.
|
| 1227 |
+
|
| 1228 |
+
### The old lever is a no-op, provably
|
| 1229 |
+
|
| 1230 |
+
The statistic that drifts is the high-band **fraction**, hi_sigma / sigma. A
|
| 1231 |
+
fraction is invariant under uniform rescaling, and a uniform rescale is the
|
| 1232 |
+
entirety of what `sigma` mode does. Driven end to end through
|
| 1233 |
+
`_condition_pin_latent` with a 12.74% band drift planted in hop 2:
|
| 1234 |
+
|
| 1235 |
+
mode=off ratio after 0.3526 (anchor 0.3128) err +12.74%
|
| 1236 |
+
mode=sigma ratio after 0.3526 (anchor 0.3128) err +12.74% x0.9651 applied
|
| 1237 |
+
mode=band ratio after 0.3127 (anchor 0.3128) err -0.04% hi x0.8339
|
| 1238 |
+
|
| 1239 |
+
`sigma` applied a real scale factor and moved the drift by nothing at all. This
|
| 1240 |
+
is stronger than §26's "corrects the wrong way": there is no gain, no strength
|
| 1241 |
+
knob and no anchor choice that makes a scale-invariant statistic respond to a
|
| 1242 |
+
scale. The lever was mis-specified, not mis-tuned. It is kept only for the
|
| 1243 |
+
workflows that saved it.
|
| 1244 |
+
|
| 1245 |
+
### The fixed point that nearly shipped
|
| 1246 |
+
|
| 1247 |
+
`match_band` first computed `k = target * sigma / hi_sigma`. That is wrong in a
|
| 1248 |
+
way that hides: scaling the high band changes the sigma it is a fraction of, so
|
| 1249 |
+
the target moves while you apply it. It landed at 0.3331 against a 0.3168
|
| 1250 |
+
target -- 5% short, in the right direction, which is the worst possible
|
| 1251 |
+
signature because it looks like it works.
|
| 1252 |
+
|
| 1253 |
+
Now it solves the orthogonal fixed point in closed form,
|
| 1254 |
+
`k = r*L / (H*sqrt(1-r^2))`, then refines two or three passes against the
|
| 1255 |
+
statistic as actually measured, because a difference of Gaussians is not an
|
| 1256 |
+
exact projection. Lands at 0.3167 against 0.3168.
|
| 1257 |
+
|
| 1258 |
+
### The fixture was also wrong, and would have hidden it
|
| 1259 |
+
|
| 1260 |
+
The first test used `torch.randn` for the latent. White noise has a high-band
|
| 1261 |
+
fraction of **0.966** -- pinned against its ceiling of 1.0, where lifting the
|
| 1262 |
+
high band moves the statistic by 0.6% and the clamp does all the "correcting".
|
| 1263 |
+
Every assertion about the lever would have been measuring the clamp. Real
|
| 1264 |
+
latents sit at 0.3643, so the fixture is now built to land near there and an
|
| 1265 |
+
assertion holds it in that regime.
|
| 1266 |
+
|
| 1267 |
+
The safety property is asserted as "the entire change lies along the high band"
|
| 1268 |
+
(cosine with `hi` > 0.99), not as "the low band is unchanged" -- re-splitting
|
| 1269 |
+
the result does not hand back the same `lo`, because the split is not a
|
| 1270 |
+
projection. The first version asserted the false one and failed correctly.
|
| 1271 |
+
|
| 1272 |
+
### One cache key narrowed
|
| 1273 |
+
|
| 1274 |
+
`pin_cond` was in every hop's key including hop 1, which has no pin --
|
| 1275 |
+
`_pin_mech_for` returns `"none"` at index 0 and the conditioning branch is
|
| 1276 |
+
`elif i > 0`. So flipping a lever discarded a byte-identical cached hop 1 and
|
| 1277 |
+
re-rendered it. That is a third of the cost of every lever A/B, on the one hop
|
| 1278 |
+
that provably could not have changed. Now keyed only from hop 2.
|
| 1279 |
+
|
| 1280 |
+
### Still unknown
|
| 1281 |
+
|
| 1282 |
+
The latent's band fraction moved 1.6% across the chain while the picture's mid
|
| 1283 |
+
band moved 8%. The decode is nonlinear so they are not expected to be
|
| 1284 |
+
proportional, but a full match to hop 1's fraction may therefore under-correct
|
| 1285 |
+
the picture. That is one A/B to find out, and it is readable off a 3-hop run:
|
| 1286 |
+
`texture_probe` reports each join separately, so two joins is two data points.
|
| 1287 |
+
If `band` shrinks the +4.2% step but does not close it, the next move is a gain
|
| 1288 |
+
above 1.0, fitted -- not guessed.
|
|
@@ -628,77 +628,115 @@ def _rebuild_latent_samples(x, parts):
|
|
| 628 |
return _latents.rebuild(x, parts)
|
| 629 |
|
| 630 |
|
| 631 |
-
def _condition_pin_latent(lat,
|
| 632 |
"""Anti-ratchet preprocessing for the latent handed to Motion-Context.
|
| 633 |
|
| 634 |
MiniMaxH3MotionContext.apply() takes `context_latent` as-is and exposes no
|
| 635 |
-
hook, so
|
| 636 |
|
| 637 |
-
|
| 638 |
-
|
| 639 |
-
|
| 640 |
-
|
| 641 |
-
|
| 642 |
-
|
| 643 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 644 |
|
| 645 |
**Per component, not per latent** (fixed 2026-08-27). Video and audio are
|
| 646 |
-
two tensors in one NestedTensor and their
|
| 647 |
each carries its own anchor. Before this, `.std()` raised on the nested
|
| 648 |
-
object and
|
| 649 |
-
|
| 650 |
-
|
| 651 |
-
|
| 652 |
-
|
| 653 |
-
|
| 654 |
-
per component -- hop 2 establishes what hops 3+ are matched against. Both
|
| 655 |
-
levers default off, in which case the latent is returned untouched.
|
| 656 |
"""
|
| 657 |
if not isinstance(lat, dict) or "samples" not in lat:
|
| 658 |
-
return lat,
|
| 659 |
x = lat["samples"]
|
| 660 |
parts = _latent_parts(x)
|
| 661 |
if parts is None:
|
| 662 |
print(f"[{TAG}] pin conditioning skipped: unrecognised latent "
|
| 663 |
f"({type(x).__name__})", flush=True)
|
| 664 |
-
return lat,
|
|
|
|
|
|
|
|
|
|
| 665 |
try:
|
| 666 |
-
cur = [
|
|
|
|
|
|
|
|
|
|
| 667 |
except Exception as e: # noqa: BLE001
|
| 668 |
print(f"[{TAG}] pin conditioning skipped ({e!r})", flush=True)
|
| 669 |
-
return lat,
|
| 670 |
-
if not all(c == c and c for c in cur):
|
| 671 |
-
return lat,
|
| 672 |
-
if
|
| 673 |
-
|
| 674 |
-
if len(
|
| 675 |
# Stream count changed mid-chain. Nothing sensible to match against.
|
| 676 |
print(f"[{TAG}] pin conditioning skipped: latent has {len(cur)} "
|
| 677 |
-
f"component(s), anchor has {len(
|
| 678 |
-
return lat,
|
| 679 |
-
|
| 680 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 681 |
out_parts, notes = [], []
|
| 682 |
-
for idx, (t, c, a) in enumerate(zip(parts, cur,
|
| 683 |
o = t
|
| 684 |
-
if
|
| 685 |
-
scale = a / c
|
| 686 |
o = o * scale
|
| 687 |
-
notes.append(f"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 688 |
if noise > 0.0:
|
| 689 |
# Per component: `.shape` on the nested object reports only the
|
| 690 |
# first component's shape, so one draw for the whole latent would
|
| 691 |
# size its noise to the video and broadcast that onto the audio.
|
| 692 |
g = torch.Generator(device="cpu").manual_seed((int(seed) + idx) & 0x7FFFFFFF)
|
| 693 |
n = torch.randn(o.shape, generator=g, dtype=torch.float32)
|
| 694 |
-
o = o + n.to(dtype=o.dtype, device=o.device) * (float(noise) * a)
|
| 695 |
notes.append(f"noise[{idx}] {float(noise):.3f}")
|
| 696 |
out_parts.append(o)
|
| 697 |
if notes:
|
| 698 |
print(f"[{TAG}] pin conditioning: " + ", ".join(notes), flush=True)
|
| 699 |
new = dict(lat)
|
| 700 |
new["samples"] = _rebuild_latent_samples(x, out_parts)
|
| 701 |
-
return new,
|
| 702 |
|
| 703 |
|
| 704 |
def _pin_mech_for(hop_index, overlap_n, prev_sampled):
|
|
@@ -1124,15 +1162,24 @@ class HandTieClips:
|
|
| 1124 |
"follows the overlap widget."
|
| 1125 |
),
|
| 1126 |
}),
|
| 1127 |
-
"pin_renorm": (["off", "
|
| 1128 |
"default": "off",
|
| 1129 |
"tooltip": (
|
| 1130 |
-
"Rescale each pinned latent
|
| 1131 |
-
"
|
| 1132 |
-
"
|
| 1133 |
-
"
|
| 1134 |
-
"
|
| 1135 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1136 |
),
|
| 1137 |
}),
|
| 1138 |
"pin_noise": ("FLOAT", {
|
|
@@ -1513,13 +1560,16 @@ class HandTieClips:
|
|
| 1513 |
pbar = comfy.utils.ProgressBar(n)
|
| 1514 |
|
| 1515 |
hop_store = None
|
| 1516 |
-
|
|
|
|
|
|
|
|
|
|
| 1517 |
pin_noise_v = max(0.0, min(0.10, float(pin_noise)))
|
| 1518 |
audio_ctx = int(audio_pin_frames) if int(audio_pin_frames) > 0 else int(overlap_n)
|
| 1519 |
-
|
| 1520 |
-
if
|
| 1521 |
print(f"[{TAG}] pin conditioning enabled: "
|
| 1522 |
-
f"renorm={
|
| 1523 |
f"noise={pin_noise_v:.3f}", flush=True)
|
| 1524 |
tone_mode = str(tone_compensate)
|
| 1525 |
tone_on = tone_mode != "off" and tone_mode in _tone.MODES
|
|
@@ -1753,7 +1803,15 @@ class HandTieClips:
|
|
| 1753 |
# has no sampler latent and falls back to AddGuide, which
|
| 1754 |
# is a different render of the same inputs.
|
| 1755 |
"pin_mech": pin_mech_pred,
|
| 1756 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1757 |
})
|
| 1758 |
# A locked shot reuses its last render even though its inputs
|
| 1759 |
# changed -- that is the point of locking. The content key would
|
|
@@ -1804,9 +1862,9 @@ class HandTieClips:
|
|
| 1804 |
elif i > 0:
|
| 1805 |
pin_latent = prev_sampled
|
| 1806 |
if pin_latent is not None:
|
| 1807 |
-
pin_latent,
|
| 1808 |
-
pin_latent,
|
| 1809 |
-
|
| 1810 |
seed=(int(seed) + i))
|
| 1811 |
cond, pin_mech_used = _pin_continue(
|
| 1812 |
cond, latent, vae, audio_vae, overlap_n,
|
|
|
|
| 628 |
return _latents.rebuild(x, parts)
|
| 629 |
|
| 630 |
|
| 631 |
+
def _condition_pin_latent(lat, anchor, mode="off", noise=0.0, seed=0):
|
| 632 |
"""Anti-ratchet preprocessing for the latent handed to Motion-Context.
|
| 633 |
|
| 634 |
MiniMaxH3MotionContext.apply() takes `context_latent` as-is and exposes no
|
| 635 |
+
hook, so every lever has to be applied to the latent before it goes in.
|
| 636 |
|
| 637 |
+
Two rescale modes, and the difference between them is the whole point.
|
| 638 |
+
|
| 639 |
+
`sigma` rescales the pin so its standard deviation matches the anchor hop's.
|
| 640 |
+
This is the original lever and **it is measurably the wrong statistic.** On
|
| 641 |
+
a 3-hop chain the pin's total sigma FELL (1.0414 -> 1.0289) while the
|
| 642 |
+
picture's mid-band energy climbed 8% and its high-band fraction rose 1.6%.
|
| 643 |
+
Matching sigma there scales the whole latent UP by 1.2%, lifting a high
|
| 644 |
+
band that was already too hot. Kept because it is what shipped, and old
|
| 645 |
+
workflows say "on".
|
| 646 |
+
|
| 647 |
+
`band` splits each spatial component into low and high and rescales only the
|
| 648 |
+
high part, so the *ratio* between them returns to the anchor hop's. That
|
| 649 |
+
ratio is what the ratchet actually moves. Still a scalar per band, so it
|
| 650 |
+
moves no structure and cannot blur or invent detail -- the property that
|
| 651 |
+
made `sigma` safe to run blind, kept.
|
| 652 |
+
|
| 653 |
+
`noise` mixes in a seeded perturbation, attacking the same ratchet from the
|
| 654 |
+
other side; measured gains reverse above 0.10, hence the widget cap.
|
| 655 |
|
| 656 |
**Per component, not per latent** (fixed 2026-08-27). Video and audio are
|
| 657 |
+
two tensors in one NestedTensor and their statistics drift independently, so
|
| 658 |
each carries its own anchor. Before this, `.std()` raised on the nested
|
| 659 |
+
object and every lever was dead -- announced once per hop as `pin
|
| 660 |
+
conditioning skipped`, which read as routine noise.
|
| 661 |
+
|
| 662 |
+
Returns `(latent, anchor)` where `anchor` is a list, one dict per component
|
| 663 |
+
-- the first pinned hop establishes what later hops are matched against.
|
| 664 |
+
Every lever defaults off, in which case the latent is returned untouched.
|
|
|
|
|
|
|
| 665 |
"""
|
| 666 |
if not isinstance(lat, dict) or "samples" not in lat:
|
| 667 |
+
return lat, anchor
|
| 668 |
x = lat["samples"]
|
| 669 |
parts = _latent_parts(x)
|
| 670 |
if parts is None:
|
| 671 |
print(f"[{TAG}] pin conditioning skipped: unrecognised latent "
|
| 672 |
f"({type(x).__name__})", flush=True)
|
| 673 |
+
return lat, anchor
|
| 674 |
+
mode = str(mode)
|
| 675 |
+
if mode == "on": # pre-2026-09-01 workflows
|
| 676 |
+
mode = "sigma"
|
| 677 |
try:
|
| 678 |
+
cur = []
|
| 679 |
+
for t in parts:
|
| 680 |
+
sig = float(t.float().std())
|
| 681 |
+
cur.append({"sigma": sig, "ratio": _latents.band_ratio(t)})
|
| 682 |
except Exception as e: # noqa: BLE001
|
| 683 |
print(f"[{TAG}] pin conditioning skipped ({e!r})", flush=True)
|
| 684 |
+
return lat, anchor
|
| 685 |
+
if not all(c["sigma"] == c["sigma"] and c["sigma"] for c in cur):
|
| 686 |
+
return lat, anchor # zero or NaN in any stream
|
| 687 |
+
if anchor is None:
|
| 688 |
+
anchor = cur
|
| 689 |
+
if len(anchor) != len(cur):
|
| 690 |
# Stream count changed mid-chain. Nothing sensible to match against.
|
| 691 |
print(f"[{TAG}] pin conditioning skipped: latent has {len(cur)} "
|
| 692 |
+
f"component(s), anchor has {len(anchor)}", flush=True)
|
| 693 |
+
return lat, anchor
|
| 694 |
+
|
| 695 |
+
# Always report the drift, even with every lever off. This is the number
|
| 696 |
+
# that says whether a lever is needed and whether one worked, and it costs
|
| 697 |
+
# nothing to read -- the alternative is inferring it from the master after
|
| 698 |
+
# a decode, which is how the wrong statistic went unnoticed for a release.
|
| 699 |
+
for i, (c, a) in enumerate(zip(cur, anchor)):
|
| 700 |
+
if c["ratio"] is not None and a["ratio"]:
|
| 701 |
+
print(f"[{TAG}] pin drift[{i}]: sigma {c['sigma']:.4f} "
|
| 702 |
+
f"(x{c['sigma'] / a['sigma']:.4f} vs anchor) "
|
| 703 |
+
f"high-band fraction {c['ratio']:.4f} "
|
| 704 |
+
f"(x{c['ratio'] / a['ratio']:.4f})", flush=True)
|
| 705 |
+
|
| 706 |
+
if mode not in ("sigma", "band") and noise <= 0.0:
|
| 707 |
+
return lat, anchor
|
| 708 |
+
|
| 709 |
out_parts, notes = [], []
|
| 710 |
+
for idx, (t, c, a) in enumerate(zip(parts, cur, anchor)):
|
| 711 |
o = t
|
| 712 |
+
if mode == "sigma":
|
| 713 |
+
scale = a["sigma"] / c["sigma"]
|
| 714 |
o = o * scale
|
| 715 |
+
notes.append(f"sigma[{idx}] x{scale:.4f}")
|
| 716 |
+
elif mode == "band":
|
| 717 |
+
o, k = _latents.match_band(o, a["ratio"])
|
| 718 |
+
if k is None:
|
| 719 |
+
# An audio component has no bands; leaving it alone is correct,
|
| 720 |
+
# not a fallback -- `sigma` on it would be a different lever
|
| 721 |
+
# applied silently under this one's name.
|
| 722 |
+
notes.append(f"band[{idx}] skipped (no spatial extent)")
|
| 723 |
+
else:
|
| 724 |
+
notes.append(f"band[{idx}] hi x{k:.4f} "
|
| 725 |
+
f"(fraction {c['ratio']:.4f} -> {a['ratio']:.4f})")
|
| 726 |
if noise > 0.0:
|
| 727 |
# Per component: `.shape` on the nested object reports only the
|
| 728 |
# first component's shape, so one draw for the whole latent would
|
| 729 |
# size its noise to the video and broadcast that onto the audio.
|
| 730 |
g = torch.Generator(device="cpu").manual_seed((int(seed) + idx) & 0x7FFFFFFF)
|
| 731 |
n = torch.randn(o.shape, generator=g, dtype=torch.float32)
|
| 732 |
+
o = o + n.to(dtype=o.dtype, device=o.device) * (float(noise) * a["sigma"])
|
| 733 |
notes.append(f"noise[{idx}] {float(noise):.3f}")
|
| 734 |
out_parts.append(o)
|
| 735 |
if notes:
|
| 736 |
print(f"[{TAG}] pin conditioning: " + ", ".join(notes), flush=True)
|
| 737 |
new = dict(lat)
|
| 738 |
new["samples"] = _rebuild_latent_samples(x, out_parts)
|
| 739 |
+
return new, anchor
|
| 740 |
|
| 741 |
|
| 742 |
def _pin_mech_for(hop_index, overlap_n, prev_sampled):
|
|
|
|
| 1162 |
"follows the overlap widget."
|
| 1163 |
),
|
| 1164 |
}),
|
| 1165 |
+
"pin_renorm": (["off", "sigma", "band"], {
|
| 1166 |
"default": "off",
|
| 1167 |
"tooltip": (
|
| 1168 |
+
"Rescale each pinned latent back toward the first pinned "
|
| 1169 |
+
"hop's, to fight the texture ratchet -- measured at +4.2% "
|
| 1170 |
+
"mid-band per join, flat inside each hop. Both modes are "
|
| 1171 |
+
"scalar rescales, so neither moves structure or can blur "
|
| 1172 |
+
"detail. "
|
| 1173 |
+
"band: match the HIGH-BAND FRACTION, the statistic the "
|
| 1174 |
+
"ratchet actually moves. "
|
| 1175 |
+
"sigma: match total spread -- the original lever, kept "
|
| 1176 |
+
"for old workflows, and measurably the wrong statistic: "
|
| 1177 |
+
"total sigma FALLS across a chain whose picture is "
|
| 1178 |
+
"baking, so it corrects the wrong way. Saved as `on` "
|
| 1179 |
+
"before 0.5. "
|
| 1180 |
+
"off reproduces chain_00038 exactly. The log prints "
|
| 1181 |
+
"`pin drift` every hop either way, so you can read the "
|
| 1182 |
+
"ratchet without changing anything."
|
| 1183 |
),
|
| 1184 |
}),
|
| 1185 |
"pin_noise": ("FLOAT", {
|
|
|
|
| 1560 |
pbar = comfy.utils.ProgressBar(n)
|
| 1561 |
|
| 1562 |
hop_store = None
|
| 1563 |
+
# "on" is what pre-0.5 workflows saved for what is now "sigma".
|
| 1564 |
+
pin_renorm_mode = {"on": "sigma"}.get(str(pin_renorm), str(pin_renorm))
|
| 1565 |
+
if pin_renorm_mode not in ("sigma", "band"):
|
| 1566 |
+
pin_renorm_mode = "off"
|
| 1567 |
pin_noise_v = max(0.0, min(0.10, float(pin_noise)))
|
| 1568 |
audio_ctx = int(audio_pin_frames) if int(audio_pin_frames) > 0 else int(overlap_n)
|
| 1569 |
+
pin_anchor = None # the first pinned hop sets what 3+ match
|
| 1570 |
+
if pin_renorm_mode != "off" or pin_noise_v > 0.0:
|
| 1571 |
print(f"[{TAG}] pin conditioning enabled: "
|
| 1572 |
+
f"renorm={pin_renorm_mode} "
|
| 1573 |
f"noise={pin_noise_v:.3f}", flush=True)
|
| 1574 |
tone_mode = str(tone_compensate)
|
| 1575 |
tone_on = tone_mode != "off" and tone_mode in _tone.MODES
|
|
|
|
| 1803 |
# has no sampler latent and falls back to AddGuide, which
|
| 1804 |
# is a different render of the same inputs.
|
| 1805 |
"pin_mech": pin_mech_pred,
|
| 1806 |
+
# Only from hop 2. Hop 1 has no pin -- `_pin_mech_for`
|
| 1807 |
+
# returns "none" for index 0 and the conditioning branch is
|
| 1808 |
+
# `elif i > 0` -- so its frames cannot depend on these
|
| 1809 |
+
# levers, and keying them in threw away a byte-identical
|
| 1810 |
+
# cached hop 1 every time one was flipped. That is a third
|
| 1811 |
+
# of the cost of every lever A/B, on the one hop nobody
|
| 1812 |
+
# needed to re-render.
|
| 1813 |
+
"pin_cond": ((pin_renorm_mode, round(pin_noise_v, 4), audio_ctx)
|
| 1814 |
+
if i > 0 else None),
|
| 1815 |
})
|
| 1816 |
# A locked shot reuses its last render even though its inputs
|
| 1817 |
# changed -- that is the point of locking. The content key would
|
|
|
|
| 1862 |
elif i > 0:
|
| 1863 |
pin_latent = prev_sampled
|
| 1864 |
if pin_latent is not None:
|
| 1865 |
+
pin_latent, pin_anchor = _condition_pin_latent(
|
| 1866 |
+
pin_latent, pin_anchor,
|
| 1867 |
+
mode=pin_renorm_mode, noise=pin_noise_v,
|
| 1868 |
seed=(int(seed) + i))
|
| 1869 |
cond, pin_mech_used = _pin_continue(
|
| 1870 |
cond, latent, vae, audio_vae, overlap_n,
|
|
@@ -46,3 +46,100 @@ def from_dict(lat):
|
|
| 46 |
if not isinstance(lat, dict) or "samples" not in lat:
|
| 47 |
return None
|
| 48 |
return parts(lat["samples"])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
if not isinstance(lat, dict) or "samples" not in lat:
|
| 47 |
return None
|
| 48 |
return parts(lat["samples"])
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
def _gauss1d(sigma, device, dtype):
|
| 52 |
+
r = max(1, int(round(3.0 * float(sigma))))
|
| 53 |
+
x = torch.arange(-r, r + 1, dtype=torch.float32, device=device)
|
| 54 |
+
k = torch.exp(-(x * x) / (2.0 * float(sigma) ** 2))
|
| 55 |
+
return (k / k.sum()).to(dtype)
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def band_split(t, sigma=2.0):
|
| 59 |
+
"""Separable Gaussian low/high split over the last two dims. -> (lo, hi).
|
| 60 |
+
|
| 61 |
+
Returns None when the tensor has no spatial extent to speak of -- an audio
|
| 62 |
+
component is a different object and blurring across it means nothing.
|
| 63 |
+
|
| 64 |
+
Replicate padding, not reflect: a latent's spatial dims are small (a
|
| 65 |
+
736x1280 frame is 46x80 here) and reflect needs the pad to be smaller than
|
| 66 |
+
the dimension, which stops being true for a large sigma on a small latent.
|
| 67 |
+
"""
|
| 68 |
+
if t.dim() < 2 or t.shape[-1] < 8 or t.shape[-2] < 8:
|
| 69 |
+
return None
|
| 70 |
+
import torch.nn.functional as F # noqa: PLC0415
|
| 71 |
+
h, w = int(t.shape[-2]), int(t.shape[-1])
|
| 72 |
+
flat = t.reshape(-1, 1, h, w)
|
| 73 |
+
k = _gauss1d(sigma, t.device, t.dtype)
|
| 74 |
+
pad = k.numel() // 2
|
| 75 |
+
lo = F.conv2d(F.pad(flat, (pad, pad, 0, 0), mode="replicate"),
|
| 76 |
+
k.view(1, 1, 1, -1))
|
| 77 |
+
lo = F.conv2d(F.pad(lo, (0, 0, pad, pad), mode="replicate"),
|
| 78 |
+
k.view(1, 1, -1, 1))
|
| 79 |
+
lo = lo.reshape(t.shape)
|
| 80 |
+
return lo, t - lo
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def band_ratio(t, sigma=2.0):
|
| 84 |
+
"""High-band sigma as a fraction of total sigma. -> float, or None.
|
| 85 |
+
|
| 86 |
+
This is the statistic the texture ratchet actually moves. Measured across a
|
| 87 |
+
3-hop chain it went 0.3643 -> 0.3673 -> 0.3702, monotone, while the total
|
| 88 |
+
sigma it is normalised by FELL 1.2% -- which is why matching total sigma
|
| 89 |
+
cannot see this and, worse, corrects the wrong way.
|
| 90 |
+
"""
|
| 91 |
+
got = band_split(t, sigma)
|
| 92 |
+
if got is None:
|
| 93 |
+
return None
|
| 94 |
+
tot = float(t.float().std())
|
| 95 |
+
if not tot or tot != tot:
|
| 96 |
+
return None
|
| 97 |
+
return float(got[1].float().std()) / tot
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
def match_band(t, target_ratio, sigma=2.0, clamp=(0.5, 2.0)):
|
| 101 |
+
"""Rescale t's high band so its high-band fraction becomes `target_ratio`.
|
| 102 |
+
|
| 103 |
+
-> (tensor, k), or (t, None) when the tensor has no bands to match.
|
| 104 |
+
|
| 105 |
+
Only the high half is scaled, so the low-frequency structure that carries
|
| 106 |
+
the scene is bit-identical and the correction is one scalar. It cannot blur,
|
| 107 |
+
sharpen unevenly, or invent detail; the worst it can do is get the gain
|
| 108 |
+
wrong, which is why `clamp` exists.
|
| 109 |
+
|
| 110 |
+
Note the fraction is against the tensor's own sigma, so restoring the ratio
|
| 111 |
+
does move total sigma a little. That is deliberate: the ratio is the drifting
|
| 112 |
+
statistic and sigma is the one that lies.
|
| 113 |
+
"""
|
| 114 |
+
got = band_split(t, sigma)
|
| 115 |
+
if got is None or not target_ratio:
|
| 116 |
+
return t, None
|
| 117 |
+
lo, hi = got
|
| 118 |
+
cur_hi = float(hi.float().std())
|
| 119 |
+
if not cur_hi or cur_hi != cur_hi:
|
| 120 |
+
return t, None
|
| 121 |
+
|
| 122 |
+
# The naive `k = target * sigma / hi_sigma` is wrong, and quietly so:
|
| 123 |
+
# scaling the high band changes the sigma it is a fraction OF, so the
|
| 124 |
+
# target moves as you apply it. Measured, it undershot by 5% on a
|
| 125 |
+
# latent-shaped tensor -- a correction that silently does most, but not
|
| 126 |
+
# all, of its job is the worst kind to ship.
|
| 127 |
+
#
|
| 128 |
+
# First guess solves the fixed point assuming lo and hi are orthogonal:
|
| 129 |
+
# k*H / sqrt(L^2 + k^2*H^2) = r, so k = r*L / (H*sqrt(1 - r^2)) ...
|
| 130 |
+
r = min(float(target_ratio), 0.999)
|
| 131 |
+
lo_sd = float(lo.float().std())
|
| 132 |
+
k = (r * lo_sd) / (cur_hi * max(1e-6, (1.0 - r * r) ** 0.5))
|
| 133 |
+
# ... then refine against the statistic as actually measured, because a
|
| 134 |
+
# difference of Gaussians is not an exact orthogonal projection. Two or
|
| 135 |
+
# three passes converge, and a latent is small enough that this is free.
|
| 136 |
+
for _ in range(4):
|
| 137 |
+
k = min(max(k, clamp[0]), clamp[1])
|
| 138 |
+
got_r = band_ratio(lo + hi * k, sigma)
|
| 139 |
+
if not got_r:
|
| 140 |
+
break
|
| 141 |
+
if abs(got_r - r) <= 1e-4 * max(r, 1e-6):
|
| 142 |
+
break
|
| 143 |
+
k *= r / got_r
|
| 144 |
+
k = min(max(k, clamp[0]), clamp[1])
|
| 145 |
+
return lo + hi * k, k
|
|
@@ -188,6 +188,71 @@ def main():
|
|
| 188 |
ck("rebuild round-trips a plain tensor",
|
| 189 |
torch.equal(lat.rebuild(t, [t]), t))
|
| 190 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 191 |
print("\nhopcache.chains -- two renders stay two renders")
|
| 192 |
mk = lambda hops: [(h, "k%d" % i, {"written": float(i)})
|
| 193 |
for i, h in enumerate(hops)]
|
|
|
|
| 188 |
ck("rebuild round-trips a plain tensor",
|
| 189 |
torch.equal(lat.rebuild(t, [t]), t))
|
| 190 |
|
| 191 |
+
print()
|
| 192 |
+
print("latents.match_band -- the lever undoes a known tilt")
|
| 193 |
+
g = torch.Generator().manual_seed(3)
|
| 194 |
+
# Latent-SHAPED, not white noise. Plain randn has a high-band fraction of
|
| 195 |
+
# 0.966 -- pinned against its ceiling of 1.0, where lifting the high band
|
| 196 |
+
# barely moves the statistic and the clamp does the "correcting". Real
|
| 197 |
+
# latents measured 0.3643, so the fixture is built to land near there, and
|
| 198 |
+
# the first assertion holds it there.
|
| 199 |
+
w = torch.randn((2, 4, 46, 80), generator=g)
|
| 200 |
+
smooth = lat.band_split(w, 4.0)[0]
|
| 201 |
+
rough = torch.randn((2, 4, 46, 80), generator=g)
|
| 202 |
+
base = smooth / smooth.std() + rough / rough.std() * 0.30
|
| 203 |
+
r0 = lat.band_ratio(base)
|
| 204 |
+
ck("the fixture sits where real latents sit", 0.20 < r0 < 0.50,
|
| 205 |
+
"ratio %.4f (measured on a real chain: 0.3643)" % r0)
|
| 206 |
+
|
| 207 |
+
# Bake it: lift the high band the way a hop's pin drifts.
|
| 208 |
+
lo, hi = lat.band_split(base)
|
| 209 |
+
baked = lo + hi * 1.25
|
| 210 |
+
r1 = lat.band_ratio(baked)
|
| 211 |
+
ck("lifting the high band raises the fraction", r1 > r0 * 1.10,
|
| 212 |
+
"%.4f -> %.4f" % (r0, r1))
|
| 213 |
+
fixed, k = lat.match_band(baked, r0)
|
| 214 |
+
r2 = lat.band_ratio(fixed)
|
| 215 |
+
ck("match_band restores it to the anchor", abs(r2 / r0 - 1.0) < 0.005,
|
| 216 |
+
"%.4f vs %.4f (k=%.4f)" % (r2, r0, k))
|
| 217 |
+
ck("and scales the high band down to do it", k < 1.0, "k=%.4f" % k)
|
| 218 |
+
|
| 219 |
+
# The safety property -- but NOT "the low band is unchanged". Re-splitting
|
| 220 |
+
# the result does not hand back the same lo, because a difference of
|
| 221 |
+
# Gaussians is not an exact projection. What is true, and what makes this
|
| 222 |
+
# safe to run blind, is that the entire change lies along the high band:
|
| 223 |
+
# no structure is added, one scalar is moved.
|
| 224 |
+
d = (fixed - baked).reshape(-1)
|
| 225 |
+
hf = hi.reshape(-1)
|
| 226 |
+
cos = float((d * hf).sum() / (d.norm() * hf.norm()))
|
| 227 |
+
ck("the entire change lies along the high band", abs(cos) > 0.99,
|
| 228 |
+
"cos %.4f" % cos)
|
| 229 |
+
ck("an audio-shaped component is skipped, not sigma-matched",
|
| 230 |
+
lat.match_band(torch.randn(2, 1024), 0.4)[1] is None)
|
| 231 |
+
ck("a zero target is a no-op", lat.match_band(base, 0.0)[1] is None)
|
| 232 |
+
|
| 233 |
+
# The measured case reproduced: sigma FALLS while the fraction RISES, so
|
| 234 |
+
# the two levers disagree about which way to correct. This is the whole
|
| 235 |
+
# argument for `band` over `sigma` -- on the real chain, matching sigma
|
| 236 |
+
# would have scaled a pin UP whose high band was already too hot.
|
| 237 |
+
shrunk = baked * (0.99 * float(base.std()) / float(baked.std()))
|
| 238 |
+
ck("sigma and band disagree in DIRECTION on this signal",
|
| 239 |
+
float(shrunk.std()) < float(base.std())
|
| 240 |
+
and lat.band_ratio(shrunk) > r0 * 1.10,
|
| 241 |
+
"sigma x%.4f but fraction x%.4f"
|
| 242 |
+
% (float(shrunk.std()) / float(base.std()),
|
| 243 |
+
lat.band_ratio(shrunk) / r0))
|
| 244 |
+
|
| 245 |
+
# And the reason `sigma` cannot be rescued by tuning: the fraction is a
|
| 246 |
+
# RATIO, so it is invariant under any uniform rescale. Whatever scale a
|
| 247 |
+
# sigma match picks, it leaves this statistic exactly where it found it.
|
| 248 |
+
# That makes `sigma` a no-op on the texture ratchet by construction, not
|
| 249 |
+
# merely a weak correction -- measured end to end through the node, drift
|
| 250 |
+
# stayed at +12.74% under `sigma` and went to -0.04% under `band`.
|
| 251 |
+
for factor in (0.5, 0.9651, 2.0):
|
| 252 |
+
ck("a uniform x%.4g rescale does not move the fraction" % factor,
|
| 253 |
+
abs(lat.band_ratio(baked * factor) / r1 - 1.0) < 1e-4,
|
| 254 |
+
"%.6f vs %.6f" % (lat.band_ratio(baked * factor), r1))
|
| 255 |
+
|
| 256 |
print("\nhopcache.chains -- two renders stay two renders")
|
| 257 |
mk = lambda hops: [(h, "k%d" % i, {"written": float(i)})
|
| 258 |
for i, h in enumerate(hops)]
|