Add LTX-2.5 DFR pipeline with generated keyframe slots
Ports DFRPipeline from the Lightricks reference. Stage 1 generates video plus
single-pixel-frame keyframe slots at a fraction of the requested resolution on
a VAE-aligned segment grid; both are spatially latent-upsampled and stage 2
re-denoises at twice that resolution with the slots re-attached and an optional
spatial detailing IC-LoRA active for that stage only. Optional temporal x2/x4
refine rounds tile the canvas at shared keyframes and densify with ancestral
Euler. With spatial_upscalings=2 a full-resolution detailing epilogue follows
the rounds.
The transformer already stored keyframes_abs_pos_embedding for load/save; this
wires it into the forward through a new video_keyframes_mask argument, which
only a keyframes-aware pipeline passes, so other pipelines are unaffected.
The epilogue denoises the whole canvas in one loop and tiles the transformer
call inside it, so every Euler step steps a canvas whose tiles have already
agreed on their overlaps. Spatial tiles blend under a trapezoidal mask, since
neither side of a height or width border holds a known answer. Temporal tiles
are cut on the keyframe seams the last refine round stitched on: both windows
reproduce a shared keyframe there, so the later one drops its run-up under a
rectangular mask rather than averaging it. Conditionings are attached once on
the whole canvas and filtered per tile at the token level, and a keyframe two
windows share is one token they both read.
The epilogue is handed its keyframes rather than asked to generate them. Each
carry plane is decoded on its own -- the VAE is causal, so a stacked decode
would bleed neighbours -- then Lanczos-stretched x2 in RGB and encoded again at
the output resolution, and pinned fully clean. Only the video latent is
spatially upsampled.
Conditioning fps is snapped to 60 above 30 rather than merely capped there, at
every stage. RoPE time is pixel_frame / fps, and the transformer is trained
around 24/25/30 and 60; a temporal round taking 24 fps to 48 lands between
those, and it shows as stutter at the latent borders. Playback fps is
unchanged, so 24 fps with one round still ships 48 fps.
A condition's index is read on the canvas num_frames asks for, and the moment
it names is carried onto each refine round's longer canvas by scaling its pixel
position by 2**round. The scaled position does not generally land on a latent
boundary, so it travels as a pixel index rather than through the public latent
index; a keyframe conditioning is appended as extra tokens instead of being
spliced into the base grid, so it does not need to.
height and width must be divisible by 2**spatial_upscalings times the VAE's
spatial compression ratio, which makes 4K 3840x2176 rather than 3840x2160. That
rule is checked ahead of the looser one every LTX-2 pipeline applies, so the
error names the divisor a DFR caller actually has to satisfy.
Four details are easy to get wrong, and each is covered by a test after showing
up as a visible seam at a tile handover:
- The ancestral step injects noise into every token, so the conditioning blend
has to be re-applied afterwards. Skipping it lets the strength-0.95 anchor
keyframes erode over the schedule, and those anchors are the only thing
pinning adjacent tiles onto the same content.
- Velocity is converted to x0 with each token's own noise level, not the scalar
schedule sigma: a token held at strength s sits at (1 - s) * sigma.
- Each tile draws its ancestral noise from a generator seeded
seed + 1000 * round + tile, kept separate from the main generator so the
draws do not consume state the next tile's initial noising reads.
- Two tiles invent the slot that falls in the later one's dropped lead-in. The
stitch keeps the earlier tile's frames there, so the earlier tile's copy is
the one the canvas holds, and the one the next round must anchor on.
