Optical Flow ​

Compute per-pixel motion vectors between consecutive video frames using the Horn-Schunck algorithm. Outputs a flow texture consumable by Vector Motion Blur, or bridgeable to a vectorField via ImageSample for particle advection.

Category: Fields Menu path: Fields > Optical Flow

Ports ​

PortTypeDirectionDescription
inimageRgba16finputCurrent frame's image (typically from VideoSource)
outimageRgba16foutputFlow field encoded as R=u, G=v per pixel

Parameters ​

ParamTypeDefaultDescription
smoothnessscalar10.0Horn-Schunck α² regularization term. Higher = smoother flow, less detail, better under image noise. Lower = sharper flow, more sensitive to noise. Typical range 1.0-100.0.
iterationsscalar32Number of HS iterative update passes. More = better convergence for larger motion. Clamped 1-256.

How It Works ​

Optical flow estimates, for each pixel in the current frame, what 2D vector it traveled from the previous frame to arrive at its current position. The Horn-Schunck algorithm does this by minimizing an energy functional:

E = ∫∫ [(Ix·u + Iy·v + It)² + α²(|∇u|² + |∇v|²)] dx dy

Where:

  • Ix, Iy are the spatial image gradients (Sobel)
  • It is the temporal gradient (current frame − previous frame)
  • u, v are the per-pixel motion vectors we're solving for
  • α² is the smoothness regularization (our smoothness param)

The first term (data term) says flow should explain the observed image motion; the second term (smoothness term) says neighboring pixels should have similar flow. The minimum is found by iterating fixed-point updates — each pass replaces u, v with a weighted average of their neighbors plus a correction from the data term.

Internally this runs as two GPU passes: one gradient pass (Sobel + difference) and an iterate pass that ping-pongs between two textures over the iterations count.

Output encoding. Per pixel, R = u (horizontal motion) and G = v (vertical motion), both in fractional UV per frame. So u = 0.01 means the pixel moved 1% of image width between the two frames. B = 0, A = 1.

Temporal Behavior ​

Optical flow is a two-frame operation — the node caches the previous frame internally per layer per node. Rules:

  • First frame / just loaded: no previous frame cached → output is zero flow.
  • Sequential playback: next frame uses the cached previous → real flow.
  • Scrub backward / skip ahead: cache invalidates → output is zero flow that frame, and re-seeds on the next sequential frame.

This is strict by design: feeding flow based on a non-sequential previous frame would produce visually confusing motion vectors. A one-frame zero-flow "reset" is cleaner than lingering artifacts.

Usage Examples ​

Vector motion blur on footage shot without it ​

VideoSource → OpticalFlow → ┐
VideoSource ──────────────→ VectorMotionBlur → Output
                              (flow_in)       (in)

The source video feeds both the flow computation and the blur target. OpticalFlow's output drives per-pixel motion-blur smearing — the classic flow-based motion-blur use case.

Particle advection from video motion ​

VideoSource → OpticalFlow → ImageSample (Luminance→VectorField bridge) → PointAdvect ← Grid(10k)

Particles flow along the video's motion field — they track moving objects naturally.

Visualize motion vectors as color ​

OpticalFlow → DrawField (or Colorize with polar ramp) → Output

Preview the flow magnitude/direction to tune smoothness and iterations.

Flow-driven displacement ​

VideoSource → OpticalFlow → UVRemap ← AnotherImage

Use the flow field as UV offsets to warp a different image by the video's motion.

Tips ​

  • Noisy footage: raise smoothness to 50-100. Low smoothness amplifies noise into flickering flow vectors.
  • Fast motion: raise iterations to 64-128. Horn-Schunck is slow to converge on large displacements; more passes help.
  • Sharp edge motion: keep smoothness around 5-10. Lower retains detail at moving edges (e.g. limb silhouettes).
  • First-frame pulse: if you see a one-frame glitch at the start of playback or after a scrub, that's the zero-flow reset. Normal.
  • The vectorField readback is budgeted at 128MB: on very large canvases (big comp × overscan) the flow raster is downsampled on the GPU before the CPU readback. Flow is smooth data, so this is visually transparent — coordinate mapping accounts for the smaller raster.
  • Performance: iterations scale GPU cost linearly. 32 iterations on 1920×1080 is ~3-5 ms on a modern GPU. Increase thoughtfully.
  • Limitation: Horn-Schunck struggles with motion larger than ~10-20 pixels per frame. For big motions, a pyramidal variant or Farneback would help — that's Phase 2.
  • VectorMotionBlur — the canonical downstream consumer for flow-based motion blur
  • VideoSource — the most common upstream source
  • ImageSample — bridges the flow texture to a vectorField for PointAdvect consumption
  • UVRemap — displacement-style consumption of a flow texture
  • DrawField — visualize the flow with heatmap/arrows (Phase 2)

© 2026 Caddis