For anyone extending it

How it works

The render pipeline, and the several ways the FFGL SDK can produce a plugin that fails silently. The second half is worth reading even if you never touch this code.

The constraint that shapes everything

FFGL 2 requires an OpenGL 4.1 core context, and macOS caps at exactly 4.1 — Apple froze OpenGL there. So there are no compute shaders, no SSBOs, and no image load/store. Every pass is a fragment shader drawing a full-screen quad into a framebuffer, and anything stateful is a ping-pong pair of textures.

That rules out the obvious implementation of a motion search, which would be a compute shader with shared memory. What is left is a hierarchical block match rendered at block resolution, one fragment per macroblock, which turns out to be both fast and a good fit for the aesthetic.

The pass graph

Per frame, in order:

1. Ingest — copy the host texture into an exact-size RGBA16F buffer, resolving the MaxUV sub-rectangle and un-premultiplying alpha. Doing this once removes two whole classes of bug from every later pass.
2. Luma + pyramid — extract Rec.709 luma and generate mips. The mip chain is the search pyramid, which is why it costs nothing to have.
3. Motion search — hierarchical block matching, coarse to fine. Described below.
4. Flow post — spatial blur, temporal inertia, freeze, quantisation, magnitude clamping and NaN rejection. The output goes into a ring of past fields so Motion Lag can reach back.
5. Control — a 1×1 state machine deciding the current mosh level from the triggers, the beat clock, the audio level and the cut detector.
6. Mosh — displace the accumulation buffer along the field and decide, per pixel, whether it may refresh from the live frame. This one decision is the entire effect.
7. Composite — wet/dry, re-premultiply, clamp, draw into the host's framebuffer.

About 61 MB of buffers per instance at 1080p. Most passes run at block resolution, so the full-resolution cost is dominated by ingest, mosh and composite.

Motion search

One fragment per macroblock, run once per pyramid level from coarse to fine. Each level reads the level above as its opening guess — bilinear magnification upsamples it for free — and searches a small window around it.

Candidates tested per block:

• The coarser level's vector for this block.
• The four neighbouring blocks' vectors. Objects are larger than one macroblock, so a neighbour has usually already found the right answer; this buys far more quality per texture read than widening the window.
• The zero vector, always — without it a static region gets dragged off by a marginally better match in noise.
• A 3×3 or 5×5 local window around the prediction.

Cost is mean absolute difference over a 4×4 sample lattice inside the block, plus a regularisation term charged in pixels: straying from the neighbours' consensus costs something, and so does motion at all. That term is what makes the field coherent rather than confetti, and charging it in pixels means the tuning holds at any resolution.

The first iteration of each frame seeds from the previous frame's finished field. Motion is continuous, so yesterday's answer is a far better opening guess than zero, and it costs nothing.

FFGL traps worth knowing

Each of these produces a plugin that fails with no error message anywhere. They cost real time to find.

SetParamInfo appends, it does not update

There is no API to rename an inherited parameter or change its declared default. Calling SetParamInfo for an index that already exists adds another record. GetNumParams then overreports, and instantiateGL — which writes every index's default before handing the plugin to the host — hits the phantom index, gets FF_FAIL, and destroys the instance.

The plugin simply never loads. Nothing is logged. This is why the mixer leaves its inherited mixVal slider alone rather than renaming it to something useful, and why there is a test that replays the host's default-initialisation walk against both plugins.

The SDK must be an object library, not a static one

plugMain is the only symbol a host looks up, and it lives in a translation unit nothing else references — the g_CurrPluginInfo global it uses is defined in a different file. Out of a static archive the linker therefore discards that object entirely and produces a plugin with no entry point.

An OBJECT library links every object unconditionally on all three toolchains, with no --whole-archive juggling. This is also why the SDK's own project files compile those sources straight into each plugin. CI checks the export on every platform now.

ffglex::FFGLFBO leaks its colour texture

Its Release() has a copy-paste error: the second guard re-tests depthBufferID instead of colorTextureID, so the colour texture is never deleted. A full-resolution texture leaks on every resize. This project uses its own RenderTarget, which also drops the depth buffer none of these passes need.

ScopedFBOBinding does not restore the viewport

It restores the framebuffer binding only. Since our passes render into reduced-size targets, returning to the host without putting the viewport back makes the host draw into a corner of its own framebuffer. ScopedViewport covers the gap.

Whole-pixel snapping is not optional

The accumulation buffer is resampled into itself every frame. With plain bilinear filtering the interpolation compounds and the image blurs to mush within a second or two. Snapping the displacement to whole pixels keeps it crisp — and is what real codecs do, for the same reason.

Testing a thing you cannot look at

42 tests run against a real headless OpenGL 4.1 core context, created through EGL on a software rasteriser so they work on a CI machine with no GPU and no display server. They drive the pipeline exactly as Resolume does.

The inputs are synthetic clips with known ground truth: a pattern translating at a known velocity must produce the matching vectors. That is the only way to regression-test a motion estimator — "does it look right" is not a test, and by the time it looks wrong you have no idea which of six passes did it.

Also covered:

• Motion compensation reconstructing a translating frame, which checks the vector sign, warp direction, pel snapping and block-centre lookup all at once — get any one backwards and the error doubles instead of cancelling.
• Cut detection, beat divisors at every setting, freeze, and motion lag against a recorded field history.
• Decay being identical over one second whether it arrives as 30 steps or 60, which is the whole point of expressing it as a half-life.
• Resize storms, GL object lifetime, and that no NaN can reach the feedback buffers — one would propagate forever and only a host restart would clear it.
• The plugin layer separately: the frame gate, trigger handling, parameter mapping and mixer input selection.

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel
ctest --test-dir build --output-on-failure

./build/tests/datamosh_tests --profile   # per-pass GPU timing