Testing

cargo test --workspace                          # everything; oracle tests skip without ffmpeg
cargo nextest run --workspace                   # the same in parallel processes (doctests: cargo test --doc)
cargo test -p kinema-h264                    # one crate
cargo test --release -p kinema-hevc          # codec tests are much faster in release
cargo xtask ci                                  # all gates (fmt, clippy, tests, layers, assets, wasm)
cargo xtask fixtures                            # pre-generate the ffmpeg fixture matrix
KINEMA_REQUIRE_ORACLES=1 cargo test --workspace   # CI: missing ffmpeg fails instead of skipping

Integration tests build one binary per crate: a crate with several test files keeps them in tests/it/, and tests/it/main.rs declares each as a module. Add a test file there with a mod line in main.rs; shared helpers live in tests/it/common/ or tests/it/support/. A test file that changes process-wide state (a global switch, pool, panic hook or fault injection) or times a workload stays a binary of its own in tests/. Select one file's tests with its module path: cargo test -p kinema-h264 --test it conformance::.

1. Kinds of tests

Commit *.proptest-regressions files so failing cases are re-run.

2. ffmpeg oracle tests

ffmpeg and ffprobe are external processes used to generate fixtures and to check results. They are never linked or shipped (AGENTS.md §2).

• The tools are found by kinema-testkit (a dev-dependency-only crate, crates/testkit), in this order: KINEMA_FFMPEG / KINEMA_FFPROBE (explicit paths; a wrong path is an error), then every directory on PATH (ffmpeg and ffmpeg.exe), then well-known install directories (/opt/homebrew/bin, /usr/local/bin, /usr/bin, C:\ffmpeg\bin, …).
• If a tool is missing the test prints SKIPPED (<test>): ffmpeg not found … and passes. Set KINEMA_REQUIRE_ORACLES=1 (CI) to make every such skip a failure. In new tests use let ff = kinema_testkit::require_ffmpeg!(); (also require_ffprobe!, require_oracles!).
• Some oracles need a recent ffmpeg: the APV tests (crates/apv/tests/it/oracle.rs) skip when ffmpeg -formats does not list apv (ffmpeg ≥ 8.0), naming the detected version.
• Fixtures are generated on first use into <workspace>/target/fixtures/<crate>/ (kinema_testkit::fixtures_dir; independent of CARGO_TARGET_DIR, so agents with private target dirs share them; KINEMA_FIXTURES_DIR overrides the root) and reused afterwards. Generators write to a per-process/thread temporary name (testkit::temp_path) and rename it into place, so concurrent tests never see half-written files. Delete the directory to regenerate. Never commit media.
• Every crate's tests/common/mod.rs delegates discovery and fixture paths to testkit, except crates/codecs/src/tests.rs, which still has its own lookup (the crate was being edited concurrently; migrate it when convenient).
• cargo xtask fixtures [crate…] pre-generates the whole fixture matrix up front (useful before a parallel test run or on a fresh machine). It runs each crate's ignored generate_fixtures test — the same generators the oracle tests call — for h264, hevc, isobmff, matroska, dnx, prores, mpeg2v, ac3 and codecs (MXF, Ogg and MPEG TS / PS), and prints one made / cached / skipped line per fixture plus a summary. The aac, opus and h264enc oracles generate small per-test signals on demand and are not part of the matrix.
• Fixture sources are synthetic: testsrc2, mandelbrot, SMPTE bars, noise and fades, sine tones. H.264 and HEVC fixtures need ffmpeg built with libx264 and libx265. VideoToolbox fixtures are generated only on macOS. The Media Foundation parity tests (crates/platform/tests/it/media_foundation.rs, Windows only) make their 640x360, 1080p and 2160p H.264 / HEVC / Main 10 fixtures with ffmpeg and skip without a Direct3D 11 video device or the HEVC / VP9 / AV1 codec extensions of the Microsoft Store. The VP9 and AV1 fixtures (libvpx-vp9, libaom-av1; 360p, 1080p, 2160p, hidden alt-ref frames, two GOPs) need an ffmpeg with those encoders.

The NVENC tests (crates/platform/tests/it/nvenc.rs and nvenc_export.rs for H.264, nvenc_rgba_input.rs for the GPU's RGB → 4:2:0 conversion, nvenc_hevc.rs, nvenc_hevc_export.rs, nvenc_hevc_probe.rs and nvenc_hevc_warm.rs for H.265, nvenc_hevc_main10.rs and nvenc_hevc_hdr_export.rs for Main 10 HDR; Windows only) skip, printing SKIPPED, without an NVIDIA GPU with NVENC (H.265: when the HEVC probe says there is no HEVC encoder; Main 10: when the Main 10 probe says there is no 10-bit encoder; on a machine that has one, a configuration NVENC refuses fails the test). The FFI layout tests in crates/platform/src/nvenc/abi_tests.rs were generated from a C program built with MSVC (cl after vcvars64.bat) against NVIDIA's MIT-licensed nvEncodeAPI.h (12.1); the header and the program are not in the repository. To regenerate them, put the header in a scratch directory and write a program that prints, for every type, field, constant and GUID of src/nvenc/ffi.rs that the asserts name (H.264, HEVC and the HEVC picture parameters with their SEI payload array), sizeof / alignof / offsetof, the enum and macro values cast to 64-bit integers and the GUIDs as 128-bit hex; for the bit-fields of the flags words, which offsetof cannot take, zero a structure, set that one field to its maximum and print the 32-bit word. Compare the numbers with the asserts (a difference is drift, which must be understood before anything is "fixed" to match) and add asserts for anything new.

Pass criteria per codec

Each codec README has the full fixture matrix and the measured results.

3. Rendering: CPU reference and GPU parity

• The CPU compositor in kinema-render is the reference. Its tests check computed values (opacity blends, dissolve midpoints, Motion placement, ½-resolution against downsampled full resolution) and that render::plan matches render_sequence.
• crates/gpu/src/tests.rs composites the same frame plan (a YUV layer plus a transformed, semi-transparent RGBA layer) on the GPU and on the CPU. It requires a 99th-percentile channel difference of ≤ 6 and a mean of < 1.5 (8-bit levels). It skips when no GPU adapter is available.
• Effects implemented only on the CPU need no GPU test: the plan pre-renders those layers on the CPU.
• Golden images (crates/golden, a test-only crate): tests/golden.rs builds small procedural projects (demo-generator footage, bars, no media files) at 320×180 and renders one frame of each through the CPU compositor: Motion transform + opacity, four blend modes (Multiply, Screen, Overlay, Difference), Gaussian Blur, Lumetri basic correction, Crop, Cross Dissolve at 50 %, Dip to Black at 25 %, Wipe at 50 %, Timecode / Clip Name burn-in text, graphic clips (also a built-in graphics template with overridden properties and per-character styles with a pinned box), and colour management: S-Log3/S-Gamut3.Cine footage interpreted into Rec. 709 (log_to_rec709) and Rec. 2100 PQ footage tone mapped into Rec. 709 (hdr_tone_map; the footage is demo frames re-encoded by the test's Encoded source). Each frame is compared with crates/golden/goldens/<scene>.png by kinema_testkit::golden: PSNR ≥ 45 dB, max abs ≤ 12, 99th-percentile per-pixel max channel difference ≤ 2 (8-bit sRGB levels; Tolerance::RENDER). On failure the actual frame and a ×8 difference image go to <workspace>/target/golden-failures/. Six vfx_* scenes cover the M5.11 effects (Corner Pin over Turbulent Displace, Ultra Key, Wonder Glow + Glint, Spacer + Stroke + Long Shadow over Gradient, VR Rotate Sphere, Channel Mix + Color Emboss + Rounded Crop); per-effect unit tests live in crates/render/src/vfx/tests.rs (identity at neutral parameters, determinism, range, a known value each, temporal/track effects through the compositor). scenes_are_distinct guards against blank or duplicate scenes.
• Video transitions (crates/render/src/transitions/tests.rs) are checked for every transition (84 modern, 21 Legacy, 7 hidden Obsolete): p = 0 is exactly the outgoing frame and p = 1 exactly the incoming one, near both ends the output is close to that frame (continuity), the midpoint differs from both, the output is finite and deterministic (also on 1×1 / odd sizes and NaN progress), every parameter visibly changes the picture, and every direction differs. Golden fingerprints (4×4 mean RGB grid at p = 0.25/0.5/0.75, ±4 levels) live in src/transitions/goldens.txt; re-bless with KINEMA_BLESS=1 cargo test -p kinema-render golden_fingerprints and review the diff. KINEMA_TRANSITION_SHEETS=<dir> cargo test -p kinema-render contact_sheets -- --ignored writes one contact-sheet PNG per folder for eyeballing (never commit them).
• Blessing: KINEMA_BLESS=1 cargo test -p kinema-golden rewrites the references (and writes a .attribution sidecar for any new one; it prints the ATTRIBUTION.md row to add). References are original work rendered by Kinema, must stay under 50 KB (enforced on bless), and every one needs its sidecar and index row (cargo xtask assets). Look at a re-blessed PNG before committing it.
• The same scenes run through plan_frame + GpuCompositor when a GPU adapter exists (gpu_matches_cpu_on_golden_scenes, same p99 ≤ 6 / mean < 1.5 criterion; skipped otherwise). This found a real bug: the GPU upload cache was keyed by pixel-buffer address without keeping the buffer alive, so a new frame allocated at a freed frame's address was drawn with the stale texture (fixed; regression test upload_cache_keeps_buffers_alive).

4. UI: control channel and screenshots

Scripted UI tests (headless)

crates/ui-egui/tests/it/scripted.rs runs the real KinemaApp under egui_kittest (build_eframe): no window, no GPU, no OS event loop. A small Driver opens the demo project, sends requests through the same control channel agents use (ControlRequest → control::handle, exactly as the TCP server does) and steps egui frames until each reply arrives. Synthetic input queued by ui.click / ui.drag / ui.key is moved into the next frame by the app's own raw_input_hook, so clicks by automation id work headless. Covered:

crates/ui-egui/tests/it/mixer_ui.rs drives the Audio Track Mixer, Audio Clip Mixer, timeline track keyframes and the Audio Gain dialog by automation id and with multi-frame pointer drags (press, move over several frames, release). With KINEMA_UI_SNAPSHOT_DIR=<dir> it also renders the window offscreen through wgpu (Harness::render) and writes mixer-*.png; this works without a visible window (for example on a locked screen, where ui.screenshot cannot capture). crates/ui-egui/tests/it/essential_sound_ui.rs does the same for the Essential Sound panel (type buttons, switches, a slider drag as one undo step, section bypass, Auto-Match, ducking, Browse presets; essential-sound-*.png). crates/ui-egui/tests/it/clip_audio_ui.rs covers the audio of video clips (#223): the linked audio's Volume, Channel Volume and Panner in Effect Controls and Properties, and the Volume line on audio clips in the Timeline (a drag is one undo step, Pen-tool keyframes, keyframe drags, a click or right-click on the line still reaches the clip; KINEMA_UI_SHOTS=<dir> writes volume-*.png).

Essential Sound engine tests (crates/engine/src/essential_sound_tests.rs) build projects from generated speech-like and tonal WAVs: Auto-Match lands within ±0.5 LU of the target (measured: 0.000 LU, with and without a repair/clarity chain) and measures a mono clip as one channel (3.01 LU below the same signal as dual mono, within 0.1 LU of a one-channel meter over the file); ducking on a dialogue + music project gives keyframes within 60 ms of the expected times (measured 20 ms) and −15.00 dB in the mix; each repair stage improves its metric through the render path (hum −39 dB, rumble −22 dB, noise floor −13 dB, sibilance −19 dB, reverb tail −10 dB); the mix is bit-identical however requests are cut and the WAV export equals it. perf_full_dialogue_chain_realtime_factor (ignored; run with --release) prints the realtime factor of all nine Dialogue effects on one clip (22× on one core).

crates/ui-egui/tests/it/export_ui.rs drives Export mode: the settings column and Summary, editing a setting turning the preset into Custom, the Preset Manager (search, favourite star, save, apply, delete), the queue panel (Send to Queue, Up/Down, Cancel, Start, Retry, Clear) with real encodes, the Export button and the header's Quick Export popup (export-*.png snapshots).

crates/ui-egui/tests/it/color_ui.rs drives colour features the same way (color-*.png snapshots): Lumetri Input/Look LUT menus and section switches, the Interpret Footage ▸ Color Management and Sequence Color Management dialogs, and the HDR scopes of a PQ sequence.

Colour science tests: kinema-color (curve round trips and published reference values per camera log curve, BT.709/BT.2087 matrices, BT.2390 EETF, gamut mapping, LUT tetrahedral vs a brute-force barycentric reference, .cube/.3dl round trips), kinema-gpu (gpu_lut_matches_cpu_tetrahedral: WGSL vs CPU, max |Δ| ≈ 2e-7), render::colorman, render::color_match, engine color_tests.rs, and export::tests::hdr_exports_signal_pq_and_hlg (PQ/HLG exports read back by our demuxer and by ffprobe: color_transfer, color_primaries, color_space, mastering display and content light side data).

Not covered headless: ui.screenshot (needs a real viewport), the wgpu monitor path (the harness runs the CPU texture path), audio output, and wall-clock playback advance (kittest frames do not advance real time). Playing sound is covered without a device: play_ahead tests drive the mixed-ahead ring with a wall-clock device (order, underruns stay in time, mixer panic, stop), and cargo run --release -p kinema-ui-egui --example bench_audio -- <project.fcproj> plays a real project's sequences through a simulated 48 kHz device and counts dropouts, mixing in the callback (direct, as before) against mixing ahead (ahead). Use Driver for new UI regressions: d.exec(command, params), d.ok(method, params), d.frames(n).

Interactive checks

For visual work, drive the real app:

cargo run --release -p kinema -- --control 9876

Then script it over the control channel or MCP: run commands, click by automation id, ui.inspect / ui.elements to assert state, and ui.screenshot {"path": "…"} to capture the window or a single panel ({"panel": "Timeline"}). Look at every screenshot. Examples are in agents.md. The headless path (kinema-cli run script.jsonl, MCP --demo) covers engine behaviour without a window.

5. Performance

cargo xtask bench (the example crates/ui-egui/examples/bench/) runs six sections, each in its own process under /usr/bin/time so peak RSS is per section: decode (every frame of 1080p and 2160p H.264, HEVC, VP9, AV1 and ProRes through the media stack), playback (the scenarios below), scrub (random seeks and playhead drags: time until the exact frame is on screen), timeline (the real app under egui_kittest with a 1000-clip / 20-track sequence: update, tessellation and wgpu render ms per frame), export (H.264 + AAC and ProRes) and project (save / open 5000 clips). Options: --sections decode,scrub, --only <substring> (fixture or scenario), --repeat N, --quick, --cpu, --hw auto|off (Settings ▸ Playback ▸ Hardware decoding; default auto, as the app), --label NAME (output target/bench/bench-<label>.{json,md}), --section NAME --json FILE (one section in-process, e.g. under a profiler). Fixtures are made with ffmpeg in target/fixtures/playback/ (or $KINEMA_FIXTURES/playback). The suite's playback section plays h264-1080, stack3 and stack3-blend at Full, h264-2160 at Full, 1/2 and 1/4 and with draft decoding (Settings ▸ Playback ▸ Draft decoding) at 1/2 and 1/4, and hevc-2160, vp9-2160 and av1-2160 at Full, 1/2 and 1/2 draft; its draft column counts the frames decoded in draft mode.

The perf tests check bit-exactness before they time anything. Results go in the crate README's performance table, with machine and thread count. Headline numbers go in ROADMAP.md. Measure on an idle machine: parallel agent builds distort timings.

Playback benchmark

cargo xtask bench-playback (the example crates/ui-egui/examples/bench_playback.rs) plays sequences headlessly through the Program monitor's own frame scheduler: the FrameServer worker pool, schedule_playback (prefetch order and stale-job dropping), its caches, and the PlaybackMeter that counts shown/dropped frames in the app. On the GPU path it also composites each plan with kinema-gpu, as the monitor does on the UI thread.

cargo xtask bench-playback                                   # every scenario, GPU path, Full and Half
cargo xtask bench-playback --scenario stack3 --res full --cpu
cargo xtask bench-playback --json target/bench-playback.json # machine-readable results

Options: --res full,half,quarter, --cpu (CPU compositor + texture conversion instead of the GPU path), --seconds, --refresh (display Hz), --workers, --repeat, --json <file>. Fixtures are made with ffmpeg in target/fixtures/playback/ (set KINEMA_FIXTURES to share one set between worktrees).

Columns: shown/drop as counted in the app (a frame is shown when its exact picture was on screen at a refresh while it was due; frames passed over without a refresh count as dropped); ontime = due frames whose job finished before they were due; lat = queue→ready per job, svc = worker time per job; cpu/j, src/j, srcC/j = worker thread CPU, source fetch (decode) wall and thread CPU per job; ui = time on the UI thread to present a frame (GPU upload and draw, or texture conversion); cpu ms/f = process CPU per frame and cores = the cores that needs at the sequence frame rate; seeks/dec = decoder restarts and samples decoded; waste = jobs for frames that were never due.

Wall-clock columns (shown/drop, ontime, latencies) depend on machine load, so each row prints the load average. CPU columns (thread and process CPU time) and the structural counters (seeks, samples decoded, wasted jobs) do not, and are what to compare between runs on a busy machine.

Results on an M4 Pro (14 cores), GPU path, 8 s plays, median of 2 alternating runs of the M4.6 baseline (commit 5170376) and the result of M4.6, on a machine shared with parallel agent builds (load average 86–207, so wall-clock columns are pessimistic; CPU ms/frame is not):

At load ~25–60 the same final build plays h264-1080, stack3 and h264-2160 at Full with 0 dropped in 3 of 3 runs (192/0) and render previews 192/0. The 4K fixture needs ~4 cores of decode per real-time second (≈160 ms CPU per frame at 170 Mbit/s); the demo project's procedural footage ~190 ms per Full-resolution frame.

Audio mixer throughput

perf_24_tracks_3_effects_realtime_factor (24 tracks, 72 inserts and a compressed submix on one core) must run at least 4x realtime in release builds (1x in debug). On Unix it times the test thread's CPU time, so it runs with the rest of the suite. Hosts without a per-thread CPU clock (Windows) fall back to wall time, which depends on load, so there it is ignored by default; run it explicitly with cargo test --release -p kinema-render --lib perf_24_tracks_3_effects_realtime_factor -- --ignored --nocapture and record the machine with the result.