HD4: a four-channel video mixer

Open HD4

HD4 is a four-channel video and audio mixer that runs in the browser. Each channel can show a camera, a video file, an image, or a Noisemaker shader. The operator prepares a channel on the preview monitor and takes it to the program output with CUT, MIX, or WIPE. HD4 also has PinP, SPLIT, QUAD, and KEY compositions, an audio mixer, MIDI control, and recording. [H1]

How it uses Noisemaker

Each channel has a ChannelRenderer, and each ChannelRenderer owns one CanvasRenderer. No other HD4 module creates a CanvasRenderer. [H2]

A shader channel compiles its own DSL program. A camera, video, or image channel compiles this program: [H5]

search synth
media().write(o0)
render(o0)

ChannelRenderer.compile() shows the load order. It loads the effect manifest once, loads the effects that the program names, and then compiles: [H2]

ChannelRenderer.compile() in channelRenderer.js
async compile(dsl) {
    if (!this._initialized) await this.init()
    try {
        const ids = extractEffectNamesFromDsl(dsl, this._renderer.manifest || {})
            .map((e) => e.effectId)
        if (ids.length) await this._renderer.loadEffects(ids)
        await this._renderer.compile(dsl)
        this._currentDsl = dsl
        this._mediaStep = this._discoverMediaStep(dsl)
        this._lastSize = null
        if (!this._renderer.isRunning) this._renderer.start()
        return { success: true }
    } catch (err) {
        const msg = typeof err === 'string' ? err : err?.message || err?.error || 'compile error'
        this.onError(msg)
        return { success: false, error: msg }
    }
}

If the compile fails, the wrapper passes the message to onError and returns it to the caller. [H2]

The wrapper finds the media() step in the program text. It does not assume that media() is step 0. [H2]

For each media frame, the wrapper first fits the source into a buffer with the channel’s aspect ratio. The cover fit fills the frame and crops the overflow. The contain fit shows the whole source on a black background. The wrapper then uploads the buffer with updateTextureFromSource('imageTex_step_N', buffer, { flipY: false }). It sets imageSize only when the buffer size changes. [H2]

Work outside the engine

ProgramCompositor owns the program output, which is a 2D canvas. It reads each channel canvas with drawImage(). It draws the transitions, the PinP, SPLIT, and QUAD layouts, the KEY overlay, the output effects, and the output fade. [H3]

The audio mixer does not use Noisemaker. It has per-channel EQ, gate, compressor, pan, delay, and AUX and reverb sends. It can switch audio when the video changes, or switch video when the audio changes. [H1]

Recording captures the program canvas at 30 frames per second with captureStream(30). It adds the audio tracks from the main bus or the AUX bus, and MediaRecorder writes the file. [H1] [H4]

Sizes and limits

  • Each channel renders at 960 × 540. The program output is 1280 × 720 by default. [H4]

  • The output resolution setting resizes only the program canvas. No code calls ChannelRenderer.resize(). [H4]

  • Four CanvasRenderer objects run at the same time, one for each channel. [H4]

  • The KEY overlay reads and writes every output pixel on the CPU in each frame, with getImageData() and putImageData(). [H3]

Sources

All links point to hd4 revision 90afc97.

[H1] (1,2,3)

README.