Shuffleset: an audio visualizer in a media player

Open Shuffleset

Shuffleset is a media streaming site. Its audio player shows cover art and a track list, and plays sets in order or shuffled. A Noisemaker visualizer fills the area behind the cover art and the track list. [S1]

How it uses Noisemaker

The visualizer is one DSL program, stored as a constant in the player module: [S1]

VISUALIZER_DSL in audio-view.js
search synth, filter, render

scope(
  color: #ffffffff,
  thickness: 3,
  gain: 1
)
  .subchain(name: "feedback loop with waves", id: "ny2d") {
    .loopBegin(alpha: 85)
    .warp(
      strength: 3.49,
      scale: -0.45,
      speed: 5
    )
    .feedback(
      blendMode: lighten,
      mix: 42.31,
      scaleAmt: 112.6
    )
    .loopEnd()
  }
  .subchain(name: "lens effects", id: "01h4") {
    .chromaticAberration(aberration: 25)
    .bloom(taps: 15)
    .lens(displacement: -0.91)
    .vignette()
  }
  .write(o0)

render(o0)

In this program, only scope() reads audio. It draws the waveform from the audioWaveform uniform, which holds 128 samples. [S4]

NoisemakerRenderer wraps one CanvasRenderer with preferWebGPU: false, so the visualizer renders with WebGL2. It loads the effect manifest, loads the effects that the program names, and compiles the program. [S2]

connectAudio() connects the player’s <audio> element to the engine:

  1. Create an AudioContext and an AnalyserNode with fftSize = 256.

  2. Connect the audio element to the analyser with createMediaElementSource(), and connect the analyser to the audio output.

  3. Create the engine’s audio state with setAudioState(), which returns it.

  4. Start the update loop. [S2]

The update loop runs on each animation frame:

NoisemakerRenderer._audioUpdate() in renderer.js
_audioUpdate() {
    if (!this._audioEnabled) return
    this._audioState.updateFromAnalyser(this._audioAnalyser, 8)
    this._audioAnalyser.getByteFrequencyData(this._fftData)
    this._audioState.setSpectrum(this._fftData)
    this._audioAnalyser.getByteTimeDomainData(this._timeDomainData)
    this._audioState.setWaveform(this._timeDomainData)
    this._audioAnimId = requestAnimationFrame(() => this._audioUpdate())
}

updateFromAnalyser() calculates the low, mid, and high bands. Its second argument is the number of frames in the rolling average, from 1 to 10. setSpectrum() and setWaveform() keep the first 128 values of each array. [S2] [S3]

Work outside the engine

The player does not connect the audio when it starts the visualizer. It connects the audio after the first play() call succeeds, or when the user presses play. The goal is to create the AudioContext during a user action, because Safari otherwise creates it suspended. [S1]

After createMediaElementSource(), all sound goes through the AudioContext. If the browser suspends the context, the player is silent. The wrapper tries to resume the context, and it adds a one-time resume on the next click, touch, or key press. [S2]

If the visualizer fails to start, the player disposes the renderer and hides the canvas. It never creates an AudioContext, so the audio element plays directly. [S1]

When the view closes, dispose() stops the update loop, disconnects the source, closes the AudioContext, and resets the audio state. Then it disposes the CanvasRenderer with { loseContext: true, resetCanvas: true }. [S2]

Limits

  • A suspended AudioContext silences playback, not only the visualizer. [S2]

  • The visualizer program is part of the player module. A new visual means a change to that module. [S1]

  • The canvas matches the CSS pixel size of its area and ignores the device pixel ratio. [S1]

Sources

S1 and S2 are the files that shuffleset.stream serves. Shuffleset’s source repository is private. S3 and S4 point to noisemaker revision 0f62657.