Turning Spotify history into generative art

I wanted to make something for the screen that is usually doing nothing: the spare monitor, the browser tab left open while music is playing, even the projector screen before I start my talk at a first-party event. A player would have been easy, but I did not want another rectangle of album art with familiar controls. I wanted the listening session itself to leave a shape.

I have been long obsessed with the live data visualization wall inside GitHub's San Francisco HQ called Gitmos: a giant display where an interactive globe and rolling stream turn global code activity into flashes of light. Whenever a developer anywhere in the world pushes code or merges a pull request, the event appears live on the wall.

GitHub Gitmos showcasing pull request data

GitHub's wall gave me the model for Audial: take a stream of activity that usually disappears into an interface and give it a persistent, visual presence. Audial does that for Spotify, turning the current track into a dithered composition. Each song gets a repeatable particle field. The album cover hands Audial its colors. Recently played songs remain around it as a constellation, while tracks in the queue appear as smaller satellites. Pausing makes the field dormant; changing tracks dissolves the old form into the next one.

The important qualification is that Audial does not listen to the song. It does not inspect a waveform, predict a beat, or decide that a track is happy, energetic, or danceable. It is a visual system built from identity, metadata, history, and context.

Why the artwork is not beat-reactive#

The first version of this idea almost wrote itself: connect Spotify, fetch audio features, and animate to the beat. That would also have turned Audial into a conventional music visualizer.

Spotify's current Web API gives an app access to the active playback state, including the track, progress, active device, context, shuffle, and repeat state. It also exposes recently played tracks and the playback queue. Its policy notes restrict synchronizing sound recordings with visual media however, and newer development-mode apps cannot depend on the old Audio Features and Audio Analysis workflow.

That constraint gave Audial its character. Audial responds to a track beginning, ending, pausing, or being replaced. The result is calmer, and it can remain on a screen for hours without constantly asking for attention.

A song begins as an integer#

Thanks to my love for math, the Spotify track ID is the stable center of the system. Audial runs it through a 32-bit FNV-1a-style hash:

let hash = 2166136261;

for (const character of trackId) {
  hash = Math.imul(hash ^ character.charCodeAt(0), 16777619);
}

That hash initializes a small seeded pseudorandom-number generator. Calling the generator produces the same sequence every time, so particle position, size, phase, orbit, and color role remain stable for a given track ID.

Audial creates 980 particles, which is essentially close to 1,000 and just enough particles to make the field feel substantial while preserving visible gaps. Most belong to the core field, where their radius is calculated as:

radius = sqrt(random()) × 0.98

The square root matters. A raw random radius would crowd points toward the center; this version distributes them more evenly by area across a disk. Roughly 14 percent of the points become an outer halo, with a separate nonlinear radius that keeps most of them close to the edge while allowing a few to escape farther out.

A second hash selects one of four transformation families. One bends the disk into a spiral. Another introduces lobes. A third creates concentric distortion. The quietest family stays close to an ellipse. The families are deliberately related: Audial has a visual language rather than an unlimited style generator.

This is deterministic, but it is not semantic. A track does not receive a spiral because its chorus sounds circular. It receives one because the hash landed in that family. Stability is doing the meaningful work here. When the song returns, so does its underlying form.

Deterministic vs. semantic generation#

The two are not opposites. A semantic system can also be deterministic if the same phrase produces the same interpretation each time. Audial separates their jobs: the hash gives a track its stable identity, while the semantic controls introduced later can steer that identity without replacing it.

The album cover gives it the yellow.. or whichever color you see#

Testing out the dither field with the obvious

The track ID can make geometry, but it cannot supply a recognizable palette in relation to the track. For that, Audial reads the album artwork into a 48-by-48 canvas and samples the pixels.

Nearly transparent pixels and colors at the extreme ends of lightness are ignored. The remaining samples are converted from RGB to HSL, grouped into 18-degree hue bins, and weighted using the following expression:

weight =
  (0.25 + saturation × 1.75)
  × (1 - abs(lightness - 0.52))

In practical terms, saturated midtones win. The most heavily weighted hue becomes the main ink. Audial looks for a second hue at least 38 degrees away for the accent, then derives a brighter spark color. Minimum saturation and fixed output lightness keep all three visible against the black-brown background.

This is closer to a tiny printmaking algorithm than computer vision. It does not know which object matters in the cover. A small red figure on a blue field will probably lose to the blue because the blue occupies more pixels. Black-and-white artwork may fall back to colors generated from the track ID. Unlike popular opinion, those are not bugs hidden behind the word “AI”; they are choices that give Audial its particular color bias because I have taste. Or I'd at least like to think so.

Metadata becomes visual grammar#

After the base field and palette exist, Spotify metadata changes the way they are drawn.

Older releases receive coarser particles, as if the image had passed through an older printing process. Track number determines the number of internal divisions. Playlist playback produces a square outer frame, an album produces a circle, and artist playback leaves two open edges. Shuffle adds controlled asymmetry. Repeating a context creates one echo; repeating a track creates two. It's very much controlled chaos, if you will.

An ode to my first Arduino Project Hub app, the first three queued tracks are reduced to small 42-particle satellites. Each satellite is already seeded from its future track ID, which lets the next songs exist at the edge of the current composition without pretending to be full previews. This also helps with dissolution of the dither field when a track ends or is changed.

These mappings are also not discoveries about music. An album is not inherently circular, and a song released in 1978 is not inherently grainier than one released in 2025. They are designed metaphors.

Audial needs rules consistent enough to learn through use, even when those rules are subjective.

Listening history leaves a constellation#

The current track is only part of a listening session. Audial fetches up to eight recently played songs and places them around the live field as ranked constellation points. The broad positions are fixed by recency, while each track ID introduces a small deterministic jitter.

That distinction is intentional:

• Recency decides the region
• Track identity decides the offset

The most recent song receives the first and most prominent node. Older songs move through the remaining anchors and leave expanding dashed shells around the current composition. Every 30 seconds, one node briefly announces itself with the track and artist, so the history can surface naturally without turning the screen into a permanent list, very much how Gitmos does with certain pull requests.

Audial also retrieves short-term top tracks and draws them as a faint background path. This is the slowest layer of the image: current playback changes in seconds, recent history changes across a session, and affinity changes over days or weeks.

Giving the dither field a direction with local AI#

There is only one place where AI enters this generative system. A field-direction input lets someone write a phrase such as “cold, fragmented signal” or “dense and radiant, almost still.” Audial runs that text through a quantized MiniLM-v2 embedding model prepared through Qualcomm AI Hub and executed locally in the browser with ONNX Runtime Web.

Why MiniLM-v2 through Qualcomm AI Hub?#

Audial does not need a chat model. It needs to answer a much smaller question: how close is a short phrase to descriptions such as “sparse and restrained” or “dense and luminous”? All-MiniLM-L6-v2 is built for sentence similarity. Its 22.7 million parameters produce a 384-dimensional embedding, which gives Audial just enough semantic structure for those comparisons without shipping a generative language model into the page.

Qualcomm AI Hub was useful at providing an on-device-oriented model implementation and a path to export and quantize it. Audial loads the resulting ONNX artifact with ONNX Runtime Web. The prompt is embedded in the browser instead of being sent to a hosted model API.

In a nutshell, Audial compares the 384-dimensional vector the model produces with hand-written positive and negative anchors for seven axes:

Seven semantic axes

The prompt and anchors are L2-normalized, so their dot product acts as cosine similarity. For each axis, Audial calculates:

axis = clamp(
  (similarity(prompt, positive) - similarity(prompt, negative))
  / max(0.08, 1 - similarity(negative, positive)),
  -1,
  1
)

The denominator compensates for anchor pairs that sit unusually close together in embedding space. The clamp prevents an extreme prompt from producing an unbounded value.

Most importantly, every resulting axis is multiplied by 0.28 before it reaches the renderer. The model can steer density, turbulence, cohesion, symmetry, warmth, motion, and halo behavior by at most 28 percent. It cannot replace the track's seeded identity.

Showcasing prompting to influence the dither field

This is not generative AI in the usual image-making sense. MiniLM never emits pixels, paths, or particle coordinates. It turns language into seven control values or knobs, and the ordinary canvas renderer does the drawing. The prompt remains on the device during inference.

Playback gives the dither field motion#

While a track plays, the dither field runs at a restrained 25 frames per second. A slow radial breath changes its scale. Per-particle phases introduce drift. Turbulence adds directed noise, and shuffle can shear the result away from symmetry. Screen compositing makes overlapping dots accumulate light.

When playback pauses, active drift stops and the field dims. Once the initial reveal is complete, the renderer can stop requesting frames entirely. This both fits the visual idea and avoids spending resources to redraw a dormant composition.

On a track change, Audial captures the outgoing structure and dither layers. A grid of jittered circular and rectangular holes, seeded from the previous track ID, cuts through that snapshot using destination-out. The old image dissolves over 1.65 seconds while the next field reveals beneath it. Even the disappearance belongs to the track that is leaving.

What Audial favors#

The app has several strong biases:

• It favors radial, celestial compositions because every track begins with an orb-like field.
• It favors saturated midtones because of the palette weighting.
• It favors recency because constellation position is chronological, not based on similarity or importance.
• It treats release year, context, shuffle, and repeat as visual metaphors rather than acoustic evidence.
• Its field-direction model inherits the language and cultural biases of MiniLM, then compresses an open-ended phrase into seven hand-authored binary scales.
• It inherits whatever Spotify's account history and short-term affinity data consider representative of the listener.

There is also a limit to the promise of “the same song creates the same artwork”, because it actually never does*. The track ID stabilizes the base particle topology, but it does not make the complete scene identical on every play. Album art, viewport size, playback state, queue, recent history, top tracks, shuffle, repeat, field direction, and controlled chaos all affect the result. The song keeps a recognizable visual signature inside a listening environment that never stands still.

What Audial remembers#

Audial is not trying to uncover a hidden mathematical truth inside a song. It works with the traces that gather around listening; the track identifier, album cover, playback context, sequence, and memory, and holds onto them long enough to give the session a shape. I wanted better visuals for otherwise stagnant screens and Audial is a good enough fit.

If you want to see what your own listening history looks like, or take apart how any of this works, Audial is open source on GitHub under the MIT license. Give it a shot, fork it, or make it your own.