Music Inpainting with Similarity-Guided Latent Diffusion

Sean Turland1,2, Eloi Moliner1, Vesa Välimäki1

1Acoustics Lab, Dept. of Information and Communications Engineering, Aalto University, Finland
2Acoustics and Audio Group, University of Edinburgh, United Kingdom

Proc. of the 27th Int. Society for Music Information Retrieval Conf. (ISMIR), Abu Dhabi, UAE, 2026

Paper (Coming Soon) Code (Coming Soon)

Block diagram of SimDPS. A similarity search over the source corpus S turns the observation Y into a retrieved guide X-tilde. The likelihood score from Y, an auxiliary likelihood score from X-tilde, and the prior score model are summed to form the update applied at each step of the reverse diffusion process, from noise X_T down to the reconstruction X_0.

Abstract

We propose a music inpainting technique combining similarity matching and a diffusion-based generative model to replace large segments of audio recordings with suitable alternatives. The proposed hybrid method, Similarity-Guided Diffusion Posterior Sampling (SimDPS), retrieves candidate replacement segments from the wider signal based on contextual similarity and uses them to guide the diffusion process toward coherent reconstructions. Being model-agnostic, the framework is compatible with any pre-trained diffusion model operating in either the waveform or latent domain. Objective evaluations demonstrate that SimDPS outperforms widely adopted RePaint and DPS methods across acoustic features and neural-embedding distance metrics. Subjective evaluations on multi-genre music datasets confirm that SimDPS enhances perceptual plausibility compared to previous diffusion-based methods and demonstrates robustness to poorly matched similarity candidates. These findings pave the way for improved music restoration and context-aware audio editing.

Listening Test — Diffusion-based Latent Inpainting on General Music

Six excerpts of varying genre were taken from the MUSDB18-HQ test split and assessed by 23 listeners in a multi-stimulus test using webMUSHRA. All diffusion-based reconstructions use the ACE-Step Latent autoencoder, run without prompts or other conditioning inputs. The masked excerpt was given as the reference for the surrounding musical context, and listeners rated how plausible each reconstruction sounded within that context. Excerpts are 15 s long with a 5-s gap; the stimuli below are the files used in the test.

Excerpt:

Listening Test — Diffusion-based Audio Inpainting on Piano

The protocol was repeated with a waveform-domain prior, an MR-CQTdiff model trained on MAESTRO at 44.1 kHz. Each 6-s piano excerpt contains a 2-s central gap, and Linear Predictive Coding (LPC) serves as the low anchor. SimDPS-Strict and SimDPS-Relaxed differ in the strength of the similarity guidance applied at inference time.

Excerpt:

Further Audio Examples

Additional latent-domain reconstructions of 15-s MUSDB18-HQ excerpts, each with a 5-s gap placed at random within the interior of the segment. Retrieval is restricted to the remainder of the same song, outside the masked interval.

Excerpt:

BibTeX

@inproceedings{turland2026simdps,
  title     = {Music Inpainting with Similarity-Guided Latent Diffusion},
  author    = {Turland, Sean and Moliner, Eloi and V{\"a}lim{\"a}ki, Vesa},
  booktitle = {Proc. of the Int. Society for Music Information Retrieval Conf. (ISMIR)},
  year      = {2026}
}