PerceptAlign

PerceptAlign generates a soundtrack for a silent video and aligns each sound to the moment its on-screen event happens. Prior video-to-audio models, including SpecVQGAN, usually produce semantically relevant audio but misalign its timing: a video of a hammer is paired with the right sound, but the impact does not coincide with the onset of that sound. PerceptAlign adds a timing signal to the generation process itself rather than aligning audio to video after the fact.

How it works

The model builds on SpecVQGAN's codebook and transformer design. A VQGAN encodes and reconstructs mel spectrograms into a discrete codebook, and a transformer autoregressively samples codebook entries conditioned on visual features. Those features are optical flow frames from denseflow and RGB features from a BN-Inception network.

The timing signal comes from a Perceptual Alignment Loss (PAL), which extends the temporal binding window the human brain uses to accept small audio-visual asynchrony. PAL upsamples the visual features by linear interpolation to match the spectrogram's temporal dimension, 212 video frames against 848 audio frames, then passes both through a local feature extractor of two Conv1D-BN-GELU blocks with L2 normalization. It computes a cross-correlation between the audio and visual embeddings across a window of offsets within each of S segments, softmax-normalizes the offsets with temperature τ, and sums them under a Gaussian-weighted temporal binding window that tolerates slight misalignment while rewarding a zero offset. PAL is the negative log of the mean weighted sum across segments.

The total objective is a linear combination of codebook cross-entropy and PAL, LTL = α·LCE + β·LPAL, so fidelity and timing are scored as separate terms. Generation runs in real time, carried over from SpecVQGAN's codebook sampling. Timing is evaluated with Dynamic Time Warping under a Euclidean distance, alongside KL divergence, Pearson correlation, and mean squared error.

Why it's interesting

PAL improves every reported metric over the SpecVQGAN baseline on both the synthetic FlashDot dataset, which pairs a flashing red dot with a 1kHz sine wave to isolate the timing task, and the open-domain VAS dataset. A cross-correlation window of K = 10 works best on both datasets even though larger expected offsets seemed to call for a wider window. Segmentation helps up to 32 segments for the 848-frame spectrograms and gives no further gain beyond that, and a τ between 1 and 3 with a 0.7-to-1.0 PAL-to-CE weighting balances the two losses.

Status

Code is at github.com/sameraslan/PerceptAlign, and the repository includes the training, evaluation, and inference pipeline along with the FlashDot dataset generator for fine-tuning on other datasets.