Abstract
“A car drives from front left to front right while a dog follows it.” Such a description specifies no particular trajectory, yet imposes a constraint on the joint motion of both sources that must hold at every frame. Existing text-to-spatial-audio methods can position and move each sound source according to its description, but none of them ensures or verifies that such inter-source relations hold. We therefore propose Rel2Traj, which parses the description into sources, relations, and anchors, jointly generates the trajectories of all sources using a relation-conditioned diffusion model, and finally renders sounds from a pretrained text-to-audio model along these trajectories to produce binaural audio. For training and evaluation, we construct a synthetic dataset from a controlled language of source motions and inter-source relations. A single frame-wise relational geometry defines both our evaluation metric, the Relation Satisfaction Rate (RSR), and an auxiliary training loss. On motion-relation combinations held out from training, Rel2Traj maintains all relation types, whereas every baseline fails on at least one. A listening test further confirms that listeners recognize the described relations. To our knowledge, this is the first text-to-spatial-audio work to model relations among moving sources as explicit geometric constraints.
Index Terms — spatial audio, sound source trajectory, inter-source relations, diffusion model, binaural audio generation
Interactive comparison
The same scene, five ways of placing it in space
Each scene below has several moving sources whose relation to each other is what the text asks for. Pick a scene, drag the view, press play, and switch system: the source clips, the renderer and the playhead all stay fixed, so the only thing that changes is the geometry. The link drawn between the two related sources is green while the geometric constraint holds and red while it fails — the strip below the view is that same signal over the whole clip, and the relation satisfaction rate is its green share. As in the paper's tables, every system except LLM-direct is conditioned on the scene's ground-truth structure (sources, relations, anchors), so the comparison isolates generation from parsing; LLM-direct reads the raw sentence.
Drag to orbit · wheel to zoom · grid cells are 1 m · the head at the origin faces front.
Relation satisfaction in this scene
RSR is the fraction of in-scope frames whose signed margin exceeds the dead zone. Ground truth is built from motion primitives with a 95% quality gate, which is why it scores 0.996 rather than 1.0.
What the relation score cannot see
A planner that optimises the declared relations can satisfy them and still produce geometry no listener would accept. Head clearance is the share of frames with a source closer than 0.5 m to the listener, and the last column is the closest any source ever gets. Our own weak column is smoothness: the diffusion model is jerkier than the ground truth.
One text, many trajectories
Diversity under constraint
A description fixes the relations, not the trajectory: “a dog follows the car” is satisfied by infinitely many pairs of paths. A generative model should return different valid ones on different draws. Tick all draws in the viewer to see the ten draws of the selected system at once; the table gives the mean pairwise displacement between draws for the scene on screen. Spread is only a virtue while the relation still holds: the independent baseline spreads more than Rel2Traj because it has no anchors to pin the sources down, and its relation score collapses at the same time. That pairing — wider spread, broken relation — is the signature of an under-determined generator, not of a richer one.
Compositional generalisation
Results per held-out combination
Each held-out cell is a motion–relation combination that never appears in training while its sibling does. For every cell, the table reports the RSR of the cell's defining relation and the scene mean, per system, over exactly the test scenes that carry that cell. Same protocol as the paper's main table: per-type mean over all relation instances and all draws, no selection.
Per inter-source relation, whole splits
The paper's per-relation breakdown, on the in-distribution validation split and the compositional test split.
What the system can be told
Relations and motions
Descriptions are written in a controlled relational language. Relations either place a source with respect to the listener or tie two sources to each other; the inter-source family is the one this work is about. Counts are relation instances across the whole dataset.
Held-out motion–relation combinations
Five combinations were frozen out of training before any model was trained. Each keeps a sibling in training, so what is tested is composition, not an unseen relation type.
Text to conditioning
The language model in the pipeline
End-to-end RSR is the trajectory score when the model is conditioned on the parse instead of the ground-truth graph, on the same held-out scenes. Note that the LLM-direct baseline in the viewer above is a different thing: there, a general model is asked for coordinates directly, with no graph, no anchors and no trajectory model.