starry0929/Separating-What-From-When

Separating What from When

Fine-Grained Temporal Control for Joint Audio-Video Generation

Yichen Liu¹ · Quanwei Zhang² · Haozhe Wang³ · Donghao Zhou⁴ · Xiaojie Li · Yang Shi² · Jiaming Liu² · Ruihua Huang² · Yingtian Zou⁵ · Daquan Zhou¹

¹Peking University · ²Qwen Business Unit of Alibaba · ³HKUST · ⁴CUHK · ⁵Shanghai Jiao Tong University

Temporal Context Routing (TCR) adds a timing term to text-attention logits so picture and speech are generated as two streams against one shared clock. This repo hosts the released LoRA (separating-what-from-when.safetensors, ~2.3 GB) and the 200 held-out test prompts (test_prompts_200.json) used in the paper. Training and inference code is at DAGroup-PKU/SEPARATING-WHAT-FROM-WHEN. Demos: project page.

Files

File	What it is
separating-what-from-when.safetensors	TCR LoRA, rank 128, trained on LTX-2.3 22B (paper 7k-step recipe)
test_prompts_200.json	200 held-out script JSON files with per-shot / per-line time_range

You still need the LTX-2.3 22B checkpoint and the Gemma 3 12B text encoder locally. One 80 GB GPU is the working setup.

huggingface-cli download starry0929/Separating-What-From-When \
  separating-what-from-when.safetensors test_prompts_200.json \
  --local-dir ./weights

Then follow the code README:

bash scripts/infer.sh \
  --checkpoint /path/to/ltx-2.3-22b-dev.safetensors \
  --text-encoder-path /path/to/gemma-3-12b-it \
  --lora-path ./weights/separating-what-from-when.safetensors \
  --output outputs/tcr_infer.mp4

Citation

@inproceedings{liu2027separating,
  title     = {Separating What from When: Fine-Grained Temporal Control
               for Joint Audio-Video Generation},
  author    = {Liu, Yichen and Zhang, Quanwei and Wang, Haozhe and
               Zhou, Donghao and Li, Xiaojie and Shi, Yang and Liu, Jiaming and
               Huang, Ruihua and Zou, Yingtian and Zhou, Daquan},
  booktitle = {Under review},
  year      = {2027}
}

License

The LoRA is trained on LTX-2.3 and is intended to be used with that backbone under the LTX-2 Community License.