README.md

---
license: apache-2.0
language:
  - en
library_name: diffusers
pipeline_tag: video-to-video
tags:
  - diffusers
  - video
  - video-to-video
  - video-editing
  - reference-image-guided
  - autoregressive-diffusion
base_model: jdopensource/JoyAI-Video-Edit
inference: false
---

# JoyAI-Video-Edit Diffusers

This repository contains the Diffusers-format release of the JoyAI-Video-Edit `0811` checkpoint. It provides the `JoyVideoEditPipeline`, transformer, causal video VAE, and scheduler in the standard Diffusers directory layout.

The MiMo-VL text/vision encoder is not duplicated in this repository. Load it separately from [`XiaomiMiMo/MiMo-VL-7B-RL-2508`](https://huggingface.co/XiaomiMiMo/MiMo-VL-7B-RL-2508), as shown below.

## Links

- [Project repository](https://github.com/jd-opensource/JoyAI-Video-Edit)
- [Paper](https://arxiv.org/abs/2608.03974)
- [Original checkpoint](https://huggingface.co/jdopensource/JoyAI-Video-Edit)
- [Online demo](https://huggingface.co/spaces/wxDai/joyai-video-edit)

## Installation

The checkpoint requires a Diffusers build containing `JoyVideoEditPipeline`. Until the implementation is available in a released Diffusers version, install the development branch:

```bash
pip install --upgrade torch transformers accelerate safetensors imageio-ffmpeg
pip install --upgrade "git+https://github.com/feice-huang/diffusers.git@add_joyvideoedit"
```

## Usage

```python
import torch
from diffusers import JoyVideoEditPipeline
from diffusers.utils import export_to_video, load_video
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration

model_id = "jdopensource/JoyAI-Video-Edit-Diffusers"
mimo_id = "XiaomiMiMo/MiMo-VL-7B-RL-2508"

processor = AutoProcessor.from_pretrained(mimo_id)
text_encoder = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    mimo_id,
    torch_dtype=torch.bfloat16,
)

pipe = JoyVideoEditPipeline.from_pretrained(
    model_id,
    text_encoder=text_encoder,
    processor=processor,
    torch_dtype=torch.bfloat16,
)
pipe.enable_model_cpu_offload()

video = load_video("input.mp4")

result = pipe(
    video=video,
    prompt="Turn the scene into a watercolor painting.",
    num_inference_steps=2,
    generator=torch.Generator(device="cuda").manual_seed(0),
    output_type="pil",
)

export_to_video(result.frames[0], "output.mp4", fps=16)
```

## Input and output notes

- The pipeline performs flow-matching denoising and does not use classifier-free guidance. It does not accept `negative_prompt` or `guidance_scale`.
- The source video frame count must be `8 * n + 1` after preprocessing.
- Output height and width must be divisible by `24`. If omitted, they default to the source video dimensions.
- `num_inference_steps=2` is the checkpoint's standard inference setting.
- Supported `output_type` values are `"pil"`, `"np"`, `"pt"`, and `"latent"`.
- `ref_image` is optional and enables reference-image-guided editing.
- The repository does not include `text_encoder`, `tokenizer`, or `processor`; these are loaded from MiMo-VL or replaced with precomputed embeddings.


## Repository structure

```text
JoyAI-Video-Edit-Diffusers/
├── model_index.json
├── scheduler/
│   └── scheduler_config.json
├── transformer/
│   ├── config.json
│   ├── diffusion_pytorch_model.safetensors.index.json
│   └── diffusion_pytorch_model-00001-of-00007.safetensors ...
└── vae/
    ├── config.json
    └── diffusion_pytorch_model.safetensors
```

## Citation

```bibtex
@article{xiao2026joyai,
  title={JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion},
  author={Xiao, Yicheng and Dai, Wenxun and Qin, Xinran and Song, Lin and Zhang, Maoquan and Xu, Hang and Chen, Yukang and Li, Yitong and Zhang, Guohui and Zhang, Yuan and Zhang, Xuying and Zhang, Tommy and Yuan, Jianlong and Li, Peihao and Lu, Shuai and Fu, Siming and Zhao, Chuyang and Han, Xin and Huang, Jie and Li, Wenbo and Ma, Guoqing and Huang, Wei and Qi, Xiaojuan and Huang, Haoyang and Duan, Nan},
  journal={arXiv preprint arXiv:2608.03974},
  year={2026}
}
```

## License

Apache License 2.0. See the [original project](https://github.com/jd-opensource/JoyAI-Video-Edit) for details.