Story

I adapted VACE to work with real-time autoregressive video generation.

Here's what it can do right now in real time:

- Depth, pose, optical flow, scribble, edge maps — all the v2v control stuff - First frame animation / last frame lead-in / keyframe interpolation - Inpainting with static or dynamic masks - Stacking stuff together (e.g. depth + LoRA, inpainting + reference images) - Reference-to-video is in there too but honestly quality isn't great yet compared to batch

Getting ~20 fps for most control modes on a 5090 at 368x640 with the 1.3B models. Image-to-video hits ~28 fps. Works with 14b models as well, but doesnt fit on 5090 with VACE.

This is all part of Daydream Scope (https://github.com/daydreamlive/scope), which is an open source tool for running real-time interactive video generation pipelines. The demo was created in scope, and is a combination of Longlive, VACE+Scribble, Custom LoRA.

There's also a very early WIP ComfyUI node pack wrapping scope: https://github.com/daydreamlive/ComfyUI-Daydream-Scope

Curious what people think.