OmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditions

Yuanhao Cai¹, He Zhang², Xi Chen³, Jinbo Xing⁴, Kai Zhang², Yiwei Hu², Yuqian Zhou²,
Zhifei Zhang², Soo Ye Kim², Tianyu Wang², Yulun Zhang⁵, Xiaokang Yang⁵, Zhe Lin², Alan Yuille¹

¹ Johns Hopkins University ² Adobe ³ HKU ⁴ CUHK ⁵ Shanghai Jiao Tong University

Paper arXiv Video Code (coming soon) 🤗 Demo

A data construction pipeline and a DiT method for controllable subject-driven video customization

Method Overview

OmniVCus can compose different input signals to customize a video. We design: (a) Lottery Embedding (LE) enables more-subject customization in inference by activating more frame embeddings with training subjects. (b) Temporally Aligned Embedding (TAE) extracts the guidance from control signals by aligning the frame embeddings of condition and noise tokens.

Table of Contents for This Webpage

(a) Main Subject-driven Video Customization Results

(a1) Single-subject Video Customization

(a2) Instructive Editing Subject-driven Video Customization

(a3) Double-subject Video Customization

(a4) Zero-shot More-subject Video Customization

(a5) Camera-controlled Subject-driven Video Customization

(a6) Depth-controlled Subject-driven Video Customization

(a7) Mask-controlled Subject-driven Video Customization

(b) Challenging Cases: Composing Different Control Conditions

(c) Model Emergence Capability: Text-to-4D

(d) Comparison with State-of-the-Art Methods

(a1) Single-subject Video Customization

The woman in IMG1 is talking to a man on a street

A giraffe from IMG1 is standing in the winter

A dog is playing with a yellow toy car of IMG1 in a living room

A girl putting on a pair of glasses that turn into the effect of IMG1

A boy of IMG1 preying in a temple

A woman of IMG1 mopping the floor in a living room

(a2) Instructive Editing Subject-driven Video Customization

(The instructive editing texts are in purple color)

(Style Transfer)

Make the stone cross as IMG1 on a stone pavement a sketch

(Color Change)

Turn the color of the toy car in IMG1 to red playing with a dog

(Removal)

Remove the blazer of the man in IMG1 standing in a summer park

(Replacement)

Swap the T-shirt of the boy in IMG1 with a blue shirt preying in a temple

(Addition)

Add a necklace to the woman in IMG1 running in the winter

(Expression Change)

Make the child in IMG1 happy doing her homework

(a3) Double-subject Video Customization

A man of IMG1 running in front the church of IMG2

A man pushing a wheelbarrow as IMG1 with oranges as IMG2 in an orchard

Turn the color of the sweater of the man in IMG1 to gray drinking a cup of coffee as IMG2

Swap the skirt of the woman as IMG1 to a gray legging and a woman as IMG2 running in a park

A man using a knife as IMG1 to cut an orange as IMG2 in the kitchen

Tomoatoes as IMG1 and sushi as IMG2 on a table

The man as IMG1 using a laptop as IMG2 in a library

Add a black coat to the man as IMG1 using a laptop as IMG2 in a library

A dog as IMG1 playing with the yellow toy car as IMG2 in a living room

A dog as IMG1 playing with the yellow toy car as IMG2 in a plaza

(a4) Zero-shot More-subject Video Customization

(Our model is trained with 2 subjects but can infer with 3 and 4 subjects)

A man pushing a wheelbarrow as IMG1 with oranges as IMG2 and grapes as IMG3 in an orchard

Turn the color of the T-shirt of the girl as IMG1 to light gray using a laptop as IMG2 at a table with a cup as IMG3 and a tomato as IMG4

A dog of IMG1 running by a stone cross of IMG3 and purple flowers of IMG4 in front of the church of IMG2

A sponge as IMG2, a cucumber in IMG3, and an orange as IMG4 in the sink as IMG1 with water flowing

(a5) Camera-controlled Subject-driven Video Customization

A church as IMG1 beside a cemetery in the winter

A woman in a mask as IMG1 holding a book and sitting on a couch in a living room

A bridge like IMG1 by the ocean

Tie the hair of the child as IMG1 doing her homework at the desk

(a6) Depth-controlled Subject-driven Video Customization

(Our model can force unmatched subjects to be filled into the depth control sequence, e.g., example 1 is man-to-woman, example 2 is woman-to-man, example 4 is orange-to-strawberry. These cases are more challenging and impressive.)

Input Depth

A man as IMG1 standing in front of an old airplane

Input Depth

Replace the shirt of the woman as IMG1 with a suit talking to her colleagues in an office

Input Depth

A boy as IMG1 looking into an open refrigerator with a tomato and a bottle of beer

Input Depth

An orange from IMG1 being dipped into melted chocolate

(a7) Mask-controlled Subject-driven Video Customization

(1) When the control mask is not complete or accurate, our model can still fill the subject and customize the video, e.g., the T-shirt of the boy in example 1.
(2) Our model can force unmatched subjects to be filled into the mask control sequence, e.g., example 3 fills a dog to the mask of a leopard and example 4 fills a church to the mask of a stone house.