Video Motion Graphs

Haiyang Liu^1,2, Zhan Xu², Fa-Ting Hong², Shin-Ping Huang², Yi Zhou², Yang Zhou²

¹The University of Tokyo, ²Adobe Research

🤗 Models (Under Legal Review) arXiv Video Code (Under Legal Review)

Video 0

Video 1

Video 2

Video 3

Video 4

Video 5

Video 6

Video 7

Application - Music to Dance Video Retrieval + Generation

Style 1

Style 2

Style 3

Style 3

Reference (29s)

Variance 1

Variance 2

Variance 3

Evaluation - Video Frame Interpolation for HMInterp (Core of Video Motion Grpahs)

Comparison with commercial models

Comparison with Open-Source models

Start

End

FILM

VFIFormer

DCInterp

ACInterp

Ours

GroundTruth

Sports

Start

End

FILM

VFIFormer

DCInterp

ACInterp

Ours

GroundTruth

Music

Start

End

FILM

VFIFormer

DCInterp

ACInterp

Ours

GroundTruth

Talk

Start

End

FILM

VFIFormer

DCInterp

ACInterp

Ours

GroundTruth

Evaluation - Ablation on Motion Guidance Interpolation

Start&End Poses

Linear Blending

Ours

GroundTruth

Ours vs. GT

Ours vs. Linear

Evaluation - Ablation on Improved Reference Decoder

Reference

Vanilla (SD)

Consistency (DALLE)

DualRef (ToonCrafter)

Temporal (SVD)

DupRef ToonCrafter

GroundTruth

Introduction Video