→

"a bear is playing a guitar"

"a monkey is playing a guitar"

"a man is playing a guitar, cartoon style"

"a man is playing a guitar, Matisse style"

→

"a man is on the swing"

"a Iron Man is on the swing"

"a woman is on the swing, Van Gogh style"

"a watercolor painting that a woman is on the swing"

→

"a Lionel Messi is dribbling a baksetball"

"a racoon is dribbling a baksetball"

"a oil painting that a man is dribbling a baksetball"

"a man is dribbling a baksetball, Renoir style"

→

"a zombie is running"

"a werewolf is running"

"a man is running, pixar style"

"a pencil sketch that a man is running"

→

"a bull is walking"

"a zebra is walking"

"a cow is walking on the snow"

"a cow is walking on the desert"

Previous

Next

Abstract

Despite the fact that text-to-video (TTV) model has recently achieved remarkable success, there have been few approaches on TTV for its extension to video editing. Motivated by approaches on TTV models adapting from diffusion-based text-to-image (TTI) models, we suggest the video editing framework given only a pretrained TTI model and a single <text, video> pair, which we term Edit-A-Video. The framework consists of two stages: (1) inflating the 2D model into the 3D model by appending temporal modules and tuning on the source video (2) inverting the source video into the noise and editing with target text prompt and attention map injection. Each stage enables the temporal modeling and preservation of semantic attributes of the source video. One of the key challenges for video editing include a background inconsistency problem, where the regions not included for the edit suffer from undesirable and inconsistent temporal alterations. To mitigate this issue, we also introduce a novel mask blending method, termed as sparse-causal blending (SC Blending). We improve previous mask blending methods to reflect the temporal consistency so that the area where the editing is applied exhibits smooth transition while also achieving spatio-temporal consistency of the unedited regions. We present extensive experimental results over various types of text and videos, and demonstrate the superiority of the proposed method compared to baselines in terms of background consistency, text alignment, and video editing quality.

Baseline Comparisons

"a Bat Man is boxing"

Source Video

Ours (Edit-A-Video)

Framewise Editing

Tune-A-Video

SDEdit

"a gorilla is riding a motorcycle"

Source Video

Ours (Edit-A-Video)

Framewise Editing

Tune-A-Video

SDEdit

Bibtex

        @misc{shin2023editavideo,
            title={Edit-A-Video: Single Video Editing with Object-Aware Consistency}, 
            author={Chaehun Shin and Heeseung Kim and Che Hyun Lee and Sang-gil Lee and Sungroh Yoon},
            year={2023},
            eprint={2303.07945},
            archivePrefix={arXiv},
            primaryClass={cs.CV}
        }