Adobe teases generative AI video tools

On Monday, Adobe announced plans to add AI video-synthesis tools to Premiere Pro and After Effects, reports Reuters. This follows an announcement last week that AI-powered text-based video editing will come to Premiere Pro in May.

Adobe's generative video effects will come courtesy of Adobe Firefly. Firefly, an image synthesis mode we previously covered in March, will expand to include video- and audio-generation tools. Premiere Pro is Adobe's flagship video-editing application, a popular industry tool for assembling TV shows and Hollywood films.

Currently, performing certain special effects or editing tasks can be labor-intensive. Adobe aims to streamline that process with generative AI assistance. In a demo reel posted on its blog, Adobe showed "Firefly for Video" generating a background soundtrack, generating sound effects, changing the visual season in a video, creating a visual effect on the word "molten," locating and brightening a person's face, and generating audio transcripts via text commands.

In its promotional blog, Adobe mentions features we might expect from its new AI tools, including some listed above (like generating sound effects and music), but also text-based color enhancements, text and logo animations, automatic storyboard creation, and creative "co-pilots" that can supposedly generate how-to's to help users master editing skills.

If these kinds of text-based video-editing tools seem familiar, it's because Runway teased similar capabilities in September, although Runway has yet to deliver on those promises with the fidelity previewed in its teaser. Similarly, it's probably wise to take Adobe's marketing video with a grain of salt until the product is released.

(And only time will tell if Adobe Firefly will ever be able to generate a video of Will Smith eating spaghetti that can rival the masterpiece devised by ModelScope, an open source video-synthesis model.)

In a separate but related announcement last week, Adobe announced it is integrating text-based video editing into Adobe Premiere Pro, which will allow editors to trim and rearrange video based on automatically detected transcripts of speech pulled from video clips. Here's how Adobe describes the feature:

Premiere Pro is the only professional editing software to incorporate Text-Based Editing, revolutionizing the way creators approach their craft by making video editing as simple as copying and pasting text. Powered by Adobe Sensei, Text-Based-Editing automatically analyzes and transcribes clips so editors, producers and assistants can simply copy and paste sentences into any order they want and instantly see them appear on the timeline. Transcripts can be searched in the transcript window to identify exact words and phrases.

Adobe initially previewed a similar text-based video-editing capability as part of "Project Blink" in October, although now Adobe says these features come from its "Adobe Sensei" framework. Whether the difference is technical or merely branding is unclear. It's worth noting that a company called Descript offers a similar feature for video and podcast editing.

Adobe says its Firefly for Video features will begin rolling out "later this year" while also allowing people to sign up to try Firefly, which is currently in closed beta testing. Meanwhile, Adobe promises that text-based video editing is coming to Adobe Premiere Pro in May.