The latest public launch of the Hunyuan Video generative AI mannequin has intensified ongoing discussions concerning the potential of huge multimodal vision-language fashions to someday create complete films.
Nevertheless, as we’ve got noticed, it is a very distant prospect for the time being, for various causes. One is the very quick consideration window of most AI video turbines, which battle to take care of consistency even in a brief single shot, not to mention a collection of photographs.
One other is that constant references to video content material (resembling explorable environments, which mustn’t change randomly when you retrace your steps via them) can solely be achieved in diffusion fashions by customization strategies resembling low-rank adaptation (LoRA), which limits the out-of-the-box capabilities of basis fashions.
Subsequently the evolution of generative video appears set to stall until new approaches to narrative continuity are developed.
Recipe for Continuity
With this in thoughts, a brand new collaboration between the US and China has proposed using educational cooking movies as a potential template for future narrative continuity programs.
Click on to play. The VideoAuteur mission systematizes the evaluation of components of a cooking course of, to provide a finely-captioned new dataset and an orchestration technique for the technology of cooking movies. Confer with supply website for higher decision. Supply: https://videoauteur.github.io/
Titled VideoAuteur, the work proposes a two-stage pipeline to generate educational cooking movies utilizing cohered states combining keyframes and captions, reaching state-of-the-art ends in – admittedly – an under-subscribed area.
VideoAuteur’s mission web page additionally contains various relatively extra attention-grabbing movies that use the identical approach, resembling a proposed trailer for a (non-existent) Marvel/DC crossover:
Click on to play. Two superheroes from alternate universes come nose to nose in a pretend trailer from VideoAuteur. Confer with supply website for higher decision.
The web page additionally options similarly-styled promo movies for an equally non-existent Netflix animal collection and a Tesla automobile advert.
In creating VideoAuteur, the authors experimented with numerous loss features, and different novel approaches. To develop a recipe how-to technology workflow, in addition they curated CookGen, the biggest dataset centered on the cooking area, that includes 200, 000 video clips with a median length of 9.5 seconds.
At a median of 768.3 phrases per video, CookGen is comfortably essentially the most extensively-annotated dataset of its variety. Various imaginative and prescient/language fashions have been used, amongst different approaches, to make sure that descriptions have been as detailed, related and correct as potential.
Cooking movies have been chosen as a result of cooking instruction walk-throughs have a structured and unambiguous narrative, making annotation and analysis a better process. Aside from pornographic movies (more likely to enter this specific area sooner relatively than later), it’s troublesome to consider another style fairly as visually and narratively ‘formulaic’.
The authors state:
‘Our proposed two-stage auto-regressive pipeline, which features a lengthy narrative director and visual-conditioned video technology, demonstrates promising enhancements in semantic consistency and visible constancy in generated lengthy narrative movies.
Via experiments on our dataset, we observe enhancements in spatial and temporal coherence throughout video sequences.
‘We hope our work can facilitate additional analysis in lengthy narrative video technology.’
The brand new work is titled VideoAuteur: In direction of Lengthy Narrative Video Era, and comes from eight authors throughout Johns Hopkins College, ByteDance, and ByteDance Seed.
Dataset Curation
To develop CookGen, which powers a two-stage generative system for producing AI cooking movies, the authors used materials from the YouCook and HowTo100M collections. The authors evaluate the size of CookGen to earlier datasets centered on narrative growth in generative video, such because the Flintstones dataset, the Pororo cartoon dataset, StoryGen, Tencent’s StoryStream, and VIST.
Comparability of photos and textual content size between CookGen and the nearest-most populous comparable datasets. Supply: https://arxiv.org/pdf/2501.06173
CookGen focuses on real-world narratives, significantly procedural actions like cooking, providing clearer and easier-to-annotate tales in comparison with image-based comedian datasets. It exceeds the biggest present dataset, StoryStream, with 150x extra frames and 5x denser textual descriptions.
The researchers fine-tuned a captioning mannequin utilizing the methodology of LLaVA-NeXT as a base. The automated speech recognition (ASR) pseudo-labels obtained for HowTo100M have been used as ‘actions’ for every video, after which refined additional by giant language fashions (LLMs).
As an illustration, ChatGPT-4o was used to provide a caption dataset, and was requested to deal with subject-object interactions (resembling palms dealing with utensils and meals), object attributes, and temporal dynamics.
Since ASR scripts are more likely to comprise inaccuracies and to be usually ‘noisy’, Intersection-over-Union (IoU) was used as a metric to measure how intently the captions conformed to the part of the video they have been addressing. The authors observe that this was essential for the creation of narrative consistency.
The curated clips have been evaluated utilizing Fréchet Video Distance (FVD), which measures the disparity between floor fact (actual world) examples and generated examples, each with and with out floor fact keyframes, arriving at a performative outcome:
Utilizing FVD to guage the space between movies generated with the brand new captions, each with and with out using keyframes captured from the pattern movies.
Moreover, the clips have been rated each by GPT-4o, and 6 human annotators, following LLaVA-Hound‘s definition of ‘hallucination’ (i.e., the capability of a mannequin to invent spurious content material).
The researchers in contrast the standard of the captions to the Qwen2-VL-72B assortment, acquiring a barely improved rating.
Comparability of FVD and human analysis scores between Qwen2-VL-72B and the authors’ assortment.
Technique
VideoAuteur’s generative section is split between the Lengthy Narrative Director (LND) and the visual-conditioned video technology mannequin (VCVGM).
LND generates a sequence of visible embeddings or keyframes that characterize the narrative circulation, much like ‘important highlights’. The VCVGM generates video clips primarily based on these selections.
Schema for the VideoAuteur processing pipeline. The Lengthy Narrative Video Director makes apposite picks to feed to the Seed-X-powered generative module.
The authors extensively focus on the differing deserves of an interleaved image-text director and a language-centric keyframe director, and conclude that the previous is the simpler method.
The interleaved image-text director generates a sequence by interleaving textual content tokens and visible embeddings, utilizing an auto-regressive mannequin to foretell the following token, primarily based on the mixed context of each textual content and pictures. This ensures a decent alignment between visuals and textual content.
Against this, the language-centric keyframe director synthesizes keyframes utilizing a text-conditioned diffusion mannequin primarily based solely on captions, with out incorporating visible embeddings into the technology course of.
The researchers discovered that whereas the language-centric technique generates visually interesting keyframes, it lacks consistency throughout frames, arguing that the interleaved technique achieves larger scores in realism and visible consistency. In addition they discovered that this technique was higher capable of be taught a practical visible fashion via coaching, although typically with some repetitive or noisy parts.
Unusually, in a analysis strand dominated by the co-opting of Secure Diffusion and Flux into workflows, the authors used Tencent’s SEED-X 7B-parameter multi-modal LLM basis mannequin for his or her generative pipeline (although this mannequin does leverage Stability.ai’s SDXL launch of Secure Diffusion for a restricted a part of its structure).
The authors state:
‘Not like the traditional Picture-to-Video (I2V) pipeline that makes use of a picture because the beginning body, our method leverages [regressed visual latents] as steady circumstances all through the [sequence].
‘Moreover, we enhance the robustness and high quality of the generated movies by adapting the mannequin to deal with noisy visible embeddings, for the reason that regressed visible latents might not be excellent because of regression errors.’
Although typical visual-conditioned generative pipelines of this type typically use preliminary keyframes as a place to begin for mannequin steering, VideoAuteur expands on this paradigm by producing multi-part visible states in a semantically coherent latent area, avoiding the potential bias of basing additional technology solely on ‘beginning frames’.
Schema for using visible state embeddings as a superior conditioning technique.
Checks
According to the strategies of SeedStory, the researchers use SEED-X to use LoRA fine-tuning on their narrative dataset, enigmatically describing the outcome as a ‘Sora-like mannequin’, pre-trained on large-scale video/textual content couplings, and able to accepting each visible and textual content prompts and circumstances.
32,000 narrative movies have been used for mannequin growth, with 1,000 held apart as validation samples. The movies have been cropped to 448 pixels on the quick aspect after which center-cropped to 448x448px.
For coaching, the narrative technology was evaluated totally on the YouCook2 validation set. The Howto100M set was used for information high quality analysis and in addition for image-to-video technology.
For visible conditioning loss, the authors used diffusion loss from DiT and a 2024 work primarily based round Secure Diffusion.
To show their competition that interleaving is a superior method, the authors pitted VideoAuteur in opposition to a number of strategies that rely solely on text-based enter: EMU-2, SEED-X, SDXL and FLUX.1-schnell (FLUX.1-s).
Given a worldwide immediate, ‘Step-by-step information to cooking mapo tofu’, the interleaved director generates actions, captions, and picture embeddings sequentially to relate the method. The primary two rows present keyframes decoded from EMU-2 and SEED-X latent areas. These photos are reasonable and constant however much less polished than these from superior fashions like SDXL and FLUX.
The authors state:
‘The language-centric method utilizing text-to-image fashions produces visually interesting keyframes however suffers from an absence of consistency throughout frames because of restricted mutual info. In distinction, the interleaved technology technique leverages language-aligned visible latents, reaching a practical visible fashion via coaching.
‘Nevertheless, it often generates photos with repetitive or noisy parts, because the auto-regressive mannequin struggles to create correct embeddings in a single cross.’
Human analysis additional confirms the authors’ competition concerning the improved efficiency of the interleaved method, with interleaved strategies reaching the very best scores in a survey.
Comparability of approaches from a human research performed for the paper.
Nevertheless we observe that language-centric approaches obtain the most effective aesthetic scores. The authors contend, nonetheless, that this isn’t the central concern within the technology of lengthy narrative movies.
Click on to play. Segments generated for a pizza-building video, by VideoAuteur.
Conclusion
The preferred strand of analysis in regard to this problem, i.e., narrative consistency in long-form video technology, is anxious with single photos. Tasks of this type embody DreamStory, StoryDiffusion, TheaterGen and NVIDIA’s ConsiStory.
In a way, VideoAuteur additionally falls into this ‘static’ class, because it makes use of seed photos from which clip-sections are generated. Nevertheless, the interleaving of video and semantic content material brings the method a step nearer to a sensible pipeline.
First printed Thursday, January 16, 2025
