Pretend you are a broke filmmaker, what do you need?
You can’t afford much so you will probably end up writing, directing and editing yourself. However, at the very least you need to figure out
What the audience will see: Cinematography, acting, lighting, color grading…
What the audience will hear: Sound design, music…
That’s the bare bone of course. Depending on your film, you might need other cool people like intimacy coordinator, weapon master, make-up artists, etc. I won’t go into details as I have literally zero experience in that department, the closest thing I had to a make-up artist is my mom applying sunscreen to my dad.
Sidebar — at the very least, you should really be figuring out how to make the audience feel something, but let’s stay pragmatic here.
The Visual Stack
AI tools can help create cinematic footage in two ways
video generation (from scratch, i.e. vision to visual)
video modification (restyle, object removal, etc)
Generating a video is easy if you know how to communicate your creative direction effectively. Simply feed the AI with a text description of your vision with an optional reference image, and voila! You now have a footage! Some advanced capabilities include
end frame control: this allows you to dictate the shot the footage should end on, which is helpful if you have a sequence of key frames that needs animation.
camera movement control: this allows for more precise controls of your camera work such as pan, tilt, zoom, etc.
motion brush: this allows you to specify motion for entities in the reference image
Video modification can come in handy when you have a video already but
want to transfer it into a completely different style
want to add, replace or remove something from the footage
The following tools are not publicly available yet. I was going to call out Sora specifically for the insufferable wait but they did launch today!
Video Upscaling
Keen readers might notice that none of the tools generates 4K footages. In fact, you often have to pay extra just to get 1080p, not to mention that some tools just straight up don’t have anything more than 720p. Are we doomed?! How can I watch anything less than 4K?! Luckily video upscalers can save the day here.
There are a couple players in the vertical as well, and you can clearly see how much of an improvement you might get. This image came from a demo of one such tool called Topaz — original image on the left and upscaled version on the right. Pretty amazing, right?
Fun Effects
Lastly, there are other models out there with some pretty fun video effects, which is particularly interesting as they really lean into the AI-ness. For instance, Midjourney’s image generation video is pretty cool, it’s like an MRI of the AI brain.
Replicate also hosts some models that lean into their own styles such as this Cog Stable Diffusion Model that creates cool interpolation among key frames.
Pika also released a suite of special video effects called Pikaffects. My favorite is this effect that can pretty much inflate anything (but your ego).
Next in line, both Luma and Sora have the ability to create infinitely looping video by morphing the video back to a previous frame. This clip below came from an OpenAI demo, where we see a mountain biker realizing that it’s really all downhill from here.
Lastly, Runway greens screen could save the day when you want to isolate certain elements of the video and overlay it on other visual elements.
Next Up
Alright, now you have everything you need…
to make a silent film, so in the next post we will cover the audio stack for AI film making, stay tuned!
In the meantime, please subscribe and make me feel needed, thanks.