Hedra AI creates the best lip sync video clips I’ve seen so far. As of this writing it’s free to use since the model it uses for lip sync, called Character-1, is in research beta.
Generating a lip sync video is really simple in Hedra. It needs some audio and a still image. Hedra combines those two pieces and makes the character in the image speak (or sing) the audio with realistic movement of the character’s mouth, eyes, face, head, and even the upper body.
It’s way better than most lip-sync tools, which just move the mouth of the character, or just a little weak and stiff movement of the eyes and face. Hedra seems to understand how bodies move when they’re talking and reportedly adapts the movement to the context of what’s being spoken or sung and the character. In my experience, it hit the mark very well.
How To Create a Lip Sync Video Clip with Hedra
If you have the audio you want to use – something you recorded or generated elsewhere – just import it. If the audio is longer than 30 seconds, Hedra gives you a warning that it may be shortened. Since I’d rather identify the most important part myself, rather than rely on AI to decide what to keep, I cut my audio down to 30 seconds before importing it.
You don’t have to bring your own audio in, however. You can type (or paste) the text you want spoken right into Hedra, choose one of their AI voices, and generate the audio right there on the same page within Hedra where you create your lip sync video project.
The character whose face we’ll see speaking this audio comes from a still image. You can import an image you already have – a picture of yourself or something you created in another AI image generator. But, you don’t have to bring in something from the outside. Hedra will generate an image for you, if you just describe the character you have in mind.
With your audio and character image set, just click the generate button and in about a minute (in my experience) your image comes to life speaking (or singing) the audio with really impressive movement of the mouth, face, eyes, head, and upper body.
I tested several different ways of generating a lip sync video in Hedra, and I was impressed with every single one. It did not take multiple attempts, patience, and determination to get a good result. It just worked!
Limitations with Hedra Lip Sync
As of this writing, you can only create lip sync videos in a square aspect ratio and they will have a Hedra watermark in the lower right corner. Hey, it’s free. It’s a research beta. So we can’t really complain about the watermark or being confined to a 1:1 ratio. The resolution isn’t stunning, but it isn’t bad either.
Folks who live in certain states are prohibited from using Hedra. When you sign up, there’s a message you have to agree to in order to continue creating your account. In addition to acknowledging you are over 18, you also have to acknowledge that you are not in one of a handful of states. I’m not listing those states here because a) I don’t remember exactly which ones, and b) the list of states could change or go away. I assume this has something to do with laws and lawyers because I can’t think of any other reason some states would be prohibited form using the platform.
Another warning I saw, and had to acknowledge, was that you can only upload images of yourself. That seems really weird to me because if I let Hedra generate the character image, it is definitely not going to create an image of me based on a text prompt. So, why can’t I upload an image of not-me that I created elsewhere? I assume, and I may be wrong, that they do not want people creating lip sync videos of celebrities or other people that actually exist. That makes sense and seems fair. Don’t make videos that portray a real person saying something they never said.
I’m no lawyer and I don’t work for Hedra, so don’t take my assumption as fact. But, I assume they are trying to prevent deep-fakes that could have negative consequences and maybe if you generate the image within Hedra it can tell by your text prompt that you’re trying to create an image of a real person and perhaps put the brakes on that, but if you import an image they don’t have the pieces in place to discern whether that the image is a non-existent AI generated non-person, or an image of a real person. Again, I’m just speculating. Follow the rules.
What Else Can Hedra Do?
Hedra’s tagline is “Video for everyone” followed by “Imagine worlds, characters and stories with complete creative control” and Their mission, according to their website, is “We are a creation lab building foundation models into products that power the next generation of human storytelling”
That all sounds like Hedra has more in the works – or at least in the plans – than lip sync. I don’t know what those other tools might be but, if they do as well as they have with Character-1 does with lip sync, I’m looking forward to seeing what they come up with.
Is Hedra Really Free?
As of this writing, there isn’t even a place in my Hedra account to put in payment information or upgrade. I fully expect that to change based on the way most other generative AI companies operate – opening their platforms for free to attract users and gain insights and training data from what those users do, then implement a subscription model for access. Again, I don’t work for Hedra and I don’t know their plans. This is just my guess based on the way others have operated.
I Made Abe Lincoln Sing the Blues!
One of the lip sync creations I made with Hedra was a 30 second clip of Abe Lincoln singing part of a blues song about attending a bad play. It’s toward the end of the video below, where I walk through how to create a lip sync video in Hedra and make several of them using audio and video generated in Hedra and audio and video imported from other sources.
