Discussions

Hello!

I'm currently working on 3D avatars with lipsync. It's a bit different than what you offer, because I already have an avatar and an audio, I would just need to align the audio to lipsync, not the whole video creation. Currently I'm using Rhubarb, but it's not very fast and it does not contain all the visemes I would like to have.
I've seen your demo videos and I found them very impressive. I would be interested in using (and paying for) your technology if it could allow me to get only the lipsync part, i.e. I would send an audio and the output would be a list of visemes/phonemes, mapped to the different time steps of the audio, something like this:
[
{"start": 0.0, "end": 0.2, "value": "viseme_sil"},
{"start": 0.2, "end": 0.4, "value": "viseme_aa"},
{"start": 0.4, "end": 0.6, "value": "viseme_sil"},
{"start": 0.6, "end": 0.8, "value": "viseme_sil"},
{"start": 0.8, "end": 1.0, "value": "viseme_RR"},
{"start": 1.0, "end": 1.2, "value": "viseme_E"},
{"start": 1.2, "end": 1.4, "value": "viseme_I"},
{"start": 1.4, "end": 1.6, "value": "viseme_sil"},
{"start": 1.6, "end": 1.8, "value": "viseme_RR"},
{"start": 1.8, "end": 2.0, "value": "viseme_sil"},
{"start": 2.0, "end": 2.2, "value": "viseme_DD"},
{"start": 2.2, "end": 2.4, "value": "viseme_I"},
{"start": 2.4, "end": 2.6, "value": "viseme_nn"},
{"start": 2.6, "end": 2.8, "value": "viseme_E"},
{"start": 2.8, "end": 3.0, "value": "viseme_RR"},
{"start": 3.0, "end": 3.2, "value": "viseme_sil"},
{"start": 3.2, "end": 3.4, "value": "viseme_nn"},
{"start": 3.4, "end": 3.6, "value": "viseme_E"},
{"start": 3.6, "end": 3.8, "value": "viseme_TH"},
{"start": 3.8, "end": 4.0, "value": "viseme_I"},
{"start": 4.0, "end": 4.2, "value": "viseme_sil"},
{"start": 4.2, "end": 4.4, "value": "viseme_SS"},
{"start": 4.4, "end": 4.6, "value": "viseme_nn"},
{"start": 4.6, "end": 4.8, "value": "viseme_E"},
{"start": 4.8, "end": 5.0, "value": "viseme_kk"}
]
I don't know if it would even be possible or how fast it would be, but I would be happy to talk about it,