DanielWolf

Hi! I just released a new version of Rhubarb Lip Sync. Until now, all discussion of Rhubarb has happened in the "Lip Sync" thread over at the Unity forum. As Rhubarb has nothing to do with Unity, I'm creating a new thread for Rhubarb here.

Rhubarb Lip Sync is a free and open-source tool that automatically generates mouth animations from audio files. You can use it with a number of software packages, including Spine. Here are some links:


The latest version 1.8.0 adds support for Ogg Vorbis files. Now Rhubarb supports both .wav and .ogg files.

I'll update this thread whenever there is news regarding Rhubarb for Spine.

Please let me know what you think. I'm always happy to get feedback!

Kullanıcı avatarı
DanielWolf
  • Mesajlar: 26

booboogaga

I will follow this :)
Kullanıcı avatarı
booboogaga
  • Mesajlar: 111

DanielWolf

Rhubarb Lip Sync 1.9 has been released. This version now comes with basic support for languages other than English. So if you're working with dialog in French, Italian, German, Spanish, or any other non-English language, be sure to give it a try!

There are also numerous fixes and improvements. Most notably, processing WAVE files is now considerably faster.

To learn more about the new multi-language support, check out this video:

Kullanıcı avatarı
DanielWolf
  • Mesajlar: 26

Erika

I'm sorry for the late reaction, but OMG! We Italians are so lucky that our language is generally spoken rather clearly, the demos looked all amazing and the phonetic recognizer is definitely a game changer for us XD
Also wow! that dubber is quite famous in Italy, what project are they working on? I'm very curious!

I've been meaning to test this in Spine for a long time, I need to try! Thanks for the update! :D
Kullanıcı avatarı
Erika

Erikari
  • Mesajlar: 3064

towfiqi

Daniel,

Great Work! from the video tutorial you shared, a one line takes around 4-5 seconds to sync. Can this be faster with a powerful machine?

Thanks
towfiqi
  • Mesajlar: 9

Harald

Amazing work indeed!

I really liked playing Thimbleweed Park - the voice acting and lip sync there were really spot an, and now I know why! :)

Keep up the great work!
Kullanıcı avatarı
Harald

Harri
  • Mesajlar: 3996

anapaulacaruso

Hey,
love the premise of Rhubarb and absolutely loved the demo but I'm struggling to figure it out how to use it, being one of those people who can animate but has no idea how to code... :think:
Do you plan to make some tutorial for Rhubarb?

thanks
anapaulacaruso
  • Mesajlar: 3

DanielWolf

Sorry to everybody for not answering! I somehow failed to subscribe to this thread... :wounded:

Thanks for the encouraging words! I'm really glad you like it. :)
Erikari yazdı:Also wow! that dubber is quite famous in Italy, what project are they working on? I'm very curious!
Guga is part of a team of adventure gamers doing a full Italian fan-dub of Ron Gilbert's Thimbleweed Park. The game contains about 16,000 lines of dialog, so this is quite a large project, but I believe they're almost done. Check out their thread in the Thimbleweed Park forums (with several demo videos)!
towfiqi yazdı:from the video tutorial you shared, a one line takes around 4-5 seconds to sync. Can this be faster with a powerful machine?
Sure, the faster the machine, the faster the processing. Also, if a recording is long enough, Rhubarb Lip Sync will process it with multiple threads simultaneously. Typically, processing speed is not an issue.
anapaulacaruso yazdı:Do you plan to make some tutorial for Rhubarb?
I realize that the documentation is rather meager right now. I'm planning on doing a video tutorial, but it may take some time. In the meantime, this document should get you started. Let me know if you have any questions!

By the way, I just released Rhubarb Lip Sync 1.10. The main change for Spine users is that Rhubarb now supports the beta of Spine 3.8. Previous versions of Rhubarb only worked with Spine 3.7.
Kullanıcı avatarı
DanielWolf
  • Mesajlar: 26

Nate

Thanks for staying on top of the latest support! We've been working on an example skeleton that shows off additive tracks for expressions and mouth shapes. We plan to hook it up with Rhubarb so it should also serve as a lip sync example.
Kullanıcı avatarı
Nate

Nate
  • Mesajlar: 11863

DanielWolf

Nate yazdı:Thanks for staying on top of the latest support! We've been working on an example skeleton that shows off additive tracks for expressions and mouth shapes. We plan to hook it up with Rhubarb so it should also serve as a lip sync example.
That sounds great! :)
Kullanıcı avatarı
DanielWolf
  • Mesajlar: 26

Bekko

@DanielWolf

hey we are going take a look at Rhubarb Lip Sync, but our work flow is a bit different, as we want to animate the spine via unity
any advice ?

tiny problem on the mac version :
For complete usage and help, type rhubarb --help

[Fatal] Application terminating with error: Invalid command line.
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.
public enum LipSync { PBM, KST, EH_AE, AA, AO_ER, UW_OW_W, F_V, L, Idle }
public LipSync lipSync;

track0.TimeScale = 0f;
if lipSync = PBM frame = 1 if lipSync = PBM frame = 2 etc...
Bekko
  • Mesajlar: 32

DanielWolf

If you want to lip-sync your skeleton through Unity, you should probably use the command-line Rhubarb tool to generate TSV, XML, or JSON files. Then you need to write some code in Unity that parses these files and switches your skeleton's attachments accordingly. You'll have to code that yourself, but it shouldn't be hard.
Kullanıcı avatarı
DanielWolf
  • Mesajlar: 26

Kombowz

If possible, please keep support for Spine 3.7 as this is the only version of spine that works for Game Maker at the moment. Also, please don't forget about that spine tutorial on how to get it all to work properly that we talked about. Like what to export, import, and where all the files should be etc. I've been stalking your YouTube waiting for it. LOL. Rhubarb Lip Sync is awesome sauce.
Kullanıcı avatarı
Kombowz
  • Mesajlar: 4

DanielWolf

Rhubarb supports both Spine 3.7 and 3.8. I have no plans to change that. But it's good to know that there are people reliant upon version 3.7.

Regarding the tutorial: I've started working on it. But beside my job and my family, I'm afraid my time is rather limited, so it'll still be some days.
Kullanıcı avatarı
DanielWolf
  • Mesajlar: 26

Kombowz

DanielWolf yazdı:Rhubarb supports both Spine 3.7 and 3.8. I have no plans to change that. But it's good to know that there are people reliant upon version 3.7.

Regarding the tutorial: I've started working on it. But beside my job and my family, I'm afraid my time is rather limited, so it'll still be some days.
I'm happy to know that Spine 3.7 will stay supported. As for the tutorial, thank you for starting it, as for finishing it, Your main job and family should always come first.. I've subscribed with notifications for when you upload it and it's available.

I appreciate everything you do, Thanks again for Rhubarb!
Kullanıcı avatarı
Kombowz
  • Mesajlar: 4

DanielWolf

It took me a while, but I finished the video tutorial I promised! It gives step-by-step instructions on how to download Rhubarb Lip Sync, setup your skeleton in spine, animate your recordings, and improve the result.

Let me know what you think!

Kullanıcı avatarı
DanielWolf
  • Mesajlar: 26

Harald

Thanks very much for creating this great in-depth tutorial video, I'm sure it will help a lot of users! Now I even learned something about the history of mouth shapes! :talk: :nerd:
Kullanıcı avatarı
Harald

Harri
  • Mesajlar: 3996

SapientX

Hi Daniel. Do you have any intention of making a real-time version of Rhubarb... perhaps as a Unity asset?
SapientX
  • Mesajlar: 1

DanielWolf

I take it that by real-time, you mean that a player speaks into a microphone and sees an animated mouth without noticeable delay. This is not something I'm planning to do in the foreseeable future. Right now, I'm focusing on improving the existing feature set for pre-recorded audio.

---

I just released Rhubarb Lip Sync 1.12. Until now, Rhubarb didn't support skinned skeletons. The new version now fully supports skinning!
Kullanıcı avatarı
DanielWolf
  • Mesajlar: 26

Misaki

Wow, that's definitely good news! Thanks for your great efforts :D :yes:
Kullanıcı avatarı
Misaki

Misaki
  • Mesajlar: 656

Mario

Super cool!
Kullanıcı avatarı
Mario

Mario
  • Mesajlar: 3035

Harald

That's great news! 8)
Kullanıcı avatarı
Harald

Harri
  • Mesajlar: 3996

jawny

Very cool tool, had fun once I got it set up. Wouldn't work properly with 4.0+ skeletons, possibly related to the curves using doubles instead of integers.

https://imgur.com/a/JVDULmF



I ended up just exporting the mouth as a 3.7 file and copying the keys over to the 4.1 skeleton.

---

Upon further testing it seems that a non-default timeline FPS was causing the issue, not the Spine version.
jawny
  • Mesajlar: 1

Nurlan

Help, please, when you press the animation button on Rhubarb-Lip-Sync Spine 2D, the rhubarb reaches six percent and no longer responds.
Nurlan
  • Mesajlar: 1


Dön Editor