Previous Thread
Index
Next Thread
Print Thread
Go To
User Showcase
Joined: Sep 2010
Posts: 6,576
dcuny Offline OP
Veteran
OP Offline
Veteran
Joined: Sep 2010
Posts: 6,576
When I posted a song a while back recorded using Sinsy, Don Gaynor had expressed hope that with this sort of technology he'd be able to sing again. While the fine folk at DynaVox haven't gotten back to him yet, I thought I might have a go at solving the problem.

You may have noticed I've been a bit absent from the board since then. blush

Here's the result: Loch Lomond (with harmony)

It's "state of the art" 1980's formant synthesis. It's not the technology I'd intended to use, but that's a rather long story. I'm still working out the bugs in the code. It's not ready for end users, but it's finally "singing."

There's a bit of post-processing on the vocals. I used the PG Vinyl plugin to reduce some of the popping on consonants, and a low-pass filter to kill some of the high-end noise. I've also run things through EZ Mix to add some reverb and compression.

Most of the phonemes are fairly acceptable, but there's an /H/ and /L/ that seem to have gone missing. I've only tweaked one of the phonemes. At 0:15 there's what sounds like a breath noise. It's actually a bug, so I lowered the volume there. (I intend to add support for breaths at some point).

The accompaniment is PJONPBA.STY (PopBalladPiano & Ac.Guitar[85RS]). I guessed at the chords from the sheet music, so there may be some clinkers in there. The harmony is just thirds above the melody, so there may be some clinkers there, too.


-- David Cuny
My virtual singer development blog

Vocal control, you say. Never heard of it. Is that some kind of ProTools thing?
User Showcase
Joined: Dec 2003
Posts: 8,987
Veteran
Offline
Veteran
Joined: Dec 2003
Posts: 8,987
I love it David! Thanki, mate!

Now you may understand the difficulty I ran into trying to overcome inertia and reluctance to change on the part of Corporations, especially from ootsiders. Very frustrating and disheartening. It's extremely difficult to keep on slugging when we see no progress. I have several dead-end projects that I am stymied on including "The iPod Project" (qv).

Last edited by Don Gaynor; 05/29/13 03:48 PM. Reason: To make a semblance of cents to Canadian readers and moderators.
User Showcase
Joined: Sep 2010
Posts: 6,576
dcuny Offline OP
Veteran
OP Offline
Veteran
Joined: Sep 2010
Posts: 6,576
Hopefully, I can shape it into something useful. I've taken a couple of days off work in an effort to get this into shape, and I've just found a major portion of the code that needs to be rewritten.

Once this is stable, I still need to set up some sort of front end for it. The current plan is to have it read MusicXML files. I've got some code I've written for a different project I should be able to re-purpose.

I also need to write some code to do dictionary lookup. I've got a nice hyphenated phonetic dictionary, so hopefully the majority of that work is already done.

But... I still need to stabilize this code.

If you're really lucky, this will give incentive to DynaVox to finish up their project, and you'll get some real voice synthesis. wink


-- David Cuny
My virtual singer development blog

Vocal control, you say. Never heard of it. Is that some kind of ProTools thing?
User Showcase
Joined: Dec 2005
Posts: 4,047
Veteran
Offline
Veteran
Joined: Dec 2005
Posts: 4,047
David,

This effort you have made into this cutting edge technology is outstanding.
I've spent thousands of hours since the 70's working on coding projects that
had never been done before. Can really appreciate the efforts you have put into
this.

Hang in there and continuing good efforts on your "projects". Note didn't say
good luck. Patience and stamina, be with you.


FrankB

Down The Street vs2015 12-03-2014
Win7, AMD QuadA8-5500,16GB,2TeraHD, Komplete 10
PG Ultra Plus 2016,Alesis 12USB, Sonar Platinum
User Showcase
Joined: Sep 2010
Posts: 6,576
dcuny Offline OP
Veteran
OP Offline
Veteran
Joined: Sep 2010
Posts: 6,576
Originally Posted By: seeker
This effort you have made into this cutting edge technology is outstanding.


Yes, cutting edge 1980's technology at it's best. wink

This is about four generations removed from current voice synthesis technology. You might recall S.A.M. (Software Automatic Mouth), which was the basis of MacInTalk. If you play with the demo on that page (decompiled from assembly into C, and then converted into JavaScript!), you can hear the familial resemblance.

Interestingly, the company that put out S.A.M. is still in business as SoftVoice, Inc., and it was their demo of Twinkle, Twinkle Little Star that convinced me that while formant synthesis might not create realistic results, it might be "good enough" for my purposes. Since they've been doing this for the last 30 years, I think their example is probably as good as this technology gets.


-- David Cuny
My virtual singer development blog

Vocal control, you say. Never heard of it. Is that some kind of ProTools thing?
User Showcase
Joined: Jun 2012
Posts: 2,888
Veteran
Offline
Veteran
Joined: Jun 2012
Posts: 2,888
Hi David,

a milestone in your efforts. Following your blog I know
what a lot of work this was and still is.

Guenter

User Showcase
Joined: Sep 2010
Posts: 6,576
dcuny Offline OP
Veteran
OP Offline
Veteran
Joined: Sep 2010
Posts: 6,576
The next step it to make this usable.

I'd like to write a UI that displays the music on a staff, and integrates with the CMU phonetic dictionary. Nothing terribly complex - just good enough to "get the job done." It'll output a .wav file, which BiaB can load.

The rest is up to Don. whistle


-- David Cuny
My virtual singer development blog

Vocal control, you say. Never heard of it. Is that some kind of ProTools thing?
User Showcase
Joined: Aug 2012
Posts: 11,636
Veteran
Offline
Veteran
Joined: Aug 2012
Posts: 11,636
David,

I commend you on the effort that this must require. Standing ovation for that, alone. Bravo.

floyd

User Showcase
Joined: May 2008
Posts: 5,063
Veteran
Offline
Veteran
Joined: May 2008
Posts: 5,063
Kudos David. What a thoughtful thing to do - not to mention all the time and effort involved. I'm sure it will bring Don and others lots of fun in making their music when you're done.

User Showcase
Joined: Sep 2010
Posts: 6,576
dcuny Offline OP
Veteran
OP Offline
Veteran
Joined: Sep 2010
Posts: 6,576
Thanks! smile


-- David Cuny
My virtual singer development blog

Vocal control, you say. Never heard of it. Is that some kind of ProTools thing?
User Showcase
Joined: Oct 2008
Posts: 18,838
Veteran
Offline
Veteran
Joined: Oct 2008
Posts: 18,838
Hi David,

This is amazing! I can't even begin to comprehend the amount of effort that must have been involved. You realize that you'll have to change your signature's saying now! "Loch Lomond" is all about 'voice control' at it's absolute cleverest smile

Regards,
Noel


MY SONGS...
Audiophile BIAB 2024
User Showcase
Joined: Oct 2008
Posts: 8,074
Veteran
Offline
Veteran
Joined: Oct 2008
Posts: 8,074
David,

I've got to be honest and say that my (very) first impression of this was that the timing of your presentation couldn't have been worse, considering the fact that Guenter has recently posted some excellent projects using a more advance synthetic voice. Comparisons are inevitable, and you are at a disadvantage being a single developer working for free without any kind of subsidy.

But the more I listen and compare your synth voice to the one Guenter is using, the more impressed I am with yours.

The biggest difference to my ear is that yours has a pronounced artifact where the syllables are joined together, whereas the other engine has figured out a way to blend the syllables more smoothly... which, in a song environment is definitely more musical.

Having dealt with artifacts when blending snippets in a DAW, I have to wonder if it would help to fade each of your separate syllables on both ends so the sharp edge that causes the artifact is less pronounced when they are joined together.

In the final analysis, all comparisons aside, what you have accomplished here is PHENOMEnal! (pun intended)

I wish you much luck with further development and I look forward to hearing more examples as time goes on.

Last edited by Pat Marr; 06/01/13 12:24 PM.
User Showcase
Joined: Dec 2011
Posts: 13,978
Veteran
Offline
Veteran
Joined: Dec 2011
Posts: 13,978
The patience of Job must have gone into that. I can't imagine the code (and I used to do a little programming). Gonna be fascinating to see how it plays out. Thanks.

User Showcase
Joined: Sep 2010
Posts: 6,576
dcuny Offline OP
Veteran
OP Offline
Veteran
Joined: Sep 2010
Posts: 6,576
Hi, Pat.

Thanks for the your response. Before getting into details, I should clarify: this project is just re-implementing what others have already done, and much better than me. From text to speech: The MITalk system (Allen) was a great resource for me. The technology is essentially abandoned for other, better approaches. I'll give an explanation why I was unable to go down that route.

Also, my immediate goal is to get Don something usable. If DynaVox finally gets him a better synthesis program, then, Hurrah!. No need for this program.


I'm aware of the Vocaloid software - I've got the Avanna as well, because it's probably got the best English accent of all the current Vocaloids.

I've also spent a lot of time looking at UTAU, a free synthetic singer written along the lines of Vocaloid.

In fact, my initial approach was exactly what you suggested: record various phonemes (using Vowel/Consonant/Vowel patterns), cross-fade them together, and use pitch shifting.

I've actually written a number of tools to do this. The stumbling block was the pitch shifting. The pitch shifting needs to shift some frequencies (the glottal pulse) and keep others fixed (the formants) or you get the "Mickey Mouse" effect.

BiaB uses the astonishingly good elastiq algorithm. I couldn't find any free libraries that gave decent results - even the RubberBand library, which has formant preservation, didn't do an acceptable job.

I tried FFT-based pitch shifting, but didn't have much luck.

I got better results with PSOLA (Pitch Synchronous Overlap and Add), but there were significant artifacts: Here's an example.


The examples I'd heard of formant-based synthesis convinced me that while it lacked realism, it was capable of creating intelligible and musical synthesis. I think you'll agree that, with some tuning, this synthesizer may not create realistic voices, but they can be understandable.


And to be honest, I've been focused on just getting the code to work. I've spent very little time on fine-tuning the phonemes. This is alpha-software, and there's lots of room for improvement.

That said, in Text-to-Speech Synthesis, Paul Taylor argues that formant-based synthesis is intrinsically un-natural because it can't capture the details of real speech, so I don't hold high hopes for it.

I've considered mixing pre-recorded audio with synthesized sounds like eSpeak, but that raises plenty of issues. And there's still the issue of handling sounds like /B/, /D/ and /G/, which are voiced and consonants. So for the moment, I'm sticking with "pure" synthesis.

I hope that somewhat explains that approach I've taken. Despite the many flaws, I figured it was time to move ahead with the project. For the moment, I'll be focusing on creating a UI.


-- David Cuny
My virtual singer development blog

Vocal control, you say. Never heard of it. Is that some kind of ProTools thing?
User Showcase
Joined: Mar 2013
Posts: 4,476
Veteran
Offline
Veteran
Joined: Mar 2013
Posts: 4,476
Hi, David !

I am no expert on these matters
which maybe makes me better judging
the end result from a listeners point of view ?
I am referring to the tune. Maybe I am
partial as this tune has a special meaning
for me. You see, once I sang this song to my
dear wife Beni at the shore of this Loch and little
did I know then, that the words held a prophecy
as "me and my true love would never be allowed
to meet again on the bonnie, bonnie banks of Loch Lomond" !
She died of cancer on October the 26 th 2011 !

I think you have done a marvellous job David !
Keep on the good work !

Cheers
Dani

Previous Thread
Next Thread
Go To

Link Copied to Clipboard
ChatPG

Ask sales and support questions about Band-in-a-Box using natural language.

ChatPG's knowledge base includes the full Band-in-a-Box User Manual and sales information from the website.

PG Music News
Convenient Ways to Listen to Band-in-a-Box® Songs Created by Program Users!

The User Showcase Forum is an excellent place to share your Band-in-a-Box® songs and listen to songs other program users are creating!

There are other places you can listen to these songs too! Visit our User Showcase page to sort by genre, artist (forum name), song title, and date - each listing will direct you to the forum post for that song.

If you'd rather listen to these songs in one place, head to our Band-in-a-Box® Radio, where you'll have the option to select the genre playlist for your listening pleasure. This page has SoundCloud built in, so it won't redirect you. We've also added the link to the Artists SoundCloud page here, and a link to their forum post.

We hope you find some inspiration from this amazing collection of User Showcase Songs!

Congratulations to the 2023 User Showcase Award Winners!

We've just announced the 2023 User Showcase Award Winners!

There are 45 winners, each receiving a Band-in-a-Box 2024 UltraPAK! Read the official announcement to see if you've won.

Our User Showcase Forum receives more than 50 posts per day, with people sharing their Band-in-a-Box songs and providing feedback for other songs posted.

Thank you to everyone who has contributed!

Video: Volume Automation in Band-in-a-Box® 2024 for Windows®

We've created a video to help you learn more about the Volume Automation options in Band-in-a-Box® 2024 for Windows.

Band-in-a-Box® 2024: Volume Automation

www.pgmusic.com/manuals/bbw2024full/chapter11.htm#volume-automation

Video: Audio Input Monitoring with Band-in-a-Box® 2024 for Windows®

We've created this short video to explain Audio Input Monitoring within Band-in-a-Box® 2024, and included some tips & troubleshooting details too!

Band-in-a-Box® 2024: Audio Input Monitoring

3:17: Tips
5:10: Troubleshooting

www.pgmusic.com/manuals/bbw2024full/chapter11.htm#audio-input-monitoring

Video: Enhanced Melodists in Band-in-a-Box® 2024 for Windows®!

We've enhanced the Melodists feature included in Band-in-a-Box® 2024 for Windows!

Access the Melodist feature by pressing F7 in the program to open the new MultiPicker Library and locate the [Melodist] tab.

You can now generate a melody on any track in the program - very handy! Plus, you select how much of the melody you want generated - specify a range, or apply it to the whole track.

See the Melodist in action with our video, Band-in-a-Box® 2024: The Melodist Window.

Learn even more about the enhancements to the Melodist feature in Band-in-a-Box® 2024 for Windows at www.pgmusic.com/manuals/bbw2024upgrade/chapter3.htm#enhanced-melodist

Band-in-a-Box® 2024 DAW Plugin Version 6: New Features Specifically for Reaper®

New with the DAW Plugin Version 6.0, released with Band-in-a-Box® 2024 for Windows: the Reaper® Panel!

This new panel offers built-in specific support for the Reaper® DAW API allowing direct transfer of Band-in-a-Box® files to/from Reaper® tracks!

When you run the Plugin from Reaper®, there is a panel to set the following options:
-BB Track(s) to send: This allows you to select the Plugin tracks that will be sent Reaper.
-Destination Reaper Track: This lets you select the destination Reaper track to receive media content from the Plugin.
-At Bar: You can select a bar in Reaper where the Plugin tracks should be placed.
-Start Below Selected Track: This allows you to place the Plugin tracks below the destination Reaper track.
-Overwrite Reaper Track: You can overwrite previous content on the destination Reaper track.
-Move to Project Folder: With this option, you can move the Plugin tracks to the Reaper project folder.
-Send Reaper Instructions Enable this option to send the Reaper Instructions instead of rendering audio tracks, which is faster.
-Render Audio & Instructions: Enable this option to generate audio files and the Reaper instructions.
-Send Tracks After Generating: This allows the Plugin to automatically send tracks to Reaper after generating.
-Send Audio for MIDI Track: Enable this option to send rendered audio for MIDI tracks.
-Send RealCharts with Audio: If this option is enabled, Enable this option to send RealCharts with audio.

Check out this video highlighting the new Reaper®-specific features: Band-in-a-Box® DAW Plugin Version 6: New Features Specifically for Reaper®

Band-in-a-Box® 2024 DAW Plugin Version 6: New Features Video

The new Band-in-a-Box VST DAW Plugin Verion 6 adds over 20 new features!

Watch the new features video to learn more: Video: Band-in-a-Box® 2024 - DAW Plugin Version 6 New Features

We also list these new features at www.pgmusic.com/bbwin.plugin.htm.

Forum Statistics
Forums66
Topics81,395
Posts732,501
Members38,441
Most Online2,537
Jan 19th, 2020
Newest Members
zagrajbarke, Ernest J, Izzy, BenChaz, Csofi
38,440 Registered Users
Top Posters(30 Days)
MarioD 195
Al-David 124
DC Ron 116
dcuny 87
rsdean 82
Today's Birthdays
CeeDee, SethMould
Powered by UBB.threads™ PHP Forum Software 7.7.5