PG Music Home Forums General Discussion Off-Topic synSinger Update

For those following the progress of this project, I'd hoped that I'd have a final version of my synthetic singer program completed back in March. However, it had a bad case of the mumbles, and I headed back to the drawing board to rewrite it. It's been eating my spare time since then.

I spent far too much time trying to get synthesized plosives and fricatives working. Then I listened to earlier versions of the program, and realized how much better they sounded using samples instead. So all that work got tossed out, and I ended up writing the code from scratch... again.

Just the other day, I got the code to where it is again reading MusicXML files and generating .wav files. There's still a lot of work to do, but the end is (hopefully) in sight.

I've been using Twinkle, Twinkle, Little Star" as my demo song, and here's the most current version, warts and all. For example, I haven't yet figured out why it can't say "world" correctly.

Twinkle, Twinkle, Little Star #15

As a change of pace, I decided to do my own version of Daisy Bell, one of the first examples of computer generated singing. It's still missing some phonemes, so I've cheated in spots. For example, the /G/ is actually /D SH/. The lyrics were automatically converted to phonemes, but I did some replacement by hand because the allophonic replacement code isn't working yet.

I also added some compression and reverb because everything sounds better with reverb. Just to let you know, it doesn't sound quite this good out of the box... But it sounds exactly as bad .

Anyway, here's synSinger singing "Daisy", as well as the 1961 version by Max Mathews, John Kelly, and Carol Lochbaum, which I found on Perry Cook's website. To make comparison easier, they're in the same key and tempo:

Daisy Bell (IBM 7094)
Daisy (synSinger)



I've always been curious about what software was used to generate the computer performance 55 years ago. It turns out that the data was hand-coded into the computer. That explains why I was never able to find any references to the text-to-singing program... It never existed!

As always, comments (positive or negative) are always appreciated.

It needs more Cowbell.

I'll confess - that's one thing I hadn't considered.

But that reminds me that I do need to add another feature - using another sound source instead of a glottal pitch. Remember the cellos voice on the MacInTalk?

I've never heard that original recording, absolutely insane that that was 1961... I also can't believe that you coded this yourself, that's crazy!! Amazing work. Your version is sounding very good. Keep us updated on how you progress with "world"!

Wow. Amazing you coded that yourself. I can't even begin grasping how you did that. Like how do you pick the right sample for a specific part of a word, or did you sample sounds and complete words? Individual letters can sound different when used in combination with other letters...

Can you please show some insight in that?

There are a couple different ways that vocal synthesis can be approached. The method that I'm using is called "formant synthesis", and is one of the oldest techniques that's been used for computer synthesis.

In English, there are approximately 40 distinct "sounds" that make up the language, are referred to as "phonemes".

There are different phonetic systems, but one of the simplest for American English is the "Arpabet", which uses plain text characters to represent phonemes. For example, the word "dictionary" would be written:

/D IH K SH AH N EH R IY/

You can play around with the the online CMU Pronouncing Dictionary to see how this works.

I use the CMU Dictionary to convert English into phonemes. If a word isn't found in the dictionary, I fall back to a public domain program called "Reciter" which guesses how to pronounce the word.

Phonemes are turned into sound by simulating the human vocal tract electronically. Before explaining that, let give give a (very simplified) explanation of how we create vocal sounds.

As air passes through the glottal folds, the folds vibrate and create sound. By controlling the tension (which in turn controls the length of the folds' opening), we can raise and lower the pitch we create. This pitch is called the fundamental frequency (F0), which we hear at the pitch of the voice.

This pitched glottal pulse (which resembled a kazoo sound) passes through our mouth. We use our tongue to create one or more resonating chambers that reinforce specific frequencies in the glottal pitch. These reinforced frequencies are called "resonances", and are what distinguishes one phonemes from another.

For example, (borrowing from the SoftVoice website), here are a number of vowel sounds, and the frequencies of resonance for the "average" male speaker in Hz:

/IY/ (beet): 270, 2300, 3000
/IH/ (bit): 400, 2000, 2550
/EH/ (bet): 530, 1850, 2500
/AE/ (bat): 660, 1700, 2400
/AH/ (but): 640, 1200, 2400
/UW/ (boot): 300, 870, 2250

In the phoneme /IY/ (as in beet), the first formant (F1) is at 270Hz, the second (F2) is at 2300Hz, and the third (F3) is at 3000Hz. Again, these formants don't alter the fundamental pitch, and remain fixed no matter what pitch you're singing.

Some phonemes are obviously more complex than that. For example, the phonemes /IY/ and /UW/ are diphthongs, and consist of two distinct targets. But I'm digressing...

To do this electronically, I generate a waveform that approximates a glottal pulse at the desired pitch, and pass it through a series of bandpass filters - one for each formant frequeny - to resonate at the desired frequencies. The output is a rough approximation of the sound.

Changing the pitch of the glottal pulse changes the pitch that's being sung. Changing the resonating filters to new values changes the phoneme that's being sung.

Some sounds (like the frication in the /F/ or the plosive in the /T/) are created by means different than described above. I used to synthesize them, but I now use digital samples because they give better results.

Very little work that I've done are my own ideas. I've borrowed heavily from the published work of Dennis Klatt, who wrote one of the first text-to-speech computer programs.

If you're curious, I'd highly recommend downloading this Formant Synthesis Demo. Click and drag in the area marked F1/F2 (formants one and two) and you'll get a good idea how this works.

Did that clear up some of the mystery?

Thanks for you answer, David. It did bring me more understanding of the subject of vocal synthesis. I will visit the sites you mentioned. Please keep us informed about your progress with this project. I find it very interesting.

Another update - the main change has been adding vibrato back into the code.

Twinkle, Twinkle, Little Star #16 (synSinger)

The various parameters of vibrato - depth, speed, minimum note length and delay before start - can be specified.

I've also worked on clearing up a number of phonemes, including the /ERL/ in "world", although the main problem with that word is the trailing /D/.

This version of synSinger is written in Lua, and isn't particularly fast - it renders audio about about half the speed of the song. I still haven't been able to spend time to figure out how to create a stand-alone executable. I need to spend some time with the squish documentation.

There are still instances where it will "squelch" when parameters change to quickly, and some of the phonemes still need more attention. But for the most part, the output seems to be fairly acceptable, although not always intelligible. Truth be told, it's only incrementally better than prior versions.

I've also created a "female" voice for synSinger by mapping phonemes formants from average male phoneme space to average female phoneme space. It also modifies some other parameters, such as raising the pitch up an octave (so "she" doesn't sound like a chain smoker), adding more breath noise, altering the glottal pulse, and modifying the formant bandwidths based on a shorter larynx. But it still sounds a bit cheesy, because female voices aren't something formant synthesis does that well:

Twinkle, Twinkle, Little Star (synSinger female voice)

As always, feedback is welcome.