Converting MusicXML to sound and humming vocals

[Eric Lafortune]

Inspired by @SteveB's MuseScore/4a, I've created two new tools to play MusicXML sheet music, from assembly code this time.

 

ConvertMusicXmlToSnd takes a MusicXML file and produces an SND file with music for our SN76496 sound chip. Features:

• Selectable instruments: piano, flute, drum, cymbal, etc.
• The instrumemts are mainly defined by their own Attack/Decay/Sustain/Release envelopes.
• Any number of input parts, staffs, voices, and chords. The tool constantly optimizes the use of the 3 available sound generators and single noise generator.

ConvertMusicXmlToLpc takes a MusicXML file and produces an LPC file with speech commands humming to the tune (more precisely "singing with non-lexical vocables") on our TMS5200 speech synthesis chip. Features:

• Selectable hum samples, already including common ones like hm, la, oh, ah, etc.
• All samples have their own speech coefficients and Attack/Decay/Sustain/Release envelopes.
• Only one voice at a time, because of the speech synthesis chip.

Both tools recognize and support the following features in MusicXML files:

• Measures and beats, with proper emphasis.
• Beats per minute, constant or variable.
• Chords.
• Ties between notes.
• Articulations: accent, stress, etc.

A sample: The Logical Song by Supertramp (shoutout to @pixelpedant who pioneered playing the music and actual vocals of this song earlier):

 

 

The quality of the vocals is debatable... This is the instrumental version:

 

 

Another sample: Seven Nation Army by The White Stripes:

 

Another sample: O Fortuna from Carmina Burana, by Carl Orff, with 12 parts, one or two staffs each, up to 4 chord notes each:

 

 

Of course, the idea is to play the music and singing from assembly code on a TI. Most conveniently, you can integrate them in a TMS video file for my compact video player and package them in a cartridge ROM. This is an example workflow:

export CLASSPATH=/path/to/videotools.jar

java ConvertMusicXmlToSnd \
  -startmeasure 1 \
  -endmeasure 87 \
  -part 'Piano'                                          -transpose 0 -attenuate 0 -instrument piano \
  -part 'Electric Piano' -staff 1 -voice 1 -chordnotes 3 -transpose 0 -attenuate 4 -instrument piano \
  -part 'Electric Piano' -staff 2 -voice 5 -chordnotes 2 -transpose 1 -attenuate 1 -instrument piano \
  TheLogicalSong.xml \
  TheLogicalSong.snd

java ConvertMusicXmlToLpc \
  -startmeasure 1 \
  -endmeasure 87 \
  -part 'Alto Saxophone' \
  -transpose -2 \
  -attenuate 2 \
  -hum ta \
  -addstopframe \
  TheLogicalSong.xml \
  TheLogicalSong.lpc

java ComposeVideo \
  TheLogicalSong.png \
  10:TheLogicalSong.lpc \
  12:TheLogicalSong.snd \
  TheLogicalSong.tms

java PackageVideoInCartridge \
  -title 'the logical song' \
  -titlewithoutspeech 'instrumental' \
  TheLogicalSong.tms \
  TheLogicalSong.rpk

mame ti99_4a \
  -ioport peb \
  -ioport:peb:slot3 speech \
  -cart1 TheLogicalSong.rpk

The above music as runnable RPK cartridge files for Mame:

TheLogicalSong.rpk SevenNationArmy.rpk OFortuna.rpk

Note that RPK files are zip files; they contain the raw multi-bank ROM files, which should work in RAM cartridges and other emulators.

 

Give them a try! You get background screens (converted with Magellan) and the option to play the instrumentals without vocals.

 

ConvertMusicXmlToSnd and ConvertMusicXmlToLpc are part of my Video Tools for the TI-99/4A. You can find the tools, documentation, and source code on Github. Notably, you can find compiled binaries (for Java 14 or higher) in the release section.

 

Please try out the tools and comment! This has been a fun and educational side-project for my game Stealth Runner.
 

[SteveB]

Incredible! I'm tongue-tied with my fingers on the keyboard ... speechless. 

 

I never thought something like this would come out of the venerable TI.

[+mizapf]

I'd love to see a way to run that on the Geneve (so I can run it on my real hardware).

 

Fantastic job, indeed!

[+9640News]

I agree as well with Michael.  Would love to see something for the Geneve as well.

 

[+OLD CS1]

5 hours ago, Eric Lafortune said:

Another sample: O Fortuna from Carmina Burana, by Carl Orff, with 12 parts, one or two staffs each, up to 4 chord notes each:

Most impressive of the batch.

[Eric Lafortune]

Thanks, everyone!

22 hours ago, mizapf said:

I'd love to see a way to run that on the Geneve (so I can run it on my real hardware).

Looking at the specifications of the Geneve, modifying the tiny player code (assembly for xas99) should be easy. The ROM contains 8K with 438 bytes of player code, and subsequent 8K blocks of video data, for the cartridge memory banks. The player code doesn't call any external subroutines.

• It starts with a cartridge ROM header → not sure what Geneve programs look like.
• It sets up registers with the SCHWT, VDPWR,... addresses → looks like the Geneve has equivalent addresses.
• It copies the main player loop to scratchpad RAM → optional; fast enough without animations.
• The player loop streams bytes from subsequent cartridge memory banks → bank-switching is optional; the music videos fit in ~40K.
• It checks low-level CRU bits to detect Vsync and the QUIT key → similar?

I won't try this myself right now, but I'll be happy to help anyone who wants to give it a go.

[+mizapf]

One thing is to convert the banks of the module into memory pages. In both cases this is just a way to map 8 K segments into the address space, so this should not be too difficult.

 

As for low-level CRU, there should be some differences. We cannot poll the QUIT key in this way, since the Geneve has a normal PC keyboard (or, well, a PC-XT keyboard) and no QUIT key. I once wrote a guide to write a Geneve keyboard driver (https://www.ninerpedia.org/wiki/Geneve_keyboard_control). I guess the key scan is more a could-have priority.

 

The video processor is a v9938; the MSB of status register 0 is VSYNC (just like the TMS9918/28).

 

The Geneve has no scratch pad RAM area, but an on-chip RAM (serving a similar purpose at F000). But any kind of memory access is much faster than on the TI, so no need to use it.

[+OLD CS1]

Thank you.  I am definitely interested in your SND tools.  I have an A/Rexx script I use to manipulate sound list binaries, which I believe is your SND format.   I have had some tribulations with VGM conversions, and with optimizing ISR sound lists to remove redundant commands (which is more related to my pseudo-array handling in Rexx.)

[Asmusr]

On 11/12/2024 at 9:49 PM, Eric Lafortune said:

The tool constantly optimizes the use of the 3 available sound generators and single noise generator.

Is there any support for using periodic noise for bass notes? 

[Eric Lafortune]

4 hours ago, Asmusr said:

Is there any support for using periodic noise for bass notes? 

Yes, there is. The instruments are currently hardcoded. Each instrument has:

• Attenuation envelopes for Attack/Decay, optionally Sustain, and Release.
• A variable or fixed divider for a tone generator or for the noise generator.

For example:

• Piano: variable tones.
• Cowbell: a fixed tone.
• Drum: fixed periodic noise.
• Snare drum: fixed white noise.
• Cymbal: fixed high-pitched white noise.
• Saxophone: variable periodic noise -- sounded suitably grating to me. 🙂

I can create a more flexible file format for instruments, but there's only so much you can do with attenuation envelopes. Suggestions are welcome.

 

On the other hand, humming voices are defined by standard LPC files (binary or text), with endless possibilities. I've manually derived the included ones from the built-in vocabulary, but the speech synthesizer remains difficult to tame. "tee" sounds more crisp than "ta", for example. Any ideas are welcome.

 

[Eric Lafortune]

8 hours ago, OLD CS1 said:

I have an A/Rexx script I use to manipulate sound list binaries, which I believe is your SND format.   I have had some tribulations with VGM conversions, and with optimizing ISR sound lists to remove redundant commands (which is more related to my pseudo-array handling in Rexx.)

My SND format is essentially binary chunks to be played at 60Hz. Each chunk is a length byte followed by SN76496 sound commands. My conversion from VGM looks something like this:

java ConvertVgmToSnd 678 music.vgm music.snd

java SimplifySndFile -addsilencecommands music.snd music_simplified.snd

I've created it for the Bad Apple demo. The conversion is very basic: it only extracts SN76496 commands, assuming they are present. The simplification made quite a difference in this case.

[+OLD CS1]

4 hours ago, Eric Lafortune said:

My SND format is essentially binary chunks to be played at 60Hz. Each chunk is a length byte followed by SN76496 sound commands.

It sounds similar: the ISR "sound list" format, described in the E/A manual, is X Y... Z, X=unsigned byte indicating the length of the row or line, Y is the data byte or bytes to send to the sound chip, Z is the unsigned byte duration of the row or line in 1/60th (or 1/50th) of a second before moving to the next.  The end of the "song" is designated by Z=0.

 

Whereas I have seen some VGMs take advantage of multiple second bytes of a sound channel command being sent to produce a frequency sweep, I have not yet seen this in a sound list.

 

4 hours ago, Eric Lafortune said:

The conversion is very basic: it only extracts SN76496 commands, assuming they are present. The simplification made quite a difference in this case.

Same.  After checking and jumping past the header, I just look for 0x50 followed by a value, and add this to a row.  Then I manage the duration when I hit one of the durations bytes 0x61, 0x62, or 0x63.  The row is "closed" on 0x66.  I also check to see that the last byte at EOF is 0x66, otherwise report that the file may be incomplete or damaged.  In a lot of cases of captures from game systems, I run into VGMs which send repetitive data to the PSG, which results in sound lists with repetitive lines or channel commands.

 

vgm2isr.rexx (multi-PSG not yet implemented)

[SteveB]

rexx?  Where did I store my MVS TSO /390 machine... ? 

[+OLD CS1]

1 hour ago, SteveB said:

rexx?  Where did I store my MVS TSO /390 machine... ? 

I am from the Amiga world.  ARexx (Amiga REXX) was my first real scripting language.

[+mizapf]

Is that similar to the OS/2 REXX as well? I seem to remember I could write some simple REXX scripts back then.

[JasonACT]

@Eric Lafortune Really nice work, I'm going to give the WAV to LPC a go for a little project I'm working on, so thanks for that too.

 

I do still recommend you don't "write to ROM @ >0000"...

 

On 12/2/2023 at 9:25 PM, JasonACT said:

I've made one more change to the player.asm code here, for the no-speech menu item, I load R11 with >4000 instead of clearing it...

 

https://www.unige.ch/medecine/nouspikel/ti99/eeproms.htm

 

I.E. Writing to console ROMs causes them to output data on the 16 bit data bus at the same time the TMS9900 is also outputting data - possibly reducing the life of the console.  >4000 should be fine, as no DSR ROM will be active.  He goes on to say this, so it's not an uncommon mistake to make...

 

 

[JasonACT]

It works really well, but I thought the TI-99's speech was using a TMS5220 (and that's what I've implemented in my device, and I also compared it to a real speech unit which I now own too).  These tools default to a TMS5200, so I made sure to configure my build scripts to set it correctly to TMS5220, but when I play back the speech (on my device) it sounds nasally with the wrong (seems higher) pitch.  Then I re-processed all my phrases again set to TMS5200 and it sounds right.

 

What chip do we have here?  The number on a real synth doesn't really say (CD25015-NL)?

[Eric Lafortune]

5 hours ago, JasonACT said:

It works really well, but I thought the TI-99's speech was using a TMS5220

The TI-99 speech synthesizer uses the TMS5200. From Wikipedia:

• TMS5200 (AKA CD2501E, internal TI name is '0285' hence chip is sometimes labeled TMC0285): [...] designed for use [...] for the TI-99/4A speech module
• TMS5220 (AKA CD2805E?): Improved version of the TMS5200, [...] (rumor) on the very last run of TI-99/4A speech modules

It's also the one in the various emulators, e.g. Mame. It's an error in Thierry Nouspikel's otherwise excellent tech pages.

 

You can simulate any mismatches (with unchanged TMS5200 LPC humming coefficients) with the tools, e.g. starting from:
 

java ConvertMusicXmlToLpc -tms5200 .... music.xml music.lpc

java ConvertLpcToWav -tms5200 music.lpc music.wav

 

[Asmusr]

Now I've had The Logical Song stuck in my head for two days. 🙂

[HOME AUTOMATION]

6 hours ago, Eric Lafortune said:

Improved version of the TMS5200, [...] (rumor) on the very last run of TI-99/4A speech modules

I Think that I saw one demonstrated in a TI-99/4A, Youtube video, some years back.

It was using the TI's, built-in vocabulary, but spoken with the "Romeo" voice. So the ROMs must have been redone as well!

Oh, would I like to get my mitts on the LPC from one of those! As well as all other Romeo, Vocab..

 

Now ...I can't even find the video.

Edited November 15 by HOME AUTOMATION
syntax

[Eric Lafortune]

10 hours ago, JasonACT said:

I do still recommend you don't "write to ROM @ >0000"...

Checking the schematics now, the ROMs don't get the Write Enabled signal indeed, so writing to address >0000 may somehow clash. I was worried about blindly writing to address >4000, because that is even more of an unknown. It may also be technically different because of the 8-bit bus.

[+OLD CS1]

Is there a path from MIDI to MusicXML?

[JasonACT]

Many thanks for the pointer to the Wiki, it's all clear now, and I can see I'm creating an instance of the 5200 class and not the 5220 class (even though the MAME files are named 5220 - so it was Thierry's site and that, that had me confused).

 

12 hours ago, Eric Lafortune said:

I was worried about blindly writing to address >4000, because that is even more of an unknown. It may also be technically different because of the 8-bit bus.

Well, the speech ports are on the 8 bit bus, so I'd think it's technically more similar (ignoring the READY line).  But even reserving a single word in scratch RAM where you can dump the unwanted speech stream would be acceptable.  To be honest, since I originally wrote my first quoted comment, I've implemented a mode I call RAMBO >4000 which pages in SAMS memory to that area (my device keeps track of all 16 device CRU lines to hide it, when needed).  So, yeah, I've got RAM there when no DSR is enabled, and would probably prefer it didn't write there now.

 

I'm very happy with the new tools too, thanks!

[Eric Lafortune]

16 hours ago, OLD CS1 said:

Is there a path from MIDI to MusicXML?

Musescore can import MIDI files, and I see a number of other open-source projects on Github. I've never tried them though. The MIDI format looks feasible to implement next to MusicXML, but it may not be worth the effort if these tools work fine. Is there a lot of MIDI music available for which it would be useful? I'm still looking for rights-free epic music (like in farCry I) for my video game Stealth Runner...

[+OLD CS1]

1 hour ago, Eric Lafortune said:

Is there a lot of MIDI music available for which it would be useful?

Literally decades of MIDI out there.  The quality of what you can find is debatable, but I have a ton of MIDIs, plus some favorites on my phone.  But, if Musescore can import MIDIs, then it might not be worth a direct MIDI-to-SND conversion.

 

Attached are a few worth looking at, IMO.  Some have more difficult instruments which might not convert well, but sound pretty good, nonetheless.  (I could not find where I stashed my Duke Nuke'em songs.)

Benny_And_The_Jets.mid battlestar_galactica.mid Crocodile_Rock.mid desperado.mid last_starfighter.mid Level 2.mid Level 3.mid mm8opening.mid Obladioblada.mid Piggies.mid sailor_moon.mid simpsons.mid WhileMyGuitarGentlyWeeps.mid Wizards and Warriors.mid

fawlty.mid