VGM Compression Tool

[+OLD CS1]

Can it convert sampled sounds from SID files?

[Tursi]

2 hours ago, OLD CS1 said:

Can it convert sampled sounds from SID files?

No. This is a 60hz playback system, so samples aren't going to work regardless of the source.

 

[Tursi]

Now that I've said that, tonight I pushed up voice2psg, which will take in a voice sample in Wave format and output a PSG file that attempts to reproduce it. It's really limited, and doesn't tend to be very clear, but for some samples, it does work, and hey, it's 60hz sample playback.

 

It's based on Artrag's Matlab concept, but several attempts at porting the code to C++ just had me walk away - every function I looked at linked to two more massive ones, and keeping the arrays straight was driving me crazy.  I did get about halfway through it all if anyone wants that code, but likely I'm going to discard that work in favor of tuning this newer tool.

 

I don't have a good example ready, but you want a very clear voice sample, with no background sounds, loud and crisp. With all that, the result should be barely recognizable in the output.

 

TI1.wav

 

Ultimately, this tool won't often be useful, but for screams and hoots it actually does pretty decently, so you can get some sounds in there if you need them.

 

 

[+FarmerPotato]

On 8/2/2020 at 6:58 AM, Tursi said:

Now that I've said that, tonight I pushed up voice2psg, which will take in a voice sample in Wave format and output a PSG file that attempts to reproduce it. It's really limited, and doesn't tend to be very clear, but for some samples, it does work, and hey, it's 60hz sample playback.

 

It's based on Artrag's Matlab concept, but several attempts at porting the code to C++ just had me walk away - every function I looked at linked to two more massive ones, and keeping the arrays straight was driving me crazy.  I did get about halfway through it all if anyone wants that code, but likely I'm going to discard that work in favor of tuning this newer tool.

 

I don't have a good example ready, but you want a very clear voice sample, with no background sounds, loud and crisp. With all that, the result should be barely recognizable in the output.

 

TI1.wav 104.18 kB · 6 downloads

 

Ultimately, this tool won't often be useful, but for screams and hoots it actually does pretty decently, so you can get some sounds in there if you need them.

 

 

Alright, YEAH!

I've heard artrag's results and liked them. Yours is really intelligible once you know what it's supposed to be saying. The intonation and slope at the ends of words comes through, its a big hint to the brain.

 

And, this style would be pretty fun as robot or boss voices in a game. 

 

Again, WOW!

 

[Tursi]

Finally back on this task, and tonight I pushed up something I've wanted forever, and one of the reasons I started this update - md2psg. MD is MegaDrive, aka the Sega Genesis. MegaDrive VGMs can now be read and output in a format loosely compatible with the PSGs used here.

 

It's far from perfect. I think the envelopes must be working, but honestly I am having a lot of trouble understanding how the MD generates the final volume on a channel at any given time, since emulators work at the sample level. What I have seems good enough as a starting point, anyway.

 

Of course, with 6 FM channels PLUS a TI PSG in there, the task of getting 10 channels playing on a TI is left as an exercise for the reader.  But that's the point of this whole toolchain - to provide the tools needed to tweak and manipulate this sort of music into something usable.

 

I've posted a video here with a number of conversions. Since the video was recorded I fixed the desynchronization issue that broke Scrap Brain Zone, and also rebalanced the volumes by default a bit, so I think the starting point is pretty decent. It also converts the tune in one pass for both chips - in the video you'll see me running the two converters separately.

 

https://open.lbry.com/@tursilion:1/VgmComp2---Sega-Genesis-to-SN-Example:f

 

Of course, the files are are pushed up to Github with binaries and docs in the dist folder. It's nearly there, just a few tools left to build (and another player if I decide I want to do it... waiting till I get that far before I delete it  )

 

[Tursi]

(Now with much improved level balance between the FM channels as well as between the FM and SN.)

 

[Tursi]

I've now finished this as far as the features I intended to code, and am moving on.

 

https://github.com/tursilion/vgmcomp2/blob/master/readme.md

 

[Asmusr]

I'm considering using the new VGM tools for my Zaxxon demo, but I have a few questions:

• What are the most important advantages or disadvantages compared to the old tools (I'm thinking of speed, space, sound quality, features)? I could probably find out be reading the whole thread again, but if I can get it from the lion's mouth...
• Which assembler do I need to assemble the code? It has macros like .size and .section and some comments incompatible with xas99. Do I have to use refs and defs or can I just assemble a bunch of files together?
• Which of the many tools do I need to convert a VGM for our PSG into playable format?

Thanks,

Rasmus

[Tursi]

31 minutes ago, Asmusr said:

• What are the most important advantages or disadvantages compared to the old tools (I'm thinking of speed, space, sound quality, features)? I could probably find out be reading the whole thread again, but if I can get it from the lion's mouth...

Less memory used, less CPU used, better compression, and much richer toolset.

 

Both compressors are non-lossy (most of the time), so sound quality isn't really a feature.

 

Downsides, more complex process, but this was a deliberate choice, as I wanted to be able to convert many more sources and edit the channels. My workflow is basically to determine the processing I want to use and put it in the makefile, so it's still one typed command anyway.

31 minutes ago, Asmusr said:

• Which assembler do I need to assemble the code? It has macros like .size and .section and some comments incompatible with xas99.

The code was written for GCC, and the assembly is hand-tuned from GCC's generated code. So I built with gas. The size macros can be deleted, and you can remove the section macros if you pay attention to the comments that explain why they are there (mostly to make moving the various bits of data in RAM easier).

 

I did the same with the previous player, but I released a version with the dseg and pseg commented out.

31 minutes ago, Asmusr said:

• Do I have to use refs and defs or can I just assemble a bunch of files together?

I'm not sure what you are asking here... like most TI code refs and defs are used to reference symbols in other files. Those two concepts go together...?

31 minutes ago, Asmusr said:

• Which of the many tools do I need to convert a VGM for our PSG into playable format?

The documentation actually summarizes the tools by what they do, just scroll down to page 8.

 

You use three tools for that process, since you won't need to do any editing.

 

My early code called our chip the PSG, so you have to live with that naming.  Thus, you would start with vgm_psg2psg. This will output four files, one for each channel. The intent of this is that each channel can be edited with the edit tools, or even by hand since they are all text files with one row per frame. There are 9 input converters, six for different VGM chips and three for other formats. Depending on the chip, the output may not be directly playable by the TI chip, but the system has several output devices as well.

 

Next you run prepare4sn and pass it the four channels. It will verify that all the data is valid for the SN (and it should be due to the source), and pack into a single file. It's still a text file, but it has all four channels in one. Anything that is out of range is clipped or dropped at this point - the intent is to use the editing tools to massage the data before this step, so that you can control what happens to that data. Since you're starting with the same chip, there's not going to be any out of range data.

 

Finally, vgmcomp2 does the actual compression.

 

At any point in the process you can run testplayer to hear what the current state of the song sounds like (roughly, the timing is hit or miss, but it reports the frame rate it's actually managing  ).

 

Looks like I didn't do a video for that basic scheme.

 

[Asmusr]

2 hours ago, Tursi said:

I'm not sure what you are asking here... like most TI code refs and defs are used to reference symbols in other files. Those two concepts go together...

I don't think I have ever used refs of defs. I think it's only relevant if you use object code. So if I want to play music and sound FX in 60Hz on the TI, which files do I need to assemble?

2 hours ago, Tursi said:

The documentation actually summarizes the tools by what they do, just scroll down to page 8.

Sorry but I never got past the first section "What is it". Then it starts referring to things that implies that you have already read the documentation. ? Well maybe not as bad as that, but I have really tried a few times to understand it.

 

Thank you for the primer, I know what to do now. ?

[Tursi]

14 hours ago, Asmusr said:

 

I don't think I have ever used refs of defs. I think it's only relevant if you use object code. So if I want to play music and sound FX in 60Hz on the TI, which files do I need to assemble?

You've never used a def to provide a start address to LOAD AND RUN? That's impressive if so.  I guess with the modern tools you can get away with it. 

 

They are very simple though. When you want to make a symbol public so that any other file can see it, DEF symbol. When you want to use a symbol from another file, REF symbol. That's it. As long as every REF has a matching DEF, you're done, the files all link together. With the standard TI assembler, the linking happens during LOAD AND RUN.

 

Keep in mind it's been 7 months since I last looked at this.

 

Anyway, start in Players\libtivgm2. Link with libtivgm2 - assuming you can link with GCC libraries it's already built. If not, then you are going to need to build CPlayerCommonHandEdit.asm, CPlayerTIHandEdit.asm, CPlayerTIHandlers.asm, CPlayerTISfx.asm and CPlayerTISfxHandlers.asm. It's all broken up so that C programs only load the code and data they actually need to use.

 

CPlayerCommonHandEdit contains shared code and data used by all the playback functions. This is mostly the decompressor, and a large comment block describes the register usage (TLDR: all of them at >8300, but they are not persistent like the old player). You don't call anything in here directly.

CPlayerTIHandEdit contains the main code SongLoop that you call every frame.

CPlayerTIHandlers contains other functions you need to call - StartSong and StopSong.

CPlayerTISfx contains the main code SfxLoop that you call every frame for sound effects. Call this before SongLoop so the mutes are set right.

CPlayerTISfxHandlers contains the other functions for SFX - StartSfx and StopSfx

 

You didn't ask, but it's worth noting that the 30hz players are different from before - instead of only functioning every other call, they process 2 of the 4 channels every call in order to spread the load. Technically this /could/ cause slight aliasing of the music (ie: a chord that starts some channels 1/60th of a second before the others), but in practice you don't really hear it. I figured if one wanted the old method, one would be smart enough to just call the 60hz player at 30hz themselves with appropriate data.

 

We can probably merge all this into a single big file that xas or asm994a can handle. Each file contains its own data and they are pretty much marked by the sections. In gcc, you can just pass the desired address for a section to the linker, that's why I did that. If we merge it into one big file, then we can just throw all the data at the top as per normal and then it's easy to find. I would attempt it, but then I would not sleep at all today . I can potentially get to it late tomorrow night tonight.

 

Although it's all written in C, Samples/TISNSFX contains a sample program that shows usage of music and sfx (it's the same program as the old sample, just updated).

 

14 hours ago, Asmusr said:

Sorry but I never got past the first section "What is it". Then it starts referring to things that implies that you have already read the documentation. ? Well maybe not as bad as that, but I have really tried a few times to understand it.

That's why I told you what page to jump to

 

Given the amount of interest I generally get, I just write references for my own use rather than tutorials.

 

AAAAANNND.. if you hate all this and just fall back on the old one, nobody will mind. The performance and memory gains are often only in the single digit percentages. I was expecting much better and the only reason I personally like this one more is the conversion and editing features let me do what I always wanted to do foreign music sources.

 

[Tursi]

All right, it took a lot longer than I expected, but I have a system that auto-generates the single-file assembly versions now, so although it's not terribly pretty, it's low impact on my side and it works.

 

Also looked at the manual and noticed it was still full of development notes in the beginning areas, so removed that, cleaned up the section headers, and added an example with explanations at the end.

 

Download the zip again, and then have a look in Players/ti_raw - you'll see, like the old version, one source file for each combination that you might be able. The readme.txt in this folder contains condensed instructions - just select the scenario you want and you'll get a reference of the basic important functions.

 

In that folder is also an SfxSNSample which is the exact case you were asking about, it's a quick and dirty port of the same old test app, re-written in assembly and using COPY directives to pull in the data and player, rather than linking. (Since I used winAsm994a, the copy directives have hard coded paths - change that before trying to build). The readme.txt in this file reiterates that and comments quickly what each file is - main.a99 is the one you want to look at for "how to use". EA#5 object files are included to see it run.

 

https://github.com/tursilion/vgmcomp2/blob/master/dist/VGMComp2.zip?raw=true

 

[Tursi]

This is a copy/paste of the example section I added to the docs:

 

To convert a VGM file containing SN compatible music (such as from the Sega Master System) for SN output, with no editing or conversion, you would follow these steps:

 

vgm_psg2psg.exe ab_psg.vgz

• My example here is the file “ab_psg.vgz”, which is a VGM containing music from Afterburner on the Master System
• This will output 4 files, one for each channel. They will use the file name “ab_psg.vgz” as a prefix, and then append information about the channel, in particular “_ton00” for the first tone channel, “_ton01” and “_ton02” for the second and third tone channels, and “_noi03” to indicate the fourth channel is noise data. All files will have the extension “.60hz” to indicate they are intended to play at 60hz.

prepare4SN.exe ab_psg.vgz_ton00.60hz ab_psg.vgz_ton01.60hz ab_psg.vgz_ton02.60hz ab_psg.vgz_noi03.60hz ab_psg.psgsn

• This program reads in the processed (if any) psg files and combines them into a single text file containing all four channels. It also enforces any chip level restrictions on frequency or volume at this point - in short this output file is now SN-valid data
• I named my output file ab_psg.psgsn - I like to use “psgXX” to remember which chip the data is formatted for, but no syntax is enforced here.
• This particular file output a warning about “47 mutes mapped” - all this means is that there were channels currently muted, but they changed their frequency. To improve compression, that change was discarded until it was needed, if ever. The report will tell you if any changes were ‘lossy’, meaning the output quality is affected. In all cases you can use other tools to control how those lossy values are handled - the prepare tool will use the most heavy handed mapping available to it.
• The output file is now ready to be compressed.

vgmcomp2.exe -sn ab_psg.psgsn ab_psg.sbf

• The -sn is passed to tell the compressor it is working on an SN file. It will do a final check for valid data but will just fail out if anything is not valid.
• We then pass the input file, and specify the binary output file. I use the extension .sbf, but this is not mandated.
• You can pass multiple input files to compress them into a single sound block. You should not mix chip types in the same compressed file.
• You will get some statistics about how well it compressed, as well as the cost in bytes per second to play back.

And on my screen, it looks like this:

 

 

Spoiler

D:\work\TI\vgmcomp2\test>..\release\vgm_psg2psg.exe ab_psg.vgz
Import VGM PSG - v20201001
Reading ab_psg.vgz - 2512 bytes
Signature not detected.. trying gzip (vgz)
Decompressed to 16293 bytes
Reading version 0x101
Refresh rate 60 Hz (735 samples per tick)
Selecting 16-bit shift register.
File 1 parsed! Processed 5042 ticks (84.033333 seconds)
Adapting user-defined shift rates...0 notes tuned.

-Writing channel 1 as ab_psg.vgz_ton00.60hz...
-Writing channel 2 as ab_psg.vgz_ton01.60hz...
-Writing channel 3 as ab_psg.vgz_ton02.60hz...
-Writing channel 4 as ab_psg.vgz_noi03.60hz...
Skipping channel 5 - no data
Skipping channel 6 - no data
Skipping channel 7 - no data
Skipping channel 8 - no data
done vgm_psg2psg.

D:\work\TI\vgmcomp2\test>..\release\prepare4SN.exe ab_psg.vgz_ton00.60hz ab_psg.vgz_ton01.60hz ab_psg.vgz_ton02.60hz ab_psg.vgz_noi03.60hz ab_psg.psgsn
VGMComp2 PSG Prep Tool - v20201006

Opened tone channel 0: ab_psg.vgz_ton00.60hz
Opened tone channel 1: ab_psg.vgz_ton01.60hz
Opened tone channel 2: ab_psg.vgz_ton02.60hz
Opened noise channel 3: ab_psg.vgz_noi03.60hz
Imported 5042 rows
0 custom noises (non-lossy)
0 tones moved   (non-lossy)
47 mutes mapped  (non-lossy)
0 tones clipped (lossy)
0 noises mapped (lossy)
** DONE **

D:\work\TI\vgmcomp2\test>..\release\vgmcomp2.exe -sn ab_psg.psgsn ab_psg.sbf
VGMComp2 Compression Tool - v20200803

Got 5042 rows... Successful!

1 songs (84.030000 seconds) compressed to 2960 bytes (35.226000 bytes/second @ 60hz)

** DONE **

 

 

ab_psg.vgz

[PeteE]

I was playing around with vgmcomp2 today and made a little something.  Actually, I found SNevenTracker by accident and then found this F-Zero arrangement, and then put them all together for your listening enjoyment on the TI.  There are no graphics because I am lazy; audio only.

f0music.bin

[Tursi]

Nice work! No worries if you don't do a viz, just open the heatmap and watch that!

 

That's a neat little tracker! Found a demo on Youtube:

 

 

For those who want to compose music but don't want to write software, there is a GUI tool (for Windows) in the archive that will convert your tune into a playable EA#5 for the TI or ROM for the ColecoVision - look for QuickPlayer.exe. It lets you include text onscreen, though that's all.  

 

 

If you're on Linux and don't mind compiling your own source, there's also "quickplayercmd" which is the command line version of the same tool.

 

 

[PeteE]

I added one more song, and put all the source files together in the zip to show how to use the vgmcomp2 player with xas99.py.  The .snm files are converted to .vgm using SnevenTracker, and the .bat file converts using the vgmcomp2 tools to .sbf.  Then the Makefile builds the cartridge using xas99.py with a hacked-up version of Tursi's player code.

f0music.zip f0music8.bin

[speccery]

Interesting stuff, need to try these out, thanks @PeteE

I just yesterday added the I2S DAC driver code to the StrangeCart codebase - it still doesn't do much, just produces a saw wave or sine wave output. But it is 16-bit stereo sound at 48kHz.

[Nick99]

@Tursi What windows version and drivers are required to get this to work?

I have tested it on Win Vista, Win 7, Win 8 and Win 10, all I get is the error message 0xc000007b, and I sat down and googled what to do to fix it, but no, still the same error.

I get the same error using TIvidconverter.

 

 

[Tursi]

15 hours ago, Nick99 said:

@Tursi What windows version and drivers are required to get this to work?

I have tested it on Win Vista, Win 7, Win 8 and Win 10, all I get is the error message 0xc000007b, and I sat down and googled what to do to fix it, but no, still the same error.

I get the same error using TIvidconverter.

All Windows versions XP and up, no drivers (except for testplayer which needs DirectX - can't remember if I linked against 8 or latest).

 

I'm kind of guessing cause you didn't give me the exact error message, or tell me what you found when you Googled... but...

 

That error number appears to be "The Application was Unable to Start Correctly", which is more useful to me. If that is indeed the error message you had associated with the number, that just means you are missing the Visual Studio Runtime code. I'm building with Visual Studio 2019, so installing this runtime should make it all work for you. Classic99 and anything else I've built in the last couple of years will require it as well.

 

https://support.microsoft.com/en-us/topic/the-latest-supported-visual-c-downloads-2647da03-1eea-4433-9aff-95f26a218cc0

 

Looking on the page, looks like they rolled 2015, 2017 and 2019 into the same redistributable - that's new and that's nice. I don't remember when I published TIvidconverter, but I expect it's under the 2015 runtime (a quick check of the dependencies supports that). My code is all x86, so I believe you need that regardless of your CPU (it will tell you if I'm wrong). And of course I'm being nitpicky, but these are libraries, not drivers .

 

This isn't unique to my software, of course, I just don't ship installers so I don't ship those parts.  These redistributables get automatically installed with many other packages.

 

[Nick99]

Thanks @Tursi, yes libraries are correct, and the error is correct that you mention. When I googled it they mentioned and linked to Visual C++ runtime package and a lot of other things, not anything helped solve the problem on my win 7 computer.

But now I will try install Visual C runtime libraries and Direct X and see if I have any success.

It all came back to me why I switched to iMac, I didn't have time to find everything that is needed to get programs to work. Of course, I've seen issues with OSX thath appears after every upgrade.

 

thanks, I give it a try and hope I can try to convert my own music to share.

Edited August 16, 2021 by Nick99

[Tursi]

Well, you're free to compile the tools for Mac. Except for testplayer, I made sure there was nothing Windows specific in any of them.

 

[Tursi]

The Smurf compilation kind of inspired me, and I've been working on the Quickplayer. I wanted to add two things - first was multiple visualizations, so I've been working on porting the old ones over to it. I just finished the ball animation one, and updated it a bit while I was at it (it works a lot better now). Once I get them all in there I'll publish a new release.

 

The other was the ability to chain, so compilation carts can be a bit more flexible. That's in there though not tested yet (I probably should test it before I get too far into putting it in all the different playbacks, but yeah).

 

Anyway, here's the video for the updated ball viz...

 

https://odysee.com/@tursilion:1/vgmcomp2newvis:0

 

[ti99iuc]

woow! great idea! I really like!

thanks

[Nick99]

Great @Tursi, it will make it more fun when it happens somthing on the screen while the music is playing. I will try it when you release the new version.

[Tursi]

Next viz is in place - this was the first one I did way back before version 1 was released - Chuck. Managed to squeeze it down to the 8k block and port to Coleco as well.

 

Two more to go, then I just need to update the command line version of the quickplayer tool, test the chaining, and write the chaining program.

https://open.lbry.com/@tursilion:1/chuckVizVgmComp2:6?r=9pDgdsqTMZDnDuNaQFPxMF43B9C3ZoZp

 

Edited September 2, 2021 by Tursi