RMT2LZSS: convert RMT tunes to LZSS for fast playback!

[VinsCool]

I can confirm everything related to current 16-bit features work perfectly in this version for me  

Test 2.rmt Test 2.xex Test 2.erti Sketch 50 V1.erti Sketch 50 V1.rmt Sketch 50 V1.xex

[VinsCool]

... and actually I take back what I said, I managed to record the audio from altirra and now the Sketch 50 sounds incorrect.
Last version was all good, however. I used literally the same files so I have no idea why this sounds different, but I suspect that's related to the bug fixes so I probably have to edit my module to fix it, so that's only a minor issue for now.
Other than that everything in it worked fine

 

[rensoup]

19 hours ago, VinsCool said:

... and actually I take back what I said, I managed to record the audio from altirra and now the Sketch 50 sounds incorrect.
Last version was all good, however. I used literally the same files so I have no idea why this sounds different, but I suspect that's related to the bug fixes so I probably have to edit my module to fix it, so that's only a minor issue for now.
Other than that everything in it worked fine

 

hmm.. seems similar to me... anymore clues ?

[VinsCool]

1 hour ago, rensoup said:

hmm.. seems similar to me... anymore clues ?

Yeah it's okay, I figured the reason this happened (mainly the first few seconds, the kick drums sounded bugged) was caused by the way I worked around the 16-bit code from the previous version.

I abused the fact that 16-bit at volume 0 wouldn't be enabled, and also not update the content of AUDF1/AUDF3, to make it simple.
What I did was manually set a frequency for the portamento to pitch from to the lowest note (C-1), with a fast portamento value, this created a kick drum, and was constant.
Now the bug fix pretty much broke my workaround, but technically it worked exactly like it should have-- my workaround was no longer necessary  

So it's all good here, I just need to edit my .rmt to fix that part.

Edited May 18, 2021 by VinsCool
typo

[rensoup]

16 minutes ago, VinsCool said:

So it's all good here, I just need to edit my .rmt to fix that part.

alright well, I guess I can add 16bit support for cmd1 & 2 !

[VinsCool]

1 minute ago, rensoup said:

alright well, I guess I can add 16bit support for cmd1 & 2 !

What I really like there with the new TableNote16 feature is that it works really damn well, I'm excited for these 2 now 

[rensoup]

On 5/19/2021 at 1:32 AM, VinsCool said:

What I really like there with the new TableNote16 feature is that it works really damn well, I'm excited for these 2 now 

There you go!

 

1.77
    -added 16bit support for envelope command 1 & 2.
 

As usual, minimal testing was done, so let's see if I got it right first try ?

[VinsCool]

Took me a bit longer to come back at this, mainly due to having been busy for most of the last few days, combined with the computer problems I finally got fixed and properly set up.

Here's my test results using 1.77 so far:

Bugs(?) identified:
- Escape key to show the raster line no longer works in NTSC.

I've actually noticed this one a few versions ago, but did not put much thought about it since I was using a crap PC, and I thought it was related to that, but now I can confirm this is generalised and exclusive to NTSC.
- Command 1 works, but picks up values from XY in the .rmt module, interpreted as 8-bit values.
- Command 2 also works, but it is much of the same, it picks up values from XY in the .rmt module, interpreted as 8-bit values.

Suggestions: 
Allow the use of EnvelopeArgXY16 for command 1 and 2, which was what I actually expected to use at first, but noticed quickly the line was ignored in the conversion.
For the X and Y values, they could be 16-bit numbers when combined, or 8-bit values per letter?

Eg: X = 00, Y = FF, so $00FF would be then used for command 1 and 2 in 16-bit mode?

 

If the note outputs a frequency of $1F79 (A-2), using command 2 with EnvelopeArgXY16= $0001 would output a frequency of $1F7A, similar to using the frequency table in an instrument.

If the note outputs a frequency of $1F79 (A-2), using command 1 with EnvelopeArgXY16= $0001 would output a frequency of $0001, acting as the absolute frequency to output, which is exactly how XY is used in 8-bit mode.

And here's some hypothetical ideas regarding a possible workaround:

If no EnvelopeArgXY16 value is read in .erti, XY could be to only used for the low byte?
Eg: If the .rmt XY value is 1F, it would be interpreted as $001F?

Or,

If no EnvelopeArgXY16 value is read in .erti, interpret XY as the lower nibble of each byte maybe?
Eg: If the .rmt XY value is 1F, it would be interpreted as $010F?

I personally think the first idea would make more sense in this situation, and could work well enough for command 2 in particular, for quick and easy fine tune effects.

Otherwise, it would be logical to always use EnvelopeArgXY16 instead, for the purpose of having valid XY values in 16-bit mode for the commands 1 and 2.

Another observation:
- Could it be possible to add support for command 0 in 16-bit mode? It may be simple enough to add since it's only a note offset, similar to the note table in an instrument.
There may be several other commands that could maybe benefit from it, but I personally don't see what would really be worth adding for the moment, so commands 0, 1 and 2 would work the best for now, 5 being already done and fully functional.  

I hope this post will be helpful, took me a little bit to write everything as detailed as I could make it.
For now, I'm getting back to attempting some music cover using these cool special POKEY sounds  

Edited May 22, 2021 by VinsCool
typo

[emkay]

I'm still not keen on doing crossquirkjuggling the 16 Bit stuff. 

There is a feature that is totally missed for "standard music creation". 

 

Imagine, you have one instrument, and you want to create a "floating" effect to this instrument. 

Well, SID Trackers offer this "trimming of registers"  by default.

 

Imagine, you have some instrument at 16 Bits, and you want to do variations. 

If you want to do variations inside a note-range, you had to create "100" different instruments. 

Doing that automatically, just needs "one" instrument  plus the "trimming" of the wanted value. 

 

It's not Portamento , and no vibrato. 

 

Imagine the envelope

 

xXXXxxx

 

or 

xXxXxX

 

or just

 

Xxxxxxxx

 

 

the upper X is the register that get allowed to be trimmed while the tune is playing. 

 

What's needed is the feature to set the X and the amount of X.

Then there had to be the starting value and the ending value to be set. The speed of changing the X had to be set , too.

 

3 features would be helpful

 

1) from low to high 

2) from high to low

3) "ring" changing

 

low-high-low-high....

 

Hoping , it is to understand

 

 

 

 

[rensoup]

4 hours ago, emkay said:

There is a feature that is totally missed for "standard music creation". 

 

Imagine, you have one instrument, and you want to create a "floating" effect to this instrument. 

Well, SID Trackers offer this "trimming of registers"  by default.

Does this "trimming" have a name in those trackers ?

 

Can you demonstrate the effect in one of them ?

 

4 hours ago, emkay said:

What's needed is the feature to set the X and the amount of X.

Then there had to be the starting value and the ending value to be set. The speed of changing the X had to be set , too.

The thing is there is no space to put those parameters anywhere because I'm guessing they'd have to be on the track line together with the note and the instruments?

 

In other words, you're asking for effects with notes, just like regular amiga/pc trackers ?

 

I know you've already asked for this but doing this with a text file (like .erti but for patterns instead of instruments) would be a gigantic pain...

 

[rensoup]

1.77b May 2021
    -fixed 16bit support for envelope command 1 & 2 (should really work now)
    -A8 player: removed HP filter check from VUMeter code 
 

On 5/22/2021 at 9:48 PM, VinsCool said:

Allow the use of EnvelopeArgXY16 for command 1 and 2, which was what I actually expected to use at first, but noticed quickly the line was ignored in the conversion.

so that should be fixed this time, I tried with your test tune and it produced different results from before ?

 

On 5/22/2021 at 9:48 PM, VinsCool said:

- Could it be possible to add support for command 0 in 16-bit mode? It may be simple enough to add since it's only a note offset, similar to the note table in an instrument.

You said it worked when the table is in note mode right ? that makes sense... 

 

What seems weird is that you said it worked in regular RMT with frequency mode with dist 6 ? Looking at the code, I don't see how that's possible because here's what happens:

 

	if (instr[chIdx].tableType == Instrument.TableType.Note)
		note += (byte)(m_songRuntime.TrackLine[chIdx].TableNoteOrFreq >> 8);

	if (note >= 61)
	{
		m_songRuntime.AudC[chIdx] = 0;
		note = 63;
	}
	freq = (UInt16)((frqtab[frqTableOffset[chIdx] + note] + m_songRuntime.TrackLine[chIdx].Shiftfrq) << 8);
	freq += m_songRuntime.TrackLine[chIdx].frqAddCmd;

	if (instr[chIdx].tableType == Instrument.TableType.Frequency)
		freq += m_songRuntime.TrackLine[chIdx].TableNoteOrFreq;
	else
		m_songRuntime.TrackLine[chIdx].OutNote = note;

 

so basically:

1. are we in table mode ? add note from note table to base note

2. make sure note is not outside range

3. convert note to frequency using 8 bit table

4. are we in frequency mode ? add frequency from frequency table to frequency computed in 3.

5. are we in table mode ? store note from 1. in variable called OutNote

 

(the frequency then goes into a temporary AUDF for 8 bit playback)

 

later on the code checks for dist 6 and if enabled:

1. set 16bit mode

2. convert OutNote to frequency using 16 bit table

3. stores frequency into both AUDF

 

Which means 16bit mode discards steps 3. 4. and the temporary AUDF

 

So if using dist 6 with regular RMT, frequency mode doesn't work, which makes sense because the note was converted using the 8 bit table.

 

Could you confirm that ?

 

(if it turns out to be right, the good news is that it could be fixed)

 

 

[emkay]

1 hour ago, rensoup said:

Does this "trimming" have a name in those trackers ?

 

Trackers might not have that by standard. It's more the control on SID for the filters or the PWM setting.

 

1 hour ago, rensoup said:

Can you demonstrate the effect in one of them ?

 

 

 

 

 

1 hour ago, rensoup said:

The thing is there is no space to put those parameters anywhere because I'm guessing they'd have to be on the track line together with the note and the instruments?

 

In other words, you're asking for effects with notes, just like regular amiga/pc trackers ?

 

I know you've already asked for this but doing this with a text file (like .erti but for patterns instead of instruments) would be a gigantic pain...

 

Yeah. I would be needed to add "parallel" calculation routines to the main tune.

Running at the same speed as the song. But it would give a lot better value to the tuning over all than to add stiff 16 bit sound. 

[rensoup]

 

well I guess I don't get it...

 

20 hours ago, emkay said:

Running at the same speed as the song. But it would give a lot better value to the tuning over all than to add stiff 16 bit sound. 

 

Whatever this effect is, it would require an editor and that's not on the cards right now, but making 16bit mode behave more consistently seems like a reasonable feature to add (you asked for 16bit portamento remember ?)

 

[emkay]

1 hour ago, rensoup said:

 

well I guess I don't get it...

 

 

Whatever this effect is, it would require an editor and that's not on the cards right now, but making 16bit mode behave more consistently seems like a reasonable feature to add (you asked for 16bit portamento remember ?)

 

Of course. And I should check the features in the converter, next.

[VinsCool]

On 5/23/2021 at 9:53 PM, rensoup said:

so that should be fixed this time, I tried with your test tune and it produced different results from before ?

Thanks! Next time I test I'll report back if I notice anything from there.

On 5/23/2021 at 9:53 PM, rensoup said:

You said it worked when the table is in note mode right ? that makes sense... 

 

What seems weird is that you said it worked in regular RMT with frequency mode with dist 6 ? Looking at the code, I don't see how that's possible because here's what happens:

Interesting observation.
When I did notice command 0 working with Dist 6, the program did make the notes using Notes mode by default in the table, which made me unsure regarding the fact Frequency mode seemed to have broken it.

I did not realise this was the explanation behind my "bug" lol
(Speaking of which, the NTSC raster bar "missing" was another moment when the problem was in fact between the chair and the monitor, still surprised I overlooked something so obvious lol)

On 5/23/2021 at 9:53 PM, rensoup said:

Could you confirm that ?

So if I understand correctly, what I experienced (command 0 disabled with Frequency Mode in tables) was actually normal and part of the original 16-bit code?
That makes sense if that's the case, since Frequency pretty much was not possible to adjust, which has now been changed with your new 16-bit improvements.
Could it be possible to make it behave in a way that Table Mode should not get involved for using commands, in this case?

I ask because in most situations that do not use 16-bit, Table Mode can be either (and even be combined!) to commands in the instruments envelopes, this was actually what I expected and did not realise I could have used command 0 only if I was in Note mode... lol
I understand the existence of the original code, but now with your new and improved 16-bit code, it's pretty much pointless to keep this limitation, since it made sure to always pick the 16-bit frequencies when required anyway  

[rensoup]

14 hours ago, VinsCool said:

So if I understand correctly, what I experienced (command 0 disabled with Frequency Mode in tables) was actually normal and part of the original 16-bit code?

yep!

14 hours ago, VinsCool said:

Could it be possible to make it behave in a way that Table Mode should not get involved for using commands, in this case?

? what? Can you rephrase that ? Or do you mean make 16bit mode work consistently with frequency tables for commands where it makes sense ? In theory it should do that already with the latest version although that's probably only useful for cmd3 and maybe 7 ?

cmd 4 & 6 mess with the FreqShift register which I think is for vibrato ?

 

[VinsCool]

2 hours ago, rensoup said:

? what? Can you rephrase that ? Or do you mean make 16bit mode work consistently with frequency tables for commands where it makes sense ?

Yeah pretty much that! So having table mode set to note or frequency wouldn't make any difference combined to whatever is under the volume envelope, this is the expected behaviour for anything not using current 16-bit code.

 

---------

Speaking of which I did more tests today with the newest version and came across some observations:

- My older Test 2 files (testing portamento, filter, EnvelopeArgXY16 and TableNote16) still work exactly as expected, so that's good!
- Commands 1 and 2 appear to work perfectly with EnvelopeArgXY16 in my newer tests!, they also work regardless of what Table mode is being used!

But here's new things I noticed now, to which had to poke around for a little while to figure what was happening:

- Command 0 still only works when in Note Table mode, trying to use it in Frequency Table mode is ignored, but it seems to make sense why this is the case based on your reply above.
So, the "issue" I mentioned in the previous version still exists, but that's not too bad so far, since I can still use the Note Table mode with it, and I can even combine them together to go to different notes, so this at least does work fine that way.  

- TableNote16 only works correctly when Portamento is enabled, this actually was the thing that made me scratch my head for at least 30 minutes.

I was so confused as to why EnvelopeArgXY16 still worked in my older tests, but not here when in Frequency Table mode for some reason.
The workaround? command 5, XY= 00, and now I can use it ?, but this really shouldn't be happening.

- Frequency Table mode (along with TableNote16) is expected to be working regardless of portamento being enabled, or not.

In this case, I did not need, or wanted to do any portamento, since I wanted to "finetune" my notes, and only that.

 

----------

 

So, to summarise what I wrote in fewer words:

- There seems to be a bug involving the portamento code attempting to use the Frequency Table mode + TableNote16, it will not work unless command 5 is set, and the Portamento bit (under Volume and Filter) is enabled.
Workaround: Using command 5 with XY= 00 will make the finetune effect work, and will not create any unwanted portamento effect.

Of course, using command 5 with EnvelopeArgXY16 on top of it will also work just fine, since the previous versions did get fixes for that specific combination.

- Command 0 does not work in Frequency Table mode, but I now know this was actually how it was originally coded into RMT's 16-bit mode. With your new 16-bit mode code, this limitation is no longer necessary, in my opinion. 

Workaround: I can actually use the Note Table mode combined to command 2 + EnvelopeArgXY16, this combination is essentially doing the exact same thing.
 

Most commands (0-1-2...) and Table Modes could technically be combined if Frequency Table mode + TableNote16 wasn't currently locked to command 5, or command 0 being locked to Note Table mode.

You were able to implement a large portion of all these features into your own player code so far, that makes me believe this might be not too difficult to improve 

----------

 

In other news, I have been working on my POKEY tuning and frequencies calculation, and yesterday I was been able to figure out a couple constants used in formulae.  

Now, I can accurately predict and calculate the real frequencies of several distortions in different modes! With this in mind, I may be able to find more possibilities, and will also save me a ton of time trying to identify pitches manually with poor accuracy.

Here's some of the constants I observed, and was able to confirm being very accurate to my known data dump:

The following was already well known:
Distortion A 64khz -> divide by 28 from Master Clock
Distortion A 15khz -> divide by 114 from Master Clock
Distortion A 1.79mhz -> Use Master Clock directly

These are the numbers I identified and confirmed being accurate, so far:
Distortion C, Buzzy Tones 64khz -> divide by 70 from Master Clock
Distortion C, Gritty Tones 64khz -> divide by 210 from Master Clock
Distortion C, Buzzy Tones 15khz -> divide by 285 from Master Clock
Distortion 2, Squareish Tones 1.79mhz -> divide by 31 from Master Clock

This is honestly a very nice discovery for me, because now I believe I may be able to easily find out even more constants like those!
And like I mentioned, using these numbers do provide very accurate results, so my future tuning tables may be a lot less frustrating to put together if I can actually find the frequencies without sitting for several hours noting down the tones 1 by 1 ?

I have no idea if that was documented before, because I never was able to find anything online, so just the fact I understand better how the Atari 8-bit system works makes me pretty happy and also excited for what will come next  

 

I'll attach the spreadsheet I was doing, and I apologise for it being a little messy right now, this will get worked on in due time, and it won't be as slow as it used to be since I can actually calculate the frequencies directly!

I have been experimenting with different tuning ratios as well, using the other sheets by Ivop, but this is not in my calculations yet, however, I do have some very promising results and will post more about these later as well.

 

Atari Tuned A+C v30.ods

[rensoup]

2 hours ago, VinsCool said:

So, to summarise what I wrote in fewer words:

??

6 hours ago, rensoup said:

In theory it should do that already with the latest version although that's probably only useful for cmd3 and maybe 7 ?

Oops... that's not actually in the latest version...  it's in the version I've got on my hd, I was waiting for your confirmation that cmd 1 & 2 are ok ?

 

3 hours ago, VinsCool said:

- There seems to be a bug involving the portamento code attempting to use the Frequency Table mode + TableNote16, it will not work unless command 5 is set, and the Portamento bit (under Volume and Filter) is enabled.
Workaround: Using command 5 with XY= 00 will make the finetune effect work, and will not create any unwanted portamento effect.

I'm not sure if I understand that one correctly... it may be fixed with the next version, let me know...

 

3 hours ago, VinsCool said:

Most commands (0-1-2...) and Table Modes could technically be combined if Frequency Table mode + TableNote16 wasn't currently locked to command 5, or command 0 being locked to Note Table mode.

You were able to implement a large portion of all these features into your own player code so far, that makes me believe this might be not too difficult to improve 

True but that would break compatibility with stock RMT ? better keep those commands even if it makes less sense at this point...

 

3 hours ago, VinsCool said:

In other news, I have been working on my POKEY tuning and frequencies calculation, and yesterday I was been able to figure out a couple constants used in formulae.  

I'm not sure if it'd make sense to have some kind of way to compute that directly in the .erti file, but perhaps it's not really relevant if you can output the table automatically from your spreadsheet

 

going to upload 1.78 soon...

[rensoup]

1.78 May 2021
    -fixed frequency table for various commands
 

[VinsCool]

16 hours ago, rensoup said:

??

?

16 hours ago, rensoup said:

Oops... that's not actually in the latest version...  it's in the version I've got on my hd, I was waiting for your confirmation that cmd 1 & 2 are ok ?

 

Yep like I said, those 2 are okay!, it was the other things I mentioned that were not quite, but not unusable regardless.

16 hours ago, rensoup said:

I'm not sure if I understand that one correctly... it may be fixed with the next version, let me know...

Basically the effect would not work unless Pormanento was specifically enabled, otherwise it was simply ignored.
This effect is not required for portamento, but portamento could use it, if that makes more sense  

I mentioned a workaround because that meant I was able to at least use the effect with this condition, with virtually no problem, other than being unnecessary under normal circumstances  

Will check this out in the new release.

16 hours ago, rensoup said:

True but that would break compatibility with stock RMT ? better keep those commands even if it makes less sense at this point...

Maybe stock 16-bit code, that is true, but stock 8-bit code supports all of these, so since more than half of that is now supported in your 16-bit improvements, I don't see any reason to not allow this if the Custom Tables and Custom Instruments are used.

16 hours ago, rensoup said:

I'm not sure if it'd make sense to have some kind of way to compute that directly in the .erti file, but perhaps it's not really relevant if you can output the table automatically from your spreadsheet

I have no idea how that could even work, so doing all that maths in the sheets first then pasting everything over the CustomNotesTables.txt will work perfectly fine for me  

14 hours ago, rensoup said:

1.78 May 2021
    -fixed frequency table for various commands

Thank you! I will verify that in more details later once I can access my computer  

 

 

 

[VinsCool]

Okay here's some new results:

- Command 0 used in both Notes Table mode and Frequency Table mode works nicely now!
- Frequency Table mode + TableNote16 still does not work unless command 5 is used and Portamento bit is enabled.
- Commands 1 and 2 still work as expected as well.

So that's getting better!

 

---
I have noticed something else however... 
Do you remember when I mentioned getting weird stutters using 16-bit notes at certain times? well now I think I found a possible explanation for it, it looks like there are steps at the start of an envelope that do get skipped randomly.

So what that means is that if certain things are expected to be 1 frame long, instead, there will be flat silence output there instead, which if odd because it is never happening at the same place.

Here's a quick comparison, the first is doing the notes in 16-bit, the other, in 8-bit with a similar envelope for reference.

Both are supposed to produce a similar results, but for some reason, it's not exactly what is happening.

Test 6.obx Test 6.rmt Test 6.erti 

That pretty much confirms I also did not imagine things this time, it's pretty easy to notice that after several loops, the 16-bit notes have random skips.
Please don't mind the channels using "null" instruments, it was just to always reset the frequencies to 00, but that did no difference there anyway.

[rensoup]

2 hours ago, VinsCool said:

- Frequency Table mode + TableNote16 still does not work unless command 5 is used and Portamento bit is enabled.

I think I see what you mean now... I simply forgot to add TableNote16 in 16bit mode!

 

2 hours ago, VinsCool said:

Do you remember when I mentioned getting weird stutters using 16-bit notes at certain times? well now I think I found a possible explanation for it, it looks like there are steps at the start of an envelope that do get skipped randomly.

No idea about that one... will investigate!

[rensoup]

On 5/28/2021 at 12:17 AM, VinsCool said:

- Frequency Table mode + TableNote16 still does not work unless command 5 is used and Portamento bit is enabled.

should be ok now ?

 

1.78b May 2021
    -fix for notetable16 not working 
 

[VinsCool]

3 hours ago, rensoup said:

should be ok now ?

 

1.78b May 2021
    -fix for notetable16 not working 
 

I confirm this is working now!
After all of this, the current 16-bit design is a huge improvement over RMT's original code

[emkay]

I post it here, as it shows the benefit of fast replay very clearly. 

Not sure who did it, but he was really an ambitious...

 

Chipster Harmony.