RMT2LZSS: convert RMT tunes to LZSS for fast playback!

[emkay]

1st some special question. 

Is it possible to just switch from RMT Player to LZSS player, to just exchange it in the assembler?

 

Let's say the RMT player is at Address 9500

The starting Address is at B300

The Song data is at A000

 

exchanging the lzss player directly with the "rmtplayr.a65" file?

 

[rensoup]

7 hours ago, VinsCool said:

If you can share the files it would help a lot. There are some quirks related to portamento and I think I know what is up.

It's caused by the way it is programmed but can be worked around.

 

I am a avid user of portamento after all

 

I could potentially fix it if there's a problem with the code

[rensoup]

4 hours ago, emkay said:

1st some special question. 

Is it possible to just switch from RMT Player to LZSS player, to just exchange it in the assembler?

Hell no... that wouldn't work at all.

 

It's a lot more involved than that (I mentioned it in the Beyond RMT thread).

 

Even if the RMT source code was available it would be a sizable job: basically it would involve removing the 6502 emulator and the 6502 asm player. Then include my C# version of the player (which would have to be converted to C++ first) and interface it with pokey.dll (which is actually Fox's ASAP).

 

The source code not being available, it's an "impossible" task.

 

The other closest alternative is to rewrite the editor around my C# player (and interface it with pokey.dll), that's a loooong and boring job...

 

[emkay]

3 hours ago, rensoup said:

Hell no... that wouldn't work at all.

 

It's a lot more involved than that (I mentioned it in the Beyond RMT thread).

 

Even if the RMT source code was available it would be a sizable job: basically it would involve removing the 6502 emulator and the 6502 asm player. Then include my C# version of the player (which would have to be converted to C++ first) and interface it with pokey.dll (which is actually Fox's ASAP).

 

The source code not being available, it's an "impossible" task.

 

The other closest alternative is to rewrite the editor around my C# player (and interface it with pokey.dll), that's a loooong and boring job...

 

Hm.

 

Isn't it that simple?

There is a range in Memory where any software or data could be placed.

Then there are the "start/stop" vectors, possibly switches for muting channels. 

Which is obviously there in the programming of the RMT-Player. Without that, it isn't possible to control the player at all

 

Possibly you misunderstood my question?

 

Let's say, if a game exists with the title and the music is running on the RMT Player.

What would be the real problem to exchange the parts in the RAM?

 

It's just: If the RMT Player cannot be "solved", why not changing the player completely? 

 

[VinsCool]

6 hours ago, rensoup said:

I could potentially fix it if there's a problem with the code

I wouldn't really call any portamento behaviour a problem, that is, if we can even see what happens at all in the tune with a "bug" lol
I know that portamento effects like to kill certain things like fine tune commands, which is pretty much the only one I have in mind there. The easiest way to avoid it is to give enough time to the portamento effect to slide to the target note then switch it off, otherwise some things can sound wrong, that is more noticeable with filter modulation, which seems to be what happened in emkay's tune.

Command 6 (never really understood how that one really worked but it manipulates the filter frequencies I think) also dislikes being used with portamento, so that could be related.

[rensoup]

21 hours ago, emkay said:

Isn't it that simple?

it's simple if you only understand half the problem ?

21 hours ago, emkay said:

Let's say, if a game exists with the title and the music is running on the RMT Player.

What would be the real problem to exchange the parts in the RAM?

 

It's just: If the RMT Player cannot be "solved", why not changing the player completely? 

Because the tracker editor interacts directly with the player, it puts your data (notes instruments,...) into the 6502 memory every time you press "play", then runs the 6502 player which in turn, decodes song lines/ patterns/ instruments and turns it into raw Pokey data.

LZSS is just a decompressor for raw Pokey data...

[rensoup]

18 hours ago, VinsCool said:

Command 6 (never really understood how that one really worked but it manipulates the filter frequencies I think) also dislikes being used with portamento, so that could be related.

It doesn't seem to do a whole lot:

 

               //  6    Add $XY value to FILTER_SHFRQ. (Whenever the new note in track is getting started, FILTER_SHFRQ is initialized to $01, so that default join filter generator frequency is higher by 1.)

 

for each channel:

 When getting a new track line: [CH].FilterShiftFreq = 1;  (emkay asked me to be able to have a custom value, so you can change it in the instrument file )

 

When updating 
                        [CH].FilterShiftFreq += [CH].envelope.commandArg;

 

 if HP filter is enabled

                AudF[CH+2] = AudF[CH] + [CH].FilterShiftFreq;

 

It's pretty simple though I don't know what kind of effect it has on sound

 

[VinsCool]

2 hours ago, rensoup said:

It doesn't seem to do a whole lot:

 

               //  6    Add $XY value to FILTER_SHFRQ. (Whenever the new note in track is getting started, FILTER_SHFRQ is initialized to $01, so that default join filter generator frequency is higher by 1.)

 

for each channel:

 When getting a new track line: [CH].FilterShiftFreq = 1;  (emkay asked me to be able to have a custom value, so you can change it in the instrument file )

 

When updating 
                        [CH].FilterShiftFreq += [CH].envelope.commandArg;

 

 if HP filter is enabled

                AudF[CH+2] = AudF[CH] + [CH].FilterShiftFreq;

 

It's pretty simple though I don't know what kind of effect it has on sound

 

Oh I see now.

So by default filter simply does the Frequency from 1 channel +01, that does match up my observation.

 

The thing that confused me was that using command 6 seemed to "roll over", in a way that incremented or decremented without control but now I think I understand what is happening.

 

It does pretty much exactly what can be done using 2 channels by manually setting the frequency and controlling it using command 2 (fine tune) in one of them.

So the commands simply get stacked together each time a new command 6 is used? That certainly explains why it has to be something like a envelope loop with a FF then a 01 somewhere, so it reminds stable.

 

I think I understand why Emkay wants to manually set it, it would make more sense to start at 0 (no shifting) and then change it to either 01 ot FF, or any value really, which would changed the direction of the Pulse waveform nicely.

 

I personally find using command 2 much more precise, but it requires to use 2 channels and 2 instruments, but that comes with a nice bonus of being able to do anything you want on the get go, for example chords sounds or special timbres that are produced at certain intervals, and also output volume with no limitation on the "modulation" channel.

 

 

Anyway, off that tangent, either of these commands don't play so nice with portamento effect, and that's pretty much the only thing I can think about the so called "portamento bug".

It is indeed a little annoying, but as long as the note has enough time to slide, deactivating the portamento slide somewhere will stabilise the sound and also most likely land on the right note if a fine tune value was used somewhere.

[emkay]

6 hours ago, rensoup said:

it's simple if you only understand half the problem ?

Because the tracker editor interacts directly with the player, it puts your data (notes instruments,...) into the 6502 memory every time you press "play", then runs the 6502 player which in turn, decodes song lines/ patterns/ instruments and turns it into raw Pokey data.

LZSS is just a decompressor for raw Pokey data...

Erm... I'm not about using any RMT parts then. 

It's about your player working in a software package . 

[rensoup]

17 hours ago, emkay said:

Erm... I'm not about using any RMT parts then. 

It's about your player working in a software package . 

 

I've said it already: my player is Raster's 6502 asm RMT player converted to C#: it does exactly the same thing in a different language

 

 

 

[_The Doctor__]

somethings have to be different though yes?

[Wrathchild]

1 hour ago, rensoup said:

Raster's 6502 asm RMT player converted to C#

Which one though, the one from his site (or archive of)?

Those were born out of the atari800 emulator and I updated these back in 2005 (a reminder to check your help/about box when running RMT).

 

What the RMT player actually needs is the Altirra equivalent
But from what you are saying, surely a C/C++ bridge to your C# code could be rustled together?

 

[Edit] Scratch this - of course it is the 'asm' i.e. code that runs on the A8.

So that's quite neat then as the resulting executable code on the A8 is dependent on things that the RMT editor 'knows' about the tune.

e.g. if it is using a 16 bit channel, if you are using a certain effect etc.

if you aren't then the code isn't generated and so the runtime of the vblank time is overall quicker.

Due to speed not being an issue on a PC's processor, I guess you have guess left all the flags in and set them from the header/defines output by RMT and all possible code paths in the asm equivalently available in the C# port?

Edited March 24, 2021 by Wrathchild

[rensoup]

22 hours ago, _The Doctor__ said:

somethings have to be different though yes?

Of course...

 

-The Pokey emulator of course. RMT uses ASAP(Pokey.dll) which was originally based on Atari800 and rewritten later. The same tune exported to an obx will sound different if played in Altirra.

 

-LZSS simplifies the compressed data if the volume is zero, it seems to cause differences but it can be disabled in RMT2LZSS.

 

-the way the values are sent to Pokey could potentially cause differences but I tried using the same method as RMT and it made no audible difference.

 

The RMT player, C# or asm, should output exactly the same data however...

[rensoup]

21 hours ago, Wrathchild said:

Those were born out of the atari800 emulator and I updated these back in 2005 (a reminder to check your help/about box when running RMT).

 

What the RMT player actually needs is the Altirra equivalent
But from what you are saying, surely a C/C++ bridge to your C# code could be rustled together?

Yes you're talking about the Pokey emulator, would be nice if Altirra was modular and we could use its Pokey emulation but it may not be that difficult to split it... It's not really that important (for now at least) because you can always export an obx and run it Altirra.

 

21 hours ago, Wrathchild said:

[Edit] Scratch this - of course it is the 'asm' i.e. code that runs on the A8.

So that's quite neat then as the resulting executable code on the A8 is dependent on things that the RMT editor 'knows' about the tune.

e.g. if it is using a 16 bit channel, if you are using a certain effect etc.

if you aren't then the code isn't generated and so the runtime of the vblank time is overall quicker.

Yes but I doubt RMT does any "on the fly switch toggling" and rebuilds the xex internally, it probably has all the flags enabled... (easy to check I guess... just output the simplest possible tune and check the xex )

 

21 hours ago, Wrathchild said:

Due to speed not being an issue on a PC's processor, I guess you have guess left all the flags in and set them from the header/defines output by RMT and all possible code paths in the asm equivalently available in the C# port?

That's right I took the player source from the 1.28 release and left all the options enabled (except playSfx and stuff like that) but the code is all cleaned up now so there's none of that stuff left.

[pps]

Hello Folks,

 

my first time I played a bit with this routines on real machine gave me some strange errors, that I finally could solve. But start from the beginning...

 

I used an old mod conversation I made to test if I could adapt the player routine from @dmsc to be able to play within my code as well. On Altirra everything sounds correct - if you selected STEREO machine only left side plays full track. On my real machine (600XL+U1MB+Simple Stereo 1(not switchable)) strange things occur: Only left side plays, that's ok, but there only 2 channels seemed to play. So the whole song sounds incorrect. To try out, if it was just my fault when I converted the routine into my code, I used RMT2LZSS .xex output to test again with the same song. Even there the same error occured.

 

1. After some more tests I finally found a simple solution: The play routine in @dmsc's routine doesn't initialize POKEY correct. I added a simple 'mva #3 SKCTL' and it runs fine now.
2. For more comfort on STEREO machines you can add an 'STA SKCTL+16' right below that and add two times 'STA POKEY+16,x' just where the other 'STA POKEY,x' are. That solves the problem.

At least the 1. option should be added into all routines for correct playback on ATARIs.

 

Here the complete source of the used lzss16 play routine with my additions a bit more indented:

;
; LZSS Compressed SAP player for 16 match bits
; --------------------------------------------
;
; (c) 2020 DMSC
; Code under MIT license, see LICENSE file.
;
; This player uses:
;  Match length: 8 bits  (1 to 256)
;  Match offset: 8 bits  (1 to 256)
;  Min length: 2
;  Total match bits: 16 bits
;
; Compress using:
;  lzss -b 16 -o 8 -m 1 input.rsap test.lz12
;
; Assemble this file with MADS assembler, the compressed song is expected in
; the `test.lz16` file at assembly time.
;
; The player needs 256 bytes of buffer for each pokey register stored, for a
; full SAP file this is 2304 bytes.
;
    org $80

chn_copy    .ds     9
chn_pos     .ds     9
bptr        .ds     2
cur_pos     .ds     1
chn_bits    .ds     1

bit_data    .byte   1

.proc get_byte
    lda song_data+1
    inc song_ptr
    bne skip
    inc song_ptr+1
skip
    rts
.endp
song_ptr = get_byte + 1


POKEY = $D200
POKEY2 = $d210	;needed for 2nd POKEY
SKCTL = $d20f	;needed for correct init

    org $2000
buffers
    .ds 256 * 9

song_data
        ins     'yourmusic.lz16'
song_end


start

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Song Initialization - this runs in the first tick:
;
.proc init_song

    ; Example: here initializes song pointer:
    ; sta song_ptr
    ; stx song_ptr + 1
	mva #3 SKCTL	;needed for correct init
	sta SKCTL+16	;init for 2nd POKEY
    ; Init all channels:
    ldx #8
    ldy #0
clear
    ; Read just init value and store into buffer and POKEY
    jsr get_byte
    sta POKEY, x
	sta POKEY2,x	;needed for 2nd POKEY
    sty chn_copy, x
cbuf
    sta buffers + 255
    inc cbuf + 2
    dex
    bpl clear

    ; Initialize buffer pointer:
    sty bptr
    sty cur_pos
.endp

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Wait for next frame
;
.proc wait_frame
    lda 20
delay
    cmp 20
    beq delay
.endp

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Play one frame of the song
;
.proc play_frame
    lda #>buffers
    sta bptr+1

    lda song_data
    sta chn_bits
    ldx #8

    ; Loop through all "channels", one for each POKEY register
chn_loop:
    lsr chn_bits
    bcs skip_chn       ; C=1 : skip this channel

    lda chn_copy, x    ; Get status of this stream
    bne do_copy_byte   ; If > 0 we are copying bytes

    ; We are decoding a new match/literal
    lsr bit_data       ; Get next bit
    bne got_bit
    jsr get_byte       ; Not enough bits, refill!
    ror                ; Extract a new bit and add a 1 at the high bit (from C set above)
    sta bit_data       ;
got_bit:
    jsr get_byte       ; Always read a byte, it could mean "match size/offset" or "literal byte"
    bcs store          ; Bit = 1 is "literal", bit = 0 is "match"

    sta chn_pos, x     ; Store in "copy pos"

    jsr get_byte
    sta chn_copy, x    ; Store in "copy length"

                        ; And start copying first byte
do_copy_byte:
    dec chn_copy, x     ; Decrease match length, increase match position
    inc chn_pos, x
    ldy chn_pos, x

    ; Now, read old data, jump to data store
    lda (bptr), y

store:
    ldy cur_pos
    sta POKEY, x        ; Store to output and buffer
	sta POKEY2,x	;needed for 2nd POKEY
    sta (bptr), y

skip_chn:
    ; Increment channel buffer pointer
    inc bptr+1

    dex
    bpl chn_loop        ; Next channel

    inc cur_pos
.endp

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Check for ending of song and jump to the next frame
;
.proc check_end_song
    lda song_ptr + 1
    cmp #>song_end
    bne wait_frame
    lda song_ptr
    cmp #<song_end
    bne wait_frame
.endp

end_loop
    rts

    run start

Hope this is usefull info for you

cu, Ralf

[pps]

On 3/21/2021 at 8:59 PM, emkay said:

1st some special question. 

Is it possible to just switch from RMT Player to LZSS player, to just exchange it in the assembler?

 

Let's say the RMT player is at Address 9500

The starting Address is at B300

The Song data is at A000

 

exchanging the lzss player directly with the "rmtplayr.a65" file?

 

Mario wants to know if he can simply use the play routine via cut+paste over rmt routine in some asm source he has for creating own intros.

 

No Mario, it's not just simple cut+paste, but I'll provide you with the sources you need soon

[ivop]

19 minutes ago, pps said:

The play routine in dmsc's routine doesn't initialize POKEY correct.

I have been bit by that bug, too, in the past

 

20 minutes ago, pps said:

STA SKCTL+16

Especially this one. That's why sid2gumby at first did not work on real hardware. Now it does

 

[VinsCool]

46 minutes ago, pps said:

At least the 1. option should be added into all routines for correct playback on ATARIs.

Would that initialisation possibly the reason why sometime it feels like a tune would often start with a frame skipped after being converted with RMT2LZSS? or that was something different?
I always thought it was lag caused by the CPU usage spike and that was normal, so it never really bothered me.

 

[pps]

3 minutes ago, VinsCool said:

Would that initialisation possibly the reason why sometime it feels like a tune would often start with a frame skipped after being converted with RMT2LZSS? or that was something different?
I always thought it was lag caused by the CPU usage spike and that was normal, so it never really bothered me.

 

If you had this issues on real machine I think, you discovered the same. On Altirra it sounds ok for me. Maybe that's caused by emulator has some differences esp. when you burst in a .xex directly.

 

As long as you just use RMT2LZSS .xex output, you'll have to wait till @rensoup corrects his code, too. Then you can compare your songs.

[VinsCool]

2 minutes ago, pps said:

If you had this issues on real machine I think, you discovered the same. On Altirra it sounds ok for me. Maybe that's caused by emulator has some differences esp. when you burst in a .xex directly.

Yeah I noticed it happened both in Altirra and real machine, but I assumed it was normal, most of the time it's unnoticeable too  

[ivop]

When I did some recordings for the Lovebyte competition, I noticed there was a difference between calling Altirra.exe bla.xex from the command line, and loading an xex file from within the GUI. The amount of cycles burnt before your code starts running is not the same.

[rensoup]

7 hours ago, pps said:

my first time I played a bit with this routines on real machine gave me some strange errors, that I finally could solve. But start from the beginning...

Good to hear that someone else found a use for it (on the A8)!

 

7 hours ago, pps said:

1. After some more tests I finally found a simple solution: The play routine in @dmsc's routine doesn't initialize POKEY correct. I added a simple 'mva #3 SKCTL' and it runs fine now.
2. For more comfort on STEREO machines you can add an 'STA SKCTL+16' right below that and add two times 'STA POKEY+16,x' just where the other 'STA POKEY,x' are. That solves the problem.

At least the 1. option should be added into all routines for correct playback on ATARIs.

That's an interesting diagnostic given that dmsc's code works in Altirra and on Vinscool's real unmodified Atari 800XL but not on your modified 600XL ?

 

It seems to me there's an issue with your extension ?

 

I don't know if SKCTL needs to be initialized but I never touched because it seems to be useful for producing different kind of sounds (like synthpopalooza demonstrated ). It was discussed a while back that it could possibly be stored as an extra channel but only a single tune ever changed it during sound playback... so it was left alone.

 

I figured I could add an option to set the SKCTL value in RMT2LZSS so that it would be possible to try different modes like synthpopalooza but... too lazy so far.

 

...Perhaps for the next version then!

 

As for 2. you mentioned that your simple stereo is not switchable ?? Doesn't that renders it incompatible with every A8 software ? (Your fix would make 50+hz tunes DLIs slower too)

 

 

[rensoup]

7 hours ago, pps said:

Mario wants to know if he can simply use the play routine via cut+paste over rmt routine in some asm source he has for creating own intros.

 

No Mario, it's not just simple cut+paste, but I'll provide you with the sources you need soon

Ah... makes more sense when you explain it!

 

[rensoup]

7 hours ago, VinsCool said:

Would that initialisation possibly the reason why sometime it feels like a tune would often start with a frame skipped after being converted with RMT2LZSS? or that was something different?
I always thought it was lag caused by the CPU usage spike and that was normal, so it never really bothered me.

I doubt that...  Are you saying it happens in the emulator ?  Which you happen to be running from that horribly slow Wine thing ?... or does it happen on your real A8 ?

 

[VinsCool]

4 hours ago, rensoup said:

I doubt that...  Are you saying it happens in the emulator ?  Which you happen to be running from that horribly slow Wine thing ?... or does it happen on your real A8 ?

 

 

11 hours ago, VinsCool said:

Yeah I noticed it happened both in Altirra and real machine, but I assumed it was normal, most of the time it's unnoticeable too