Unit testing VCS programs

[DirtyHairy]

This may seem a little bit weird, but a few colleagues of mine and me have been experimenting with unit testing VCS programs. The backstory:

 

A few weeks ago, we had a coding challenge during our slacktime --- implementing a binary clock. For the laughs, I decided to propose a VCS implementation, and that way, a colleague and me ended up implementing a crude binary clock on the VCS. He didn't have any experience with the VCS before, and we ended up with a decent result after a few hours. After that, we wondered about possibilities to unit test our code.

 

The last two days, we had slacktime again, and we put together a way to do unit tests for VCS code using 6502.ts. The tests are written in Typescript (a typed variant of javascript), and the code is assembled during testing using a javascript build of DASM that was done by @zeh. The tests run the binary in 6502.ts and can modify variables, jump to labels, wait until execution has reached a certain label and assert the contents of memory. The addresses are taken straight from the symbols in the assembly.

 

A typical test looks like this:

 

setup(async () => {
    runner = await VcsRunner.fromFile(path.join(__dirname, 'bitclock.asm'));
});

test('handles day-change properly (unit test)', () => {
        runner
            .boot()
            .cld()
            .jumpTo('AdvanceClock')
            .writeMemoryAt('hours', 23)
            .writeMemoryAt('minutes', 59)
            .writeMemoryAt('seconds', 59)
            .writeMemoryAt('frames', 49)
            .runUntil(() => runner.hasReachedLabel('ClockIncrementDone'));

        strictEqual(runner.readMemoryAt('hours'), 0);
        strictEqual(runner.readMemoryAt('minutes'), 0);
        strictEqual(runner.readMemoryAt('seconds'), 0);
        strictEqual(runner.readMemoryAt('frames'), 0);
    });

and the result looks like this:

 

 

At the moment, there is no (easy) way to do more specific assertions against the hardware, but adding those would be pretty trivial (register values, sprite positions, etc.). Input could be covered by providing helpers to trigger joysticks, paddles and switches at specific cycles.

 

If you like to check it out, the code lives on github: https://github.com/6502ts/mayflower-binaryclock . If you feel adventurous, you can install nodejs and yarn and then do "yarn install" and then "yarn test" on the command line to run the tests. This should work on any OS, but the specifics of installing those packages vary, of course.

 

I am curious: is this something that you would consider using for developing homebrews? What testcases would you consider, and what assertions would you like to have?

Edited December 5, 2020 by DirtyHairy

[+Andrew Davie]

Interesting.

 

I'd pepper my whole code with what are effectively "ASSERT(...)" statements, like I do with my high-level language coding.

Never make an assumption when you can verify something is actually the case.

 

It would be useful for making sure variables are within valid ranges, that they're not stomped, that parameters to functions are within valid ranges (without having to actually check them in the code)...    

Would be nice to have conditional watches on values of labels.
i.e, run until label == value, or until label is accessed.

 

[scrottie]

Absolutely.  I was doing something similar for https://github.com/scottwalters/2600lava before wound up working too much and not being able to work on fun things.  In that case, making sure that computations checked the timer frequently enough to not overshoot was critical, so I was doing my own cycle counting in the single-stepper callback of the 6502 emulation.  But that also made it a lot easier to get every kind of game logic right.  Highly recommended.

[scrottie]

Oh, and same thing... no hardware registers unless I implemented, and the only thing I implemented at all was the timer.

[DirtyHairy]

On 12/5/2020 at 1:26 PM, Andrew Davie said:

It would be useful for making sure variables are within valid ranges, that they're not stomped, that parameters to functions are within valid ranges (without having to actually check them in the code)...   

I have added a 'trap' method that allows to add assertions that are executed every time a trap condition is met:

    test('frame size is 312 lines / PAL', () => {
        let cyclesAtFrameStart = -1;

        runner
            .runTo('MainLoop')
            .trapAt('MainLoop', () => {
                if (cyclesAtFrameStart > 0) {
                    assert.strictEqual(runner.getCpuCycles() - cyclesAtFrameStart, 312 * 76);
                }

                cyclesAtFrameStart = runner.getCpuCycles();
            })
            .runTo('MainLoop')
            .runTo('MainLoop');
    });

The condition can be specified as a function instead of a label, which allows pretty much arbitrary conditions.

 

The emulator has an event system, and events are emitted every time the bus is accessed. In principle, this allows to register handlers during testing that throw an Exception when an undesired access happens. Currently, this requires knowledge of the API of the underlying emulation components, but we can add a friendlier API for using those events on the runner.

 

On 12/5/2020 at 1:26 PM, Andrew Davie said:

Would be nice to have conditional watches on values of labels.
i.e, run until label == value, or until label is accessed.

That's already possible. If you look at 'runUntil' in my first example, the first argument is a function. This function is executes after every instruction and can query arbitrary conditions. An example would be

   test('it takes 50 frames to count one second', () => {
        let frameNo = 1;

        runner
            .runTo('MainLoop')
            .writeMemoryAt('hours', 0)
            .writeMemoryAt('minutes', 0)
            .writeMemoryAt('seconds', 0)
            .writeMemoryAt('frames', 0)
            .trapAt('MainLoop', () => frameNo++)
            .runUntil(() => runner.readMemoryAt('seconds') === 1, 100 * 312 * 76);

        assert.strictEqual(frameNo, 50);
    });

Somehow I got the feeling that the examples are becoming a little bit contrieved ? I feel that I might have to develop a game (or collaborate with someone) in order to further develop this.

On 12/5/2020 at 1:26 PM, Andrew Davie said:

I'd pepper my whole code with what are effectively "ASSERT(...)" statements, like I do with my high-level language coding.

Never make an assumption when you can verify something is actually the case.

An interesting possibility would be a small DSL for expressing assertions that would allow embedding assertions as comments into the code. The runtime could read the assertions from the DASM listing file and execute them whenever the line following the assertion is executed.

Edited December 6, 2020 by DirtyHairy

[DirtyHairy]

On 12/5/2020 at 8:41 PM, scrottie said:

Absolutely.  I was doing something similar for https://github.com/scottwalters/2600lava before wound up working too much and not being able to work on fun things.  In that case, making sure that computations checked the timer frequently enough to not overshoot was critical, so I was doing my own cycle counting in the single-stepper callback of the 6502 emulation.  But that also made it a lot easier to get every kind of game logic right.  Highly recommended.

Nice, so there is a 6502 emulator in Perl ?

On 12/5/2020 at 8:42 PM, scrottie said:

Oh, and same thing... no hardware registers unless I implemented, and the only thing I implemented at all was the timer.

The nice thing here is that 6502.ts is a full-fledged VCS emulator, so the information is already there. I only need to add a proper API to extract them from the emulator during testing.

Edited December 6, 2020 by DirtyHairy

[EvoMikeUK]

It would be great to create a NUNIT style framework for Atari Dev Studio and bBasic.