Jump to content
IGNORED

Removers Library - a.out Binary Emulation Dropped From Linux Kernel


BitJag

Recommended Posts

23 hours ago, lachoneus said:

@ggn I don't know if you have been following this, but does the bug that Seb mentions in the above post apply to rln as well?

I don't think so. At least I use rln to link Raptor Basic+ binaries which consist of files output by gcc, plus plain assembly files without any issues that we've come across lately. Did you switch to Seb's linker because you came across some bug in rln? (sorry, I kind of lost the discussion after one point!)

Link to comment
Share on other sites

14 minutes ago, ggn said:

I don't think so. At least I use rln to link Raptor Basic+ binaries which consist of files output by gcc, plus plain assembly files without any issues that we've come across lately. Did you switch to Seb's linker because you came across some bug in rln? (sorry, I kind of lost the discussion after one point!)

No worries.

 

Using rln to link my program and then using the open_console() function from the remover's library results in rln failing (see attached build log).  Seb addressed this problem by having me switch to jlinker, and this seems to the resolve the issue with the open_console() function.  But now I am having issues with programs that playback MOD files crashing in virtual jaguar, when I use rmac/jlinker tool chain.  If I comment out the functions for playing modules, the program runs fine.

 

Comparing this with the rmac/rln toolchain, building the same program with the MOD music results in a successful build, and it runs just fine in virtual jaguar.

 

In short, at the moment I have to choose between having music (rmac/rln) or having the console functions (rmac/jlinker).  It seems that rln is the problem when it comes to the open_console() function, and it seems that jlinker is the problem when it comes to playing back MOD files. 

 

In regards to jlinker not liking MOD file playback anymore. The only guess I can  make is that we made modifications to some assembly files pertaining to the routines to play MOD files in the removers library, in order for rmac to successfully build sound.s, paula.s and display.s.  And these changes, when using jlinker, some how break the program when the program attempts to use functions for MOD file playback.  Or maybe rmac compiles the assembler files into object files in such a way that rln can link correctly, buy jlinker cannot for some reason.

 

I am not sure, its just strange that rln can handle the modified rmvlib files just fine, but jlinker cannot, assuming that the modified assembly files are the issue.

open_custom_console_bug_build_log.txt

Link to comment
Share on other sites

Ok, first of all sorry to @lachoneus for such a huge time gap with this issue. I thought you'd sorted the problem out so I didn't pursuit the matter further :(. If I knew it was an issue with the linker I'd have looked into it much earlier. Anyway!

 

I must say I'm really not in touch with the rln source code but I gave it a shot anyway. I think I fixed the issue but it remains to be verified by you for correctness. If that works for you we'll check it into the rln source tree properly. So without further ado:

 

Open rln.c, go to line 2460. It should read:

 

        if (objName[0] == 0x20)

Change that into:

 

        if (objName[0] == 0x20 || objName[0]=='/')

 

Recompile and try linking with rln. I'm sure it should link without errors now, but if this a) will be the case for you too, b) works as advertised.... it remains to be seen.

 

So give this one a try and let us know what happens!

 

(for the curious: rln was skipping object files whose filename was bigger than 16 characters. Anyone from the linux camp who mocks Bill Gates for the infamous 640k quote..... I call that pot, kettle, black!)

Edited by ggn
  • Like 1
  • Thanks 1
Link to comment
Share on other sites

@ggn The console functions in rmvlib seem to be working with your latest change rln.  Thank you!

 

Unfortunately, it seems that rln is manifesting the same problem as jlinker when it comes to playing back mod files (this is probably a big assumption to make, but it is an interesting coincidence that they are both having problems with mod file playback). I have spent the last couple hours exploring this to make sure that it wasn't a problem with my C code.

 

To make sure that it was just amiga module playback code that was causing the problem, I created a simple program that focuses module playback only.  I have built and tested this same program code in 3 completely independent tool chains to ensure that there isn't something wierd being share between tool chains or some other unknown causing problems from previous builds.  Here are the results.

 

rln build - atari-mint-cross-compiler/rmac/rln/jlibc/rmvlib(modified for rmac's sake) - program builds, but crashes when attempt to run in virtualjaguar or on real hardware

 

jlinker build - atari-mint-cross-compiler/rmac/jlinker/jlibc/rmvlib(modified for rmac's sake) - program builds, but crashes when attempt to run in virtualjaguar or on real hardware (the behavior on real hardware is a bit different from the rln build. After uploading to the skunkboard with jcp, the screen goes black for a second, but then drops back to the green skunkboard screen.  The rln build stops on the black screen and the skunkboard needs to be reset manually)

 

aln build - seb-built-gcc-m68000-crosscompiler/madmac/aln/jlibc/rmvlib(unmodified) - program builds, and plays as expected in virtualjaguar and on real hardware

 

I have included the COF files for byte comparison and testing on your end.  I have also attached build logs for each program for each build environment.  I have also included an archive with the source code for my program so it can be quadruple checked by someone else other than myself. 

 

I have also included my tool chain setup script, in case there is a problem there.  To build rln vs. jlinker is a matter of commenting in/out a few lines, these sections are clearly marked if you want to try running this yourself.  I don't have a script to build the old aln environment, if you want this, I can provide a link and instructions for setting up this environment.

 

Please let me know if you need anything else from me to hunt down the problem, and thanks again for your time.

 

 

@SebRmvPinging Seb on this as well.  Thanks again Seb for helping out with this.

 

aln_build_log.txt jlinker_build_log.txt rln_build_log.txt aln_modExample.cof jlinker_modExample.cof rln_modExample.cof mod_example.zip RMAC_RLN_JLIBC_RMVLIB_Setup_Scripts_For_Distribution.zip

Link to comment
Share on other sites

Okay, there's something really screwy happening here.

 

I started comparing your aln linked binary with the rln one, going object file by object file. When modExample.o is linked in, the .cof files start to diverge massively. It's not just some offsets plugged wrong, we're talking about whole instructions inserted and code shifted around. I even spotted a NOP instruction in there (if it's not data, hard to tell at the moment). Looking at the command lines you use to build each binary I see you're using different builds of gcc (m68k-atari-mint-gcc vs m68k-aout) which is probably responsible for this. In order to have an apples-to-apples comparison you will need to use the same compiler otherwise there's so many different things between the two .cof files that the issues will be lost inside the noise.

 

Also I see that you use the same compiler (Vincent's) for both rln and jlinker binaries. So it could be that this is related.

 

I guess you can use the exact same .o files that you use on your aln binary with rln and jlinker and see what happens. If that works then we can focus on the compiler doing something weird.

 

[EDIT] Just for the record I ran your script that installs and patches all the tools necessary, but I think that didn't include mk68-aout gcc

Edited by ggn
  • Like 1
Link to comment
Share on other sites

I took the object files from my old tool chain m68k-aout/mac/aln and tried linking them with rln and jlinker.  Both cases seem to fail.  I have uploaded both build logs and the object files, so you can double check my work and/or in case you want to give it a try yourself.

 

While I was doing this, I had a thought about my old tool chain.  My old tool chain was running an old version of jlibc/rmvlib, (jlibc 0.5.9 & rmvlib 1.3.6).  So I downloaded and rebuilt both libraries and with m68k-aout/mac rebuilt the newest versions of the libraries and then rebuilt mod example program, and it works fine.  I did this thinking that the newer version of jlibc/rmvlib might be part of the problem, which is not the case.

 

Since we are thinking it is the cross compiler, do you need the binaries for gcc from both tool chains (Vincent's & m68k-aout) for comparison?

mod_example_m68k-aout_object_files.zip

Link to comment
Share on other sites

6 hours ago, ggn said:

Just for the record I ran your script that installs and patches all the tools necessary, but I think that didn't include mk68-aout gcc

The old tool chain, that I have been using for about a decade now, is not built from scratch each time I reinstall my OS or move to a new system.  I stole this already built tool chain from and Michael Hill's old image and have been using it since.  I simply archive the whole thing since it is in one folder, and then unarchive it on the new system.  This is because I haven't taken the time yet to learn how to build my own cross compiler.  It looks like the day that I will need to learn to do this is coming sooner rather than later...  

 

The script I sent you is just for Vincent's cross compiler, and then the environment setup and library/binary build for rmac/rln/jlinker/jlibc/rmvlib.

Link to comment
Share on other sites

Ok, thanks! I'll try these out and see if they give a hint. I wouldnl't expect them to produce a proper executable because my money is on rmac/jlinker having bugs (if they can't link such a "trivial" example). Using the same object files will simply help me with byte comparing the .cof files.

Link to comment
Share on other sites

Right, so I spent some time on visual studio and diffing the object files as they're linked in, comparing them with the aln binary. Apart from objects being generated using different gcc I couldn't find anything bad (although it was hard to judge). So then out of curiosity I dragged the produced .cof file to virtualjaguar.

 

And that... worked? I then dragged the aln binary to compare, and sure enough, that also works the same way as the rln linked one. I then tried with both modExample.o variants, and they also seem to work fine.

 

So... I'm not sure what to assume here! At least I'm glad rln works fine, but why doesn't it work for you?

 

To try and figure out what's happening, I uploaded my build folder here. It contains the aln and rln coff files so you can firstly check out and see if they work for you. Then you can try building the coff yourself. To build it here I used:

 

rln -a 4000 x x -rq -v -v -v -w -e -o modExample.cof crt0.o modExample.o  gfx_data.o data_def.o sound_data.o display.o interrupt.o sound.o rmvlib.a fb2d.o lz77.o jlibc.a libgcc.a

 

Try that out and see if it produces an identical binary as the one in the .zip file. To switch between the two modExample.o files I keep them both under another filename and use something like "cp modExample_new_gcc.o modExample.o" or "modExample_old_gcc.o" and then re-invoke rln to save me some time.

 

I am now very curious to read your findings!

 

build.zip

Link to comment
Share on other sites

Thank again for looking into this.

 

Everything you gave me builds and runs fine.  I mimicked your process of collecting all the built .o files into a single folder and then manually linking them to see if my Makefile's might have a problem. I did this with my rmac/rln toolchain.  All the .o files and all the .a files are produced with rmac and vincents gcc.  After doing this, I couldn't get a working program.

 

So, I did a byte for byte comparison with the .o files you uploaded and my latest build of jlibc and rmvlib (built with rmac).  The following files are different.

 

jlibc.a

rmvlib.a

sound.o

protracker.o

 

I replaced these one at a time, with a .o/.a file made with mac (not rmac), and then tried linking and running the program.  It turns out that the only file I need to replace to get a working program is sound.o.  The sound.o file that rmac produces seems to be at least part of the issue.  I don't know if this is because we need to manually modify sound.s to get rmac to build the object file or not, or if this is something to do with rmac itself.  Its strange that I don't need to switch out display.o as well, since we make manual changes to that file to get it to play nicely with rmac as well.

 

I am not sure where you sourced the sound.o file you used in the archive you uploaded, because it doesn't seem to match any of the rmac built files or mac built sound.o files that I have.  I need to double check this though.

 

I uploaded an example with all the .o/.a files, with two sound.o files.  One compiled from rmac, and one compiled mac (they are marked as such in the filename).  I used the same linking command that you posted, but just switched out which sound.o to point to.  All the other .o/.a files were built using the latest version of rmac, and with vincent's gcc cross compiler (a.k.a. all the new stuff).

 

Thanks again for your help on this, it seems we are closing in on the issue.

rmac_vs_mac_sound_object_file.zip

Link to comment
Share on other sites

Wow, that was a bizarre trip for me.

 

I'll really spare you the details as it was mostly me fumbling at random and remembering bits of what I did a couple of months ago, so here it is: another tentative rmac fix!

 

In a freshly checked out rmac copy, open file object.s and go to line 176. There should be an "else" in that line, followed by a switch/case construct. Simply delete the "else" and rebuild. I've only tested this with sound.o and not a full rebuild of the removers' lib, but I'm hoping it should be fine :).

 

As for what sound.o I originally used - I can't really say! I was copying object files around from your zip files and probably got lucky :).

 

 

So anyway, here's hoping that this fixes everything and we can put this thread into the history closet :)

  • Like 1
Link to comment
Share on other sites

Everything so far seems to be working great.  I will keep testing jlibc/rmvlib functions to make sure there aren't too many leaks in the ship :)

 

I also tested linking the exmaple program with jlinker with this modification for rmac, and it is working as well.

 

Here is the git repository with the scripts I use to set everything up automatically, in case anyone else is interested in testing it out.

https://github.com/lachoneus/ubuntu-rmvlib-install-scripts

 

@ggn and @SebRmv , thank you, thank you for your help and your time! 

  • Like 1
Link to comment
Share on other sites

4 hours ago, SebRmv said:

Ok, so everything is fine now? Great, thanks to you

It seems so.  I am going to be doing allot more work in the next few months with Crescent Memories before release with this tool chain.  It doesn't cover every facet of rmvlib, but it should be a good opportunity to see how well everything is working between rmac/rln/jlibc/rmvlib.

 

The setup script still has to make several modifications to several source files in rmvlib.  I am fine doing these changes when I run the script, but would it hurt to make these changes to rmvlib at the source?  These manual changes are executed between lines 112 - 127 in my setup script, you can see those here https://github.com/lachoneus/ubuntu-rmvlib-install-scripts/blob/master/rmvlib_install.sh . I don't expect this to be changed in rmvlib for my sake, but I wanted to see if this would be something you would be willing to change before we put this thread to bed.

 

30 minutes ago, ggn said:

Ok! So the patch has been peer reviewed, approved and pushed to the main rmac tree. So no hacks needed now!

Just made the changes to the setup script and ran it.  Working great!

 

Like my request to Seb above, there is still one manual modification that is made to rmac's expr.c file to get rmac to play nice when building rmvlib.  This was one of the first adjustments that we made when this thread started, and I wanted to look into the possibility of making this a permanent change to rmac, if it doesn't cause any problems for anyone else or the long term development of rmac.

 

At line 55 in the setup script it changes line 467 in expr.c from.  I believe this change was implemented to accommodate the changes made to rmvlib that I mention above to Seb.

 

*a_attr = cursect | DEFINED;

to

*a_attr = DEFINED;

 

Thanks again!

 

 

Link to comment
Share on other sites

Long story short, this is just the tip of the iceberg for that issue. The fix I have provided you there is just a kludge in a long series of kludges on that part of the assembler. Thus, it has grown organically to a point that nobody fully grasps how it exactly works. Therefore nobody wants to touch it. So our next big task is to burn everything to the ground and rewrite that function (we opened a ticket for it here). We want to make it happen but we simply lack the time/courage for now.

 

So unfortunately you either have to patch the offending sources in the library or path the assembler itself (personally I'd go for the first option).

  • Like 1
Link to comment
Share on other sites

  • 2 years later...

FYI, the remaining a.out support in the Linux kernel is on the chopping block. I've noted the MAC and ALN use case on the Linux Kernel Mailing List (LKML):

 

https://marc.info/?l=linux-kernel&m=164710765611526&w=2

 

Don't want to knock all the great work on rmac and rln, but mac and aln still do a few things they don't, and they are the only way to compile some old Atari source code without modification. Feel free to chime in if you still use ALN or MAC. As noted in my LKML message, I have patches locally that fully re-enable a.out that I'll submit if there's interest upstream.

  • Like 1
Link to comment
Share on other sites

2 hours ago, 42bs said:

Why not just use an old kernel with a.out support for old tools in a.out format in a VM? If it is only to compile old sources?

Just because of the trouble. I have all my other files and stuff on my regular machine. If I have to boot a VM and copy it all over there, or reboot my machine (I generally code on a machine I use as a general purpose server, so I'd be killing a bunch of other stuff like my local audio and video streams) just to see if I've found a bug in rmac/rln or whatever, I'm liable to just give up and have a beer instead.

 

That said, I don't know what the kernel folks will say. They're likely to have similar suggestions if the code is actually causing them any trouble.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...