Compiling Altirra from source

[ivop]

Now that @phaeron released his latest 4.00-test43 source code, including the RMT libraries, more than ever, I would like to compile Altirra from source. Instead of asking Phaeron directly, I thought it would be nice to have a compilation guide here on AtariAge

 

Has anybody, besides Phaeron, ever compiled it from source?

Which compiler do I need?

Is it free?

Does it run on WINE?

How hard do you think it would be to cross-compile with i686-w64-mingw32-gcc?

 

 

Edit: answering two of my own questions:

 

There's a free version of Visual Studio at https://visualstudio.microsoft.com/free-developer-offers/

 

Does it work on WINE? https://appdb.winehq.org/objectManager.php?sClass=application&iId=892

NO!

 

 

Edit2: found two vcproj2cmake projects (1, 2) but both are very old. Now that VS integrates CMake more and more (https://docs.microsoft.com/en-us/cpp/build/cmake-projects-in-visual-studio?view=msvc-160, would it be possible to generate them from within VS? 

Edited October 3, 2021 by ivop

[ggn]

Hi,

 

Phareon has instructions inside the source code, just open src\BUILD-HOWTO.html.

 

But to answer your questions: Yes, for the main compiler you do need Visual Studio 2019, it's free, and it will not compile with other compilers currently (gcc/mingw/clang etc). I think your best bet would be to spin a VM and compile under that.

[phaeron]

If you want to try to cross-compile the RMT plugins, you're on your own, but I can try to provide some context.

 

Any project or documentation that refers to vcproj instead of vcxproj is ancient. The .vcproj format is the old project format that was only used through VS2008 to they rewrote the C++ project system on top of MSBuild in Visual Studio 2010. There's practically nothing in common between the formats and old converters can't be used.

 

Altirra's build uses .vcxproj and thus MSBuild-based, and there are customizations to reduce the boilerplate that VS normally dumps into the project files. I don't believe that the Visual Studio CMake integration attempts to convert, or at least not anything beyond the simplest configurations. The good news is that only a subset of the project needs to be built for the RMT plugins, so some of the trickier ones can be avoided. You will still need YASM integration for the system library. Beyond that, it's just compile a few static libs, and then two DLLs.

 

The source will mostly compile under Clang in ms-compatibility mode as set up by clang-cl. There are a couple of places where Clang is not compatible with Visual C++ with regard to Windows-specific calling conventions and intrinsics, but I think those might only be in places that aren't used for the RMT plugins. The catch is that IIRC the default Clang for Windows toolchain still uses the MSVC runtimes and still needs the VC++ toolchain and Windows SDK to build. You also need a recent version of Clang because Altirra uses features up to C++17.

 

MinGW is practically guaranteed not to work, as it's based on GCC, which insists on doing many things the GCC way even when building for Windows. It also frequently has out of date Windows API headers.

 

[rensoup]

6 hours ago, ivop said:

Has anybody, besides Phaeron, ever compiled it from source?

Which compiler do I need?

Is it free?

Yep, I did a while back, with Virtualbox+ Win10 + Visual Studio. I can't remember which but one of the MS product had a 30 days limit.

For win10. You don't need a hack or anything but you must skip some registration step (the option appears at some point during install)

I had to use the online VS install.

[ivop]

7 hours ago, phaeron said:

If you want to try to cross-compile the RMT plugins, you're on your own, but I can try to provide some context.

Thanks you. So, as I thought, automatic conversion is out of the question. Generating CMake files is not possible or not mature enough. And even if it was, cross compiling with clang or gcc is mostly impossible

 

3 hours ago, rensoup said:

Yep, I did a while back, with Virtualbox+ Win10 + Visual Studio. I can't remember which but one of the MS product had a 30 days limit.

For win10. You don't need a hack or anything but you must skip some registration step (the option appears at some point during install)

I had to use the online VS install.

I think this will be an easier way. Virtualbox I'm familiar with (and qemu, vmware, ...), but I have to look where to legally download these Microsoft products (OS and VS).

Edited October 4, 2021 by ivop

[rensoup]

22 hours ago, ivop said:

I think this will be an easier way. Virtualbox I'm familiar with (and qemu, vmware, ...), but I have to look where to legally download these Microsoft products (OS and VS).

? microsoft.com ?

[dmsc]

Hi!

Quote

Thanks you. So, as I thought, automatic conversion is out of the question. Generating CMake files is not possible or not mature enough. And even if it was, cross compiling with clang or gcc is mostly impossible

 

I think this will be an easier way. Virtualbox I'm familiar with (and qemu, vmware, ...), but I have to look where to legally download these Microsoft products (OS and VS).

In fact, Visual Studio command line tools *do* run on Wine, it is the installer and the graphics tools that don't run.

 

You can:

- Download a Windows 10 plus Visual Studio VM from Microsoft: https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/

- Mount the image under Linux, and copy the visual studio command line tools - look under "Program Files/Microsoft Visual Studio/2019/BuildTools/" for the binaries and "Program Files/Windows Kits" for the headers and libraries.

- Look for the "vcvars.bat" command, it sets the environment variables depending on the target architecture and windows version. Alternatively, just open a command visual-studio command prompt on windows and copy all the variables.

 

A while ago, I managed to compile a chunk of Altirra using VS2017 on Linux, I made the following bash script to run the compiler setting the correct variables:

#!/bin/bash

# Set to Linux path for the compiler root:
LNX_VSPATH='/opt/ms/vs2017'

# The same path as seen by wine:
VSPATH='Z:\opt\ms\vs2017'

VCPATH="$VSPATH\\VC\\14.12.25827"
KITPATH="$VSPATH\\Kit\\winv6.3"
UCRTPATH="$VSPATH\\Kit\\10.0.10240.0\\ucrt"

INCLUDE="$VCPATH\\include;$KITPATH\\include;$UCRTPATH\\include"
LIB="$VCPATH\\lib;$KITPATH\\lib;$UCRTPATH\\lib"

# Suppresses "0009:err:msvcrt:demangle_datatype Unknown type s"
# not needed on all wine versions
WINEDLLOVERRIDES=api-ms-win-crt-conio-l1-1-0,ucrtbase=n

export INCLUDE
export LIB
export WINEDLLOVERRIDES

exec wine "$LNX_VSPATH/VC/14.12.25827/bin/cl.exe" "$@"

 

Now, you can execute the compiler from the command line:

$ vs2017-cl
Microsoft (R) C/C++ Optimizing Compiler Version 19.12.25834 for x86
Copyright (C) Microsoft Corporation.  All rights reserved.

usage: cl [ option... ] filename... [ /link linkoption... ]

 

I copied just what was necessary, about 300MB in total from the 20GB image.

 

Have Fun!

 

EDIT: Also, I just googled this, looks easy to use: https://github.com/mstorsjo/msvc-wine

 

Edited October 5, 2021 by dmsc

[ivop]

I have looked into docker. Well, it'll be a cold day in hell before I run that stuff as root. And adding myself to the docker group is a huge security risk. Basically it makes your user account almost root.

 

Then, as suggested on their site, I tried running:

./vsdownload.py --dest <dir>

under a test account, but it kept crapping out with Python errors. I hate Python with a passion

 

So, next I'll look into @dmsc's suggestion, and otherwise I'll just run Microsoft's VM under VirtualBox. Good to see they supply these images, fully prepared for development.

Edited October 6, 2021 by ivop

[dmsc]

Hi!

On 10/6/2021 at 12:40 PM, ivop said:

I have looked into docker. Well, it'll be a cold day in hell before I run that stuff as root. And adding myself to the docker group is a huge security risk. Basically it makes your user account almost root.

 

Then, as suggested on their site, I tried running:

./vsdownload.py --dest <dir>

under a test account, but it kept crapping out with Python errors. I hate Python with a passion

Had a little time today - I suspect the error is that you ran it under python-2, but the script is for python-3, just try (for Debian or Ubuntu):

sudo apt install python3-simplejson python3-six msitools gcab

python3 vsdownload.py --cache $(pwd)/cache/ --dest  /opt/vs2019/

I added the "cache" option so the script does not try to download all again if it fails...

 

To install a somewhat smaller set of packages, I tried this:

python3 vsdownload.py --dest /opt/vs2019 --skip-recommended --cache $(pwd)/cache/  \
                      Microsoft.VisualStudio.Workload.VCTools \
                      Microsoft.VisualStudio.Component.VC.Tools.x86.x64 \
                      Microsoft.VisualStudio.Component.Windows10SDK.19041

sed -i -e '35,36 s/ arm arm64//' install.sh 

bash install.sh /opt/vs2019

The above installs over 3GB of data, including multiple compilers (CL and CLANG), on 32 bit and 64 bit x86 hosts, for multiple targets (32 and 64 bits, x86 and ARM), debuggers, .NET, etc. A lot of those could be removed.

$ vs2019/bin/x64/cl
Microsoft (R) C/C++ Optimizing Compiler Version 19.29.30136 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

usage: cl [ option... ] filename... [ /link linkoption... ]

 

Have Fun!

 

[ivop]

Indeed, my /usr/bin/python points to python2. So I tried python3:

 

Spoiler

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "vsdownload.py", line 412, in _downloadPayload
    six.moves.urllib.request.urlretrieve(payload["url"], destname)
  File "/usr/lib/python3.7/urllib/request.py", line 276, in urlretrieve
    block = fp.read(bs)
  File "/usr/lib/python3.7/http/client.py", line 461, in read
    n = self.readinto(b)
  File "/usr/lib/python3.7/http/client.py", line 505, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.7/ssl.py", line 1052, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.7/ssl.py", line 911, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "vsdownload.py", line 577, in <module>
    downloadPackages(selected, cache, allowHashMismatch=args.only_download)
  File "vsdownload.py", line 390, in downloadPackages
    downloaded = sum(task.get() for task in tasks)
  File "vsdownload.py", line 390, in <genexpr>
    downloaded = sum(task.get() for task in tasks)
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
socket.timeout: The read operation timed out

 

Before I also got the occasional:

 

URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
URLError: <urlopen error [Errno 101] Network is unreachable>
timeout: The read operation timed out

 

$ cat /etc/debian_version

10.10

 

My network and DNS are fine.

 

Thanks for the --cache option BTW. That saves a ton of time.
 

Edited October 9, 2021 by ivop