Jump to content
IGNORED

NetBSD gains support for the Atari partition scheme


Recommended Posts

TL;DR I can transfer files to and from my UltraSatan's SD card even though the disk is NOT formatted with an ordinary MBR.


Hey gang! So, I'm a NetBSD developer. The other day, I was able to extend NetBSD's dk(4) driver to be able to recognize the old, plain TOS hard disk partition map. This allows me to mount the SD card from my UltraSatan, which was formatted with the ICD Pro driver, on a modern PC.


I made a video that explains it all, and you'll get to see a quick demo, too! The code is not committed to NetBSD just yet (pending some code review). If I stare at the code really hard then I *might* be able to get it working in Linux as well.

 

 

  • Like 5
  • Thanks 3
Link to comment
Share on other sites

Nice! Btw Linux did contain similar code but it is (AFAIK) either disabled or not longer working in recent kernels. So if you could improve that as well, that would be awesome!

  • Like 1
Link to comment
Share on other sites

8 hours ago, mikro said:

Nice! Btw Linux did contain similar code but it is (AFAIK) either disabled or not longer working in recent kernels. So if you could improve that as well, that would be awesome!

AFAIK, it was removed as the filesystems used by AWS/Cloud distros were being misidentified as TOS. Something like that :-)

Link to comment
Share on other sites

Posted (edited)

So I actually just went ahead and plugged the SD card into my Linux laptop (Debian/x86_64 "bullseye") and the 2 partitions showed up as /dev/mmcblk0p1 and /dev/mmcblk0p2, it Just Worked. That's Linux 5.10, though. Linux 6.8 is A Thing now so maybe something's happened in the meantime that broke support for it. I'd need to dig through the Git history.

 

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/block/partitions/atari.c

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/block/partitions/atari.h/

 

I have a theory. There unfortunately aren't any unique magic numbers to be found inside the TOS partition scheme, AFAICT. This means when the kernel is trying to determine "does this block device represent a TOS-formatted hard disk?" you can't give a definitive y/n answer, you kinda have to guess. My NetBSD code does this, even. If TOS support is removed like you've mentioned, then  my hunch is that the current code is prone to mis-identifying some non-Atari devices to be inappropriately identified as Atari, and the simplest fix is to just never consider TOS partitions altogether.

Edited by dressupgeekout
Link to comment
Share on other sites

  • 1 month later...
  • 6 months later...

Hi dressupgeekout

 

I've tried to test this. I've booted NetBSD-10.99.12-amd64-live.img on two different laptops now, both with built in SD card sockets, and NetBSD failed to detect or mount my Atari ICD partition on my SD card when plugged into both and when I tried using a USB SD card reader too.

 

Should your Atari partition code be present and enabled in the NetBSD 10.99.X test images or do we need to build the NetBSD kernel ourselves to use your code?

 

I'm trying to mount a 250 MB ICD BigGEM partition on a 32 GB microSD card. I have created and formatted the partition using ICD Pro 6.2. I'm also using a UltraSatan like yourself.

Should NetBSD be able to mount Atari partitions attached to a USB SD card adapter or will it only work with 'proper' SD card controllers, the sdmmc type?


This would be an infinitely better way for me to transfer files to my Atari vs zmodem or floppies if I can get it to work like in your video.

Thanks

Edited by danm
  • Like 1
Link to comment
Share on other sites

On 10/29/2024 at 5:10 AM, danm said:

Should your Atari partition code be present and enabled in the NetBSD 10.99.X test images or do we need to build the NetBSD kernel ourselves to use your code?

Hey, thanks for giving it a shot. Yes, you'll need to build your own kernel at the very least, because the Atari partition code is not enabled by default in the GENERIC configuration -- even the bleeding edge "10.99" version. You'll see here that `DKWEDGE_METHOD_TOS` is commented out.

 

https://github.com/NetBSD/src/blob/trunk/sys/arch/amd64/conf/GENERIC#L195-L202

 

Fortunately, building a custom kernel is very easy, and you don't even need NetBSD to do it.

 

https://netbsd.org/docs/guide/en/chap-build.html#chap-boot-cross-build-kernel

 

You'll know it's working when the console (or dmesg) shows that a dk(4) wedge has been assigned to it: you'll see a message that looks something like: 'dk0 at ld0: "ATARI_GEM_01"' or similar. It shouldn't matter that you're using a USB adapter to read the SD card.

I'm surprised that a 32GB card works in your UltraSatan, I was only able to get what I wanted with an old 4GB card.

(edit: it's a 4GB card, not a 2GB one.)

Edited by dressupgeekout
  • Like 1
Link to comment
Share on other sites

Thanks for the explanation dressupgeekout! I'll be building a NetBSD kernel with DKWEDGE_METHOD_TOS enabled and report back soon.

 

Does your driver support partitions bigger than 32 MB? I've read online that the Linux kernel used to support mounting Atari style partitions but someone said it only worked with GEM (up to 32 MB) but not BIGGEM (bigger than 32 MB) partitions? I've had no luck mounting Atari partitions with modern Linux kernels.

 

I would presume that DKWEDGE_METHOD_TOS will remain disabled by default for NetBSD 11 and later releases? Its hard to deny its is a pretty niche feature so I can't expect that they'll decide to enable this by default.

Have you got 4 MB RAM? Have you tried FreeMiNT on your ST? I have bash and vi(m) running nicely under FreeMiNT on my ST here. I'm pretty sure you need a Falcon or a TT to run NetBSD/Atari so FreeMiNT is the next best thing for us owners of lesser STs.

Edited by danm
Link to comment
Share on other sites

On 10/31/2024 at 2:28 AM, danm said:

Does your driver support partitions bigger than 32 MB? I've read online that the Linux kernel used to support mounting Atari style partitions but someone said it only worked with GEM (up to 32 MB) but not BIGGEM (bigger than 32 MB) partitions? I've had no luck mounting Atari partitions with modern Linux kernels.

My driver does support "BIGGEM" ("BGM") drives, although I must admit I haven't actually tested it myself. But, having looked through the "Atari Hard Disk File Systems Reference Guide," I don't see any reason why it wouldn't work. However, the driver currently does NOT support "extended partitions" ("XGM")

 

"GEM", "BGM" and "XGM" are identifiers in the root sector, see page 42: https://info-coach.fr/atari/documents/_mydoc/Atari_HD_File_Sytem_Reference_Guide.pdf
 

On 10/31/2024 at 2:28 AM, danm said:

I would presume that DKWEDGE_METHOD_TOS will remain disabled by default for NetBSD 11 and later releases? Its hard to deny its is a pretty niche feature so I can't expect that they'll decide to enable this by default.

Yeah. As a NetBSD developer, I could vouch for enabling it by default if I really wanted to, but I don't think that's wise. This old partitioning scheme frustratingly doesn't have any unmistakable markers or magic numbers. It doesn't make sense to risk having ordinary partitions be mistaken by NetBSD as an Atari partition just for the sake of the occasional Atari geek.

 

On 10/31/2024 at 2:28 AM, danm said:

Have you got 4 MB RAM? Have you tried FreeMiNT on your ST? I have bash and vi(m) running nicely under FreeMiNT on my ST here. I'm pretty sure you need a Falcon or a TT to run NetBSD/Atari so FreeMiNT is the next best thing for us owners of lesser STs.

No, my 1040STF is has the stock 1 MB of RAM. I don't think the newest versions of FreeMiNT will be useful with such little memory. And yes, from what I've read, TT030 is the real way to go to run NetBSD/atari if you can manage it. Despite NetBSD's commitment to retro platforms, NetBSD requires more RAM than most people realize.


https://wiki.netbsd.org/ports/atari/

  • Like 2
Link to comment
Share on other sites

I think I'll be fine without XGM. I just need to be able to mount a ~250 MB BGM partition.

 

That's a shame about the lack of magic numbers to id the Atari partitions.

 

4 MB is the minimum amount of RAM to run FreeMiNT but you can boot it straight into bash and run vi etc like that and I think its painfully cool for some reason, especially being able to run the same binaries on my Amiga and ST. It is very handy being able to mount DOS and ext2 partitions when you are using EmuTOS / FreeMiNT too.

 

I've spent a good few hours over the past couple of days trying to install NetBSD and build a kernel with your feature enabled. I was unable to get NetBSD amd64 10 or 10.99 to boot on my UEFI only laptop so I had to resort to installing NetBSD 10 i386 on my ancient Pentium 4 to get it to successfully (BIOS) boot.

 

`make depend` for the kernel worked but I've had a couple of errors trying to build the current kernel manually on my P4. I have not yet tried build.sh so maybe I'll have more luck cross compiling a i386 kernel under Linux? I thought I would have more luck by trying to build it manually under NetBSD itself but maybe not.

 

It would be great if you could upload a customised bootable NetBSD amd64 USB image somewhere, that way we wouldn't need to install it or bulld a new kernel to be able to use it. Or maybe just upload amd64 and i386 kernel images for those who can install NetBSD.

Thanks

Edited by danm
Link to comment
Share on other sites

I thought I'd check out build.sh to cross compile a NetBSD kernel but the docs are a bit vague.

 

You pointed me towards https://netbsd.org/docs/guide/en/chap-build.html#chap-boot-cross-build-kernel . It seems to indicate build.sh is in /usr/src so I think I found the same build.sh within ftp://ftp.NetBSD.org/pub/NetBSD/NetBSD-10.0/source/sets/src.tgz

However, when I run:
 

./build.sh list-arch



because I wanted to know the correct options to build an i386 kernel, I get the error:


 

ERROR: src/share/mk is missing; please re-fetch the source tree

*** BUILD ABORTED ***


 

Which NetBSD tarballs do I need to extract and where to use build.sh? I presume I don't need to download and extract xsrc.tgz if I only want to build a kernel?

 

You obviously can't just build.sh without extracting some other archives first. Can I basically extract the tarballs anywhere as long a I extract them all in the same dir?

I'm trying to run build.sh under Debian 12 amd64.

Edited by danm
Link to comment
Share on other sites

Yeah, you didn't fetch all of the sources. At the top of Chapter 33 of the NetBSD guide is a pointer to Chapter 32 which explains how to obtain all the source tarballs.

https://netbsd.org/docs/guide/en/chap-fetch.html#chap-fetch-tarball

 

You could also clone the Git mirror if you prefer. https://github.com/netbsd/src
 

3 hours ago, danm said:

Can I basically extract the tarballs anywhere as long a I extract them all in the same dir?

Yes, correct. It doesn't have to be `/usr/src`.

I will admit the documentation could be better. I anticipate your build.sh command line would look something like:

 

./build.sh -Uu -mi386 tools kernel=MYKERNEL
Link to comment
Share on other sites

On 11/5/2024 at 9:45 AM, danm said:

I've spent a good few hours over the past couple of days trying to install NetBSD and build a kernel with your feature enabled. I was unable to get NetBSD amd64 10 or 10.99 to boot on my UEFI only laptop so I had to resort to installing NetBSD 10 i386 on my ancient Pentium 4 to get it to successfully (BIOS) boot.

 

`make depend` for the kernel worked but I've had a couple of errors trying to build the current kernel manually on my P4. I have not yet tried build.sh so maybe I'll have more luck cross compiling a i386 kernel under Linux? I thought I would have more luck by trying to build it manually under NetBSD itself but maybe not.

 

It would be great if you could upload a customised bootable NetBSD amd64 USB image somewhere, that way we wouldn't need to install it or bulld a new kernel to be able to use it. Or maybe just upload amd64 and i386 kernel images for those who can install NetBSD.

Every build of NetBSD is a cross-build -- the instructions are actually pretty much the same no matter whether you're doing a native compile or a cross-compile. 'build.sh' internally takes a few shortcuts if your host OS is itself NetBSD, however.

You know, I think it does make sense for me to upload a pre-built kernel or two with DKWEDGE_METHOD_TOS enabled. That's probably the next best thing since I really can't enable the feature in NetBSD by default.

For what it's worth, I've also been inspired to upload some of my earlier work in this field: https://github.com/dressupgeekout/ahditools

 

Beware, it doesn't do everything that I want. But I think I could get back into the swing of things and make it work the way I originally intended!

Link to comment
Share on other sites

Thanks for telling me the exact build.sh command I should try. I'll give build.sh another go and see if I have any more luck.

 

It would be good if you could upload i386, amd64 and aarch64 kernel images and/or bootable NetBSD images. I didn't have any luck booting NetBSD 10 and current on my RPi4. I didn't spend long trying to troubleshoot it, I didn't get my USB UART adapter out etc. I didn't see anything out of my HDMI when I tried to boot it. Maybe the I'd have more luck with a RPi 3? The RPi machines are probably the platform that people are most likely to have around apart from PCs and Macs so its a shame I had no luck with my RPi 4.

 

I have successfully used this script to mount multi partition Atari hard disk images under Linux:

 

https://www.seniorlinuxadmin.co.uk/atari-fs.html

 

I don't think it would work under NetBSD because it uses losetup which I don't think has a NetBSD version. It is of course much nicer being able to mount the partitions directly (under NetBSD) rather than having to create images and use hacky scripts to transfer files.

 

Edited by danm
Link to comment
Share on other sites

It works!

I managed to successfully cross compile the NetBSD 10.99.12 i386 kernel under a Debian 12 incus container with DKWEDGE_METHOD_TOS enabled. I have tested this new kerne feature with a STFM running TOS 1.02 that I used to format a 250 MB ICD Pro BGM partition on a 16 GB Transcend SDHC card which I was able to successfully mount this as well as read and write files to it OK under NetBSD.

I have attached a gzip compressed NetBSD i386 kernel image in case anyone else manages to install NetBSD and would like easy access to their Atari hard disks. I tested it by plugging my Atari formatted SD card into a USB SD card adapter which I plugged into my Pentium 4 that I had installed NetBSD 10 i386 on. After installation of NetBSD 10 i386, I replaced the /netbsd kernel image file with the one attached simply by replacing the file. No updating of bootloaders is required when updating a NetBSD kernel, just a reboot.

 

It's good to know that we have a fully open source way to access Atari partitions on (not so) modern computers, at least those that can run NetBSD, without resorting to using hard disk images or special tools.

I have documented the process of how I cross compiled a custom NetBSD kernel.  I should probably get this guide tidied up, formatted and add it to the Atari wiki one day. The official NetBSD docs for cross compiling a kernel are here:

 

https://netbsd.org/docs/guide/en/chap-build.html#chap-boot-cross-build-kernel

 

Cross compiling a NetBSD (current) kernel under Debian Linux

 

Cross compiling NetBSD involves extracting several large tarballs into the /src dir. If you already have files within /src, then doing the following could be  risky and potentiallu make a mess of your file system or at least overwrite any files you may have stored under /src. For this reason it is recommended that you follow these instructions within a new container or virtual machine to avoid breaking, overwriting or making a mess of your main OS. I prefer to use incus, a fork of LXD, and Debian to create containers under Linux but feel free to use whichever container, hypervisor or VM program or OS you prefer.

 

Create a new incus container:

 

incus launch images:debian/12 Debian-NetBSD-cross

 

Log into it:

 

incus shell Debian-NetBSD-cross

 

Create a directory to store the NetBSD source tarballs and build files:

 

mkdir -p ~/src/netbsd && cd ~/src/netbsd

 

Install the Debian build tools, zlib dev files and the ftp command:

 

sudo apt update && apt install -y ftp build-essential zlib1g-dev

 

Download the NetBSD current source tarballs:

 

ftp -i ftp://ftp.NetBSD.org/pub/NetBSD/NetBSD-current/tar_files/src/

 

After logging in to the NetBSD ftp site, at the ftp> prompt, run:

 

mget *.tar.gz

 

Type `quit` to leave the ftp client after all the tarballs have transferred.

 

You can extract all the tarballs using a script like:

 

#!/bin/sh
for file in *.tar.gz
do
tar -xzf $file -C /
done

 

 

I coped this into a file called extract.sh then I ran:

 

sh ./extract.sh

 

Next we'll create a custom i386 kernel config file:

 

cd /src/sys/arch/i386/conf
cp GENERIC MYKERNEL

 

Edit MYKERNEL as required. In this case we want to enable Atari disk partition support so we uncomment the line:

options     DKWEDGE_METHOD_TOS

 

Now you can build the kernel:

 

/src/build.sh -Uu -mi386 tools kernel=MYKERNEL

 

If you had built an i386 kernel using a config called MYKERNEL you should find a netbsd kernel at:

 

/src/sys/arch/i386/compile/obj/MYKERNEL/netbsd

 

To copy the kernel out of my incus Debian container and onto my host machine I ran:

 

incus file pull Debian-NetBSD-cross/src/sys/arch/i386/compile/obj/MYKERNEL/netbsd .


All you have to do then is copy the kernel into place, ideally after backing up your old kernel first. On my pretty old i7 laptop with 16 GB RAM, it took just under 30 minutes to build the NetBSD toolchains and then just under 20 minutes to build the NetBSD kernel.

 

If you are using a FAT formatted disk to transfer the new kernel to your NetBSD computer, note that NetBSD has the funny quirk of enumerating "foreign" filesystems start starting from e, as opposed to starting from a like under Linux. So, if your USB drive is registered as sd0 when you plug it in, the command to mount the first partition on it under /mnt would be:

 

mount -t msdos /dev/sd0e /mnt

 

After copying the new kernel into place and booting into it, attach your Atari disk. You should see NetBSD print a kernel message that mentions ATARI_GEM_00 or similar. If so, you should be able to mount your first GEM partition using a command like:

 

mount -t msdos /dev/dk0 /mnt/

 

netbsd.gz

Edited by danm
  • Like 1
  • Thanks 1
Link to comment
Share on other sites

I installed NetBSD on an old Pentium 4 because it was the only PC I had to hand that could boot a BIOS type NetBSD install, as opposed to a UEFI based install of NetBSD. It might've just been something peculiar with my more modern laptop but more likely is that I was making a mistake with my disk partitioning. For some reason I was unable to get NetBSD 10 to boot on my UEFI only laptop so YMMV with NetBSD 10 if you can only boot via UEFI. Hopefully you'll have more luck than me.

You should be able to boot NetBSD i386 on your 64 bit PC/Laptop provided it can boot BIOS style, non-UEFI operating systems. Sometimes you need to enable some options called something like CSM aka Compatibility Support Module options in UEFI BIOSes that support MBR based OS booting. The Dell laptop I was trying to install NetBSD amd64 UEFI on doesn't have CSM support.

 

I'm so deep down the filesystems, booting and build systems rabbit hole now...

Edited by danm
Link to comment
Share on other sites

I'm glad you were able to get it to work! I'm actually very happy that my code really works for someone out there in the world who is not me.

18 hours ago, danm said:

I have successfully used this script to mount multi partition Atari hard disk images under Linux:

 

https://www.seniorlinuxadmin.co.uk/atari-fs.html

 

I don't think it would work under NetBSD because it uses losetup which I don't think has a NetBSD version. It is of course much nicer being able to mount the partitions directly (under NetBSD) rather than having to create images and use hacky scripts to transfer files.

Yes, that script is not portable. But NetBSD actually does have a feature just like losetup, it's called vndconfig(8). It lets you do things like:
 

# Mount the disk image file as if it were a real disk
vndconfig vnd0 MyDisk.img

# Confirm there are Atari partitions present
dkctl vnd0 listwedges

# Mount a partition
mount /dev/dk1 /mnt

 

and so on. I actually just used vndconfig to confirm for myself that partitions of type "BGM" actually do work with my driver. In Hatari, I used ICD Pro to format a disk (image) with a single partition 64 MB in size, and NetBSD correctly identified it as a big partition.

Furthermore, I was able to backport the DKWEDGE_METHOD_TOS code onto NetBSD 10.0. If anyone builds the kernel from my special branch below, then you can have this feature in a stable, official version of NetBSD rather than the bleeding-edge 10.99.

https://github.com/dressupgeekout/netbsd-src/tree/netbsd-10-release-w-dkwedgetos

  • Like 1
Link to comment
Share on other sites

Cool to hear you've backported it to NetBSD 10! Yes, I do think its useful as an UltraSatan owner. Let's see if anyone else is tempted to install NetBSD mainly for this feature?

 

I did find something online about vndconfig but I didn't try it out. It seems like its a bit easier to use than Linux's losetup, comparing your example command to the losetup equivalent. Seems there's probably no need for a dedicated script to mount Atari disk images under NetBSD if it only requires running those three commands.

 

I have tried NetBSD a few times before this but I'd never tried building a custom kernel no had I ever experimented with removable disks under NetBSD before today. Its great that we have a modern, open source UNIX that still supports m68k, powerpc, sparc and i386 etc, platforms which Linux has mostly or fully dropped support for.

 

I'd like to see the Netbsd installer gain ZFS root support, thats the big advantage that FreeBSD has had over it for some time for me.

Edited by danm
Link to comment
Share on other sites

Have you successfully installed NetBSD 10 on a UEFI PC? Did you have to modify the default suggested UEFI partitioning?

 

Am I right in thinking its only possible to build the NetBSD kernel by extracting the sources to the default location beneath /src? This is why I have suggested people use a container or VM , just in case they already have other stuff in /src that they don't want to risk being overwritten when extracting the NetBSD source tarballs.

 

I have updated https://www.atari-wiki.com/index.php?title=Linux_Disk_Commands with a new section about mounting TOS partitions that mentions NetBSD as a (non Linux but free software) alternative option to the script I posted previously and mtools but it is currently awaiting approval by the Atari wiki mods.

Edited by danm
Link to comment
Share on other sites

18 hours ago, danm said:

Have you successfully installed NetBSD 10 on a UEFI PC? Did you have to modify the default suggested UEFI partitioning?

Yes, I was able to get it to work, but unfortunately it requires that the user understand a few details about how UEFI actually works under the hood. It is too easy to make mistakes with the installer, in my opinion.

18 hours ago, danm said:

Am I right in thinking its only possible to build the NetBSD kernel by extracting the sources to the default location beneath /src? This is why I have suggested people use a container or VM , just in case they already have other stuff in /src that they don't want to risk being overwritten when extracting the NetBSD source tarballs.

Did you mean to say '/usr/src' ? Either way, no, that is simply not true. As long as all of the source code is extracted into the same place, 'build.sh' can be executed from literally anywhere -- and as an unprivileged user, too. If it doesn't work as advertised, then it's a bug.

 

Neither of these are relevant to Atari matters, though.

Edited by dressupgeekout
  • Like 1
Link to comment
Share on other sites

None of these are relevant to Atari matters but I wanted my mini guide above to cover stuff all these little gotchas that the official NetBSD docs don't mention, like foreign (FAT) disks starting enumeration from device e instead of a. This point seems to be totally missing from the NetBSD guide and hence all the Linux users would be trying to mount their disks on /dev/sd0a, which is what it would be logically using NetBSDs own device naming scheme. There should be a section in the NetBSD guide on how to mount USB/FAT disks that explains this quirk IMO as FAT disks are still very common. Most people of this forum might have some Linux experience but have likely never used NetBSD before so these are the sorts of things I wanted to note.

Its a shame that the NetBSD installers support for UEFI isn't better. Its fair enough that the user should be expected to know how UEFI works IF they want to multi boot different OSs from the same disk but when you let the NetBSD installer use a whole disk and delete/replace all of that disks partitions with ones it creates itself then a UEFI install should "just work", presuming most of the relavant hardware is supported by the kernel.

Good to hear you can install the source files anywhere. This isn't made clear in the official docs as the extraction script they give tells you to untar them all to /, hence my recommendation to do this within a VM or container. The current tarballs extract themselves into /src when using the suggested commands.

I shall stop with the OT waffle now as none of this is your fault. Thanks for your help dressupgeekout!

Edited by danm
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...