Atari Cassette Preservation?

[bbking67]

I am interested in volunteering for the preservation project... the Atari scene is badly in need to a "trusted good dump" of all original commercial software from diskette tape and cartridge.

 

/bbking67

[Kr0tki]

I once had the idea to digitalize the analog tapes, then burn them to a CD and hack an old CD drive's audio output to the SIO port.

Already done.

 

The preserved image can then be processed by tools like WAV2CAS or whatever, but the initially recorded image should at least be archived.

I hope you are thinking only about tapes with audio tracks - for other tapes keeping a CAS image is enough, as its contains all data needed to recreate such tape.

 

I know there are 'CAS' archives, but what about 'WAV' archives? (or any other suitable lossless audio format).

The "Tape Preservation Project" at AtariArea has a few MP3s and FLACs.

[walter_J64bit]

I've started whith a 1010 tape drive, I've got some tapes that aren't listed, I'll post soon!

 

[walter_J64bit]

Look here http://www.atariage.com/forums/topic/107630-jump-man-cas/page__fromsearch__1

[F#READY]

 

The preserved image can then be processed by tools like WAV2CAS or whatever, but the initially recorded image should at least be archived.

I hope you are thinking only about tapes with audio tracks - for other tapes keeping a CAS image is enough, as its contains all data needed to recreate such tape.

 

I know there are 'CAS' archives, but what about 'WAV' archives? (or any other suitable lossless audio format).

The "Tape Preservation Project" at AtariArea has a few MP3s and FLACs.

 

Thanks for your comments Kr0tki.

 

No, I mean _all_ tapes!

How can you be sure that the CAS image contains all the data?

 

Call me paranoia, but MP3 is not a valid archiving format since it's lossy. FLAC would be ok, but it also depends on the recording method / equipment.

 

Ofcourse I understand the need for a more compact form to use for e.g. emulators.

 

My point is, preservation is for the very long future where you need to minimise the risk of loosing a single bit of information.

Processing raw material (e.g. using wav2cas) introduces a risk which can not be reverted without the raw image.

[Kr0tki]

No, I mean _all_ tapes!

How can you be sure that the CAS image contains all the data?

A CAS file actually contains more data than the source audio file - it contains decoded bytes of data.

 

An audio file contains unprocessed series of signals, which may contain noise, random drops in signal level, and other glitches that might break the loading process on some of the lesser-quality Atari tape recorders. The quality of such audio file can be verified only by checking whether a tape recreated from such file loads correctly - and even if it loads, it may be only by chance, because existence of the aforementioned glitches might break the loading process when it is performed again, or on another tape recorder.

 

Converting an audio file to CAS decodes the raw audio signal to bytes of data - similarly to what POKEY does while loading - and during this process the data is actually verified for correctness. If the conversion ended with success, the resulting image:

- contains no glitches that could prevent the tape from loading; and

- has no frame errors involving invalid lengths of start bits, stop bits, etc.

Correctness of the resulting CAS image can be additionally verified by computing checksums of each block of data.

 

Sometimes preserving a tape may involve manual intervention. I have designed software for preserving of all kinds of tapes (even copy-protected ones) in CAS format (see the signature). Using this software I've managed to recreate CAS images of a few tapes that could no longer load on a physical computer. Restoring these tapes involved manual removal of glitches in the source signal and then verifying checksums of the resulting blocks - but it resulted in CAS images with better quality than the source signal. This experience allows me to conclude: CAS Images Are Better.

 

Call me paranoia, but MP3 is not a valid archiving format since it's lossy. FLAC would be ok, but it also depends on the recording method / equipment.

Are you aware that fidelity of a Type I compact cassette is lower than that of a 128kbps MP3 file?

 

My point is, preservation is for the very long future where you need to minimise the risk of loosing a single bit of information.

Processing raw material (e.g. using wav2cas) introduces a risk which can not be reverted without the raw image.

As explained above, it's actually the opposite way.

Edited January 8, 2012 by Kr0tki

[Rybags]

IMO a proper preservation process would normalize the amplitude level and perform correction of such glitches/spikes.

 

Maintain the timing, and correct the waveforms to match an "ideal" situation, ie - fix frequency drift and employ waveform decay as would be expected from a top quality recording.

 

Such a process should also lend itself well to compression, maybe even something specifically designed for such a project, assuming it's aim is preservation above emulation use.

 

The thing is - there's no tape out there that's not aged and lost quality. Unless someone kept it in ideal conditions and created a WAV in the early 1990s which I'd say isn't very likely for 99% of what's out there.

[bbking67]

In spite of the fact that MP3 may be higher fidelity than the tape, the object of preservation is to capture the source material in the best possible fidelity, and obviously MP3 will degrade it slightly. For end users who just want to play the tape it will be "good enough", but I agree totally that the archival format for the audio portion of dual track tapes needs to be lossless (any lossless format or WAV should be fine). Storage is no longer much of an issue and I can't imagine these to be very long.

 

The problem with tape and other analog sources is that you are already losing a generation by making a copy, and that's bad enough. If you then convert to MP3 then you are effectively downsampling.

 

/bbking67

[F#READY]

No, I mean _all_ tapes!

How can you be sure that the CAS image contains all the data?

A CAS file actually contains more data than the source audio file - it contains decoded bytes of data.

 

An audio file contains unprocessed series of signals, which may contain noise, random drops in signal level, and other glitches that might break the loading process on some of the lesser-quality Atari tape recorders. The quality of such audio file can be verified only by checking whether a tape recreated from such file loads correctly - and even if it loads, it may be only by chance, because existence of the aforementioned glitches might break the loading process when it is performed again, or on another tape recorder.

 

Converting an audio file to CAS decodes the raw audio signal to bytes of data - similarly to what POKEY does while loading - and during this process the data is actually verified for correctness. If the conversion ended with success, the resulting image:

- contains no glitches that could prevent the tape from loading; and

- has no frame errors involving invalid lengths of start bits, stop bits, etc.

Correctness of the resulting CAS image can be additionally verified by computing checksums of each block of data.

 

Sometimes preserving a tape may involve manual intervention. I have designed software for preserving of all kinds of tapes (even copy-protected ones) in CAS format (see the signature). Using this software I've managed to recreate CAS images of a few tapes that could no longer load on a physical computer. Restoring these tapes involved manual removal of glitches in the source signal and then verifying checksums of the resulting blocks - but it resulted in CAS images with better quality than the source signal. This experience allows me to conclude: CAS Images Are Better.

 

 

Good points! I try to clarify...

 

Ok, let's assume the WAV file is correct (more about this later). The WAV file then contains all the information to convert to CAS, so the information was already available in the WAV file.

But I agree completely that the original form (WAV) isn't the ideal format for the user of even for further processing.

 

Don't get me wrong, I fully acknowledge your skills on processing and preserving tapes. My only concern was (is) that you _need_ to save the original source if your goal is long-term preservation.

 

In case you want to preserve a corrupted tape, you save the audio signal in the highest lossless quality and then the process of restoration can begin.

Let's say you find another tape which is also corrupt, you still have a better chance of _reconstructing_ the original from the two corrupted ones.

 

Verification is indeed a very important point. That's where the tools can help out. Still, using a tool is also a human intervention step. The tool is created and can contain bugs. Some tools are better than others.

Maybe in the future we will have tools which can automate even more steps from the restoration process.

In all cases we need the material most close to the source which can be preserved for a long time.

 

So, in short, I think CAS files are ok to store next to the original signal (WAV or whatever is decided). Also for the user who is not really concerned with preservation, CAS is fine.

 

Hope this makes sense?

[bbking67]

my comments are foe audio tracks only. i think .cas is okay for data as long its compatible.

[Kr0tki]

My only concern was (is) that you _need_ to save the original source if your goal is long-term preservation.

Would you mind not telling me what I need to do? I can imagine that each of us understands "preservation" differently, but telling people what they need is bordering on rude.

 

In case you want to preserve a corrupted tape, you save the audio signal in the highest lossless quality and then the process of restoration can begin.

Let's say you find another tape which is also corrupt, you still have a better chance of _reconstructing_ the original from the two corrupted ones.

Sure, keeping recordings of corrupted tapes is useful, I'm doing it myself; however once you manage to create a correct CAS image, storing them becomes pointless - a CAS image contains all data needed to restore the tape in its original form. (Well, unless you count such features like for example non-standard shape of the signal's sine wave as important enough to preserve. Obviously I don't.)

 

The tool is created and can contain bugs. Some tools are better than others.

The advantage of existence of multiple (two three, including a8cas-util.pl) CAS imaging software developed independently is, we can rule out probability of them having errors quite easily, by cross-comparison.

 

In all cases we need the material most close to the source which can be preserved for a long time.

Here's another one. What's up with this telling people what they need?

Edited January 9, 2012 by Kr0tki

[F#READY]

My only concern was (is) that you _need_ to save the original source if your goal is long-term preservation.

Would you mind not telling me what I need to do? I can imagine that each of us understands "preservation" differently, but telling people what they need is bordering on rude.

 

In case you want to preserve a corrupted tape, you save the audio signal in the highest lossless quality and then the process of restoration can begin.

Let's say you find another tape which is also corrupt, you still have a better chance of _reconstructing_ the original from the two corrupted ones.

Sure, keeping recordings of corrupted tapes is useful, I'm doing it myself; however once you manage to create a correct CAS image, storing them becomes pointless - a CAS image contains all data needed to restore the tape in its original form. (Well, unless you count such features like for example non-standard shape of the signal's sine wave as important enough to preserve. Obviously I don't.)

 

The tool is created and can contain bugs. Some tools are better than others.

The advantage of existence of multiple (two three, including a8cas-util.pl) CAS imaging software developed independently is, we can rule out probability of them having errors quite easily, by cross-comparison.

 

In all cases we need the material most close to the source which can be preserved for a long time.

Here's another one. What's up with this telling people what they need?

 

Ok, I'm sorry my wording offends you. My English is probably not good enough to clarify what I mean. The sentences "you need/we need" are not intended to tell you/us to do anything.

 

Let me then finish to say I'm going to store the original WAV together with metadata about the processed/extracted CAS file(s). This may be useful, maybe not.

[Shawn Jefferson]

I've never looked at the CAS format, but would an extension be possible that reflects where the audio track would have been and then an mp3 (or more than one) be attached to the cas file?

[Rybags]

http://a8cas.sourceforge.net/format-cas.html

 

Probably not.

 

The easiest implementation in emulator or SIO2xx might be to just search for an .mp3 or other audio file of the same filename.

[Mclaneinc]

Forgive as I'm tired but doesn't Altirra already do the audio support by treating it as an audio file as well?

 

Or am I off track and its a diferent subject...

[www.atarimania.com]

It does and works very well with what has been archived so far.

 

--

Atari Frog

http://www.atarimania.com

[Mclaneinc]

Ah, glad I wasn't too sleepy then

[ivop]

Let me add a few things, as I have the impression that some of you misunderstand each other. If not, then please ignore me.

 

For emulation purposes, .CAS is fine and those can always be extracted from wave files.

 

As for preservation, think hundreds of years. If .CAS files get corrupted over the years (degradation of storage media) and there's no error-free file available anymore, the information is lost forever. Wave files on the other hand contain a lot of redundancy. A bit of corruption is less likely to destroy all of the information.

 

A test for this might be adding some random noise to a CAS file and to a WAV file (i.e. flipping bits all over the place, or inc/dec randomly, store zero here and there). The CAS file will be totally useless. The WAV file will still convert fine to .CAS.

 

Of course this has its limits too, e.g. too much noise or a few seconds completely missing. Multiple copies at different locations are needed too, to secure preservation even further.

Edited January 11, 2012 by ivop

[bbking67]

ivop, I'm not totally with you on that. If the .CAS is a digital copy, there are many ways to ensure its survival. backing up on different media types and of course there has to be some continutity in terms of curation. Obviously, for data having the .CAS + a .WAV (or any lossless audio format) copy of the data track would be ideal. For the audio track on those data cassettes that use them, a .WAV (or other lossless format) is the way to go, but a convenient .MP3 could also be created for use with emulators, etc.

[ijor]

I hope you are thinking only about tapes with audio tracks - for other tapes keeping a CAS image is enough, as its contains all data needed to recreate such tape....

An audio file contains unprocessed series of signals, which may contain noise, random drops in signal level, and other glitches ...

Converting an audio file to CAS decodes the raw audio signal to bytes of data - similarly to what POKEY does while loading - and during this process the data is actually verified for correctness.

 

I agree with Kr0tki that saving the original audio is usually overkill. It might make sense in some cases (copy protected tapes, tapes with custom turbo formats, audio tracks, etc), but probably not worth in the general case.

 

Verification is indeed a very important point. That's where the tools can help out. Still, using a tool is also a human intervention step. The tool is created and can contain bugs. Some tools are better than others. Maybe in the future we will have tools which can automate even more steps from the restoration process. In all cases we need the material most close to the source which can be preserved for a long time.

 

Everything is a human step, and everything might fail. Even saving the WAV file is an human step that involves tools, and conceivable, it could fail as well. The point is, what are the chances that something will get wrong on the "digitalization" (conversion to CAS, or similar) process. And what are the chances in comparison to other types of failures.

 

I would say that these tools are so reliable, they were used so many times with so many different tapes, there are so many verifications (including playing and testing the game), etc; that chances for something going wrong are really not significant. They would never be zero, there is always the chance that something would go wrong, but again even saving the audio is not 100% foolproof either.

 

For emulation purposes, .CAS is fine and those can always be extracted from wave files... As for preservation, think hundreds of years. If .CAS files get corrupted over the years (degradation of storage media) and there's no error-free file available anymore, the information is lost forever. Wave files on the other hand contain a lot of redundancy. A bit of corruption is less likely to destroy all of the information.

 

Beg to differe Ivo, but this doesn't make much sense to me. Digital data doesn't degradate (that's one of the reasons the world has gone digital). Digital data stays exactly the same no matter how many times you copy, and no matters how much time you keep it. Of course, the physical media that supports the digital file might degradate (or get damaged, or destroyed, or lost), and the data might be corrupted (say, virus).

 

But corruption or media failure would probably affect an analog audio as much as a digital CAS file. Files are usually stored compressed, a single bit corruption might mean the whole archive is gone. A virus might corrupt the operating system metadata. A hard disk crash means (usually) that everything is gone. You don't gain too much by storing the original audio, at least not in this sense.

 

Yes, of course, redundancy with multiple copies in multiple locations is fundamental. But in this case, probably there is nothing better than make it public. A public archive gets naturally diseminated all over. It is very hard to achieve more redundancy than that.

 

One thing that might be useful though, is some kind of hashing (SHA or whatever) for keeping a "checksum/crc" of the original files.

 

Btw, Ivo, thanks for "your" Pokey donation to visual6502

Edited January 11, 2012 by ijor

[ivop]

For emulation purposes, .CAS is fine and those can always be extracted from wave files... As for preservation, think hundreds of years. If .CAS files get corrupted over the years (degradation of storage media) and there's no error-free file available anymore, the information is lost forever. Wave files on the other hand contain a lot of redundancy. A bit of corruption is less likely to destroy all of the information.

Beg to differe Ivo, but this doesn't make much sense to me. Digital data doesn't degradate (that's one of the reasons the world has gone digital). Digital data stays exactly the same no matter how many times you copy, and no matters how much time you keep it. Of course, the physical media that supports the digital file might degradate (or get damaged, or destroyed, or lost), and the data might be corrupted (say, virus).

 

Yes, and that's exactly the problem with digital data. Current storage solutions are not built for the ages like, say, engraving in stone or titanium plates

 

But corruption or media failure would probably affect an analog audio as much as a digital CAS file. Files are usually stored compressed, a single bit corruption might mean the whole archive is gone. A virus might corrupt the operating system metadata. A hard disk crash means (usually) that everything is gone. You don't gain too much by storing the original audio, at least not in this sense.

 

Most compression formats do not handle even slight corruption, i.e. a few bits being off. That's why most compression formats are not suitable for long (and I mean Looong) time storage on weak storage media. An uncompressed wave file is way more error-resilient. Basically, if you increase the entropy, you lose more data if a bit gets lost.

 

Btw, Ivo, thanks for "your" Pokey donation to visual6502

 

You're welcome

[Kr0tki]

Most compression formats do not handle even slight corruption, i.e. a few bits being off. That's why most compression formats are not suitable for long (and I mean Looong) time storage on weak storage media. An uncompressed wave file is way more error-resilient. Basically, if you increase the entropy, you lose more data if a bit gets lost.

Sure, disks fail sometimes, but it is not an issue related to file format. Adding redundancy to a file format would unnecessarily grow the cost of a) designing the format; b) developing software that works with the format; while not adding anything beneficial above other thinkable solutions, such as, I don't know, copying the damn file.

EDIT:

Ok, I'm sorry my wording offends you. My English is probably not good enough to clarify what I mean.

I don't know, maybe I overreacted, sorry for that.

Edited January 11, 2012 by Kr0tki

[ivop]

I am not saying CAS files have no use, they definitely have. But for long time preservation one needs something better, too, IMHO. I reread a statement above saying that digital data stays the same, no matter how many times you copy it. That's not true. Each time you copy a file, send it over a network, or do something similar, there is a possibility that it gets corrupted without being noticed. Sure, for the short term, just copying the damn file will suffice, but what'll happen in hundred years when we're all dead? This is where redundancy and error-correction codes and the like come in.

 

And, paraphrasing Freddy here, having a copy as near to the original as possible leaves open the possibility to reprocess the original in different ways in the future. If you discard this near-original signal this won't be possible if the originals (i.e. the tapes) have degraded even further.

Edited January 11, 2012 by ivop

[Shawn Jefferson]

That's easy to solve. Use a hash so you can verify the file.

 

So there is already a solution to run say, States and Capitals or Speed Reading in emulation or with sio2pc and get the audio too?

 

[Kr0tki]

Each time you copy a file, send it over a network, or do something similar, there is a possibility that it gets corrupted without being noticed.

This is getting ridiculous. Even disregarding the fact that hard disks and network protocols contain checksums to detect such problems - you can always copy the file twice and compare the contents. Or you can store the file's checksum in a separate place. Having a checksum in a file doesn't give higher level of redundancy than having that checksum separated, or keeping two copies of a file.

 

Sure, for the short term, just copying the damn file will suffice, but what'll happen in hundred years when we're all dead?

You mean that somehow the arcane skill of keeping two copies of a file will be forgotten THAT quickly?

 

 

So there is already a solution to run say, States and Capitals or Speed Reading in emulation or with sio2pc and get the audio too?

Yes, use Altirra or Atari800-a8cas.

Edited January 12, 2012 by Kr0tki