Yep, I would put the FP Rom at 95% odds on to blame. You could try pulling the OS card out then pry it out slightly and reseat it.
Alternatively the problem might be in memory select logic which then becomes a bit more complex to chase down.
I'd otherwise suggest/provide a small typein program to test if the FP is there, but I can't think of anything you could run in Basic in the machine's current state that won't just crash as everthing else has.
Do you have the means to transfer downloaded programs to bootable media for the Atari?