So a few weeks ago our trusty NAS’s hard drives decided to be unhappy. No worry I thought as I shut down the NAS and replaced the HD with a spare. Unfortunately, the HD enclosure decided that it time was over and never came online again.
No worries I thought, I’ll just order a replacement. The replacement duly arrived and I transferred the hard disks to the replacement unit. All was well till the following Sunday when I woke up with I/O errors when I ssh’d to the machine. Checking MDADM showed that all 3 drives had detached from the enclosure. Luckily after a reboot everything came back and I decided to monitor it and make sure everything was backed up.
Backups were running and I thought that even if it all went down I would be able to restore from back-ups. I would be safe or so I thought. Sure enough, the NAS did the same thing the following weekend. This time sadly the raid5 array did not come back up. To compound matters the backups or the machines home directories consisted entirely of symlinks. Luckily I had a single backup of my home directory. Siobhan was not so lucky. We also lost the contents of an old hard disk from the laptop from my pre-transition days.
We recovered what we could and rebuilt. This time I decided to try out using LVM to run the raid rather than doing it at the disk level with MDADM which wasn’t particularly successful. Each time I set up a test RAID1 volume it would complain about a bad superblock on reboot. At that point I decided enough was enough with using ARM SBCs with their older Linux kernels and just decided to replace the SBC with a second-hand desktop PC.
A new PC was acquired from Ebay and arrived that week. It’s a lot bigger than the old SBC but it runs the latest debian and doesn’t consume huge amounts of power. It’s a bit overkill but has been stable so far which is the most important thing.