Fileserver

Before I left for San Diego, my fileserver's northbridge fan and heatsink became separated. It's really worse than a divorce, in many ways, because things are all fucked up, but there's still hope, which means uncertainty, lots of work, and very careful handling of electronics. OK, so I don't know much about divorces.

The lack of heat transfer away from the chip kept my fileserver from booting. Since I've already complained about my loud northbridge fan, I chose to take the opportunity to replace that noisy piece of hardware. I already had a spare fan from somewhere, so I went to the hardware store to find screws long enough to secure it. At the hardware store, I had the bright idea of comparing the screw holes of the two fans and found that they didn't line up. That's when I made the coolest discovery of the week: Krazy Glue comes in a small bottle with a tiny brush built into the lid! I glued the fan to the heatsink, and it stuck almost immediately.

When I got home, I couldn't find my isopropyl alcohol. I was able to get some last Friday night, though, and I set out to clean the bottom of the heatsink and the top of the northbridge heat spreader. I remember isopropyl alcohol working much better than it did. I had to cut the thermal paste off the heatsink, for the most part. Perhaps thermal paste is just that horrible compared to thermal grease. The important thing is that I could now keep my northbridge chip cool.

Doh! The power cord for the fan doesn't reach! I had to pry the fan off the heatsink and re-glue it, rotated ninety degrees. A little thermal grease later, and I was ready to plug my motherboard back into the rest of the computer.

Everything is working mostly. The only weird things are related to mdadm and the raid, I think. The automatic resync operation seems to fail. The first night, I had "watch -n 1 cat /proc/mdstat" running, but I woke up to stack traces being dumped to the screen every second instead of the nice md status. I tried again, but came back later to find the same stack traces showing up every once in awhile. So, I loaded the graphical interface to try and catch more information - but the error stopped. The resync also stopped at 8% and refused to go any further.

I decided that some write activity might wake it up, and started up some bittorrents. They worked well for a few tens of MB, but now my fileserver seems to have rebooted to an initramfs prompt and automatically started a resync. It's currently at 6.9%. I'll see what happens at 8% before I go to bed, and will update as things progress.

Update: 1:49 AM: resync is at 9.6% and going strong.
Update: 9:21 AM: resync is at 64.9% and going strong.
Update: 12:14 PM: resync is at 85.9% and going strong. I'm getting bored waiting for this.
Update: 3:16 PM: resync is done. Things have rebooted, and I will begin some bittorrent stress testing shortly.
Update: 10:23 AM: Fileserver is still up after a couple days of heavy I/O. I'm happy!

Fileserver drama

It's been an exciting few days here in Geekland. I successfully installed my two new hard drives. That involved routing cables in an ugly manner, but was otherwise uneventful. I was not able to replace my north bridge fan, though - the wiring for the current one goes under the heatsink, but to remove that I think I need to remove the motherboard from the case. Anyway, at that point, I was all ready to grow my raid5 array.

But wait! How could this be? My 4-disk raid5 array is only running with 3 active disks¹. It would appear that sometime on December 17, a power outage or similar caused a hard drive to be marked as failed. I should really set up some sort of notification. Well, I took the opportunity to learn all about recovering a dirty raid array. Good ol' mdadm was marvelous!²

With a fully functioning, clean, 4-disk raid5 array, it was time to grow the array. I called upon mdadm once again³ and got my two new drives added as hot spares. Just one more command⁴ to grow the array --

What's this? Linux and mdadm require versions 2.6.17 (2.6.19 according to some) and 2.4.1 and later, respectively, in order to grow a raid5 array? Surely Ubuntu, the most user-friendly linux version available will have a convenient upgrade mechanism - well, sort of. Upgrading reported a few errors, but I was running 2.6.20 and mdadm⁴ wasn't throwing an error anymore. Huzzah, my raid array was growing!

Kernel Panic? Aiee! Oh god oh god oh god. 2TB of data lost! Please let this reboot erase this bad dream... Eep! /dev/md0 no longer exists! There has to be a way to fix this - it's linux! Why, of course! I can always rely on mdadm. Why, once you reassemble the array⁵, it goes right on growing! Kernel Panic, again? Screw this.

After tiring of kernel panics and screaming "Aiee!" I downloaded an installation CD for the latest Ubuntu distribution. I had to reformat /root and /boot (and opted to format /home while I was at it), but I had a clean installation. In fact, it made everything easier. Once I reinstalled the mdadm package, /dev/md0 magically reappeared and was growing once again. It's now 56.7% done growing. After that, I need to resize the ext3 file system⁶, but I think that will go smoother.

At least I know the data is still there (at least mostly). After another while reinstalling and configuring samba and mounting /dev/md0, I have successfully watched an episode of TV. Indeed, I can still use my 2TB of file storage while it's growing into 3.5TB! If it weren't for a faulty upgrade, I probably wouldn't have had to reboot except for the hardware installation (and I admit that can even be avoided given proper cable planning in the case). I'm still amazed that mdadm can handle a bad disk, adding two disks, a faulty OS upgrade, kernel panics interrupting a reshape, reassembling unclean disks and making them clean again, resuming an interrupted reshape operation from a different version, and almost all while allowing the drive to remain accessible. Simply amazing.

Commands to remember:

1. cat /proc/mdstat
2. mdadm --add /dev/md0 /dev/sdd1
3. mdadm --add /dev/md0 /dev/sde1 /dev/sdf1
4. mdadm --grow /dev/md0 -n 6
   
5. mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sde1 /dev/sdf1
6. resize2fs /dev/md0 (theoretically)