New Parts for File Server

I fixed my file server a couple of weeks ago. I now have my 3.5TB of storage space available again! If you'll recall, I've fixed my file server a few times before...

I started trying to fix my file server a couple of months ago by reinstalling Ubuntu. Unfortunately, this would hang at about 5% on the formatting/partitioning step. I suspected a faulty system hard drive, but didn't get around to trying another one until early October. I tried an old (from back in high school, for at least part of the machine) 40GB drive. The installation went through, but the system wouldn't boot from that drive. Since the drive had been in my garage for a number of years, I didn't particularly trust it. So, I went to Micro Center and picked up a 500GB hard drive that was on sale. It, too, allowed the installation to complete but would not boot up. It should be noted that the system hard drive has to be an IDE hard drive because I use all the SATA ports for the RAID array.

I figured it was most likely a faulty motherboard at this point. I went to newegg.com and picked out a cheap motherboard and processor (I always prefer to buy them together to ensure I have the right socket and so I don't have to mess with reapplying thermal grease whenever possible). Along with my new camera, the parts arrived in just a few days. I really do love Newegg.

There's not much else to tell. The installation went well, the raid was recreated without much hassle. I did learn a new command: mdadm --scan --detail /dev/md0 >> /etc/mdadm/mdadm.conf. I don't remember having to run this command or having to mess with mdadm.conf at all, in fact, in the past. I'm not sure what has changed, but in order to get the raid device to persist after rebooting, I had to do this. I'm just happy it all works.

And it does all work! I no longer have that noisy north bridge fan (this motherboard doesn't have a north bridge fan to get noisy)! Everything seems much more stable and efficient. I'm really quite pleased with the new motherboard. I'm also happy with the new camera. I took some pictures of the file server, which I can't believe I haven't done before!

The old, very dusty, motherboard.

My newly installed motherboard. So pretty.

The entire case. This is my first Micro ATX motherboard, and it is surprisingly tiny.

Raid Write Intent Bitmaps

During my last raid crash, I couldn't seem to get the damn thing to resync. I tried multiple times, but it would always lock up. I almost took the drastic step of borrowing a raid array to back things up as much as possible and then rebuild from scratch! However, in my many hours of research into potential solutions, I found that most people prevented the problem using a little thing called write intent bitmaps.

As far as I know, they are not available with hardware raid solutions, such as dedicated cards or those built into motherboards. Linux software raid, however, makes it ridiculously easy. Once I finally did get things resync'd (dumb luck, really. I didn't try anything new - it just started working), I immediately turned on the write intent bitmap feature.

mdadm --grow --bitmap=internal /dev/md0

This may reduce write performance slightly (up to 10%), but given that something is funky with my machine and it crashes during most full resyncs, I'm all for the saved resync time. Since I've turned this on, I haven't had a single problem with the array or the machine. There have been a couple blown fuses due to a space heater on the same circuit, but the machine came up and stayed up just fine, barely requiring a few seconds to resync. I can't tell you how happy I was.

So, switch on over to mdadm, add an internal bitmap, and rejoice in the short resync times.

Fileserver

Before I left for San Diego, my fileserver's northbridge fan and heatsink became separated. It's really worse than a divorce, in many ways, because things are all fucked up, but there's still hope, which means uncertainty, lots of work, and very careful handling of electronics. OK, so I don't know much about divorces.

The lack of heat transfer away from the chip kept my fileserver from booting. Since I've already complained about my loud northbridge fan, I chose to take the opportunity to replace that noisy piece of hardware. I already had a spare fan from somewhere, so I went to the hardware store to find screws long enough to secure it. At the hardware store, I had the bright idea of comparing the screw holes of the two fans and found that they didn't line up. That's when I made the coolest discovery of the week: Krazy Glue comes in a small bottle with a tiny brush built into the lid! I glued the fan to the heatsink, and it stuck almost immediately.

When I got home, I couldn't find my isopropyl alcohol. I was able to get some last Friday night, though, and I set out to clean the bottom of the heatsink and the top of the northbridge heat spreader. I remember isopropyl alcohol working much better than it did. I had to cut the thermal paste off the heatsink, for the most part. Perhaps thermal paste is just that horrible compared to thermal grease. The important thing is that I could now keep my northbridge chip cool.

Doh! The power cord for the fan doesn't reach! I had to pry the fan off the heatsink and re-glue it, rotated ninety degrees. A little thermal grease later, and I was ready to plug my motherboard back into the rest of the computer.

Everything is working mostly. The only weird things are related to mdadm and the raid, I think. The automatic resync operation seems to fail. The first night, I had "watch -n 1 cat /proc/mdstat" running, but I woke up to stack traces being dumped to the screen every second instead of the nice md status. I tried again, but came back later to find the same stack traces showing up every once in awhile. So, I loaded the graphical interface to try and catch more information - but the error stopped. The resync also stopped at 8% and refused to go any further.

I decided that some write activity might wake it up, and started up some bittorrents. They worked well for a few tens of MB, but now my fileserver seems to have rebooted to an initramfs prompt and automatically started a resync. It's currently at 6.9%. I'll see what happens at 8% before I go to bed, and will update as things progress.

Update: 1:49 AM: resync is at 9.6% and going strong.
Update: 9:21 AM: resync is at 64.9% and going strong.
Update: 12:14 PM: resync is at 85.9% and going strong. I'm getting bored waiting for this.
Update: 3:16 PM: resync is done. Things have rebooted, and I will begin some bittorrent stress testing shortly.
Update: 10:23 AM: Fileserver is still up after a couple days of heavy I/O. I'm happy!

Fileserver drama

It's been an exciting few days here in Geekland. I successfully installed my two new hard drives. That involved routing cables in an ugly manner, but was otherwise uneventful. I was not able to replace my north bridge fan, though - the wiring for the current one goes under the heatsink, but to remove that I think I need to remove the motherboard from the case. Anyway, at that point, I was all ready to grow my raid5 array.

But wait! How could this be? My 4-disk raid5 array is only running with 3 active disks¹. It would appear that sometime on December 17, a power outage or similar caused a hard drive to be marked as failed. I should really set up some sort of notification. Well, I took the opportunity to learn all about recovering a dirty raid array. Good ol' mdadm was marvelous!²

With a fully functioning, clean, 4-disk raid5 array, it was time to grow the array. I called upon mdadm once again³ and got my two new drives added as hot spares. Just one more command⁴ to grow the array --

What's this? Linux and mdadm require versions 2.6.17 (2.6.19 according to some) and 2.4.1 and later, respectively, in order to grow a raid5 array? Surely Ubuntu, the most user-friendly linux version available will have a convenient upgrade mechanism - well, sort of. Upgrading reported a few errors, but I was running 2.6.20 and mdadm⁴ wasn't throwing an error anymore. Huzzah, my raid array was growing!

Kernel Panic? Aiee! Oh god oh god oh god. 2TB of data lost! Please let this reboot erase this bad dream... Eep! /dev/md0 no longer exists! There has to be a way to fix this - it's linux! Why, of course! I can always rely on mdadm. Why, once you reassemble the array⁵, it goes right on growing! Kernel Panic, again? Screw this.

After tiring of kernel panics and screaming "Aiee!" I downloaded an installation CD for the latest Ubuntu distribution. I had to reformat /root and /boot (and opted to format /home while I was at it), but I had a clean installation. In fact, it made everything easier. Once I reinstalled the mdadm package, /dev/md0 magically reappeared and was growing once again. It's now 56.7% done growing. After that, I need to resize the ext3 file system⁶, but I think that will go smoother.

At least I know the data is still there (at least mostly). After another while reinstalling and configuring samba and mounting /dev/md0, I have successfully watched an episode of TV. Indeed, I can still use my 2TB of file storage while it's growing into 3.5TB! If it weren't for a faulty upgrade, I probably wouldn't have had to reboot except for the hardware installation (and I admit that can even be avoided given proper cable planning in the case). I'm still amazed that mdadm can handle a bad disk, adding two disks, a faulty OS upgrade, kernel panics interrupting a reshape, reassembling unclean disks and making them clean again, resuming an interrupted reshape operation from a different version, and almost all while allowing the drive to remain accessible. Simply amazing.

Commands to remember:

1. cat /proc/mdstat
2. mdadm --add /dev/md0 /dev/sdd1
3. mdadm --add /dev/md0 /dev/sde1 /dev/sdf1
4. mdadm --grow /dev/md0 -n 6
   
5. mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sde1 /dev/sdf1
6. resize2fs /dev/md0 (theoretically)
   

Dungeons and Dragons

I was featured in yesterday's Internet Video of the Day, which was quite fun. I am indeed playing Dungeons and Dragons for the first time on Saturday. I'm really looking forward to it. My friend's wife helped me create an initial character and told me about some of the basics. Since then, I've been reading up in more detail on the D&D Wiki.

I still have quite a few questions, and I'm sure some things will end up being like the Free Parking in Monopoly - things aren't always done according to the rules. It should be a fun experience regardless.

In other nerdy news, my hard drives arrived! I think I will install them on Sunday. I'm quite excited. I just hope I don't lose all my data when I try and expand the raid array.

Computer Upgrades

I placed an order on Newegg yesterday for two more 750GB hard drives for my fileserver. I'm very excited because they arrive tomorrow! Newegg is fast! I'm thinking of making a trip up to Microcenter at some point in the next couple of days to get a new north bridge fan. I figure as long as my machine is going to be off while I add new hard drives and the like, I might as well get rid of the one loud fan that's in the box. I still need to figure out exactly how to expand a linux raid5 array, but that should be fun. Also cool: my hard drives were each part of a combo deal, so I get two free USB drives - one is 2GB and the other is 4GB. Perhaps I'll actually use one sometime.

I also did some research for my next primary machine while I was shopping. I've picked out a likely case and memory, but beyond that I ran into some troubles. There wasn't any clear best choice on processor/motherboard/graphics combinations. In the end, I decided that since I won't be buying a new machine until at least June, that I'll postpone any research until then, too.

The only lasting decision I made was to name my new computer Nora. In fact, this will be the first computer I will have named. I chose the name when it immediately popped into my head when Newegg asked me what I should name the wish list of parts I'd created. So, I'm not sure what the significance is yet, but Nora has some potential to be special.