During my last raid crash, I couldn't seem to get the damn thing to resync. I tried multiple times, but it would always lock up. I almost took the drastic step of borrowing a raid array to back things up as much as possible and then rebuild from scratch! However, in my many hours of research into potential solutions, I found that most people prevented the problem using a little thing called write intent bitmaps.
As far as I know, they are not available with hardware raid solutions, such as dedicated cards or those built into motherboards. Linux software raid, however, makes it ridiculously easy. Once I finally did get things resync'd (dumb luck, really. I didn't try anything new - it just started working), I immediately turned on the write intent bitmap feature.
mdadm --grow --bitmap=internal /dev/md0
This may reduce write performance slightly (up to 10%), but given that something is funky with my machine and it crashes during most full resyncs, I'm all for the saved resync time. Since I've turned this on, I haven't had a single problem with the array or the machine. There have been a couple blown fuses due to a space heater on the same circuit, but the machine came up and stayed up just fine, barely requiring a few seconds to resync. I can't tell you how happy I was.
So, switch on over to mdadm, add an internal bitmap, and rejoice in the short resync times.
Showing posts with label linux. Show all posts
Showing posts with label linux. Show all posts
Thursday, March 5, 2009
Wednesday, March 5, 2008
Fileserver drama
It's been an exciting few days here in Geekland. I successfully installed my two new hard drives. That involved routing cables in an ugly manner, but was otherwise uneventful. I was not able to replace my north bridge fan, though - the wiring for the current one goes under the heatsink, but to remove that I think I need to remove the motherboard from the case. Anyway, at that point, I was all ready to grow my raid5 array.
But wait! How could this be? My 4-disk raid5 array is only running with 3 active disks1. It would appear that sometime on December 17, a power outage or similar caused a hard drive to be marked as failed. I should really set up some sort of notification. Well, I took the opportunity to learn all about recovering a dirty raid array. Good ol' mdadm was marvelous!2
With a fully functioning, clean, 4-disk raid5 array, it was time to grow the array. I called upon mdadm once again3 and got my two new drives added as hot spares. Just one more command4 to grow the array --
What's this? Linux and mdadm require versions 2.6.17 (2.6.19 according to some) and 2.4.1 and later, respectively, in order to grow a raid5 array? Surely Ubuntu, the most user-friendly linux version available will have a convenient upgrade mechanism - well, sort of. Upgrading reported a few errors, but I was running 2.6.20 and mdadm4 wasn't throwing an error anymore. Huzzah, my raid array was growing!
Kernel Panic? Aiee! Oh god oh god oh god. 2TB of data lost! Please let this reboot erase this bad dream... Eep! /dev/md0 no longer exists! There has to be a way to fix this - it's linux! Why, of course! I can always rely on mdadm. Why, once you reassemble the array5, it goes right on growing! Kernel Panic, again? Screw this.
After tiring of kernel panics and screaming "Aiee!" I downloaded an installation CD for the latest Ubuntu distribution. I had to reformat /root and /boot (and opted to format /home while I was at it), but I had a clean installation. In fact, it made everything easier. Once I reinstalled the mdadm package, /dev/md0 magically reappeared and was growing once again. It's now 56.7% done growing. After that, I need to resize the ext3 file system6, but I think that will go smoother.
At least I know the data is still there (at least mostly). After another while reinstalling and configuring samba and mounting /dev/md0, I have successfully watched an episode of TV. Indeed, I can still use my 2TB of file storage while it's growing into 3.5TB! If it weren't for a faulty upgrade, I probably wouldn't have had to reboot except for the hardware installation (and I admit that can even be avoided given proper cable planning in the case). I'm still amazed that mdadm can handle a bad disk, adding two disks, a faulty OS upgrade, kernel panics interrupting a reshape, reassembling unclean disks and making them clean again, resuming an interrupted reshape operation from a different version, and almost all while allowing the drive to remain accessible. Simply amazing.
Commands to remember:
But wait! How could this be? My 4-disk raid5 array is only running with 3 active disks1. It would appear that sometime on December 17, a power outage or similar caused a hard drive to be marked as failed. I should really set up some sort of notification. Well, I took the opportunity to learn all about recovering a dirty raid array. Good ol' mdadm was marvelous!2
With a fully functioning, clean, 4-disk raid5 array, it was time to grow the array. I called upon mdadm once again3 and got my two new drives added as hot spares. Just one more command4 to grow the array --
What's this? Linux and mdadm require versions 2.6.17 (2.6.19 according to some) and 2.4.1 and later, respectively, in order to grow a raid5 array? Surely Ubuntu, the most user-friendly linux version available will have a convenient upgrade mechanism - well, sort of. Upgrading reported a few errors, but I was running 2.6.20 and mdadm4 wasn't throwing an error anymore. Huzzah, my raid array was growing!
Kernel Panic? Aiee! Oh god oh god oh god. 2TB of data lost! Please let this reboot erase this bad dream... Eep! /dev/md0 no longer exists! There has to be a way to fix this - it's linux! Why, of course! I can always rely on mdadm. Why, once you reassemble the array5, it goes right on growing! Kernel Panic, again? Screw this.
After tiring of kernel panics and screaming "Aiee!" I downloaded an installation CD for the latest Ubuntu distribution. I had to reformat /root and /boot (and opted to format /home while I was at it), but I had a clean installation. In fact, it made everything easier. Once I reinstalled the mdadm package, /dev/md0 magically reappeared and was growing once again. It's now 56.7% done growing. After that, I need to resize the ext3 file system6, but I think that will go smoother.
At least I know the data is still there (at least mostly). After another while reinstalling and configuring samba and mounting /dev/md0, I have successfully watched an episode of TV. Indeed, I can still use my 2TB of file storage while it's growing into 3.5TB! If it weren't for a faulty upgrade, I probably wouldn't have had to reboot except for the hardware installation (and I admit that can even be avoided given proper cable planning in the case). I'm still amazed that mdadm can handle a bad disk, adding two disks, a faulty OS upgrade, kernel panics interrupting a reshape, reassembling unclean disks and making them clean again, resuming an interrupted reshape operation from a different version, and almost all while allowing the drive to remain accessible. Simply amazing.
Commands to remember:
- cat /proc/mdstat
- mdadm --add /dev/md0 /dev/sdd1
- mdadm --add /dev/md0 /dev/sde1 /dev/sdf1
- mdadm --grow /dev/md0 -n 6
- mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sde1 /dev/sdf1
- resize2fs /dev/md0 (theoretically)
Monday, April 16, 2007
File server fixed!
I'm almost positive that my file server is fixed! I am very excited. I can now fill up the remaining 600GB (of 2048GB).
What was the problem, you ask? Well, it would appear that the voltage applied to the DIMM sockets was set at 1.9V instead of 2.1V. I'm not sure if I should blame the RAM's SPD data, or the BIOS for ignoring said data. I'm betting it's bad data on the RAM, because the Corsair Memory website listed both 1.9 and 2.1 as the tested voltage in two different places. They appear to be confused.
Either way, manually setting the DIMM voltage to 2.1 allowed memtest86 to run for a full 24 hours, which is hasn't been able to do for a long time. It's time to plug my hard drives back in, boot up, and transfer some very large files!
What was the problem, you ask? Well, it would appear that the voltage applied to the DIMM sockets was set at 1.9V instead of 2.1V. I'm not sure if I should blame the RAM's SPD data, or the BIOS for ignoring said data. I'm betting it's bad data on the RAM, because the Corsair Memory website listed both 1.9 and 2.1 as the tested voltage in two different places. They appear to be confused.
Either way, manually setting the DIMM voltage to 2.1 allowed memtest86 to run for a full 24 hours, which is hasn't been able to do for a long time. It's time to plug my hard drives back in, boot up, and transfer some very large files!
Friday, March 9, 2007
Linux, Ubuntu, and My File Server
My file server has been ill recently. It's been freezing. It has other problems, too, like a loud northbridge fan, but these seem minor in comparison to failing to function at all.
Well, after having it freeze on the login screen, without the screen saver on, I noticed some screen corruption. This says to me that it's graphics related. It's hard to tell if it's drivers or hardware, or a combination. I'm not a fan of manufacturers overclocking their boards by default. Instability is the result. One explanation for the suddenness of the increase in freezing frequency is miscellaneous updates to Ubuntu. Or, perhaps my GPU is just gradually melting and becoming less stable. The problem here is that when booting in Windows, things remained unfrozen for three days before I tried Ubuntu again.
Either way, I figured a good way to test the graphics card problem would be to run in command line mode for awhile, and see if things freeze. Getting Ubuntu to run in command line mode isn't terribly easy, though. Most linux distros have runlevel 2 mapped to command line mode. Ubuntu has levels 2-5 all mapped to the same standard experience. So, I had to do an "apt-get install rcconf" and "sudo rcconf" and then find a site that talked about what each service does.
It makes me feel inadequate and a bit sad that I'm such a linux newbie. I think I'd be embarrassed if I ever had to have conversations about linux with a real techie. Luckily, at work, I have the excuse "I work on .NET, which is Windows only. So, I need an answer for Windows. Oh, I have a linux file server at home, though." I am simply ashamed.
Also, I've had this damn thing for many months, now, and I still haven't even installed apache. I meant to have this thing functioning as an FTP server, website, and collaboration tool for family software projects. I need to get on this, and soon. Once I buy a house, anyway.
Well, after having it freeze on the login screen, without the screen saver on, I noticed some screen corruption. This says to me that it's graphics related. It's hard to tell if it's drivers or hardware, or a combination. I'm not a fan of manufacturers overclocking their boards by default. Instability is the result. One explanation for the suddenness of the increase in freezing frequency is miscellaneous updates to Ubuntu. Or, perhaps my GPU is just gradually melting and becoming less stable. The problem here is that when booting in Windows, things remained unfrozen for three days before I tried Ubuntu again.
Either way, I figured a good way to test the graphics card problem would be to run in command line mode for awhile, and see if things freeze. Getting Ubuntu to run in command line mode isn't terribly easy, though. Most linux distros have runlevel 2 mapped to command line mode. Ubuntu has levels 2-5 all mapped to the same standard experience. So, I had to do an "apt-get install rcconf" and "sudo rcconf" and then find a site that talked about what each service does.
It makes me feel inadequate and a bit sad that I'm such a linux newbie. I think I'd be embarrassed if I ever had to have conversations about linux with a real techie. Luckily, at work, I have the excuse "I work on .NET, which is Windows only. So, I need an answer for Windows. Oh, I have a linux file server at home, though." I am simply ashamed.
Also, I've had this damn thing for many months, now, and I still haven't even installed apache. I meant to have this thing functioning as an FTP server, website, and collaboration tool for family software projects. I need to get on this, and soon. Once I buy a house, anyway.
Subscribe to:
Posts (Atom)