Ok if you followed along so far you know that I’m a big fan of software RAID. Most of the time I end up doing RAID1. That’s the simplest form of RAID - basically you have two disks - they mirror each other. The advantages are that it’s easy to setup and you immediately get the benefit of having some redundancy. The bad news is you lose half the total drive space (2 150GB drives = 150GB RAID1)
There are other options - namely RAID5. The primary advantage of RAID5 in my circumstance is that you get N-1 disk space (so 3 400GB drives = 800GB RAID5). That’s assuming of course that you actually RAID the entire drive as single partitions - my partition scheme is a little more complicated than that.
I currently have a fileserver in my home office running Dapper. It has two 400GB drives. They each have 4 partitions (/,boot,swap,freezer (that last partition is the file server portion)). I’m running out of space (blame a heck of a lot of raw DV Video/photos from my recent wedding). So I thought it would be cool if I could upgrade from what I have to what I want.
You may want to read this if you want to see approx how I got this setup in the first place.
Prep
Since the server I’m working on has a cdrom and is sitting next to me, I went the easy route for part of this process and simply booted it off of a cd so I could fully unmount the current RAID array and modify it. If you don’t have this luxury - you are on your own - it’s probably totally possible to do this remotely but a lot scarier. Especially because in my case I don’t have backups for a lot of this data.
The other thing that is important to know is that this is some cutting edge stuff. You need to have access to a 2.6.17 kernel (The current kernel at the time I wrote this). My Dapper cds did not. I found that Knoppix.org version 5.0.1 has that version. Which is nice because I didn’t find any other cds versions that did have it.
Second, you need a very very modern version of mdadm(2.5.2). It has just recently added the code to do what I’m about to (namely grow a RAID5 array dynamically). I couldn’t find a package for it - but it was dead easy to download,compile, and install under Knoppix.
Basically, you should burn yourself a knoppix disk. Boot onto it, download and install mdadm and the rest will take a lot of time but it will go smoothly.
Getting Started
I bought a new drive. It’s 500GB (instead of 400GB). That will just mean I have 100GB of extra space left over that won’t be part of any raid partition. I ended up buying the bigger drive because there was a special where you could get it for $120 - that seemed like a price too good to pass up.
I took down the file server and installed the new hard drive. I’m doing all of this on IDE so
/dev/hda -> RAID1 Disk 1
/dev/hdc -> RAID1 Disk 2
/dev/hdd -> New disk
Partitioning
You’ll notice in my comments before that the RAID drives have 4 partitions. If I partitioned the new drive the same way I would never be able to access the extra 100GB of drive space - which frankly just seems wrong. To compensate I ended up resorting to extended partitions. Here’s what the partition table looks like
Disk /dev/hdd: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/hdd1 1 4 32098+ fd Linux raid autodetect
/dev/hdd2 9 71 506047+ fd Linux raid autodetect
/dev/hdd3 76 2508 19543072+ fd Linux raid autodetect
/dev/hdd4 2513 60801 468206392+ 5 Extended
/dev/hdd5 2513 48641 370531161 fd Linux raid autodetect
/dev/hdd6 48646 60801 97643070 83 Linux
You’ll notice that I’ve set most of the partitions to fd (for raid) except for the last one - that’s the left over drive space.
I’m just doing this to be careful
mdadm --zero-superblock /dev/hdd
If the drive had ever have RAID on it before, this wipes it out. It is a handy command to have if you end up re-using drives.
Start Creating Arrays
I had to create these arrays with higher level numbers so I could get everything setup before I switched over from the current RAID1 array.
It turns out that you can’t boot from a RAID5 array. The bootloader simply can’t reconstruct the array properly - so your boot partition must be RAID1 (I didn’t find this out until very late in the game.) As a result, I ended up just adding my new drive’s (hdd1) first partition as a spare RAID1 partition for booting purposes.
mdadm --create /dev/md5 --verbose --level raid5 --raid-devices=2 /dev/hdd2 missing
mdadm --create /dev/md6 --verbose --level raid5 --raid-devices=2 /dev/hdd3 missing
mdadm --create /dev/md7 --verbose --level raid5 --raid-devices=2 /dev/hdd5 missing
cat /proc/mdstat
md7 : active raid5 hdd5[0]
370531072 blocks level 5, 64k chunk, algorithm 2 [2/1] [U_]
md6 : active raid5 hdd3[0]
19542976 blocks level 5, 64k chunk, algorithm 2 [2/1] [U_]
md5 : active raid5 hdd2[0]
505920 blocks level 5, 64k chunk, algorithm 2 [2/1] [U_]
This shows that the arrays are up - but they are degradded. I’ll get to adding another drive once I’ve copied over data from the current RAID1 array.
mkswap /dev/md5
mkfs.ext3 /dev/md6
mkfs.ext3 /dev/md7
Setup MDADM
To make sure things survive the reboot properly - I regen’d my /etc/mdadm/mdadm.conf file.
Basically - do the following:
sudo echo "DEVICE partitions" > /etc/mdadm/mdadm.conf
sudo mdadm --detail --scan >> /etc/mdadm/mdadm.conf
Mount & Copy
Basically I just mounted the filesystem from the new RAID devices and copied over everything onto it.
mkdir /mnt/new_raid
mount /dev/md6 /mnt/new_raid
mkdir /mnt/new_raid/freezer
cd /mnt/new_raid
tar -C / -clspf - . | tar -xlspvf -
cd freezer
tar -C /freezer -clspf - . | tar -xlspvf -
I ended up leaving this part overnight.
Scary Part
This part is a little nerve racking but it went well enough. You want to make sure your computer is on a ups and that it does not crash during this process - I’m guessing it would be bad.
I basically moved each partition over one at a time. Process is very very simple. I stopped the RAID1 arrays by using:
mdadm --stop /dev/md0
mdadm --stop /dev/md1
mdadm --stop /dev/md2
mdadm --stop /dev/md3
Now for each partition (except for the md0 which is /boot) I did the following
(I believe I did the super block thing but I can’t remember exactly atm - so if it gives you an error just skip it - the goal is to make sure that the drive forgets it was part of another RAID array)
mdadam --zero-superblock /dev/hda3
mdadm --add /dev/md5 /dev/hda3
wait for rebuild
mdadam --zero-superblock /dev/hdc3
mdadm --add /dev/md5 /dev/hdc3
wait for rebuild
e2fsck -f /dev/md5
wait for disk check
resize2fs /dev/md5
cd /mnt
mkdir md5
mount /dev/md5 md5
Assuming the above went smoothly you should now have a RAID5 device up with twice the diskspace of the old RAID1 partition. You just do that for each of the other partitions and once you are done reboot.
Also note, that the rebuild times depend a lot on the size of your partition and the speed of you processor and drive. In my case, it took 8-10 hours to churn through the rebuild for the 330GB partition. The disk check probably took an hour. And the resize took 30 minutes.
Once this was all complete I updated the /etc/mdadm/mdadm.conf file on the new RAID5 partition. I modified the file to rename md4 -> md0 , md5->md1, md6->m2,md7->md3. Then I rebooted.
The beauty of this is there is room for one more drive in my file server. I know now that I can easily upgrade one more time and add more space if needed.
Have I mentioned how much I love software RAID?
Left Over
Since I had that extra 100GB on hdd left over - I ended up backing up the /boot and / onto it during this whole process. Once the server is back in production I’m thinkin I may put some of /var system on here since I don’t need deb or logs raided.
Props
Main props goto Ferg’s Gaff His article showed the path - now I’ve added some more fluff to fill it out.
You can also read here. This a gentoo wiki - but it has some great stuff related to using a modern version of MDADM to minimize rebuild time. The bad news it is required 2.4.1 mdam (available in edgy - easily rebuilt in dapper) and a 2.6.16 or greater kernel (Sorry currently Dapper is at 2.6.15 - so I guess this will have to wait).
