Random Musings of an Insane Mind

We received a client machine a while back. At some point they needed more disk space and sent us a pair of drives with instructions for mirroring data from the existing machine to the new drives.

When I looked at the drive I noticed something odd:

md4 : active raid1 sdb8[1](S) sda8[0]
      1942145856 blocks super 1.2 [1/1] [U]

While this was a Raid 1 set, notice that the indicator shows that sdb8 is a Spare (S) and that the Raid 1 set has 1 of 1 drive and isn’t broken. This usually occurs when someone is doing an in-place migration to a larger drive and creates a Raid 1 partition with a single device and forces it online with the intention of later adding the second drive. Had the primary drive failed, they would have experienced data loss.

To fix it:

mdadm --remove /dev/md4 /dev/sdb8
mdadm --grow /dev/md4 --raid-devices=2 --force
mdadm --add /dev/md4 /dev/sdb8

and we see the resulting md status:

md4 : active raid1 sda8[0]
      1942145856 blocks super 1.2 [2/1] [U_]

After the reconstruction we now have a properly configured Raid 1 set.

I installed Linux Mint on a machine, but, wanted to use Raid 1 for the drive. However, even through the custom installation with both drives in place, I saw no way to configure Raid on installation. Since we do this sort of thing quite frequently, I figured I would write a quick guide detailing the proces.

apt-get install mdadm
cfdisk /dev/sdb

When you install mdadm, it’ll ask you if you want to boot if the primary boot partition is degraded, i.e. one of the primary drives has failed. You will want to answer Yes to this now, but, can change this later. Since we are creating a Raid-1 partition that is already degraded, it’ll prevent your system from booting. Create your partitions

mdadm --create --run --metadata=0.90 --force --level=1 --raid-devices=1 /dev/md127 /dev/sdb1

We create the raid partition using the old metadata 0.9.0 just in case you ever use a kernel that doesn’t have an initrd.

mkfs -t ext4 /dev/md127
mount /dev/md0 /mnt
rsync -aplx / /mnt/
rsync -aplx /dev/ /mnt/dev/
vi /etc/default/grub

Uncomment the line that says: GRUB_DISABLE_LINUX-UUID=true so that it is enabled.

dpkg-reconfigure grub-pc

Make sure grub is written to the second drive.

vi /mnt/etc/fstab

change reference from /dev/sda1 to /dev/md127

Halt the machine, remove the primary drive, make sure grub boots into the raid volume properly. sdb will now become sda. If on boot, grub complains that it cannot find the root disk, or, goes into rescue mode:

insmod normal
normal

Then, it should boot. You might need to hit e to edit the command line to set the linux kernel option for root to root=/dev/md127

After the system comes up, run

dpkg-reconfigure grub-pc

to get everything reconfigured properly. It’ll say that it detected a drive that wasn’t in the boot sequence and prompt to rewrite grub. Why mint changes the device to md127, I don’t know.

After you’ve done that, halt, reconnect the old ‘sda’ as ‘sdb’ and bring the system up.

At this point, you are booting off the raid set and we just need to make a few changes to the raid configuration, then, add the other drive to the raid set.

cfdisk /dev/sdb

Change the partition type from 83 to FD. You might need to reboot if your controller doesn’t properly handle the ioctl change and/or tells you that /dev/sdb1 is too small to join the array.

mdadm --grow /dev/md127 --raid-disks=2 --force
mdadm --add /dev/md127 /dev/sdb1
echo 40000 > /proc/sys/dev/raid/speed_limit_min
echo 100000 > /proc/sys/dev/raid/speed_limit_max
watch cat /proc/mdstat
/usr/share/mdadm/mkconf > /etc/mdadm/mdadm.conf
dpkg-reconfigure mdadm

The last command will show you the progress as the partition is being mirrored. Once this finishes, you should be set.

Reboot your machine and it should come up running Raid 1 on the boot drive.

One of our client systems has a Raid 1 setup using two 1 Terabyte drives. Last night, Debian’s consistency check launched, but, his system was doing some heavy disk IO due to some scripts that were being processed and the system was estimating close to 1000 hours to complete the check.

md3 : active raid1 sdb8[1] sda8[0]
      962108608 blocks [2/2] [UU]
      [===>.................]  check = 15.1% (145325952/962108608) finish=60402.6min speed=224K/sec

To abandon the check, we issued:

echo idle > /sys/block/md3/md/sync_action

Which allowed the machine to skip the rest of the test. While I don’t like disabling the checks, we’ll reschedule this one to do the check after they are done doing their work.