Vat of Knowledge

I’ve been working with HP Storage Mirroring v. 5 lately, in an effort to create replication redundancy between two HP StorageWorks AiO 1200 servers. Seems to work very well, but be careful how you configure it, and be sure where you are replicating to. I’m going in to storage pretty green, and have been picking up bits and pieces with regard to Microsoft’s iSCSI implementation.

We were looking to replicate our file shares from one AiO to the other, and the only storage space that we had to hold the replica ended up being an iSCSI virtual hard disk (VHD). Not good, it caused all kinds of corruption and problems, since this VHD was an iSCSI target for two clustered SQL servers. The SQL database itself was only using 6 GB out of a 2 TB volume, so there was indeed plenty of space, but I don’t recommend trying to replicate to a locally mounted VHD on the NAS. It started to replicate, and then the target folders were being created, then simply disappeared. After deleting the replications sets, it was not possible to re-claim the lost space since the folders had disappeared of course. Our plan of action was fairly straight forward. Since we did not need 2 TB of usable space strictly for SQL, I created a new iSCSI target of around 400 GB.

During a maintenance window, I disconnected both iSCSI initiators, and set up the 400 GB iSCSI target and reattached the primary node’s iSCSI initiator. I then formatted the 400 GB partition and used robocopy to copy the data from the 2 TB to the 400 GB. I then removed the 2 TB VHD from the target, and re-assigned the 400 GB partition drive letter on the primary node to match the old drive letter, and re-created the physical disk resource. I then brought the second node online and all data was intact. After re-mounting the old 2 TB VHD, I was able to see the missing folders! Which was beside the point anyway, since we just removed the VHD anyway to claim more space on the array it was on.

We want to not only replicate file shares, but iSCSI VHD’s as well. This has posed some problems as well. There is one document I have found at http://h71028.www7.hp.com/ERC/downloads/4AA2-0555ENW.pdf that describes how to set up replication for VHD’s as well as recover them. Do not try this at home! I first tested the quorum drive, which seemed to replicate fine. I then went on to replicate one of our SQL VHD’s of around 230 GB and it screwed up. After failing, it disconnected the iSCSI initiators and I found out at that time that the quorum drive had become corrupt as well. Fortunately all the SQL data was fine, but I ended up having to do disaster recovery to repair the quorum for both cluster nodes to get the cluster back up again.

I had to build a new iSCSI VHD and assign it to the target, connect the primary node and format the new Q: drive. I then had to start the cluster service using the /fixquorum switch to even get the cluster admin running again. I then had to create the new Q: drive physical disk resource using the new Q: drive. I then had to use the clusterrecovery.exe (click to download) to move the old Q: drive resource to the new Q: drive resource, which moves the signatures to the new disk, etc. After completing that the tool renames the old Q: resource to Q: Drive (old), and it can be removed once you get both cluster nodes back online. Fun stuff…