From time to time I think that we all sometimes overlook the value of virtualization. I know that I do, and it came home to me today. We have a problem with storage, it seems we can’t get enough! Over the summer we migrated from an older 7TB fiber channel SAN, to a new 13TB iSCSI SAN. Since then we have watched as more and more of our storage has been consumed. We hit a crisis mode this past weekend when the 3.5TB lun we had allocated for user data, quite literally filled up. Feel free to yell at me for not being more mindful or more proactive, it’s a crazy situation and I don’t know if I had to do it over, I would make the same choices.
Since we run a Storage Server cluster we run a report, and based on those we can tell you based on file ownership who our top consumers are. Needless to say the same handful of users are typically on top. In particular one user’s consumption ballooned by maybe 300%, their data quite literally occupied 1/6th of the entire 3.5TB! After going back and forth about possible solutions, we decided that the data should remain on the server, but be moved to a different lun. The only problem is none of our other luns had enough space to handle that amount of data, so we had to extend one of our luns. In case you’re curious about how long it took to extend, I started the process yesterday morning and it was still going when I left last night, but was done when I got in to my office this morning.
The fun part is what happened next. From Microsoft, presenting iSCSI luns to a server as dynamic volumes is not supported. From experience we can confirm that it is very painful, especially on reboots. Carson tracked that down one day while I was on vacation, I don’t think he’s documented that so I’ll bug him. All of the luns on our cluster are Basic disks, and as you know you cannot extend a basic disk. That is unless you know how to work DiskPart! We used this tool early on when we were troubleshooting computer imaging issues, and one of the nifty things it can do is extend the size of a disk.
We are just a trifle leery of fiddling with the drives on the cluster since it’s in production and at the core of most of the services we provide. This is where virtualization steps in, thankfully we implemented a VMware cluster a few years ago, and most of our server infrastructure is now virtualized. I have a machine, which I named “Schmoopy”, that I tend to abuse quite regularly so Carson pointed out that you can extend the size of a virtual disk from the VMware interface.
So we fired up the VMware Infrastructure client, browsed to Schmoopy and added a disk. Keep in mind this was all done while the server was running. We verified that the disk showed up in Disk Management, we then initialized the basic disk and formatted it. Back in the VMware interface we then changed the size of the disk from 8GB to 20Gb, Schmoopy reported that it now had a chunk of free space at the end of the newly created disk. We fired up the command line and followed the instructions from an article we found on the Windows IT Pro website. We then went back to Disk Management and verified we had a newly available 20GB drive. The process went just as smoothly on the production server and the user’s data took the rest of the day to move!
It was most certainly the highlight of our day, and like I said earlier, it really made me aware of how valuable virtualization can be for everything we as IT Pro’s do!