Small Business Upgrade to Virtualization...

[Bacon8tor]

Alright all, so I have been thinking about doing this log for an upgrade I am performing at work. Background I am all self taught, only been into IT for about 4 years now, actually started working in the IT world for about 1 year and a half, luckily the company I have been working at for 5 years(started off as a telemarketer) gave me the opportunity to be an IT assistant for sometime, then my boss and only other person in the IT department got a new job and left with onyl 2 days notice. So I dived in head first with a hefty pay raise, man let me tell you about the mess I ran into. 

 

The Old Hardware: 

5 Windows servers:

2x Dell Poweredge 2950 2x Dell 850

1x Dell xeon tower(not sure of model) 

2x Dell Poweredge 2850 

 

All of these windows servers are running Windows Server 2003 r2 . . . . . . . .  ummmm WTF . . . . . . EXTREMELY OUTDATED!!!!

I do have one server which is set up as a file share that has win 2008 r2(why i have no idea). 

 

on to the rest 

11

1x Dell PowerEdge 

4x Dell SC1425 

2x Dell 1950

4 Built PC's not sure of exact specs all have Xeons

 

haha my worst nightmare now.... the network....

Spoiler

 

 

 

 

 

This is a F*$&*#$&# MESS . 

 

talk about a heart attack. Ok now they got the defibrillator out.....  what ever the 2 it ppl that were in the position before me were smoking out of there pipe must have been some crazy shit. following a cable in here is like threading a 20 gauge wire thru a needle. I am constantly afraid i will accidentally unplug something while trying to see where a cable is going. there are cords unplugged not sure why or where they go.... Now reason I have left it like this is because all of the switches must go. They are a HUGE Bottleneck they are all 10/100 switches. yes thats right no gigabit not even server to server communication. RIDICULOUS.  

 

 

Hardware Upgrade Plans

What My original plans were to convert all servers into virtual machines on an Esxi Host with a San or NAS attached making managing the machines like a million times easier. I Have never done this before and only experience with virtual machines was using vmware player to test/make specific development computers/also as a work computer on my home pc.So at first I reached out to a few IT companies to maybe assist with the upgrade as hardware matching can be extremely difficult with Esxi hosts, but after getting a few quotes over $150,000 and knowing that the company wouldnt fork over nearly that much money, I decided for myself(great learning experience) and the company that I had to just take care of this. One of the main problems with Outsourcing it was the companies would only use the vary newest models of whatever brand Dell/hp/cisco, one went as far to try and sell me a technology that cisco just released a couple weeks before and then couldnt even get pricing for it. So I went to VMwares compatibility guide and started looking around at some host. I have like the dell machines they are fairly easy to work on , fairly easy to find parts, Looking for something with similiar features that the ESxi host that other companies had recommended, I found a Dell R710 Hex-core with 48 gb of ram installed for $400 thru pcserverparts on ebay, they had a 5 star rating and offer a 30 day guarantee. 

 

Next was the Nas 

 

I had determined I needed about 8 TB of usable storage, but also needed redundancy, also performance would be nice although not super needed. I opted for RAID 10 in sets of 4x 1 tb drives giving me just under 2Tb(1.8 to be exact) of usable storage. 

Next was the case 

On one of linus' video he had a norco 4224 which looked like what i needed and it was fairly priced on newegg but I opted for just the 20-bay version Norco-RPC-4220

 

At this point I was unsure what OS for the NAS i wanted to use, unfortunately vmware Vsan software would not have worked because that uses the drives in the Esxi host as a san, I wante a seperate unit. FreeNas , i was just unsure about using it in a enterprise enviroment, although like I said demand on the drives is fairly small. 

I looked at buying a isx systems machine with TrueNAS , basically a enterprise version of Freenas+support, but I wanted to make this machine and support..pshhhh i got this. 

also there was Open-e's DSS v.7, I tested this out withand it seemed vary similiar to freenas but was super basic. 

 

I opted for FreeNAS, had some experience I knew it could do everything I needed, and even more than I needed. 

 

so this is the hardware for the NAS 

 

http://pcpartpicker.com/p/cQqLvK 

here is the PSU - Newegg

HBA (LSI) card -  Newegg

and some Sff-8087 cables 

Once I actually got all these pieces 

Spoiler

 

Im going to start a new post with the building, will be up shortly 

 

[Mr.Meerkat]

This is going to be an interesting change instead of the usual $5k full WC loop build logs  

 

And for the servers, I hope you get what you wanted for your business in the end  

[Bacon8tor]

Alright so building the NAS, I kinda forgot in building this the R710 comes all built and ready just have to install OS. 

 

So in typically Optimistic fashion I just went straight for the kill and just installed everything in the case without testing it. 

also side note I am using a spare antec hcg-900 cause I did not have  the redundant psu's yet . 

Installation went very easy and absolutely smooth, everything installed fine wasnt really worried about clearance on anything nothing was big and bulky. 

Spoiler

Once I booted the machine up POST'd the first time, I was jumping for joy, no memory issues, everything was recognized. 

One thing I will note about the cables to the backplane for the hardrives 

Spoiler

as you can see space is extremely tight. what I did was remove this bracket that the 120mm fans are attached to plugged in all the (f&*$) molex adapters. then put the bracket back on(big mistake forgot about the SFF-8007 plugs) . 

 

Yup once i got the HBA i had to remove the fans again and run all the 8087 cables. 

 

Spoiler

 

 

Once I got the HBA installed I moved on to installing FreeNAS, which I ran into some problems with the usb drive mounting when trying to installed I tried every usb port and several usb sticks of varying size nothing worked, well poop. 

Then I thought hmmm cd should work, and IT DID! wooohoo got freenas installed on the 16gb flash drive that conveniently had a nice spot on the motherboard WHICH was AWESOME i didnt have to worry about the usb drive sticking out. I had originally bought some usb header to usb that I was planning on just leaving dangling in the case, this solution was better. 

Next install all hard drives in cages. 

self explanatory, 

 

One thing That kinda stumped me at this point, and prolly my inexperience with using HBA's ,

But I had the hardest time finding a 5 port HBA i could not find a reasonable one, so I got a 4 port and figured 4 backplanes into the HBA then 1 backplane to 4 sata ports on the MOBO. 

Once I got into freeNas it was only seeing drives attached to the HBA...????? at this point I just rearranged the cables so the the 4 top bays were into the HBA and the last row is not connected. this last most bottom row I was going to use for hot spare drives, incase I am not around if a drive fails, I can feel a little easier and buys me a little more time. also I did not have enough HDD to fill the bays up anyway i started with 12 drives. leaving 2 rows or 8 drive bays open. 

 

then all worked well, Notice I am not using any sort of Hardware raid as freenas states not to use hardware i made sure the HBA does not do raid. setup the Raids so that the each row of drives is in a raid 10. 

 

got it hooked up to the network and installed in rack... next the Dell R710 install. 

(might be a day or so Just got the exact same parts for the backup nas as I was writing this so time to work now...... I Guess)

 

 

[Bacon8tor]

20 minutes ago, Mr.Meerkat said:

This is going to be an interesting change instead of the usual $5k full WC loop build logs  

 

And for the servers, I hope you get what you wanted for your business in the end  

Thanks, I figured not to many get this chance, or document it. I am surely happy, and will make my job/life alot easier. 

[crosstiger]

Good luck mate! Ask if you need a beer to cool down

 

[unijab]

How do you plan to setup the drives in pools? 

What ZFS level?

[Bacon8tor]

2 hours ago, unijab said:

How do you plan to setup the drives in pools? 

What ZFS level?

So since each row has 4 drives. I set up each one in a RAID 10 within freeNAS, I found the easiest way is to use the wizard it has some presets that will setup a drive configuration although I have to put 4 drives in use the wizard put the nest 4 in and use the wizard, etc. the poor image below kinda shows you. Then I make a zvol size depending on how big the virtual machine I plan to put on it. add that as an extent within freeNAS then I add the iSCSI target to that extent, Within the ESXi host you add the iSCSI target. So far thats how I found the best way to do it is. seems to be working amazing. 

Then I plan to use the backup nas as such, it will Rsync with the main NAS every hour, if I ever need to take down the primary NAS i will switch the IP's on the backup nas to the Primary nas reboot esxi host and all should be well( THIS IS JUST A THEORY, I have found no way to do this otherwise.) 

[connor201]

As a fellow systems administrator I would never buy production hardware (the R710) off ebay buying new has the advantage of next day replacement parts from dell and it is a great thing. Also with virtualization you want to be careful you do not want to put all your eggs in one basket you traditionally want to run 2 ESX hosts with primary DC on one and a Secondary DC on the other, by running all on one host you run into the problem that if something happens to the hardware all of your hosts are down you may want to look into the idea of n+1 virtualization. Also are you going to be doing a P2V for the conversion or are you going to be recreating the servers from scratch? 

[Bacon8tor]

2 hours ago, connor201 said:

As a fellow systems administrator I would never buy production hardware (the R710) off ebay buying new has the advantage of next day replacement parts from dell and it is a great thing. Also with virtualization you want to be careful you do not want to put all your eggs in one basket you traditionally want to run 2 ESX hosts with primary DC on one and a Secondary DC on the other, by running all on one host you run into the problem that if something happens to the hardware all of your hosts are down you may want to look into the idea of n+1 virtualization. Also are you going to be doing a P2V for the conversion or are you going to be recreating the servers from scratch? 

I completely agree. I am planning on using 2 host and 1 back up. I initially bought just 1 r710 and 1 nas. To make sure it actually worked and was possible for me to do . since i got it up and running last week this week i ordered the 2nd host and nas. Then ill order back up stuff. 

I am converting most of the linux machines p2v the windows machines i will not cause i have to upgrade them to win 2012. So ill migrate data as much as i can from those avoid having to recreate domain controller and what not. 

I have about 4 linux machines already converted these were some of the smaller machines. They converted with little problems . i did convert 1 windows machine kinda for funsies. 

[unijab]

Why not ZFS2 instead of raid10?

[Bacon8tor]

On 4/3/2016 at 5:40 PM, unijab said:

Why not ZFS2 instead of raid10?

My understanding is you get better performance with RAID10 since with a zfs it is using that single parity disk or 2 but this limits the speed to that one drives max write/read speed

[AlexanderWeeks]

Is there any update to what happened? Reading all of this rather late, but want to figure out if all was resolved, or if there is still a gargantuan mess on your hands.

[Bacon8tor]

13 hours ago, AlexanderWeeks said:

Is there any update to what happened? Reading all of this rather late, but want to figure out if all was resolved, or if there is still a gargantuan mess on your hands.

Yea sure sry I didnt update this . . .  Kinda forgot and have been busy. 

 

So far I have 3 Esxi host setup , I purchased Dell R710's from ebay, and before all the other IT Admins start saying to buy from manufacturers to get warranties and replacement parts quick, yes I know this but I did not have that option and needed to find something. Yes I took a HUGE gamble on buying from ebay but I also did not buy from a random seller I did my research, the seller was PC server parts http://stores.ebay.com/PC-Servers-and-Parts-Inc they have a very good rating as a seller and I actually called them before hand and got some more information about the machines, which was nice they helped and were professional as I could ask. Not to mention I paid $430 for a dell R710 with 2x Intel Xeon X5650 Hex-core , 48GB of Ram , dell Perc 6/i . and upon arrival was extremely well packed in a double box with the expanding spray foam( I forgot the name)  once I received and setup the first machine I had the business buy another for my second host. Both installed ESXi without a hitch , setup IPMI on both allowing me to power on and off remotely. 

 

For the NAS' so Im a little drunk now and I dont want to take the time to find the exact parts and I dont believe they are in a previous post so Ill update that later. But I bought Norco 20 bay Hotswappable case , ASRock server LGA1151 mobo , forget what kind of xeon and 16gb of ram , LSi HBA card that has 4 SFF8087 connections . working within the norco case as stated in the previous post was a BITCH , really jsut the part of plugging in the stuff to the backplane the molex and sff8087 were hard to get especially with my large hands.  Other than that the Machines have been operating smoothly come to find out that the ASrock board I purchased in the install manual show you how to install FREEnas over the IPMI , I didnt not do this but neat, and also sweet that the board obviously is compatible with freeNAS, Yup if you didnt knwo I am using FreeNAS in a enterprise environment.  And dont regret it one bit its working amazing, and my options arent tied like going with Vsan or Open-e(this one IMO is a clear rip of freenas but shittier and payware, with half the options).  

 

Network, this is the most recent problem I have been dealing with, first off I am not the best Network Admin , I mean I got the basics down but when trying to diagnose an issue deep within a switch , I am lost , predicting what a network would do is something that I can not do. I purchased a Cisco Managed Gigabit switch, from amazon for about $230. if youve read up I believe I mention all the switches in the business being 10/100 , ridiculous!!!!!!!!! But when I introduced the gigabit switch into the network I was unable to switch the still physical machines to the new switch and was only able to move the ESxi host to the new switch but then bridged(again a networking term I am prolly misusuing but I made a short cable to go from one switch to another.) This resulted in a ridiculous bottle neck on the internal network in loading some internal web pages from linux machines on the VM. (check out the first post with the nightmare pics) I found a temp fix for some reason whether its the fact that the internal network was taking a long time to resolve the hostname . not sure, but if the user jsut used the actual IP address of the server hosting the webpage it loaded almost immediately where as if you used the FQDN(fully qualified domain name)  it could take up to 10 minutes!!!!!!! I always was looking at the vm to see what the network was like on that and it wasnt being stressed or even at its limit telling me the bottle neck had to be within the 2 switches, so actually this morning I switched all ESXi host to the old 10/100 switch for sanity sake and people could load what they needed. 

 

 

This is why I am drunk.... lol .... Today for some reason the internet was extremely slow .. im talking dialup speed. . at best .. 1 mb down 2 up ..... we are suppose to have a 50 down 50 up connection ( which is bad enough for a business to have and I am working on getting this bandwidth up'd) so after countless staring at the switches realizing I had done nothing that would cause this I called our ISP which is century link(I HATE CENTURY LINK , had them at home and they suck) and submitted a ticket, so far they have sent emails saying its being evelaveted to another group or higher tier. I had to contact them , I was xtremely stressed at this point finally getting someone to read the notes and inform me of what was going on whether they saw an issue on there side or not. and they were seeing a loss in connection between them and my modem. SO phew it wasnt my doing(so far at this point).. I contacted century link at 11AM MST and its now 3:48PM MST still have not heard from century link, and Im drunk of a sixer of Stella Artois...... 

 

I will try to take some pics of the server room now as I have taken down about 7 or 8 physcial machines, the networking is still garbage but withing the next month I hope to have that and the Windows server upgraded to 2012 r2 at this point I may be able to have an actual weekend off, what a thought. . .  glad some are interested in this I will definitely try to keep this updated. 

 

Biggest challenge so far: Figuring out the correct Microsoft licenses --> CALS SUCK FUCKING BILL GATES DICK <_ block it if you want but we all know its true

[Mr.Meerkat]

3 minutes ago, Bacon8tor said:

-snip-

pls...this is a tad too long...1000+words in a single post/update...ermmmm (considering I'm a person who scan reads find this too long to read, I wonder how other normal readers would find this...), I may as well be looking at one of my high school essays right now...hehehe

[Bacon8tor]

13 minutes ago, Mr.Meerkat said:

pls...this is a tad too long...1000+words in a single post/update...ermmmm (considering I'm a person who scan reads find this too long to read, I wonder how other normal readers would find this...), I may as well be looking at one of my high school essays right now...hehehe

Fuck that you dont want to skim it or read it then go on, I am not re writing this for your pleasure

[Heesleemer]

23 minutes ago, Mr.Meerkat said:

pls...this is a tad too long...1000+words in a single post/update...ermmmm (considering I'm a person who scan reads find this too long to read, I wonder how other normal readers would find this...), I may as well be looking at one of my high school essays right now...hehehe

you want to read my senior thesis and my potential masters dissertation for me? The former is only 40 pages. 

[Bacon8tor]

37 minutes ago, Heesleemer said:

you want to read my senior thesis and my potential masters dissertation for me? The former is only 40 pages. 

whats it on? the senior thesis? or dissertation for that matter?

[Heesleemer]

13 hours ago, Bacon8tor said:

whats it on? the senior thesis? or dissertation for that matter?

Lol well my sarcasm derailed this. 

The Future of the United States Navy's carriers: UAVs, and Chinese Anti Access and Area Denial capabilities.