NSX Distributed Firewall and Log Insight

Working with NSX Distributed Firewall I needed a way to help me audit the impact of a proposed firewall rule on specific systems. In the physical firewall world I would normally create a rule for the traffic and allow it, but log the allow. As I dug into Syslog for the distributed firewall I discovered that firewall log messages are sent to the same Syslog collector as other ESXi messages, firewall messages can't be directed to a security specific syslog server.

I honestly hadn't spent much time with Log Insight, but I knew we had the lab servers pointed to a Log Insight box so I began to dig around with it. I was honestly impressed with how intuitive and easy to use LogInsight is. The field extraction functionally of Interactive Analytics let's you quickly slice and dice massive amounts of log data exactly as you would like to see it, and I didn't have to learn another query language. Everything "Just Works" and I was able to isolate firewall logs down to specific systems and rule ID's quickly.

Since there doesn't appear to be an official NSX Content Pack for Log Insight yet I wrote a quick community content pack for Distributed Firewall. It can be downloaded from the VMTN Community Forums here.

The pack extracts the Rule ID, Pass / Drop, Rule ID, Source IP, Source Port, Destination IP, and Destination Port fields. It includes a dashboard with a few charts, and some queries to get you started in interactive analytics to slice and dice the syslog data.

Instructions on importing the content pack are here, once it is imported you can select the Distributed Firewall dashboard by clicking on the dashboard menu at the top left.

The built in queries included on the dashboard are :

• Count of drop events over time grouped by Rule ID
• Count of drops grouped by Source IP
• Count of drops grouped by Destination Port, Destination IP
• Count of Allow-Log events over time grouped by Rule ID
• Count of Allow-Log events grouped by Source IP
• Count of Allow-Log events grouped by Destination IP, Destination Port

For my use case of analyzing the impact of a rule on the environment I would start with the "Count of Allow-Log events grouped by Rule ID" query. Then select the rule ID I'd like to analyze, and add a filter for a specific source IP if I needed to narrow it down further.

Once I have my lab rebuilt I'll add a short video demo.

Read More

Homelab Build : Dell dcs6005 / 6105 FreeNAS and ESXi Lab

As I prepare to change jobs one of the things I will miss about my current employer is the LAB (capitalized because a lab this awesome deserves it). Nexus 7k, 6k, 5k, two UCS fabric, UCS blades, and UCS C460s all backed by a VNX plus whatever storage they are beta testing for EMC.

I'll probably never be able to replicate the level of lab I had access to outside of VCE or Cisco, but I need some type of home lab to continue my work with VCAC, LogInsight, NSX and to continue preparing for my VCDX defense. Two 16GB MAC minis with a synology would make a nice, quiet, cool, power efficient homelab with high wife acceptance. Unfortunately I need more RAM than that, and the cost can get quite high.

I looked into Intel NUCs and white boxes, but the best value for me turned out to be older Dell "cloud systems" boxes that are wholesaled on eBay. These boxes are a 2U chassis designed to house four individual servers allowing shared web-hosting companies to drive server density. They don't have the intelligence, IO flexibility and blade removal capability of a real blade chassis and are simply designed to provide cheap density.

There are two main flavors of these boxes, the C6100 which is Intel powered and the dsc6005 / C6105 that is powered by the AMD 6 Core Opteron 2419 EE. The Intel powered option has gone up in price, but there are currently a flood of the AMD powered boxes selling on Ebay for good prices. For $479 I got a chassis with three dual socket servers with each server having two Six-Core Opterons and 32GB of RAM. There are 12 3.5in drive bays on the front, and each server is wired to four drive bays.

Dive into the detailed build out after the break. 

Read More

VCAP5-DCD Exam Experience and Prep Tips

As soon as I passed the VCAP5-DCA I scheduled the VCAP5-DCD exam. I scheduled the DCD exam for 30 days after the VCAP5-DCA exam.

The registration process for the DCD exam is identical to the DCA exam. You need pre-approval from VMware and a registration code in order to register for the exam on the Pearson-Vue site. The exam is given at the "Pearson Professional" center which could impact exam scheduling. There are only a few Pearson Professional exam centers in each region, and they are quite busy with non-IT exams.

I went into the exam without an expectation of passing. I was hoping for a passing score, but honestly I got burned by the DCA so bad I didn't expect to pass this. About half way through the exam I was feeling better, but I was still surprised with I clicked finish and got my score and passed. Unlike the DCA you get your score immediately when you finish the exam.

Just like the DCA exam time management plays a big role in the DCD exam. This exam has multiple choice, drag and drop, and "visio like" questions. What worked for me from a time management perspective was going straight through the exam answering all of the non-visio questions, then going back and answering the visio questions. You do need to make sure you have enough time left to tackle the visio questions, I had 105 minutes left when I started the visio questions and 30 minutes left when I finished and started reviewing other questions.

This is a difficult exam to prepare for. It feels very "cumulative", not just from a VMware perspective but from an IT Infrastructure perspective. If you have years of experince with large scale IT projects you will have a cumulative knowledge that will help you with this exam. I would take this exam after the VCAP5-DCA, there is some crossover material on the blueprint for this exam that could help.

Tips :

• Use good test taking strategy like any multiple choice test. Eliminate answers that can't be correct. Flag questions you don't know in case you run into the answer later.
• Track your time. Make sure you get all of the easy, quick, points out of the way first. Make sure you keep enough time for the visio questions. 

Prep Materials -

• Pluralsight - Designing VMware Infrastructure by Scott Lowe
• VCAP5-DCD Certification guide by Paul McSharry 

If you have the cumulative knowledge from years of IT projects the Pluralsight course and exam guide should be all you need to prepare for the exam. If you are newer to IT or haven't worked in a large environment with a formalized design process this exam could be very difficult. 

Read More

VCAP5-DCA Exam Experience and Prep Tips

I had trouble registering for the exam. The process is to request authorization from VMware, then once authorized register for the exam with Pearson Vue. My account got wonky and it took more phone calls that it should have to get registered for the exam. 

The exam is only offered at Pearson "Professional" centers, so isn't offered at the normal IT training places that offer testing. In a metro of 1M people I probably have 8 testing centers where I could have taken the VCP, but only two that offer the VCAP. I registered for the March 18th, and the date snuck up on me really fast.

Exam Day Experience

I don't drink coffee, but on the morning of the exam I was careful with the Diet Coke. I didn't want to be bogged down from lack of caffeine, but I also didn't want to be jittery. The other consideration is that time is precious on this exam; If you have to take a bio break the clock is still running. I normally eat lunch early and my exam was scheduled to run until 1:45. Since no food or drinks are allowed I ate a snack in the test center parking lot. 

Signing in at the test center is like the normal VMware process, except a little more formal. I got some exam center rules to read over, presented my ID, and was seated for the exam. 

My biggest enemy on the exam was time. Make sure you don't go down the rabbit hole on a question and spend too much time. I numbered out the questions on my whiteboard. If I didn't know how to do the task without going to the docs I drew a circle next to the number. If i started the question but I needed to work ahead I while a process finished I marked it with a slash, so I would remember to return to the question. Once a question was complete I made another slash to make an X. I found this helpful to keep track of my progress. When I went back to a circle I marked it with a slash or X as I made progress. 

Don't waste time, If a question is taking too much time move on. Make sure you get all of the points you know before you start spending time figuring out something you don't know. 

This was the first time I've run out of time during an exam. At the end of the exam I was working back through my circle questions, but had just started on them. I feel like I hit enough to pass, but I also wouldn't be surprised if I just missed it and have to test again. The worst part will be waiting for the results.

Prep tips after the break. 

Read More

Building a Nested ESXi Lab on VMware Workstation

If you are studying for the VCAP5-DCA you definitely need a lab. If you are studying for the VCP probably need a lab unless you are in vCenter all day at work. Nested virtualization runs one hypervisor upon another; so a nested ESXi lab runs the ESXi hypervisor on another hypervisor like VMware Workstation. So why build a nested lab instead of a physical lab?

Flexibility - A nested lab on workstation is going to provide more flexibility than a physical lab.  I have both, and I love having the ability to create another ESXi host in minutes by cloning it from a template. I can also turn off my 5.0 lab I am using to study for the VCAP5-DCA and turn on my 5.5 lab and show a coworker a new feature.

Cost - The cost of a nested lab can be cheaper than the cost of a physical lab, especially if you have box that you can simply upgrade the RAM in. When building a computer to run a nested lab the cost could be similar or more than buying used servers from eBay, but the power consumption should be much less. Building a low power solution like Intel NUC or MAC Mini's combined with a Synology will cost more than building a nested lab.

Portability - A small nested lab can run on a laptop allowing you to study on the road.

What do I need to build a nested lab?

Computer - One that supports VT-x (or the AMD version). Preferably one that supports EPT; without EPT support you will be limited to running 32bit guest virtual machines inside your nested ESXi instances. The 32bit restriction isn't a big deal, but it would be nice to not have to deal with it. If you are unsure of the virtualization features of your processor you can look it up at the Intel or AMD site. You should be aware that these features may not be on by default, you will need to check in the BIOS. 

RAM - Lot's of RAM. Did I mention RAM? Can you afford any more RAM? With ESXi 5.0 8GB of ram would allow you to get two ESXi hosts, vCenter, and an openfiler running. ESXi 5.5 brings higher minimum RAM requirements with all of the new features, 16GB really becomes the new minimum for two hosts, a vCenter, and an openfiler. If you want to lab larger scenarios like SRM or NSX you will need 32GB and up. 

I have a a Dell Precision T7500 Workstation with 48GB of RAM I jumped on when an engineer from our HPC group upgraded to newer model. It has an older processor, the Intel Xeon E5507, but it is quad core and supports VT-x with EPT so it meets my needs.  

VMware Workstation - Fusion will work as well, but I like the interface and memory overcomitment of Workstation. If you have your VCP they were providing workstation license keys upon passing, I'm not 100% sure if they still are. If you are a VMUG Advantage subscriber one of the benefits is a discount on the Workstation license. 

Dive into configuration after the break.

Read More

Multi-NIC vMotion on ESXi 5.5

What is Multi-NIC vMotion?

Multi-NIC vMotion allows you to send multiple vMotion streams (even when migrating a single virtual machine), and if configured properly provides a higher overall throughput to the vMotion process. The configuration is straightforward, the vMotion service is configured on multiple VMkernel adapters, and each VMkernel adapter is associated with a single physical interface. With multiple gigabit interfaces the throughput of the vMotion migration scales linearly with amount of adapters used, with 10G you will very likely hit some other performance barriers once you have more than two 10G interfaces involved. 

This feature was added in ESXi 5.0 and remains mostly unchanged. In my experience it is not a heavily used feature, although I depend on it in our production environment and have been running it since early 2012 in 5.0 and now 5.5.

Why do I need it?

There are two scenarios where the available throughput for vMotion can come into play. The first is simple, hosts with a large amount of RAM and a very large number of VMs. The second can be a little more complicated, virtual machines under heavy load that are dirtying memory pages very rapidly. 

The benefit of mulit-NIC vMotion when dealing with large hosts with many virtual machines is obvious. We have many hosts with 1TB of ram, and a few hosts that have 2TB of ram. Placing these hosts into maintenance mode would take FOREVER over a single gig interface, and quite some time over a single 10G interface. 

The second use case is for migrating a large single VM under heavy load. During vMotion the guests memory is copied, then a delta is copied containing the pages that were changed or "dirtied" during the first copy. If the guest is dirtying memory pages faster than they can be copied this will cause Stun During Page Send to kick in and slow the rate that the guest OS can dirty the memory pages so the copy can finish. This stun has caused us some problems with big databases servers, and the only way to avoid it is to throw more bandwidth at vMotion. 

Dive into configuration after the break.

Read More

Removing the Nexus 1000v

The other day I needed to remove three nexus 1000v distributed switches from one of our lab vCenter environments in order to prepare for NSX testing. Removing the Nexus 1000v should be a fairly straightforward process. In my case it seemed like the supervisor modules had become self aware and knew I was trying to kill them.

The first step is to use the migration tool and migrate all virtual machine and VMkernel networking to a standard or VMware distributed switch.

Once all VM and VMkernel networking is migrated the next task is to remove the hosts from the distributed switch object. Click Inventory, then Networking, and select the 1000v you wish to remove from the list of distributed switch objects. Click on the Hosts tab, right click on the host, and select Remove from vSphere Distributed Switch. Repeat for each host.

Read More

Convert an older Linksys wireless router to a wireless bridge.

I've got a couple of devices at the house that don't have integrated wifi (Dish Hopper and Onkyo Receiver), but it would be nice to get them on the network. I can't fish Cat5 to the location easily and the wifi add-on adapters from Dish and Onkyo are expensive.

My solution to this problem was to take my old Linksys WRT54GL running DD-WRT firmware and convert it from an AP to a wireless bridge that can connect these two wired devices to my home wireless (provided by my Asus RT-NSSU). This solution is free since I have the Linksys sitting around; I recently replaced it with the Asus.

1. The first thing you need is an old wireless router (secondary router). I used a WRT54GL from linksys, an 802.11G box. There are several "aftermarket" software versions for these routers but I know this will work with DD-WRT. DD-WRT is easy to install and there is plenty of information to help you get it on your router if you are still running the stock software. 
2. I needed to make a couple of changes to the primary Asus router to let the secondary router connect to the wireless. On the 2.4Ghz wireless settings I changed the authentication method to "WPA-Auto-Personal" and the encryption to "TKIP+AES". This allows less secure older WPA clients to connect.
3. Connect to the secondary router LAN port, and access the admin page.
4. Reset the secondary router to factory defaults. 
5. Change the IP address. It will default to 192.168.1.1, which is probably in use by your primary router. 
6. Navigate to the Wireless and Basic Settings tab. Change the wireless mode from the default of AP to "Client Bridge" and click Apply Settings. Configure the network name field with the wireless SSID used on your primary router. Click Apply Settings.
   
7. Navigate to the Wireless Security tab. Set the security mode to WPA Personal, and the encryption to TKIP+AES. Enter the shared key in use on the wireless network.
   
8. The wireless light on the secondary router should turn solid green, and at this point you should be able to pull DHCP from the primary router and access the internet while plugged into the LAN ports on the secondary router. It should be ready to hook up your non-wifi devices. 

NOTE : I had trouble using WPA2 with AES, I had to set the primary router to WPA Auto and the secondary router to WPA to get this to work. This does expose the network to having the key compromised. I don't consider this a risk at in my environment but it is something you should be aware of. 

Read More