VMware’s NSX Distributed Firewall Review- Very cool NOT enterprise ready toy

OK folks, the word is out. NSX firewall (Distributed Firewall – DFW) has some seriously cool features

• Security groups
• Security tags
• Dynamic inheritance of policies on VMs, and group membership
• Inheritance hierarchy of firewall rules so you can build subzones
• Micro-segmentation…very cool…per host firewalls
• Ability to monitor traffic on a per nic basis for every host
• REST interface
• Integration into NSX environment for common monitoring (you can see rules and objects in logs)
• Integration into NSX for object lists (object definitions for vSphere items exist for you to consume)
• No separate upgrade for firewalls, part of the whole NSX upgrade package.
• Allow rule inheritance (haven’t tried it but shows potential, lot better then strictly Global/Local like in CP)

…..but will never cut it in an enterprise environment.  It just doesn’t scale, you need an army of analysts to maintain. What I call a ‘write once’ firewall, you write the rules once and never go back to maintain it. It’s times like this that remind me how mature CP management and logging are (and forget about all the bugs).

SUMMARY: Security composer very cool just don’t use it. Easy to write rules, a nightmare to maintain.

I just wish Vmware and CP could merge their firewall efforts. DFW has some really cool ideas, they just need CP’s 25+ year enterprise management mindset.

Here is what I know….

(WORK IN PROGRESS, These are my notes as I am learning. I may update them as I verify the good/bad/ugly. Put grain of salt….)

• vSphere NSX web client is a pretty poor shaky bug ridden java mucus patch, they must of hired some Cisco CSM developers.
  
• Rule administration is done from several different screens so you jump around a lot. Firewall, Security Composer, vCenter VM summary, vCenter VM manage firewall rules, security tags…are all related to rule administration. All in different windows.
• If you are creating a rule in service composer and need to add a unknown service, you have to kill the whole rule creation process and first create the service and then go back and redo your work creating the rule. Not true for the firewall management window, you can create a new service.
• If you want to create a single rule for a single SRC/DST it is easy to use the Firewall manager. But be aware that there is only 1 humongous rule base for all your rules. If you split the one’sies from the more complex rulesets in Security Composer, then you will have 2 rulebases to manage. Write once, no win-win. The micro-segments are broken into SECTION headers but they all still appear as one huge database.
• Basic problem with all new products that don’t scale. Indirection. In order to make a product scale you can’t have tons of base objects. For example in firewalls you can’t have JUST IP addresses in your rules. There would be millions of rules. So you use host objects WebServerA=1.1.1.1 and groups WebServersGroup includes WebServerA/B/C….. and hierarchy of groups EnterpriseServes include WebServersGroup. This is called indirection. You have to go through several references before you can get to 1.1.1.1Well DFW SecurityGroups are very special when it comes to indirection. #1 it can be deeply hierarchical with references to other SecurityGroups and Object groups and #2 it is DYNAMIC, changes at run time. Very powerful and dangerous at same time.With any product, when you see indirection like this you have to look at the back end support. If someone says “1.1.1.1 is slow, fix it” how do you find 1.1.1.1 if its buried in layers of indirection? Lots of new fancy tools have indirection to build the schema, but few good products have tools to unwind and find the base objects. DFW does not have this capability. It cannot search through a hierarchy of SecurityGroups. So debugging is painful. I have spent hours with NSX support looking through backend databases trying to figure out where an object is all defined. Very painful.Write Once Firewall. You write it once, but you can never modify debug it.

  This is where CheckPoint shines. Unwinding indirection is CP’s sweet spot. They have many tools to search for buried objects and manage them. THUS the reason it is my #1 pick for enterprise management.

• Security composer forces you to write rules as:
  # 1 FROM: specificzone TO: whatever
  you always have to specify a zone in the FROM/TO. You can’t specify:
  #2 FROM: any  TO: any
  I think they do this because in the end the compile all rules for NSX into one huge rulebase. So if they allows #2, then it would override all the rules 2+
  NOTE: There is a APPLY TO parameter to make this work. Kind of hidden.
• Because there is 1 humoungous rulebase that applies to all VMs its easy to break micro-segmentation. Imagine you have a zone rule:RULE: 100: FROM ZONEA TO ZONEB DENYYou will have to look through all the rules ABOVE rule 100 to make sure your RULE 100 works as you think it should. Imagine 1 day after you put in RULE 100, someone might create RULE 1 which overrides your microsegmentation.RULE: 1     : ANY ANY ANY ANY DENY
  RULE: 100: FROM ZONEA TO ZONEB DENYImagine trying to do a rule cleanup? After a couple years pulling out RULE 1 would be a nightmare.The problem is rules are in 1 huge database and not broken into individual rulebases for each VM.Write once firewall.
• Some tables you can sort, others not. Security composer you can’t sort.
• Scoping is complex. It is based on the physical world: datacenters and portgroups. So your objects are visible either
  • Globally
  • Per datacenter (group of ESX systems)
  • Per portgroup (group of network ports)
• In reality you want scoping to be based on a logical security constructs
  • Per firewall
  • Per firewall zone (network interface(s))
  • Per management domain
  • Globally
• You can’t comment and log rules by default you have to open a column view window and tell the GUI to insert column and then enter comments and logging option. You can comment when you initially create in the wizard, but after that it is a hassle.
• Security composer: Can’t write comments in individual rules. So making modifications, can’t put in ticket numbers and who/why/owners/ticket numbers/etc.
• Assigning security tags – If you are assigning a security tag to a new VM in a production system, you can’t put a change control comment anywhere. Once you assign the tag, its in production. Hopefully nothing blows up because you won’t be able to track it (although there is an audit function which is nice, just no comments or SR #s etc.) There is a audit function, haven’t tried that yet.
• Security composer: Can’t put section titles on rules.
• HOLD ON THIS NOT SURE: PERFORMANCE: VMware will talk about how the DFW is embedded within the hypervisor so its fast. What they won’t tell you is that the rulebase for ALL the VMs are in 1 rulebase. So if you have 100,000 rules in NSX, then each VM has 100,000 to evaluate PER interface. You can’t assign 6 rules to VM1, 10 rules to VM2, etc.
  NOTE: There is a APPLY TO parameter to make this work. Kind of hidden.
• In FIREWALL, the policy creation works as normal policy editors do. The SRC/DST can be IP, groups, objects, etc. In Security Composer you can only reference Security Groups. So if rule is SRC: vm1 DST: group_of_ips  then you have to embed group_of_ips in a security group which means more clicking and windows. Takes forever to create new rules in security composer.
• No ‘where located’ button to find objects in rules/groups, etc.
• Can’t/limited copy and paste drag/drop policies/objects/services so you can build templates and re-use them. You can copy rules, one at a time.
• Hard to determine relationship between tags and security groups. You can easily build the relationship, but hard to maintain and delete because you can’t understand the relationship. Security groups->Tags is dynamic filter so you are guessing if the tag is included and Tag->Security Group I haven’t figured out how to determine the relationship. Right now have to use naming schema with common names which doesn’t really scale.
• If you are looking at Security Policies, you can’t tell which security policies are tied to which VMs, only security groups. So if you are cleaning up unused rulesets and trying to tell if a Security Policy is actually enforced on any VMs, there is no way to easily tell. You have to visit each VM and look at the ruleset. True, this is all dynamic, but there is no history to show if the policy has been used recently, etc. Write once firewall.
• You can append policies together, but the priority of rule order has to be done on a global basis. You get a list of ALL the policies and you have to UP|DOWN re-arrange them in the window. It should be pick a policy and then chain them together.
•  You never know what all the rules are for a single VM. All the rules for the whole vcenter are in one big long list with ‘assign to VM’ tags on them you so you have to go through them 1 at a time…and there could be thousands. There is one place that lists most of the rules…but not all of them. Only the service composer rules not the firewall rules.
• VM->Monitor->Firewall Rules will show some of a VMs rules, but not all of them. And if the names are long that’s all you get is This_Grou…….. with dots. AND no comments, And you can’t expand the groups, and and and and. Horrible. Write once firewall.
• Security composer – you can’t negate individual objects, you have to negate all the objects in the src/dst column.
• A security policy can be applied to a single VM or a group. When you are editing a policy you can’t tell if you are modifying a policy for a single VM or a group of VMs.
• Sometimes you click on things to open them up, sometimes you have to hunt for the little teeney weenie insey binsey icon…and then try to figure out which icon does what. Other views the object is not clickable. OR you have to highlight something first before you can edit (rule editor to insert a new rule).
• When in service composer and you want to add a service for port 8080, it only searches for names and not port numbers. So you have to guess what the service is if you don’t know it.
• There is minimal policy search. In CP, you can search a policy for port 80, and it will show you all the rules where port 80 appears….even if its embedded in a group. You can in DFW, but you just get the rule headers and not the highlighted items that makes it easy to find like in CP.
• When in Service composer and the SOURCE is P…[finish this]
• When hovering over objects: sometimes you get information about the object in a popup, sometimes not. Sometimes you can expand (firewall), sometimes you can’t (Security Composer, vCenter). Sometimes you click on it (services) sometimes you don’t. Like security groups you really want to know what the description is but you can’t see the info, you can’t click on it so you see a security group name “VERYSECURE_X009383″ and you have to try and figure out what they meant. You HAVE to have a great naming scheme where the name is long and descriptive THISISSECUREGROUPFORSNMPSERVERSINCHINA”.
• Long object names are chopped and even if you expand the column you can’t tell what they are. So if you are looking at rules and you want to find all the CSX_001_AD…………             different AD servers, you aren’t sure if the AD………. is a server or an ADsomething_Network or ADsomethingname. You just get the …… So you have to hover over and it tells you. But you have to look at each one one at a time. You can’t review in bulk. Write once firewall.
• From Service Composer you cannot dynamically create IPs, ranges, services, etc. These all have to be pre-created and THEN used in service composer. In Firewall you can dynamically create these. So to create a service, you first have to use Firewall and edit an old rule and create a new service…but not use it just create it. THEN go into Security composer and use the service. Lots of mouse clicks. In CP it is just 1 or 2 clicks.
• No drag and drop
• Security tags can be created but not edited in vSphere. Have to edit and manage in NSX manager.
• This may seem trivial but huge: Grouping is very basic. CP does a great job of allowing you to group by name, color, drag and drop, etc. DFW you must edit one object at a time. So imagine adding 100 objects to a group…good luck
• SyslogLogging is not stateful…you see every UDP packet. It is overwhelming and hard to search for what happened. Syslog on steroids.
• When review the rules in security composer, you can’t expand the all objects to understand what is inside the groups, services, etc. You have to open the objects separately. Takes at a minimum 5 clicks PER object you want to review.
• Security Composer: If you explicitly add a host into a group, the sum of these does not show up in the summary window. So the group looks empty but it actually has members. Would be a bummer in a rule review to toss it! Write once firewall.
• You can’t stage a policy and then push it during a change window. In firewall rule creation you can insert a new rule and get a “publish now” button…but that is almost like “OK” button. You can SAVE your current config and push later, but that config is for ALL the policies. you can’t stage individual VMs or groups of VMs. So if someone is in the middle of modifying rules in 1 of the 200 sections you will be pushing ALL rules include the 1/2 built ones. In service composer what you change takes effect immediately. if you change security tags rules change immediately (with no possibility to comment) So cross your fingers you changed things correctly, no review process. Write once firewall.
• Some times you create the rule and then have to ‘publish it’..in Security Composer. BUT if you just update a group with new IPs, then it gets automatically pushed out..no ‘publish it’. CHANGE CONTROL!
• If you change policy that has dynamic tags associated with it, you will have a fun time finding what policies are assigned to what VMs you are about to blow up. ‘Write once’. Small scale will work, large scale it is scary.
• Logging Tool – Log Insight : Somewhat cool but SLOOOOOOOOWWW and kludgey to use compared to SmartLog. a 24 hour rules in SmartLog takes seconds. In Log Insight might be 20+ minutes.
• No timeouts on rules. You have to comment the rule and then use rule reviews to delete. Write once firewall
• You can’t expand/collapse rules base, have to expand/collapse each section by hand.
• You can’t freely edit the section title names. They are auto generated for you. So you have to name your security policy with nice names and that is what is used to ID the section titles.  Makes it very hard to figure out where rules are located in the sections, or what the section does. Especially on large rule bases. Remember, the rules are all in 1 LOOOOONG list, you have have 100000 rules, so section titles and searching is critical to finding things. Write once firewall.
• You can’t free text dynamically search the rulebase. Have to fill in filter and then hit apply. Some of the choices are limited, like you can only search for 1 service at a time.
• You can’t search or sort IPsets/services by numbering, only names. You can’t even do a web page search for text, it is disabled. So if you are trying to see if anyone defined port 1234 you have to search through a list of 65535 service items (ok i am exaggerating, maybe 65534) and ranges. Anyways its a long list and will get longer. If a new admin wants to create port 1234 he will just add in Yet Another port 1234 with new name. In 10 years you will have 100 ports @ 1234 all with different names. Write once firewall.
• You can’t create a new host using using the DNS name to resolve to an IP address. You have to enter both separately. So this breaks the tie between DNS and IP address.  If IP addresses change, you can’t go in and click ‘resolvebyIP’. You have to type in new IP address. Write once firewall.
• No NextGen capabilities like allowing User->Server rules.
• You can adjust the placement of the rules in the rulebase,but you have to use this metric number ‘this rule has weight of 5000’ and hope you have all your other weights correct.
• In Security Composer you can have rules log, but in the rule list there is no logging (or comment) column so you can’t tell if you set the flag or not.
•  Because DFW rules are aggregated into one long list that applies to each VM, if you mistakenly insert a ANY ANY ANY ALLOW NOLOG(default) rule (we do this regularly in rule reveiw/cleanup or when building a new site and need to know what traffic is being passed by firewall), you will allow access to ALL your VMs…and good luck finding the problem. Well, it won’t be a problem because everything will work even better than before!
• Rules like:SecurityComposer:
  FROM: SecurityGroupA
  TO: ANYFirewall:
  From: 1.1.1.1
  TO:GeorgeGarageShopGet inserted into EVERY VM. Why? Because the ANY means that all VMs have to evaluate if the packet can enter/leave the VM both on the source VM and dest VM. The 2nd firewall rule does NOT have any objects that tie it to a specific VM so it has to be evaluated by all.SO: Be very careful how you craft these rules because you might think they impact 1 VM, but they may impact ALL VMs.

  ALSO: Bit of performance impact. All VMs will be evaluating this rule even though they may never hit on it.

• If a Security Group statically includes a IPSET, you can’t tell…the security group looks empty unless you open it up and looks at the details. Why do you care? If you are doing cleanup and you see an empty security group, delete it…OOOPS but it is not empty! You need to have a management interface that shows the relationships between objects so you don’t delete things that are in use. WHERE USED in CheckPoint. Another example of “Write Once” firewall.
• Security Composer: In the SRC or DST field one of them has to be a generic POLICYSECURITYGROUP (which you dynamically apply to a security group). You can’t have a rule “FROM: 1.1.1.1 TO: 2.2.2.2 ACTION: Allow” or a specific security group “FROM: HR_security_group TO: SALES_security_group ACTION: allow”. This has to be done with the FIREWALL rulebase. WHY: because negation of objects doesn’t work with the generica POLICYSECURITYGROUP. So you can’t build rules like: FROM: NOT POLICYSECURITYGROUP TO: Sales_security_group ACTION: Allow.
• Security Composer: Basically you can only work with groups and not individual IP addresses (well you can, but you have to bundle into security group). The group is your security zone and how it interacts with other security zones. So if you have mostly zone-to-zone type rules, then SC works. If you have zone to individual-ip OR zone to misc-group-of-random-IPs, then Security Composer is probably not your friend. you might want to create sections in the firewall rulebase. Unfortunately now you have 2 different rulesets for the same zone. So there is no win, hard to maintain. Don’t get me wrong I like the dynamic nature and Securty Composer but it forces you to architect the rules differently.
• When building groups of services, Service Groups, you only get to see the names and not the port numbers. So when you build a group and you want to include port 123, you have to know the name is AREYOUSTUPIDTHISISNTP_POOL_BLAHBLAH_LOSER. You can’t sort or search by port number or see if its UDP/TCP.
• FYI: when you upgrade NSX prepare yourselves for an outage no matter what they say
• Rule numbers that show up in log insight and syslog are GUID and  not the same sequential ones that show up in the GUI. You have to enable GUID in the rule GUI. The GUID seem to be random based on when you entered the rule, so you can’t tell if rule 1812 preceeds/follows 1811. Very difficult to map between the logs and the rule base. Let’s say you are trying to find when a rule is dropped. Rule GUID #1002 drops it. But rule #1893736 allows it. So does 1002 precede/follow 1893736??? Have fun looking through the WHOLE rulebase. Remember, the rule filter will always label rules with #1.So basically you have 5 rule ids. 1) Rule number in Firewall 2) Rule number in security composer 3) Rule number in manage VM 4) Rule GUID 5) Rule number when you filter rules. Why complain…5 for the price of 1!!!They need to have both rule IDs in log insight. Remember that Log Insight searches are painfully slow and its the only source for GUIDs (well, syslog but its ugly). So hunting down rules is just torture.Write Once Firewall.
• Rule GUID does not show up in VM summary of firewall rules. Write once firewall.
• Rule hit counts exist, but ours don’t work. Not sure who’sproblem.
• So when Log Insight decides to return information, you get the GUID and search your rulebase via a filter. Unfortunately you only get the single rule  and its labeled #1 (no matter what its real number is) and you can’t see the whole rule section so you can’t see the context of the rule with the rules above and below.
• The vsphere indexed search is fast, but you cannot search for everything like items in service composer. There are so many menus, you wind up looking through 100 tabs find and object called ‘SOMESTRANGENAMEYOUNEVERSAWBEFORE’  Needs to be extended.
• Log Insight is an IP based SEIM tool. It can’t use security tags or security groups as a filter. When you are doing analysis you want to ask:FROM SECURITYGROUP-A TO SECURITYGROUP-B
  FROM SECURITYTAG-A TO SECURITYTAG-BShow me all the flows.CheckPoint SmartLog is tightly integrated with its management DB. You can query for any objects in the mgt database. AND it is 100x faster.
• Can’t copy multiple rules and use for templates. Write once firewall.
• So you decide to just make changes to the rules and not publish them. Your buddy comes to work and gets a call that says XXX doesn’t work. Buddy looks at the rules and the rule exists and the traffic should flow. How can you tell what has been pushed to the firewalls and what is being staged? Good luck with that. Write once firewall.
• HOLD CHECKING ON THIS: Different NSX controllers have different policies. So if you have similar policies between them, good luck keeping them separated. Make sure you are editing the right one. Same problem with CP when you have multiple policies in a SmartCenter.
• No ‘disable’ rule in Firewall (verify, might only be security composer)
• We are having problems with disappearing vm objects. VMs get modified or removed…but they are still in firewall rules which prevents us from pushing policy. Write once firewall.
• Be aware that NSX and VCENTER are two products jammed into one interface. So the user/admin is still split between two interfaces in the GUI. So if you add a user, you have to add their permissions in 2 spots. Which doesn’t sound bad but because of the poor interface design you wind up clicking a lot.
• DR Danger!: Try taking a snapshot or a Netbackup/snapshot and restore the VM…Is something missing??? Perhaps the security tag? Geez what did label that tag as…hmmmmm.. Write once firewall.
• We added a new subnet. Our security groups trigger off of subnets. So a VM can be included into a security group based on its IP address. We have 100’s of security groups. Which of the 100’s of security groups did this new subnet impact? Have fun going thru them all and looking at each dynamic rule. Write once firewall.
• Security groups can include other security groups. No big deal. Easy to do. X includes Y. X is parent Y is child. What if you wanted to find all the groups that Y is the child? Pretty funny huh? Write once firewall.
• So I get a call “Dreez, what rules are applied to XXX0027 VM. Hey let me look that up…NOT”
  • #1 you can only find the security composer rules
  • #2 If you are searching for RULES that apply to a VM (SRC or DST field), and that VM is  is buried in a group within a group within a group….go fish…search does not work
  • Write once firewall. Putting in the rules is easier then looking them up.
• You can only open 15 ports in a single rule
• Firewall rule Searches are from 1980s. They are not indexed like smartlog or the rest of vcenter. You have to click 5 times to do 1 search. Like if you search for a VM id, you can’t enter it you have to let the GUI fill it in for you. Then it will return ALL rules where that VM lands, like the ANY rules, the encompassing subnet rules, NOT the rules with GROUPS in them….Basically most of the rules. Write once firewall.
• When you add a rule, the default entry is ANY ANY ANY allow. Hope someone does not forget to fill it in
• Let’s say you have a VM with 10 security tags. The tags allow it to belong to 10 security groups. The 10 security groups pull in 10 policies to be applied to that VM. Confused yet?So you get a ticket to add VMXXX to  talk to VMYYYY on port 999. Now this rulebase was created in the civil war and no one remembers what the application does. Which of the 10 rule sets do you want to add the new rule to? If you add this rule, what other set of VMs will you modify because they might also use the same policy?Point is that this approach to micro segmentation works on Day 1, but is difficult to modify on Day 10,000 because the policies are not tightly bounded to the VM. The policies are in a pool that can be used by ANY VM. In a way they are like CheckPoint Global Policies. When you modify it, you impact MANY firewall policies, not just one.So microsegmentation with tags and groups need guardrails to prevent geriatric degradation (I just made that up, sounds kind of official huh?).
• I was asked to find out why LDAP was dropping. Well LDAP rule was named MSAD so couldn’t search for the name or the number 389. Then it was buried in a group. Then it was buried in another group. I couldn’t expand like in CP, so I had to search through several rules expanding several groups one at a time. Then I also had to do this for the SRC and DST groups buried in security composer security groups. Can’t expand. One at a time. It took me about 4 hours, admitedly I had several balls in the air.
• You’ll love this one. This is the only true way of knowing which rules are on a VM.
  1. find host vm is running on
  2. turn on ssh on host
  3. log in
  4. summarize-dvfilter | less
  5. /vm-name
  6. find network interface
  7. copy network interface name
  8. vsipioctl get rules -f <nicname> | less
  9. have funincredidible….
     Write once firewall.

• Beware, I have seen 2 instances where DFW just ‘forgets’ to load policy…and the world is wide open. You won’t find out about it because there is no alert or red sirens. Logging mostly works,  applications are working, no customers are calling so what is the problem? Yeah very scary. Like adding a new logical network, but the firewall wasn’t protecting it. If you do a:

  vsipioctl getrules
  and it comes up empty…you have a problem.

• I just figure out something about many systems as well as DFW. Let’s say you have a rule that reads:From: 1.1.1.1   TO 2.2.2.2  PORT: 80and someone calls you to debug this and says “Do you have a rule that allows 1.1.1.1 to go to 2.2.2.2 on port 80??”. The rule is blatantly obvious. What if the rule is:From: SecurityGroupX To: SecurityGroupY PORT:ManyFunkyPortsThis is obviously more difficult to find. Why? Because:SecurityGroupX->HostGroupZ->ClientHostObject->1.1.1.1There are so many levels of indirection between SecurityGroupX and 1.1.1.1. Almost like a hierarchy of Groups, the final object is hard to find. In addition its dynamic do to the nature of security groups. So one second this relationship exists and the next second its gone because VM 1.1.1.1 got removed from the group for some reason.

  So although security groups and tags are great for building rulesets, debugging them is a bitch because the tools do not exist to allow you drill down into these rule objects. You have to manually expand them through various screens. Write once firewall.

  CheckPoint on the other hand does a great job of allowing you to expand and drill down.

• So you are working with your policy that is 1000000 rules large because for the whole environment there is only 1 rulebase. You try to push policy, but get errors “VMXXX is unknown object”. WTF? Turns out the VMXXX was deleted by someone else. So somewhere buried in your policy are rules with VMXXX. Have fun finding them and deleting them before you are allowed to push policy. The bad thing is that it is not isolated to a single VM, or policy. Deleting a single VM effects the whole platform. Image Amazon AWS. You delete 1 VM and you can’t push policy at all. Write once firewall.
• So a security advisory comes out to not use Shared Global Addrset. No problem, all firewalls get security advisories. Let’s research this a little:
  
  Oh sorry, we didn’t document that. Anyways I’m sure it will eventually show up, but the documentation is pretty poor. If you want to see great documentation check out AWS, wow.

NSX DFW…definitely enterprise OUT,

dreez