HTTPS Inspection – The Dark Side

So been playing with HTTPS Inspection on Squid and CP for URL filtering lately and am starting to understand why people avoid it. This is not a CP problem, but an industry problem. Sort of like when IPSEC came out, your IPSEC never talked to my IPSEC. Only HTTPS Inspection has been around since I had hair and I’m not sure we are any closer to nirvana then we were before.

So basically – HTTPS Inspection is where the proxy is a Man-In-The-Middle between a client and a server.  The proxy can then decrypt and inspect all data in the tunnel. Google will give tons of details so I won’t bore you. The key here is the certificates. How does the client get WELLSFARGO.COM server certificate if the proxy is in the middle and acting as the client? Well, the proxy generates a fake WELLSFARGO.COM certificate and sends it to the client. Here is a good diagram and slide show of how the fake cert is created and sent to the client. You might also want to research HTTPS CONNECT where the client connects to the proxy server for the HTTPS tunnel.

All sounds easy-peasy huh??? Not so quick…Before embarking on this journey I tried to google what are all the problems of forcing clients to install a new cert and proxy through a HTTPS Inspecting gateway. Only got bits and pieces. So here is my list of the fun times I am having doing HTTPS Inspection:

 

  • How do you work with humans that don’t even know what a cert is? Much less a proxy? How many times do you get the same phone call “What is a cert? Proxy what?”. Yeah deployment was automated via AD and magic just happened….right. Problem is there are too many pieces and unique components..proxies, certs, AD, Linux, OpenSSL, JavaCertStores, pinned certs, that automation will only get 80% and the others are YOUR phone call.
  • Client desktops are locked down. Have fun installing those certs or fixing problems for the client in Zambia speaking Bemba on a locked down desktop.
  • Not all servers use Windows so installing certs via command line on TinyLinux version .00000003  will be a great time.
  • Some applications use Java cert store or OpenSSL cert store or Bob and Jimbo’s cert store version 0.45. So make sure you install the right cert into the right cert stores.
  • Some clients apps just will not use fake signed certs signed by the proxy server.  They are pinned  and will only use the original cert and not one generated by the proxy server. 
  • BS you say. My ORG has AD or other friggin magic and we already distributed an ORG public cert to all our clients. That’s all you have to do is setup the proxy to sign certs as the ORG CA and wham-bam thank you mam its done. OK, so that means giving the ORG private key to the proxy server. Let me say that again….the ORG PRIVATE key. How do you control use of that private key? How do you know the proxy admin is not deep into hookers and blow and accidentally sends the private key to all his Russian friends? And malware these days are looking for private keys like crazy…is the proxy server secured???
  • No problem you say, I’ll just buy some fancy FIPS 140-2 certified  $1billion Hardware Security Module HSM with 126 pretty blinking lights and hook it up and push the GAZUNGA easy button to store all the private keys. HAHAHAHAH. Oh you make me laugh geek boy. Yet more technology on technology on technology. Have fun with that when it breaks. Interoperability, supportability, debugging, etc.
  • Speed. Generating fake certs and decrypting SSL kills CPUs. No problem you say, my vendor has XL5Billion hardware accelerator developed by some Nigerian guy for the KGB. HAHAHAHAHAH. You so funny. See above.
  • After your vendor wines and dines you with hookers and blow and single malt, in your drunken stupor ask them to allow you to log into their support site and type in HTTPS inspection or SSL decryption.

 

So my cost/benefit analysis is pretty negative right now as I work through all these HTTPS issues. I’ll keep you updated.

 

 

 

How To Change CMA/Domain Name – Preserve SIC

Our local MDS god figured this out. Renaming CMA/Domains

Verified on R77.10.

Note this procedure preserves SIC.

Check Point says this procedure is OK as long as the global policy doesn’t change under you.

 

  1. Remove CLM from FW cluster object logging.
  2. Old logs:
    1. User Tracker on CLM to issue log switch.
    2. Back up CLM logs (optional).
    3. Make sure to use “p” option to preserve log file timestamps:

“cp –p 2016* /var/temp-logs/”

  1. Delete CLM via MDG.
  2. Take CMA backup:
    1. mdsenv cma-xyz
    2. cd $FWDIR/bin/upgrade_tools
    3. ./migrate export /var/export_cma-xyz.tgz
  3. Delete entire old domain containing CMA.
  4. Create new domain & CMA with new names.
    1. Make sure GUI-clients is “any”.
    2. Use same IP address as old CMA so FW still talks to same CMA IP.
    3. Don’t start the CMA till after the import below.
  5. Import CMA objects using file /var/export_cma-xyz.tgz.
    1. Click “continue” to the global policy warning.
  6. Assign new CMA to appropriate global policy.
  7. Create CLM with new names.  Copy logs back in.
  8. Tell cluster to send logs to new CLM.
    1. Push policy
    2. Install database.

PA Daily Operations update – From The Trenches

Firewalls have been around for 25+ years and at this point to me is a firewall is a firewall no matter what the label on the box says. I am totally mystified why organizations randomly jump from one vendor to another based on technology alone. I know licensing costs soar, support sucks, platforms are unstable, etc. But in the end a firewall is a firewall and the grass doesn’t get much greener after you make the switch. You probably get 1 year of cheap hardware and kiss ass support before you swirl to the bottom of the toilet as the new vendor pulls in new customers and forgets all the fireworks they promised would fly from your behind.

So yes I am talking the PA vs. CP debacle. PA is a fine firewall (if you don’t turn on all the misc junk). CP is a fine firewall (if you don’t turn on all the misc. junk). PA is a massive marketing machine the earth has never before seen. CP has an incredible enterprise management and logging infrastructure that can’t sell snow to Eskimos.

I just spent 3 hours in a PA hands-on class. Been 5 years since touching one. My reaction : painful.  Why? Because from the marketing rah-rah I was expecting fireworks would fly from my behind. The reality is: Its just another firewall. The web interface reminds me of all the other freeware java/ajax shakey hope-to-god-this-works GUI firewall interfaces. Or maybe they hired a bunch of ex-Cisco CSM programmers and sent them to Web development school for 6 weeks.  I mean its OK, but primitive compared to CP for an enterprise environment. Their logging just sucks compared to SmartLog.

I just don’t get it, but kudos to their marketing machine.

I have friends at another  enterprise corp that spent millions and countless hours to switch. Years ago they had about 300-400 CP firewalls and ~5-10 firewall people. NOW:

  • ~50 firewall bodies
  • PA management and logging is OK but definitely not as good as CP
  • It is stable
  • App control is starting to be deployed and mostly works sometimes
  • Various 3rd party analysis tools don’t work like Tufin, Firemon, etc. so rule reviews are difficult
  • EXPENSIVE
  • Support is hot/cold

So millions were spent and countless hours were toiled and they went from Point A to Point A with 5x more bodies. How did that increase security? How did that lower costs?

Summary: Don’t drink the Kool-aid. Understand what your end goal is, don’t just go from Point A to Point A. Spend those funds on more important security projects that have a cost/benefit.

 

 

 

 

Palo Alto Threat Detection review from the trenches

So I have a friend of mine XXX who has been through several iterations/implementations of IPS, DLP, Firewalls, Threat Detection because someone drank Vendor YYYY cool-aid. XXX is much like me — dealing with CheckPoint can sometimes be a pain and its getting real old but CP management and logging (SmartLog) keeps us with the home team.

XXX’s mgt drank the Palo Alto kool-aid so XXX brought me up to date on the good/bad/ugly of PA’s threat detection environment.

So here it is in my words with XXX’s review:

PROS:

  • Scoping for objects/rules is great: firewall,zone,global. Wish Checkpoint had this
  • Licensing is easier
  • Solid as a rock, good quality
  • IPS between Palo and CheckPoint is about the same

CONS:

  • Logging cannot compare to SmartLog. Some cryptic form of regex
  • Trying to correlate logs in centralized logging is very difficult – each log type Firewall, URL, Threat, etc has its own log item and the only way to tie them together is session ID which is reused about every 2 weeks.  Very difficult when there are multiple firewalls that use the same session id.
  • They don’t even have a true DLP it is called Data Filtering.  It will not take full regex entries therefore false positive rate is very high for SSN and CC
  • Wildfire only scans specific file types and is far less than FireEye.  It also will only scan a file that is 10 mb or lower so some files can get through.  [Getting exact numbers from FireEye.]
  • Rules are easy to enter where it becomes difficult is if you want split responsibilities between network and security.  In order to enable URL filtering, IPS, data filtering they need to be added to a rule.
  • no “Where Used” function until last release
  • Expensive!
  • Checkpoint has more of a true DLP, Palo has data filtering
  • Support has been poor
  • FYI: XXX LOVES!!! FireEye. Every firm I’ve worked with has said the same. What SmartLog is to me, They feel about FireEye

Summary: Detection systems are weak and forensics capabilities (Log searches/correlation) is even weaker.

VMware’s NSX Distributed Firewall Review- Very cool NOT enterprise ready toy

OK folks, the word is out. NSX firewall (Distributed Firewall – DFW) has some seriously cool features

  • Security groups
  • Security tags
  • Dynamic inheritance of policies on VMs, and group membership
  • Inheritance hierarchy of firewall rules so you can build subzones
  • Micro-segmentation…very cool…per host firewalls
  • Ability to monitor traffic on a per nic basis for every host
  • REST interface
  • Integration into NSX environment for common monitoring (you can see rules and objects in logs)
  • Integration into NSX for object lists (object definitions for vSphere items exist for you to consume)
  • No separate upgrade for firewalls, part of the whole NSX upgrade package.
  • Allow rule inheritance (haven’t tried it but shows potential, lot better then strictly Global/Local like in CP)

…..but will never cut it in an enterprise environment.  It just doesn’t scale, you need an army of analysts to maintain. What I call a ‘write once’ firewall, you write the rules once and never go back to maintain it. It’s times like this that remind me how mature CP management and logging are (and forget about all the bugs).

SUMMARY: Security composer very cool just don’t use it. Easy to write rules, a nightmare to maintain.

I just wish Vmware and CP could merge their firewall efforts. DFW has some really cool ideas, they just need CP’s 25+ year enterprise management mindset.

Here is what I know….

(WORK IN PROGRESS, These are my notes as I am learning. I may update them as I verify the good/bad/ugly. Put grain of salt….)

  • vSphere NSX web client is a pretty poor shaky bug ridden java mucus patch, they must of hired some Cisco CSM developers.
    2016-04-01_10-04-20
  • Rule administration is done from several different screens so you jump around a lot. Firewall, Security Composer, vCenter VM summary, vCenter VM manage firewall rules, security tags…are all related to rule administration. All in different windows.
  • If you are creating a rule in service composer and need to add a unknown service, you have to kill the whole rule creation process and first create the service and then go back and redo your work creating the rule. Not true for the firewall management window, you can create a new service.
  • If you want to create a single rule for a single SRC/DST it is easy to use the Firewall manager. But be aware that there is only 1 humongous rule base for all your rules. If you split the one’sies from the more complex rulesets in Security Composer, then you will have 2 rulebases to manage. Write once, no win-win. The micro-segments are broken into SECTION headers but they all still appear as one huge database.
  • Security composer forces you to write rules as:
    # 1 FROM: specificzone TO: whatever
    you always have to specify a zone in the FROM/TO. You can’t specify:
    #2 FROM: any  TO: any
    I think they do this because in the end the compile all rules for NSX into one huge rulebase. So if they allows #2, then it would override all the rules 2+
  • Because there is 1 humoungous rulebase that applies to all VMs its easy to break micro-segmentation. Imagine you have a zone rule:RULE: 100: FROM ZONEA TO ZONEB DENYYou will have to look through all the rules ABOVE rule 100 to make sure your RULE 100 works as you think it should. Imagine 1 day after you put in RULE 100, someone might create RULE 1 which overrides your microsegmentation.RULE: 1     : ANY ANY ANY ANY DENY
    RULE: 100: FROM ZONEA TO ZONEB DENYImagine trying to do a rule cleanup? After a couple years pulling out RULE 1 would be a nightmare.The problem is rules are in 1 huge database and not broken into individual rulebases for each VM.Write once firewall.
  • Some tables you can sort, others not. Security composer you can’t sort.
  • Scoping is complex. It is based on the physical world: datacenters and portgroups. So your objects are visible either
    • Globally
    • Per datacenter (group of ESX systems)
    • Per portgroup (group of network ports)
  • In reality you want scoping to be based on a logical security constructs
    • Per firewall
    • Per firewall zone (network interface(s))
    • Per management domain
    • Globally
  • You can’t comment and log rules by default you have to open a column view window and tell the GUI to insert column and then enter comments and logging option. You can comment when you initially create in the wizard, but after that it is a hassle.
  • Security composer: Can’t write comments in individual rules. So making modifications, can’t put in ticket numbers and who/why/owners/ticket numbers/etc.
  • Assigning security tags – If you are assigning a security tag to a new VM in a production system, you can’t put a change control comment anywhere. Once you assign the tag, its in production. Hopefully nothing blows up because you won’t be able to track it (although there is an audit function which is nice, just no comments or SR #s etc.) There is a audit function, haven’t tried that yet.
  • Security composer: Can’t put section titles on rules.
  • HOLD ON THIS NOT SURE: PERFORMANCE: VMware will talk about how the DFW is embedded within the hypervisor so its fast. What they won’t tell you is that the rulebase for ALL the VMs are in 1 rulebase. So if you have 100,000 rules in NSX, then each VM has 100,000 to evaluate PER interface. You can’t assign 6 rules to VM1, 10 rules to VM2, etc.
  • In FIREWALL, the policy creation works as normal policy editors do. The SRC/DST can be IP, groups, objects, etc. In Security Composer you can only reference Security Groups. So if rule is SRC: vm1 DST: group_of_ips  then you have to embed group_of_ips in a security group which means more clicking and windows. Takes forever to create new rules in security composer.
  • No ‘where located’ button to find objects in rules/groups, etc.
  • Can’t/limited copy and paste drag/drop policies/objects/services so you can build templates and re-use them. You can copy rules, one at a time.
  • Hard to determine relationship between tags and security groups. You can easily build the relationship, but hard to maintain and delete because you can’t understand the relationship. Security groups->Tags is dynamic filter so you are guessing if the tag is included and Tag->Security Group I haven’t figured out how to determine the relationship. Right now have to use naming schema with common names which doesn’t really scale.
  • If you are looking at Security Policies, you can’t tell which security policies are tied to which VMs, only security groups. So if you are cleaning up unused rulesets and trying to tell if a Security Policy is actually enforced on any VMs, there is no way to easily tell. You have to visit each VM and look at the ruleset. True, this is all dynamic, but there is no history to show if the policy has been used recently, etc. Write once firewall.
  • You can append policies together, but the priority of rule order has to be done on a global basis. You get a list of ALL the policies and you have to UP|DOWN re-arrange them in the window. It should be pick a policy and then chain them together.
  •  You never know what all the rules are for a single VM. All the rules for the whole vcenter are in one big long list with ‘assign to VM’ tags on them you so you have to go through them 1 at a time…and there could be thousands. There is one place that lists most of the rules…but not all of them. Only the service composer rules not the firewall rules.
  • VM->Monitor->Firewall Rules will show some of a VMs rules, but not all of them. And if the names are long that’s all you get is This_Grou…….. with dots. AND no comments, And you can’t expand the groups, and and and and. Horrible. Write once firewall.
  • Security composer – you can’t negate individual objects, you have to negate all the objects in the src/dst column.
  • A security policy can be applied to a single VM or a group. When you are editing a policy you can’t tell if you are modifying a policy for a single VM or a group of VMs.
  • Sometimes you click on things to open them up, sometimes you have to hunt for the little teeney weenie insey binsey icon…and then try to figure out which icon does what. Other views the object is not clickable. OR you have to highlight something first before you can edit (rule editor to insert a new rule).
  • When in service composer and you want to add a service for port 8080, it only searches for names and not port numbers. So you have to guess what the service is if you don’t know it.
  • There is minimal policy search. In CP, you can search a policy for port 80, and it will show you all the rules where port 80 appears….even if its embedded in a group. You can in DFW, but you just get the rule headers and not the highlighted items that makes it easy to find like in CP.
  • When in Service composer and the SOURCE is P…[finish this]
  • When hovering over objects: sometimes you get information about the object in a popup, sometimes not. Sometimes you can expand (firewall), sometimes you can’t (Security Composer, vCenter). Sometimes you click on it (services) sometimes you don’t. Like security groups you really want to know what the description is but you can’t see the info, you can’t click on it so you see a security group name “VERYSECURE_X009383” and you have to try and figure out what they meant. You HAVE to have a great naming scheme where the name is long and descriptive “THISISSECUREGROUPFORSNMPSERVERSINCHINA”.
  • From Service Composer you cannot dynamically create IPs, ranges, services, etc. These all have to be pre-created and THEN used in service composer. In Firewall you can dynamically create these. So to create a service, you first have to use Firewall and edit an old rule and create a new service…but not use it just create it. THEN go into Security composer and use the service. Lots of mouse clicks. In CP it is just 1 or 2 clicks.
  • No drag and drop
  • Security tags can be created but not edited in vSphere. Have to edit and manage in NSX manager.
  • This may seem trivial but huge: Grouping is very basic. CP does a great job of allowing you to group by name, color, drag and drop, etc. DFW you must edit one object at a time. So imagine adding 100 objects to a group…good luck
  • SyslogLogging is not stateful…you see every UDP packet. It is overwhelming and hard to search for what happened. Syslog on steroids.
  • When review the rules in security composer, you can’t expand the all objects to understand what is inside the groups, services, etc. You have to open the objects separately. Takes at a minimum 5 clicks PER object you want to review.
  • Security Composer: If you explicitly add a host into a group, the sum of these does not show up in the summary window. So the group looks empty but it actually has members. Would be a bummer in a rule review to toss it! Write once firewall.
  • You can’t stage a policy and then push it during a change window. In firewall rule creation you can insert a new rule and get a “publish now” button…but that is almost like “OK” button. You can SAVE your current config and push later, but that config is for ALL the policies. you can’t stage individual VMs or groups of VMs. So if someone is in the middle of modifying rules in 1 of the 200 sections you will be pushing ALL rules include the 1/2 built ones. In service composer what you change takes effect immediately. if you change security tags rules change immediately (with no possibility to comment) So cross your fingers you changed things correctly, no review process. Write once firewall.
  • Some times you create the rule and then have to ‘publish it’..in Security Composer. BUT if you just update a group with new IPs, then it gets automatically pushed out..no ‘publish it’. CHANGE CONTROL!
  • If you change policy that has dynamic tags associated with it, you will have a fun time finding what policies are assigned to what VMs you are about to blow up. ‘Write once’. Small scale will work, large scale it is scary.
  • Logging Tool – Log Insight : Somewhat cool but SLOOOOOOOOWWW and kludgey to use compared to SmartLog. a 24 hour rules in SmartLog takes seconds. In Log Insight might be 20+ minutes.
  • No timeouts on rules. You have to comment the rule and then use rule reviews to delete. Write once firewall
  • You can’t expand/collapse rules base, have to expand/collapse each section by hand.
  • You can’t freely edit the section title names. They are auto generated for you. So you have to name your security policy with nice names and that is what is used to ID the section titles.  Makes it very hard to figure out where rules are located in the sections, or what the section does. Especially on large rule bases. Remember, the rules are all in 1 LOOOOONG list, you have have 100000 rules, so section titles and searching is critical to finding things. Write once firewall.
  • You can’t free text dynamically search the rulebase. Have to fill in filter and then hit apply. Some of the choices are limited, like you can only search for 1 service at a time.
  • You can’t search or sort IPsets/services by numbering, only names. You can’t even do a web page search for text, it is disabled. So if you are trying to see if anyone defined port 1234 you have to search through a list of 65535 service items (ok i am exaggerating, maybe 65534) and ranges. Anyways its a long list and will get longer. If a new admin wants to create port 1234 he will just add in Yet Another port 1234 with new name. In 10 years you will have 100 ports @ 1234 all with different names. Write once firewall.
  • You can’t create a new host using using the DNS name to resolve to an IP address. You have to enter both separately. So this breaks the tie between DNS and IP address.  If IP addresses change, you can’t go in and click ‘resolvebyIP’. You have to type in new IP address. Write once firewall.
  • No NextGen capabilities like allowing User->Server rules.
  • You can adjust the placement of the rules in the rulebase,but you have to use this metric number ‘this rule has weight of 5000’ and hope you have all your other weights correct.
  • In Security Composer you can have rules log, but in the rule list there is no logging (or comment) column so you can’t tell if you set the flag or not.
  •  Because DFW rules are aggregated into one long list that applies to each VM, if you mistakenly insert a ANY ANY ANY ALLOW NOLOG(default) rule (we do this regularly in rule reveiw/cleanup or when building a new site and need to know what traffic is being passed by firewall), you will allow access to ALL your VMs…and good luck finding the problem. Well, it won’t be a problem because everything will work even better than before!
  • If a Security Group statically includes a IPSET, you can’t tell…the security group looks empty unless you open it up and looks at the details. Why do you care? If you are doing cleanup and you see an empty security group, delete it…OOOPS but it is not empty! You need to have a management interface that shows the relationships between objects so you don’t delete things that are in use. WHERE USED in CheckPoint. Another example of “Write Once” firewall.
  • Security Composer: In the SRC or DST field one of them has to be a generic POLICYSECURITYGROUP (which you dynamically apply to a security group). You can’t have a rule “FROM: 1.1.1.1 TO: 2.2.2.2 ACTION: Allow” or a specific security group “FROM: HR_security_group TO: SALES_security_group ACTION: allow”. This has to be done with the FIREWALL rulebase. WHY: because negation of objects doesn’t work with the generica POLICYSECURITYGROUP. So you can’t build rules like: FROM: NOT POLICYSECURITYGROUP TO: Sales_security_group ACTION: Allow.
  • Security Composer: Basically you can only work with groups and not individual IP addresses (well you can, but you have to bundle into security group). The group is your security zone and how it interacts with other security zones. So if you have mostly zone-to-zone type rules, then SC works. If you have zone to individual-ip OR zone to misc-group-of-random-IPs, then Security Composer is probably not your friend. you might want to create sections in the firewall rulebase. Unfortunately now you have 2 different rulesets for the same zone. So there is no win, hard to maintain. Don’t get me wrong I like the dynamic nature and Securty Composer but it forces you to architect the rules differently.
  • When building groups of services, Service Groups, you only get to see the names and not the port numbers. So when you build a group and you want to include port 123, you have to know the name is AREYOUSTUPIDTHISISNTP_POOL_BLAHBLAH_LOSER. You can’t sort or search by port number or see if its UDP/TCP.
  • FYI: when you upgrade NSX prepare yourselves for an outage no matter what they say
  • Rule numbers that show up in log insight and syslog are GUID and  not the same sequential ones that show up in the GUI. You have to enable GUID in the rule GUI. The GUID seem to be random based on when you entered the rule, so you can’t tell if rule 1812 preceeds/follows 1811. Very difficult to map between the logs and the rule base. Let’s say you are trying to find when a rule is dropped. Rule GUID #1002 drops it. But rule #1893736 allows it. So does 1002 precede/follow 1893736??? Have fun looking through the WHOLE rulebase. Remember, the rule filter will always label rules with #1.So basically you have 5 rule ids. 1) Rule number in Firewall 2) Rule number in security composer 3) Rule number in manage VM 4) Rule GUID 5) Rule number when you filter rules. Why complain…5 for the price of 1!!!They need to have both rule IDs in log insight. Remember that Log Insight searches are painfully slow and its the only source for GUIDs (well, syslog but its ugly). So hunting down rules is just torture.Write Once Firewall.
  • Rule GUID does not show up in VM summary of firewall rules. Write once firewall.
  • Rule hit counts exist, but ours don’t work. Not sure who’sproblem.
  • So when Log Insight decides to return information, you get the GUID and search your rulebase via a filter. Unfortunately you only get the single rule  and its labeled #1 (no matter what its real number is) and you can’t see the whole rule section so you can’t see the context of the rule with the rules above and below.
  • The vsphere indexed search is fast, but you cannot search for everything like items in service composer. There are so many menus, you wind up looking through 100 tabs find and object called ‘SOMESTRANGENAMEYOUNEVERSAWBEFORE’  Needs to be extended.
  • Log Insight is an IP based SEIM tool. It can’t use security tags or security groups as a filter. When you are doing analysis you want to ask:FROM SECURITYGROUP-A TO SECURITYGROUP-B
    FROM SECURITYTAG-A TO SECURITYTAG-BShow me all the flows.CheckPoint SmartLog is tightly integrated with its management DB. You can query for any objects in the mgt database. AND it is 100x faster.
  • Can’t copy multiple rules and use for templates. Write once firewall.
  • So you decide to just make changes to the rules and not publish them. Your buddy comes to work and gets a call that says XXX doesn’t work. Buddy looks at the rules and the rule exists and the traffic should flow. How can you tell what has been pushed to the firewalls and what is being staged? Good luck with that. Write once firewall.
  • HOLD CHECKING ON THIS: Different NSX controllers have different policies. So if you have similar policies between them, good luck keeping them separated. Make sure you are editing the right one. Same problem with CP when you have multiple policies in a SmartCenter.
  • No ‘disable’ rule in Firewall (verify, might only be security composer)
  • We are having problems with disappearing vm objects. VMs get modified or removed…but they are still in firewall rules which prevents us from pushing policy. Write once firewall.
  • Be aware that NSX and VCENTER are two products jammed into one interface. So the user/admin is still split between two interfaces in the GUI. So if you add a user, you have to add their permissions in 2 spots. Which doesn’t sound bad but because of the poor interface design you wind up clicking a lot.
  • DR Danger!: Try taking a snapshot or a Netbackup/snapshot and restore the VM…Is something missing??? Perhaps the security tag? Geez what did label that tag as…hmmmmm.. Write once firewall.
  • So I get a call “Dreez, what rules are applied to XXX0027 VM. Hey let me look that up…NOT”
    • #1 you can only find the security composer rules
    • #2 If you are searching for RULES that apply to a VM (SRC or DST field), and that VM is  is buried in a group within a group within a group….go fish…search does not work
    • Write once firewall. Putting in the rules is easier then looking them up.
  • You can only open 15 ports in a single rule
  • Firewall rule Searches are from 1980s. They are not indexed like smartlog or the rest of vcenter. You have to click 5 times to do 1 search. Like if you search for a VM id, you can’t enter it you have to let the GUI fill it in for you. Then it will return ALL rules where that VM lands, like the ANY rules, the encompassing subnet rules, NOT the rules with GROUPS in them….Basically most of the rules. Write once firewall.
  • When you add a rule, the default entry is ANY ANY ANY allow. Hope someone does not forget to fill it in
  • So a security advisory comes out to not use Shared Global Addrset. No problem, all firewalls get security advisories. Let’s research this a little:
    shared addr
    Oh sorry, we didn’t document that. Anyways I’m sure it will eventually show up, but the documentation is pretty poor. If you want to see great documentation check out AWS, wow.

NSX DFW…definitely enterprise OUT,

dreez

 

Redirecting NSX firewall syslogs into SmartLog

 

So we know that NSX DFW is a cool toy, but its logs are invaluable for debugging and forensics. Wouldn’t it be cool if would could see DFW logs along with SmartLogs???So our need is to have single-pane of glass security; Enter vSec and SmartLog.

After 100 VMs we knew that using DFW would just not work. In addition the logging was painful syslog based. So we decided to use the best management and logging tools on the market….and it just so happens this is one of the things CP does extremely good. Management and logging. Today’s talk is on “Having Fun with Logging”.

So we needed to get the DFW syslogs into SmartLog. How to do that. Well, first thing you do is pour salt on the documentation.  Its OK, but only gives you the basics. I have decoded this documentation and will show you how to send DFW logs to SmartLog and make it go a lot faster. By default, you can just turn it on with factory defaults… but will overwhelm your SmartLog server and all the data will be in the ‘info’ field and not parsed out. So it took me a couple months to figure out how to do all the parsing efficiently so as not to bury our log server.

And now for the rest of the story…..

NSX DFW LOGS

syslog-ng-arc

NSX is the part of vMWare vSpere that runs its SDN…including the firewalls..Distributed Fire Walls (DFW). These firewalls send their syslog to their eSXI hosts which we forward to a custom syslog-ng server on a single VM.  We customized this syslog-ng server to parse DFW logs into CSV format….no easy task but doable. Why????

why-syslog-ng

Once in CSV, syslog-ng forwards the logs to ‘syslog’ on currently a UTM 4800 with 4 cores. This is a custom syslog built by CP to accept syslog and convert to CP native log format. ‘syslog’ forwards the converted logs to fwd which enables SmartLog and Tracker to read the logs.

Simple enough…….ok, then read no further.

syslog-ng config

Syslog-ng allows you to parse through logs and reformat them. You can do some simple parsing with the basic configuration file, but the fun starts with more complex patterns with the pattern matching databases. Basically you need to convert this:

rawsysl

into this:

csvrules

Now all you people are tons smarter than this old balding has-been…so I am not going to explain syslog-ng internals to you all. You will have to read the documentation. But I will give you the overview of what I did with it.

syslog-ng has two parts

  • General filtering config – Most people use this just for simple filtering of logs
    and re-formatting of logs and redirecting logs to host files, DB’s, other log servers.
  • Pattern Database – More complex parsing where you can take pieces of the input line, parse out specific items with regex, and insert those regex patterns into variables/macros to be used later in the General Filtering Config.

General Filtering Config: First thing you need to do only filter on DFW logs and nothing else.  NSX puts labels on these logs and you will have to look in the raw log file for it.

filter

Second thing you have to do is get rid of the TERM log entries (terminate connections for TCP, for UDP you’ll see zillions of these going to 5355 some sort of local subnet DNS resolution. Unfortunately in cloud world we have huge subnets so all the VMs are doing this local link DNS resolution). 75% of the DFW logs are TERM entries so can be ignored.

Pattern Database: Next part is a bit harder. You have to build a pattern database that will parse raw DFW log entries and match each field up to a named macro. For example below is a snippet of a DFW log entry matching pattern. You can see the ‘action’, ‘domain’, ‘protocol’ fields are 3 of many fields that would match a DFW log entry from above.

dfwpatternparser

This can be a delicate task, but there is a tool to help debug … ‘pdbtool match ‘. You can type in a sample line and see if the your pattern database will match it:

syslogng-parserdatabase-dfwparsing-macros

General Filtering Config: Next, DFW labels ICMP protocol as ‘PROTO’. So I just translate into normal geek lingo ‘ICMP’.

dfw-cmp

General Filtering Config:Next is the fun part. We finally get to output CSV to the CP syslog server. Notice how the MACRO names like ‘size’ and ‘protocol’ are used to fill in the CSV?

syslogng-parserdatabase-dfwparsing-csvformat

General Filtering Config: And here is the ‘main’ section that drives all the above phases:

syslogng-parserdatabase-dfwparsing-main

so restart you syslog-ng server:

/etc/init.d/syslog-ng restart

and off you go!

At this point you can ‘tcpdump -X -n s0 port 514’ to verify that in fact the logs are formatted correctly and heading over to CP land.

SYSLOG Testing

Just in the off case my code sucks, here is how you test it. There are syslog generators out there that you can use: I used this one and here you can see how I generated traffic:

sysloggenerator-2

One key thing is the “dfwpktlogs–” is what triggers the regex’s to fire in the next section. Do NOT use ‘:’, you have to use ‘–‘ or for some reason the regex won’t parse correctly. More below…

 

CP SYSLOG MDS Config

So now the logs are flowing to port 514 on SmartCenter Log server. So you have to make sure you enable the syslog process (NOT syslogd) to listen.

syslogdashboard

 

This will start the syslog process on port 514 on SmartCenter.

syslog514

On a MLS, you have to restart the whole MLS for the config to kick in. On MLS, a syslog daemon will be started PER domain log server IP address for every domain SYSLOG is configured to accept SYSLOGs for that domain.

mls-syslog

and syslogng logs will flow magically into Tracker. Actually they will magically flow, but only fill in the INFO field and will not parse …. yet.

Oh yeah, when debugging you can get the syslog process to re-read the syslog config with:

  1. mdsenv <DOMAIN>  # MDS only
  2. fw kill fwd; fwd -n &

CP SYSLOG Parsing

OK, now it gets fun.

So there is this CP tool: Eventia Log Parser that looks pretty cool. You feed it your SYSLOGS and magic parsing configuration pops out the other end:

 

elp

Yeaaaaaaah…NO. Doesn’t work and the config you output is so complex the SYSLOG engine will run at 100%. Both ELP and SYSLOG were written about the time after the Civil War and are on version 1.0…Typical CP V1 code so don’t get your hopes up. Let me know if you have better luck, I spent days on this.

So I decided to generate the syslog parsing config myself. I saw what ELP attempted and then came up with my own.

Remember that SYSLOG-NG is sending CSV formatted logs:

csvrules

and CP SYSLOG is taking them in and turning them into SmartLog readable:

syslogentry

So let’s begin.

The debug process is this:

  1. mdsenv <DOMAIN> if on MDS
  2. Edit the syslog rules with your stuff
  3. ‘fw kill fwd; fwd -n &’ restarts the syslog daemon
  4. Use the syslog generator
    1. Forget all this parsing junk, just get the SYSLOG-NG to dump into the CP SYSLOG and see the results in TRACKER in the ‘info’ field
    2. Try and get the REGEX rule to fire on ‘dfwpktlogs–‘
    3. Try and get field #1 to parse and fill in some random text field for debugging
    4. Try and get field #1 to output into the official SmartTracker field
    5. Go to #2 for field #2/3/4/5….

So far I have given you enough information to do #4.1. Let’s work on #4.3.

This may look simple…but it took me days and days to fine tune this. The ELP generated config was pages and pages of REGEX expressions. I boiled down to 1 line:

cp-regex-syslog

Even if you aren’t a regex geek, you can see that I am parsing the CSV file into 9 fields.

REGEX GEEK OUT (don’t read this if you aren’t a regex geek):

FYI: They don’t implement FULL regex matching! Example: You can only have 9 matching field patterns, I couldn’t get it to recognize ‘:’, you can’t use lazy searches ‘.+?’. It is some limited hack that you will never figure out because there is no documentation other than examples.  AND their regex performance is horrible, so I tried to avoid using ‘.*’ because it is greedy and will scan the whole line and then backtrack. Instead I used the ‘[^,]’ which searches for all characters NOT ‘,’, which is the same as ‘(.*),’…but doesn’t have the backtracking.

 

============= END GEEK OUT===============================

So what is important is the ‘dfwpktlogs–‘.  When the regex sees this pattern, it will fill in the 9 columns of information from the CSV formatted input line. Now there are 3 different types of actions that will be taken depending on the results of a REGEX match:

Rules:

  • NO MATCH on regex: Result->log entry with data in INFO field
  • MATCH: but no log entry, you matched the  ‘dfwpktlogs–‘ but trying to print out a number into Tracker or Data type error, wrong field name, syntax error on field names, etc.
  • MATCH and () field hits: can use index_value(1) to fill out fields (coming next)

OK so next you will try and parse field #1 which is the first ‘([^,]+)’. This is the PASS|DROP field from the CSV. What I did is I first captured the field and then dumped it into a random text field. This told me that 1) I captured it correctly 2) I am able to write to SmartTracker.

csv-field

Here is the add_field that does this. add_field adds the field to tracker. Field_name is the Tracker field name..in this case it is the ‘rule_name field” field_index is CSV field 1 (from 1-9). Field type is the type of the Tracker field. But beware, I didn’t always get this right and the log entries would just dissappear so I had to experiment. I also looked into:

$FWDIR/conf/syslog/CPdefined/*.C files for examples to see what their field types are.

You can also look into

trackerfields

to see the Tracker field names and types. Its a bit kludgey but between CPDefined examples and this file you’ll get close.

Here you can see ‘rule_name” is the name of the Tracker field defined in svt_fields.C.

trackerfieldnames

 

So now you have 1) You captured field #1 and 2) You wrote it to ‘rule_name’ field. You can now verify that you captured the field you intended to. Once you verify this, you can then write it to the REAL field with:syslog-tracker-real-field

Here we are writing the PASS|DROP to the ‘action’ field in tracker:

cp-action-field

HOLD ON DREEZ: How in the heck did PASS|DROP get converted to Accept|Drop???? you ask.

Grasshopper, I introduce the dictionary file…

dict-file

This is a second file in the same directory that will do transforms for you. Here you can see “PASS” being converted to ‘accept’.

DREEZ you ask: How do I know what to convert to what?

Oh yes grasshopper. You read the CP documentation…NOT!!! HAHAHAHAHAH. Dreez makes a big funny. HAHAHAHAH. Remember grasshopper, this documentation was written in the Civil War and even now the Lord Developers rarely talk to their lowly documentation peasants slaving in the fields trying to identify nuggets of information to feed their ignorant customers. (If you want real documentation see AWS documentation, you can see where developers talk to the documentation team). That’s what phone support is for —- DUH!!!!

Oh yes, I easily diverge, forgive me.

No grasshopper you do what us old people have been doing for centuries. You look at examples in

$FWDIR/conf/syslog/CPdefined/*.C

OR:

On your log server: fw log

Will print out your logs with the field names in them.

To get a feel of what the valid field types are.

So now that you know how to do field #1, now you go through all the fields one at a time….or……

You can just use my template as a starter!!!

 

PERFORMANCE

So the first time we did this, we used CP’s syslog ELP generated config on one of our big servers and the server went to 100%++++. Unfortunately the ‘syslog’ process is single threaded so it had no where else to go.

So with 1100 VM’s….each with its own firewall…sending syslog to my config described above to a 4800 (slow) based management station…performance was much better but not ideal. The 4800 has a quad core Q9400 on it. Syslog process is a bit busy but at least not 100%. It will average between 11% and 60%, and burst to 90% now and then.

Now remember that on MLM, each domain/cma/clm has its own ‘syslog’, so you can manually distribute the load amongst multiple logging domains. But I would hate to do domain design based on log loads. For example, at one company 2 of 12 domains had 90% of our logging. So should we split them up further because of logging? Not sure.

Your mileage may very….

 

SUMMARY

Everyone is excited about this because SmartLog is what our org (and all the other orgs I’ve been at) live and die by. Centralized single-pane of glass easy to use and fairly fast security monitoring. It is the gem of all of CP’s products. And it mostly sometimes works!!!

Now we can send logs from other tools like other firewalls, URL filtering, FireEye, etc into this and use SmartLog to get quick answers. I am keeping both flat syslog files as well as sending them to mongodb and SmartLog. YES: SEIM tools exist…but either they don’t work, too expensive or at capacity, etc. ‘grep’ is free. ‘mongodb’ is free, etc.

 

 

LOGOUT!

dreez

 

 

The World is Changing and so am I

The Cloud. Its Coming. Torpedo is in the tube and you are a WWII lumbering oil tanker. You will be automated like a $15 minimum wage McDonalds burger flipper.

I switched gigs and and am now working a cloud security gig. My Check Point role is diminished 50%…but worse I no longer have access to my #1 diamond support guy Taylor who fed me all the good juicy techie fixes that I shared with you all.

SOOOOO, not sure what the future blog will look like at this point, but the CP focused techie posts will be reduced and you’ll see more Cloud security chatter. Worst case you still have the archives.

Administrator Audit Made Easy – Create CSV of MDS user permissions

Darn auditors want to know who has what permissions in MDS……but want it in a spreadsheet! What’s up with that old technology?

Here it is, a matrix of users and their permissions.

adminperms

Python Program #2: Adminparser

NOTE: Goes hand in hand with my Cparser module.

Hopefully this will be easier with the R80 REST interface.

Audit OUT!

dreez

Convert any CheckPoint .C file into Python List

Killing two birds  with 1 pebble. Learning some python and automating our admin audits.

This is the core of it. Converts any .C file into a Python list. So you can use this to parse through your objects, rulebases, users, admin lists, etc.Once converted you can create GUIs, other parsing tools (like I will use for admin user deltas)

Download here: Cparser.py

cpadmins

Zero Downtime Upgrade between major versions WITH/OUT dynamic routing

Good news:Can be done possible
Bad news: This is work in progress, hope to update with pictures. If you call CP support, they might be able to fish up the document.

Overview:

  1. Go through CP steps for zero time upgrades. But don’t take them toooooo seriously or you will have surprises. Make sure you do these steps.
  2. Run the upgrade on the standby – DO NOT REBOOT
  3. If you have to copy fwkern.conf from the ACTIVE member ..do it now
  4. control_bootsec – install initial policy and makes sure that the default filter (bricks the firewall) is not loaded. Run from UPGRADE file system, not old file system.

    cd /opt/CPsuite-R77/fw1/bin

    bash

    control_bootsec

  5. Reboot standby
  6. Standby comes back up “Active Attention” – no problem has no cluster policy
  7. In dynamic routing, if you have “Wait for Clustering” enabled. Disable it. Let the routed startup without a cluster
  8. Start/Stop routed:
    tellpm process:routed
    tellpm process:routed t
  9. On mgt server change policy to latest version  R77.10/20/30 and push to upgraded member (uncheck mark in policy install for cluster push). Upgraded member now knows it has to be part of a cluster. It will go to READY state, waiting for the failover
  10. Use this script to export the routes off the ACTIVE firewall onto the Standby firewall. It will turn them into STATIC routes. NOTE: There is no ‘save config’ at the end. This are only temporary until the system reboots and get real OSPF routes. Make sure you differentiate between dynamic routes that will go away on reboot and real static routes that will be kept on reboot.
  11. Reboot the READY firewall just to clear out the cobwebs.
  12. Run the ospf script on the READY firewall. This will load all the OSPF and STATIC routes onto the firewall. NOTE: YOu will have to decide if you want to keep/delete the STATIC routes. You might have to SAVE CONFIG on the static routes if you want to keep them.
  13. Do a netstat -an | wc -l and fw tab -t connections -s to metric the routes and states
  14. Do a ‘cphaprob stat’ to get the IP and ‘number ID’ of the ACTIVE member.
  15. Now on the READY member PULL the state table from the ACTIVE member.cphaprob stat   –
    Retrieve the cluster NUMBER and sync IP of the ACTIVE membercphacu start <Active Member IP> <Cluster member Number>  –
    So if active was 1.1.1.1 and number 2 in cluster:
    cphacu start 1.1.1.1 2
    Will pull the state table from the ACTIVE onto the READY member. This is like the OLD fcu command…but snazzier somehow.
  16. Do a netstat and fw tab -t connections and make sure the numbers are about the same on both members
  17. On the ACTIVE member – drum roll.
    cphaprob stop
  18. On the DOWN member STOP the routing daemon because you don’t want it to fight with the new ACTIVE member. This is where the checkpoint cluster and routing teams never broke bread and coordinated cluster & routing activity and you have to do it manually.tellpm process:routed
  19. The READY member will now go to ACTIVE
  20. On ACTIVE member check out the state tables and network tables again. OSPF should be populating. Check the neighbor status to see if OSPF neighbors are negotiating. If not, they are stuck, then stop and restart. No worries you have static entries until you reboot.clish> show ospf neighbors
    clish> show route ospftellpm process:routed         ##### stop
    tellpm process:routed t        #### start
  21. You are over the hump, congrats
  22. Upgrade the OLD system
  23. Copy fwkern from the standby if required
  24. Reboot
  25. Push policy to both members
  26. Reboot both (to clear out static network entries and cobwebs)
  27. Done
blog.lachmann.org

Michael Endrizzi's - St. Paul MN - CheckPoint blog on topics related to Check Point products and security in general.