Thought I’d add a little humor to the blog.
Someone read my posts and sent this to me.
Greetings Home Boys,
Been digging deep into NSX and SDN and gave several classes. I think I cracked the nut so decided to share with you.
Beware of snake-oil SDN people, I’m not convinced they know what they are talking about. Make sure you don’t replicate your existing environment into The Cloud. Painful.
So been playing with HTTPS Inspection on Squid and CP for URL filtering lately and am starting to understand why people avoid it. This is not a CP problem, but an industry problem. Sort of like when IPSEC came out, your IPSEC never talked to my IPSEC. Only HTTPS Inspection has been around since I had hair and I’m not sure we are any closer to nirvana then we were before.
So basically – HTTPS Inspection is where the proxy is a Man-In-The-Middle between a client and a server. The proxy can then decrypt and inspect all data in the tunnel. Google will give tons of details so I won’t bore you. The key here is the certificates. How does the client get WELLSFARGO.COM server certificate if the proxy is in the middle and acting as the client? Well, the proxy generates a fake WELLSFARGO.COM certificate and sends it to the client. Here is a good diagram and slide show of how the fake cert is created and sent to the client. You might also want to research HTTPS CONNECT where the client connects to the proxy server for the HTTPS tunnel.
All sounds easy-peasy huh??? Not so quick…Before embarking on this journey I tried to google what are all the problems of forcing clients to install a new cert and proxy through a HTTPS Inspecting gateway. Only got bits and pieces. So here is my list of the fun times I am having doing HTTPS Inspection:
So my cost/benefit analysis is pretty negative right now as I work through all these HTTPS issues. I’ll keep you updated.
I’m blown away. I’m stunned. I’d sell my kids schoolbooks to use it (I don’t have kids). It is my inner glow.
In the past year I have used:
And R80 blows away all the other vendors.
And get this….I think they even tested it before they released it. I know, even I am stunned. OK, there are still some bugs and dealing with CP arrogance is a pain but R80 makes it all OK. They actually thought about the user experience and enhanced its enterprise management capabilities to allow scaling. It is true art.
TESTED: Just basic SmartDashboard on R77.30 gateway. I did not test MDS or R80 gateway which are coming out soon.
[ IN PROGRESS ]
I’ll update this as I use it more, but so far kudo’s. For large environments you might want to wait until more bugs are ironed out but for smaller installations you will never look back.
Inner Glow YAAAAH!
Our local MDS god figured this out. Renaming CMA/Domains
Verified on R77.10.
Note this procedure preserves SIC.
Check Point says this procedure is OK as long as the global policy doesn’t change under you.
“cp –p 2016* /var/temp-logs/”
Firewalls have been around for 25+ years and at this point to me is a firewall is a firewall no matter what the label on the box says. I am totally mystified why organizations randomly jump from one vendor to another based on technology alone. I know licensing costs soar, support sucks, platforms are unstable, etc. But in the end a firewall is a firewall and the grass doesn’t get much greener after you make the switch. You probably get 1 year of cheap hardware and kiss ass support before you swirl to the bottom of the toilet as the new vendor pulls in new customers and forgets all the fireworks they promised would fly from your behind.
So yes I am talking the PA vs. CP debacle. PA is a fine firewall (if you don’t turn on all the misc junk). CP is a fine firewall (if you don’t turn on all the misc. junk). PA is a massive marketing machine the earth has never before seen. CP has an incredible enterprise management and logging infrastructure that can’t sell snow to Eskimos.
I just spent 3 hours in a PA hands-on class. Been 5 years since touching one. My reaction : painful. Why? Because from the marketing rah-rah I was expecting fireworks would fly from my behind. The reality is: Its just another firewall. The web interface reminds me of all the other freeware java/ajax shakey hope-to-god-this-works GUI firewall interfaces. Or maybe they hired a bunch of ex-Cisco CSM programmers and sent them to Web development school for 6 weeks. I mean its OK, but primitive compared to CP for an enterprise environment. Their logging just sucks compared to SmartLog.
I just don’t get it, but kudos to their marketing machine.
I have friends at another enterprise corp that spent millions and countless hours to switch. Years ago they had about 300-400 CP firewalls and ~5-10 firewall people. NOW:
So millions were spent and countless hours were toiled and they went from Point A to Point A with 5x more bodies. How did that increase security? How did that lower costs?
Summary: Don’t drink the Kool-aid. Understand what your end goal is, don’t just go from Point A to Point A. Spend those funds on more important security projects that have a cost/benefit.
So I have a friend of mine XXX who has been through several iterations/implementations of IPS, DLP, Firewalls, Threat Detection because someone drank Vendor YYYY cool-aid. XXX is much like me — dealing with CheckPoint can sometimes be a pain and its getting real old but CP management and logging (SmartLog) keeps us with the home team.
XXX’s mgt drank the Palo Alto kool-aid so XXX brought me up to date on the good/bad/ugly of PA’s threat detection environment.
So here it is in my words with XXX’s review:
Summary: Detection systems are weak and forensics capabilities (Log searches/correlation) is even weaker.
OK folks, the word is out. NSX firewall (Distributed Firewall – DFW) has some seriously cool features
…..but will never cut it in an enterprise environment. It just doesn’t scale, you need an army of analysts to maintain. What I call a ‘write once’ firewall, you write the rules once and never go back to maintain it. It’s times like this that remind me how mature CP management and logging are (and forget about all the bugs).
SUMMARY: Security composer very cool just don’t use it. Easy to write rules, a nightmare to maintain.
I just wish Vmware and CP could merge their firewall efforts. DFW has some really cool ideas, they just need CP’s 25+ year enterprise management mindset.
Here is what I know….
(WORK IN PROGRESS, These are my notes as I am learning. I may update them as I verify the good/bad/ugly. Put grain of salt….)
This is where CheckPoint shines. Unwinding indirection is CP’s sweet spot. They have many tools to search for buried objects and manage them. THUS the reason it is my #1 pick for enterprise management.
ALSO: Bit of performance impact. All VMs will be evaluating this rule even though they may never hit on it.
and it comes up empty…you have a problem.
So although security groups and tags are great for building rulesets, debugging them is a bitch because the tools do not exist to allow you drill down into these rule objects. You have to manually expand them through various screens. Write once firewall.
CheckPoint on the other hand does a great job of allowing you to expand and drill down.
NSX DFW…definitely enterprise OUT,
So we know that NSX DFW is a cool toy, but its logs are invaluable for debugging and forensics. Wouldn’t it be cool if would could see DFW logs along with SmartLogs???So our need is to have single-pane of glass security; Enter vSec and SmartLog.
After 100 VMs we knew that using DFW would just not work. In addition the logging was painful syslog based. So we decided to use the best management and logging tools on the market….and it just so happens this is one of the things CP does extremely good. Management and logging. Today’s talk is on “Having Fun with Logging”.
So we needed to get the DFW syslogs into SmartLog. How to do that. Well, first thing you do is pour salt on the documentation. Its OK, but only gives you the basics. I have decoded this documentation and will show you how to send DFW logs to SmartLog and make it go a lot faster. By default, you can just turn it on with factory defaults… but will overwhelm your SmartLog server and all the data will be in the ‘info’ field and not parsed out. So it took me a couple months to figure out how to do all the parsing efficiently so as not to bury our log server.
And now for the rest of the story…..
NSX is the part of vMWare vSpere that runs its SDN…including the firewalls..Distributed Fire Walls (DFW). These firewalls send their syslog to their eSXI hosts which we forward to a custom syslog-ng server on a single VM. We customized this syslog-ng server to parse DFW logs into CSV format….no easy task but doable. Why????
Once in CSV, syslog-ng forwards the logs to ‘syslog’ on currently a UTM 4800 with 4 cores. This is a custom syslog built by CP to accept syslog and convert to CP native log format. ‘syslog’ forwards the converted logs to fwd which enables SmartLog and Tracker to read the logs.
Simple enough…….ok, then read no further.
Syslog-ng allows you to parse through logs and reformat them. You can do some simple parsing with the basic configuration file, but the fun starts with more complex patterns with the pattern matching databases. Basically you need to convert this:
Now all you people are tons smarter than this old balding has-been…so I am not going to explain syslog-ng internals to you all. You will have to read the documentation. But I will give you the overview of what I did with it.
syslog-ng has two parts
General Filtering Config: First thing you need to do only filter on DFW logs and nothing else. NSX puts labels on these logs and you will have to look in the raw log file for it.
Second thing you have to do is get rid of the TERM log entries (terminate connections for TCP, for UDP you’ll see zillions of these going to 5355 some sort of local subnet DNS resolution. Unfortunately in cloud world we have huge subnets so all the VMs are doing this local link DNS resolution). 75% of the DFW logs are TERM entries so can be ignored.
Pattern Database: Next part is a bit harder. You have to build a pattern database that will parse raw DFW log entries and match each field up to a named macro. For example below is a snippet of a DFW log entry matching pattern. You can see the ‘action’, ‘domain’, ‘protocol’ fields are 3 of many fields that would match a DFW log entry from above.
This can be a delicate task, but there is a tool to help debug … ‘pdbtool match ‘. You can type in a sample line and see if the your pattern database will match it:
General Filtering Config: Next, DFW labels ICMP protocol as ‘PROTO’. So I just translate into normal geek lingo ‘ICMP’.
General Filtering Config:Next is the fun part. We finally get to output CSV to the CP syslog server. Notice how the MACRO names like ‘size’ and ‘protocol’ are used to fill in the CSV?
General Filtering Config: And here is the ‘main’ section that drives all the above phases:
so restart you syslog-ng server:
and off you go!
At this point you can ‘tcpdump -X -n s0 port 514’ to verify that in fact the logs are formatted correctly and heading over to CP land.
Just in the off case my code sucks, here is how you test it. There are syslog generators out there that you can use: I used this one and here you can see how I generated traffic:
One key thing is the “dfwpktlogs–” is what triggers the regex’s to fire in the next section. Do NOT use ‘:’, you have to use ‘–‘ or for some reason the regex won’t parse correctly. More below…
So now the logs are flowing to port 514 on SmartCenter Log server. So you have to make sure you enable the syslog process (NOT syslogd) to listen.
This will start the syslog process on port 514 on SmartCenter.
On a MLS, you have to restart the whole MLS for the config to kick in. On MLS, a syslog daemon will be started PER domain log server IP address for every domain SYSLOG is configured to accept SYSLOGs for that domain.
and syslogng logs will flow magically into Tracker. Actually they will magically flow, but only fill in the INFO field and will not parse …. yet.
Oh yeah, when debugging you can get the syslog process to re-read the syslog config with:
OK, now it gets fun.
So there is this CP tool: Eventia Log Parser that looks pretty cool. You feed it your SYSLOGS and magic parsing configuration pops out the other end:
Yeaaaaaaah…NO. Doesn’t work and the config you output is so complex the SYSLOG engine will run at 100%. Both ELP and SYSLOG were written about the time after the Civil War and are on version 1.0…Typical CP V1 code so don’t get your hopes up. Let me know if you have better luck, I spent days on this.
So I decided to generate the syslog parsing config myself. I saw what ELP attempted and then came up with my own.
Remember that SYSLOG-NG is sending CSV formatted logs:
and CP SYSLOG is taking them in and turning them into SmartLog readable:
So let’s begin.
The debug process is this:
So far I have given you enough information to do #4.1. Let’s work on #4.3.
This may look simple…but it took me days and days to fine tune this. The ELP generated config was pages and pages of REGEX expressions. I boiled down to 1 line:
Even if you aren’t a regex geek, you can see that I am parsing the CSV file into 9 fields.
REGEX GEEK OUT (don’t read this if you aren’t a regex geek):
FYI: They don’t implement FULL regex matching! Example: You can only have 9 matching field patterns, I couldn’t get it to recognize ‘:’, you can’t use lazy searches ‘.+?’. It is some limited hack that you will never figure out because there is no documentation other than examples. AND their regex performance is horrible, so I tried to avoid using ‘.*’ because it is greedy and will scan the whole line and then backtrack. Instead I used the ‘[^,]’ which searches for all characters NOT ‘,’, which is the same as ‘(.*),’…but doesn’t have the backtracking.
============= END GEEK OUT===============================
So what is important is the ‘dfwpktlogs–‘. When the regex sees this pattern, it will fill in the 9 columns of information from the CSV formatted input line. Now there are 3 different types of actions that will be taken depending on the results of a REGEX match:
OK so next you will try and parse field #1 which is the first ‘([^,]+)’. This is the PASS|DROP field from the CSV. What I did is I first captured the field and then dumped it into a random text field. This told me that 1) I captured it correctly 2) I am able to write to SmartTracker.
Here is the add_field that does this. add_field adds the field to tracker. Field_name is the Tracker field name..in this case it is the ‘rule_name field” field_index is CSV field 1 (from 1-9). Field type is the type of the Tracker field. But beware, I didn’t always get this right and the log entries would just dissappear so I had to experiment. I also looked into:
$FWDIR/conf/syslog/CPdefined/*.C files for examples to see what their field types are.
You can also look into
to see the Tracker field names and types. Its a bit kludgey but between CPDefined examples and this file you’ll get close.
Here you can see ‘rule_name” is the name of the Tracker field defined in svt_fields.C.
So now you have 1) You captured field #1 and 2) You wrote it to ‘rule_name’ field. You can now verify that you captured the field you intended to. Once you verify this, you can then write it to the REAL field with:
Here we are writing the PASS|DROP to the ‘action’ field in tracker:
HOLD ON DREEZ: How in the heck did PASS|DROP get converted to Accept|Drop???? you ask.
Grasshopper, I introduce the dictionary file…
This is a second file in the same directory that will do transforms for you. Here you can see “PASS” being converted to ‘accept’.
DREEZ you ask: How do I know what to convert to what?
Oh yes grasshopper. You read the CP documentation…NOT!!! HAHAHAHAHAH. Dreez makes a big funny. HAHAHAHAH. Remember grasshopper, this documentation was written in the Civil War and even now the Lord Developers rarely talk to their lowly documentation peasants slaving in the fields trying to identify nuggets of information to feed their ignorant customers. (If you want real documentation see AWS documentation, you can see where developers talk to the documentation team). That’s what phone support is for —- DUH!!!!
Oh yes, I easily diverge, forgive me.
No grasshopper you do what us old people have been doing for centuries. You look at examples in
On your log server: fw log
Will print out your logs with the field names in them.
To get a feel of what the valid field types are.
So now that you know how to do field #1, now you go through all the fields one at a time….or……
You can just use my template as a starter!!!
So the first time we did this, we used CP’s syslog ELP generated config on one of our big servers and the server went to 100%++++. Unfortunately the ‘syslog’ process is single threaded so it had no where else to go.
So with 1100 VM’s….each with its own firewall…sending syslog to my config described above to a 4800 (slow) based management station…performance was much better but not ideal. The 4800 has a quad core Q9400 on it. Syslog process is a bit busy but at least not 100%. It will average between 11% and 60%, and burst to 90% now and then.
Now remember that on MLM, each domain/cma/clm has its own ‘syslog’, so you can manually distribute the load amongst multiple logging domains. But I would hate to do domain design based on log loads. For example, at one company 2 of 12 domains had 90% of our logging. So should we split them up further because of logging? Not sure.
Your mileage may very….
Everyone is excited about this because SmartLog is what our org (and all the other orgs I’ve been at) live and die by. Centralized single-pane of glass easy to use and fairly fast security monitoring. It is the gem of all of CP’s products. And it mostly sometimes works!!!
Now we can send logs from other tools like other firewalls, URL filtering, FireEye, etc into this and use SmartLog to get quick answers. I am keeping both flat syslog files as well as sending them to mongodb and SmartLog. YES: SEIM tools exist…but either they don’t work, too expensive or at capacity, etc. ‘grep’ is free. ‘mongodb’ is free, etc.
The Cloud. Its Coming. Torpedo is in the tube and you are a WWII lumbering oil tanker. You will be automated like a $15 minimum wage McDonalds burger flipper.
I switched gigs and and am now working a cloud security gig. My Check Point role is diminished 50%…but worse I no longer have access to my #1 diamond support guy Taylor who fed me all the good juicy techie fixes that I shared with you all.
SOOOOO, not sure what the future blog will look like at this point, but the CP focused techie posts will be reduced and you’ll see more Cloud security chatter. Worst case you still have the archives.
Michael Endrizzi's - St. Paul MN - CheckPoint blog on topics related to Check Point products and security in general.