YAMDS – Yet Another MDS

One thing I like about Indeni and the new R80 (haven’t seen it, just the gossip) is the MDS will be merged with SmartCenter so that one can see all your firewalls (not a domain at a time). This takes us closer to the single-pane of glass security management solution that CheckPoint excels at. I’d like to see ALL my firewalls in one window not just a domain at a time. Indeni is similar in that it let’s you monitor all your firewalls from an enterprise view which I like (haven’t seen it in a while, but use to anyways).With SmartMonitor you only get a domain at a time and then you only get 1 firewall at a time, not even a cluster so its somewhat limited. (And remember if you have SecureXL on, the traffic stats are horked).

Anyways I diverge. So until R80 comes out with the REST API, I am working on building my own enterprise MDS that is web based. It will allow you to start Putty sessions on ALL your MDS firewalls and SmartDashboard on ALL your MDS firewalls so you don’t have to go into each domain.

Phase 1: Dump MDS – v2 – 2.4.2015 (yeah I know my code is a hack, wish I had more time)

This script filters theMDS for all your firewalls and puts them into a CSV

<Domain,fwname, mode,  IP_Address,  Software_Versions,  HW_type,d ns_name>

mode={CLUSTER, INLINE,MONITOR(Layer2Firewall)}

which I then import into a SQL database and go from there.  So I thought I’d share with you this tool because you can use it to dump into your asset tracking or script databases to access all your firewalls.

This script is cool because it gives you the hardware type and version numbers for all your firewalls. This took a bit of ‘awk’ munging to do because clusters are weird and R77.10 does clusters differently.

MDSDUMP OUT!

dreez

VSX & CoreXL Training- You’ll love the price

So Oct 13th 2013 I went VSX & CoreXL crazy. I saw the world going virtual, so I wanted to figure out how to balance VSX across millions of processors with CoreXL and how to fine tune it. (I’m sure some sales guy would love to sell that appliance). So I spent 9 months geeking out on Linux internals, talking to developers, etc. I was trying to put a class togther for the VSX/CoreXL geeks of the world. We got some interest and lots of emails but it was a money loser trying to get 10 people into a room in the same city for the right price.

WHY?

Because there are so few CoreXL geeks in the world. 99% of the people just want it to work by clicking on buttons in the GUI and then call support when it blows up. This course is 80% deep dark Linux OS…..boring, geeky, complex, scary.

SOOOOOOOOOO. We decide to just share it for FREE!!!!

I stopped development on May 2014 so the slides are probably 80% complete but you’ll probably grep some info from them.

CoreXL Slides – Version 5 – 5/24/2014

Maybe someday I’ll put together a class and finish the slides, but for now enjoy.

CoreXL OUT!

dreez

PS: Special thanks to the Chicago Check Point crew  Doug Shumaker, Jason Taplitz, Rich Comber and the 10 other Men of Check Point – for their listening to me drone on for hours and their critique.

LEA spreads the load

So we have these older log servers (common in most places I’ve worked) that need to be upgraded but of course its more fun trying to squeeze more blood from a turnip then playing politics of buying a $8K log server.

Every place I’ve worked seems to constantly drop logs

and I’ve never figured out why. So our super duper diamond guy Taylor had an idea from R&D to offload the FWD logging process from all its LEA work (sending logs to SEIM, Smartevent, Tufin), and let it just handle gateway logs.

So the magic command is sk91343

mdsenv  DOMAIN
mdsstop_customer DOMAIN
$CPDIR/bin/cpprod_util CPPROD_SetValue FW1//6.0 Spawn_LEA 4 1 1
mdsstart_customer DOMAIN

and the fun starts.

So normally FWD is sucking hard at the CPU and you see numbers like 190% CPU time (sorry don’t have any snapshots of it) for FWD.

So check out what happened when we spawned off the LEA handing.

So normally you see a parent FWD with internal threads (not processes). The green boxes are the internal thread IDs and the red boxes are the parent process id. So the example below has 3 internal threads 15259, 15260,15261.

here you can see the hierarchy better

 

When we issued the magic command, the lea processes became their own FULL processes and NOT internal threads. Their process id and thread id are the same, this is how you tell.

 

 

 

So now check out the CPUs, instead of a couple individual CPUs at 100% and the rest at 0%, now you have them all at 20%-60% (and this is low).  This is because the LEA processes are fighting for processors when before they were buried inside an single individual FWD with the whole FWD fighting for a processor and the FWD process had the processor at 100% doing 10 different tasks.

You can see smartlog and LEA sucking hard, while FWD instead of being at 110% is now at 14%.

Unfortunately this did not solve the problem of dropped logs and slowed down SmartLog and SmartTracker. So we may back out.

But it was cool to see that logging is not sucking the CPU, its LEA and SmartLog indexing that keeps a log server busy.

Morale of story: Spend more money on log servers than MDS’s.  Only split out LEA if you have a ton of CPUs.

LEA OUT!

dreez

 

Who needs SecureXL when you can turn off AntiBot?

This is a weird one that I tracked down, but can’t take credit for fixing.

Our AntiBot and AntiVirus sites were whining about things being slow after we installed firewalls with AB and AV. Well, you know how users whine all the time – who listens to them anyways? So they can’t get to Oprah’s home page, big deal.

Well one site was slow but this one single DNS name would NOT resolve. The request hit the client side of the firewall and never came out the DNS server side….for only this ONE DNS name.

Yeah….weird huh?

In addition, they are in DETECT only mode…so what could go wrong?

Well check out what zdbug says

Paste that into support.checkpoint.com and you get sk81320 which leads you to the “Speed Up DNS” button

Turns out AB/AV does reviews of DNS names before releasing the connection. You’ll see the client side nslookups have 2 second timeouts most the time. Have to do ‘background’ reviews and not ‘hold’ the connection. Geez louise.

Problem solved in R77.20 and now this is the default config

Geez I hate those random problems.

Thanks Todd! for finding this,

dreez