LEA spreads the load

So we have these older log servers (common in most places I’ve worked) that need to be upgraded but of course its more fun trying to squeeze more blood from a turnip then playing politics of buying a $8K log server.

Every place I’ve worked seems to constantly drop logsdroplogs

and I’ve never figured out why. So our super duper diamond guy Taylor had an idea from R&D to offload the FWD logging process from all its LEA work (sending logs to SEIM, Smartevent, Tufin), and let it just handle gateway logs.

So the magic command is sk91343

mdsenv  DOMAIN
mdsstop_customer DOMAIN
$CPDIR/bin/cpprod_util CPPROD_SetValue FW1//6.0 Spawn_LEA 4 1 1
mdsstart_customer DOMAIN

and the fun starts.

So normally FWD is sucking hard at the CPU and you see numbers like 190% CPU time (sorry don’t have any snapshots of it) for FWD.

So check out what happened when we spawned off the LEA handing.

So normally you see a parent FWD with internal threads (not processes). The green boxes are the internal thread IDs and the red boxes are the parent process id. So the example below has 3 internal threads 15259, 15260,15261.

internal threads

here you can see the hierarchy better

 

basic process

When we issued the magic command, the lea processes became their own FULL processes and NOT internal threads. Their process id and thread id are the same, this is how you tell.

 

 

 

lea processes

lea parent child

So now check out the CPUs, instead of a couple individual CPUs at 100% and the rest at 0%, now you have them all at 20%-60% (and this is low).  This is because the LEA processes are fighting for processors when before they were buried inside an single individual FWD with the whole FWD fighting for a processor and the FWD process had the processor at 100% doing 10 different tasks.

You can see smartlog and LEA sucking hard, while FWD instead of being at 110% is now at 14%.

top

Unfortunately this did not solve the problem of dropped logs and slowed down SmartLog and SmartTracker. So we may back out.

But it was cool to see that logging is not sucking the CPU, its LEA and SmartLog indexing that keeps a log server busy.

Morale of story: Spend more money on log servers than MDS’s.  Only split out LEA if you have a ton of CPUs.

LEA OUT!

dreez

 

Advertisements
Post a comment or leave a trackback: Trackback URL.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

blog.lachmann.org

Michael Endrizzi's - St. Paul MN - CheckPoint blog on topics related to Check Point products and security in general.

%d bloggers like this: