How to make VSX go fast

Got this from a little bird, so can’t take credit.

Use case was datacenter pushing 24Gb through a VSX chassis.

Can it do it?

I was told they got 22Gb on a 21700 through a single VS using this configuration:

1. 21700/21400 has 3 PCIe bus’s on it. Each PCIex16 bus supposedly handles 16Gb in 1 direction.
   
   
2. Config
   R77.10 – firewall blade only

   4 port 10Gb bond with two ports used on 10Gb line card one and two ports used on 10Gb line card two. Have to separate on two different PCIe buses so don’t overload single PCIe bus.

   VSLS Cluster (2 members) with 6 virtual systems created

   Layer 3+4 bond distribution algorithm

   Only one VS used to pass firewall traffic

   Single firewall rule – ANY-ANY-ANY-Accept –Log

   CoreXL enabled and set for 2 instances for the VS under test

   Hyperthreading not enabled

    

3. MultiQ enabled and set for 12 RX queues (apply to both members). NOTE: MultiQ only works on receive and not transmit.

   cpmq set rx_num ixgbe 12

4. fw ctl affinity -s -d -fwkall 4 
5. cpmq reconfigure 
6. Reboot 21700

   Follow these steps on both 21700VS cluster members

   1.            Create the $PPKDIR/boot/modules/simkern.conf file:

    

   [Expert@HostName]# touch $PPKDIR/boot/modules/simkern.conf

    

   Note: If this file already exists, then there will be no impact from ‘touch’ command.

   2.            Enable SecureXL parameter ‘sim_requeue_enabled’:

    

   [Expert@HostName]# echo ‘sim_requeue_enabled=1’ >> $PPKDIR/boot/modules/simkern.conf

   3.            Check that SecureXL parameter was added:

    

   [Expert@HostName]# cat $PPKDIR/boot/modules/simkern.conf

   4.            Reboot the machine to apply the changes.

    

    

   Run test from appliance idle state.  Between tests, please run:

    

   fwaccel off

   fw tab –t connections –x –y

   fwaccel on

    

   This will clear the connection table and avoid out-of-state errors in future tests.

    
    

Manually restoring SIC files

Every now and then SIC just tends to disappear so we manually reset. Got this from a colleague (who taught me all I don’t know which is a ton) when they lost SIC on several firewalls  because their restores didn’t work when they tried to upgrade to R77.10 and had massive failures and had to revert. THEY:

– Replaced registry file $CPDIR/registry/HKLM_registry.data
– Also  $CPDIR/conf/sic_cert.p12
– Had to go through several backups to find ‘good’ SIC keys. VERY disconcerting. 

My demo box. $CPDIR/registry/HKLM_registry.data

Be careful out there those of you living on the sharp upgrade end.

dreez

Setting Affinity for VSX

Discovered this cool thing while setting affinities for CoreXL in VSX. I have not seen this documented anywhere and it is so critical to setting the affinities correctly. Not sure why they don’t document this so.

When setting affinties for VSX, there is a specific hierarchy:

1. (V)s – is the king
2. (P)rocess – is the queen
3. (I)instance – is the jack

When you set affinity for the the VS, you are setting affinity for ALL!!! the threads of the VS in user space including the helper threads (FWD,CPD,VPD):

When you set affinity for the fwk INSIDE of a VS, you are setting the (P) process affinity for ALL of the instances of that (P)process which is the fwk:

And finally when you set the affinity for the fwk instance or as I call it the VSi instance, you are setting the affinity for a specific thread of the fwk (above). NOTE: you have to be in the VS environment to set the instance affinity.

You can see the results in the output below.  Each of the affinities indicate what command set its affinity:

and guess where this information is stored???

Cool huh???

And if you really are a Linux affinity geek you’d know what this is all about:

VSX over and OUT!

You’d see a lot more of this if you took my class!

dreez

CoreXL – VSX implementation

This is kinda hacked but work in progress as I get more info.

I am comparing CoreXL in R75.40VS of a VS standalone gateway (VPG) to a VSX gateway. Look at this.

The VS standalone gateway is configured for 4 CPUS. I then used cpconfig to enable CoreXL for 4 firewall instances. Check out the process list. OK this is cool. I setup 4 fw instances and got 4 worker threads.

Now with a VSX gateway, I setup in VS0 corexl for 4 firewall instances:

NOW check out the process list. Notice the ‘worker’ thing goes away and the VS0 process gets a ‘-i 4’ for the number of firewall instances inside that process?

The other VS1 is a switch and you can’t change the instances. But VS2 I want to change:.. but you have to do it in SmartDashboard.

Wonder what happens when I change it??

You guessed it!!! Changes the ‘-i 2’ to reflect the SmartDashboard config

Who Cares???

So this is my guess. On Standalone you map real processes to real cores. IN VSX gateway, Each VS gets a process “fwk” and internally they do internal process threading to simulate CoreXL based on the “-i” parameter.

Fact or Fiction?? Sure wish the documentation would talk about this stuff!!! In fact the documentation calls it all “firewall instance” no matter if its a VS standalone process, VSX gateway process, VS process, VS internal thread.. Tomato Tomaaaato.

Why do I care??? Because what do I map to a CPU. The OS process? or the Process Thread?? They are all instances???

…..Ongoing……

So let’s take it a bit further…

I want to make one of the 2 instances from VS2 to a specific core??? Can I do that? Does it make sense??

Well I guess I can!!! So does that mean that an additional process got generated???

NO! Same process list as before.

So what can I say….Not sure. Is it still an internal thread? You can’t assign an internal thread to a CPU that I know about.

Still a mystery but a very interesting one!

dreez

610000000000 appliance

 

I just saw this for the new 61000 appliance

http://www.checkpoint.com/products/61000-appliances/index.html

http://dl3.checkpoint.com/paid/77/CP_R75.40VS_61000_Security_System_Admin_Guide.pdf?HashKey=1382581795_feb21a171931b7fba729f7d80bcca412&xtn=.pdf

Manual is pretty poor. I was trying to understand how this is different than any other platform.  Looks like they are trying to do a ESX or Xen type VSX implementation.  So the host is like ESX and NOT GAIA with all these special ASG commands.  Then you run VSX as guests on top of it. But you still are in Unix and can see the file system.   Just guessing.

Anyone had any experience?

dreez

 

 

 

 

VSX DMS/CMA architecture

So CP documentation says that the DMS/CMA for the physical VSX gateway should be different than the DMS/CMA for the VSs themselves.

Which makes sense. You really should do this because when it comes to:

1) Assign permissions to your DMS, you want the super duper admins in charge of the
physical chassis and the sub-humans in charge of the regular VSs.

2) Decommissioning: One thing the I feel CP sucks at is deleting and moving and renaming objects. Either you can’t (GLOBAL OBJECTS), or if you do you get 1000 errors and you have to GUIDBEDIT from 1am to 7am on a Saturday morning with huge sweat stains in your armpits. If you decide to decommission a VSX physical gateway, you should isolate it into its own DMS. and put the VSs in another VS. That way its easier to delete and re-create…..even if the whole thing blows up.

So How does this all work???? Well the basics are the physical chassis goes into 1 CMS/DMS and the VSs go into a separate one.

So first create the VSX gateway in one CMA/DMS. Like here you see I created TestGW into the HQ_Domain CMA/DMS

So any policy will only be adminstrated from the HQ domain admins.

They I create Virtual Systems, in the HQ_VSs_DMS CMA/DMS that reside in the TestFW physical VSX gateway:

So you can see that the VS was created in a separate DMS.

So now the part that sucks is that the MDS does not really track in a hierarchical manner what VSs are related to what VSX gateways. As you can see above the Test VS is not under the TestGW. Duh.

I’ve written the P1 developers about it. Supposedly the new wiz bang P1 will cure cancer and grow hair on my head and solve this problem, but I’m not holding my breath. I’ll give it until 1/1/2015 until all the bugs are worked out until I see it solve this problem.

S

But only a week ago I was drinking wine and eating baguettes and cheese at my campsite. What was I thinking?

Firewalls Rule!

dreez

Set VSX environment in /bin/sh scripts

One time a Unix demigod yelled at me in public because I was writing my shell script in bash instead of /bin/sh. You’d think I drew a comic strip of {can’t speak of this religious figure because it has incited wars}. I mean it was like religion. I’ve been damaged goods ever since. Now I write my scripts in sh just because I’m afraid he’ll be lurking around the corner.

So in VSX and maybe its a GAIA thing, if you write your scripts in /bin/sh, you’ll notice you can’t access ‘vsenv’. The reason is the profile didn’t execute the /etc/profile.d/vsenv.sh script that inserts VSX functions into your /bin/sh environment.

So your scripts need to include:

#!/bin/sh

./etc/profile.d/CP.sh

. /etc/profile.d/vsenv.sh

Forgive me for my sins,

dreez

Updated my cheat sheet and random notes

Welcome back from the holidays people.

Here in Minneapolis its 40’s and raining and ugly. Kinda like London fog. It is suppose to be -30F below and 20 inches of snow!

So I decided to sit inside and work on updates to my cheat sheet. I now have GAIA and VSX in my sheet.

Midpoint Training Directory

Check Point cheat sheet

Notice that I rarely use GAIA unless I have to. I’m a died in wool SPLAT person. So my GAIA commands are limited. The only reason to use GAIA is if you are a routing geek and do dynamic routing on the firewall….which is insane in my opinion….but to each their own.

VSX on the other hand is really cool. I’m thinking everything will be in VSX someday.

I’m trying to build this uber SmartLog server. Running into problems that I hope to resolve and share. I’ve crashed R75.45 and found really obvious bugs where the counters don’t work. They sent me patches so maybe in R75.60??? you’ll get them too. Just a heads up.

Have a great 2013!!!

dreez

VSX VSLS vs HA

OK, I admit I’m a neewbie with VSX so there may be an easy answer. But I’m sharing these in case others are struggling…or they know the answer.

So I setup a VSX cluster and for the life of me I could not remember if I set it up as a VSLS or HA. I went to the ClusterXL page and guess what, no matter what you choose this page cannot be modified and it always says the same thing:

and cphaprob stat is always set to “active standby” when you’d think it’d be “active active”.

Wellll…look at the cluster mode: VS Failover!!! Ah Haaa, what does that mean Dr. Spock?

SOOOOOO where is the command to figure out what ClusterXL mode I’m really in?

I can’t find it. But this will tell you:

VSX design considerations – Eggs In One Basket

VSX is pretty cool as VMware is cool. You can spin up new firewalls in a right-click and have them running in minutes vs dealing with ISO installs and physically installing a system.

Downside: Eggs In One Basket

Obviously if the VSX chassis has an issue, then all VSs inside of it are impacted. Like VMware. If the host has problems, then all the guests have problems. Like marriage, if the wife is not happy…the man is not happy.

But I diverge.

So I am creating a list of VSX design considerations to think about when deploying VSX. I’ll update as life goes on. These are mostly related to MDM, haven’t thought about SmartCenter:

1) Put the policies for all your VSX gateways/chassis in a single domain (VSXDOMAIN) that only admins can get to. These policies are pretty simple like ssh, web, control connections. This way some newbie can’t modify chassis related parameters and rules that impact the whole chassis. You can also then assign specific administrative permissions to  VSXDOMAIN so that only superduper users have access to it. Also prevents newbies from doing an “Install Policy on All Gateways” and installing some random policy from one of the VSs onto your VSX gateway. Boy would that suck or what?

Or maybe more out there but has happened, is that when you mix VSX and VS config information in one domain, then the underlying file system is also hierarchical. If you delete the VSX domain file system, you delete all your VS config information. This is not true if you separate them.

2) Propagate routes: Note that when you update routing tables in SmartDashboard when you save the policy, the routes are immediately injected into the  running VSs even if you don’t do a SAVE. So you might want to think about limiting the propagation of routes if it will be a problem.

Check out this listing of files that changed after I saved the dashboard. THe files contain mostly routing information. These files are also updated with dinking with interfaces and spoofing. Oh yeah, in theory all you have to do is a “Save Policy”, but in reality you probably have to do a push policy. For example routing is suppose to be independent and not need a install policy, but you may run into spoofing issues and so you have to push policy to make it all work. Interfaces and spoofing you have to push policy to make work. I would say for major interface updates like bonding, vlans, give it the old reboot too.

3) VSs are attached to VSX gateway objects. This is a permanent binding, you can’t undo. If you need to rename the VSX gateway object, you basically can’t. You have to have downtime on all VSs as you re-create VSs on new VSX gateway objects.

4) Think about the things that are shared; CPU, time, file system like the /etc directory. All these shared items impact the whole chassis. So if someone deletes the /etc directory or a critical file like resolve.conf, you lose the whole platform. So create least privilege around who can modify these shared items.

5) Along with #4 above. Arp tables. VSX interacts with many clients so it creates huge arp tables. Check out sk43772.

6) Along with #4, watch out for SmartMonitor, cpstat fw -f os, top, fw ctl pstat. They all lie about CPU utilization. VSs share CPUs, but the reporting tools only report on the average of ALL the CPUs.  If the system is configured to use 2 of 12 CPUs, you get the average of all 12 even though the 2 are running at 100%. So your average will be say…. 80% idle.

7) I guess the biggest issue is scope. VSX is fantastic for small little used firewalls.Major gateways with 10s thousands of users or major ecommerce gateways….no. Why? 1) Shared platform – Any virtual system shares the platform. Eggs in one basket.If your lab firewall all of a sudden overwhelms the CPU, then your ecommerce firewall is toast. If an admin types in the wrong command, toast. If  you add interfaces and something goes wrong, toast. 2) Going backwards – VSX is really cool, but it is difficult to go backwards. Yes you can delete virtual systems….but you sit there with your fingers cross praying to gods you didn’t even know exist. And if it blows, get ready to hand edit internal databases. I’ve done it. Sucks. Check Point HAS to fix their DMS database from flat files to a SQL lite database. Right now an object name appears in 100 different places. If the delete or modify goes wrong, it only updates 50 of the 100 and then you hunt for the rest. And pray none are in a binary file.  3) Upgrades. If you never done a VSX upgrade then you won’t appreciate what it takes and how delicate it can be. You really don’t want to do this on a $1million/minute gateway. With ANY virtual system. Just one mans opinion.

That’s what I know for now, stay tuned I’m sure this will get updated as life goes on.

dreez