Making LDAP/Identity Awareness SmartDashboard User Picker Go Faster…And even Fix it

So our SmartDashboard user picker keeps breaking. So turns out for Yet To Be Determined Reason (YTBD) the User Picker gloms onto an LDAP server specified in a random LDAP AU. I haven’t figured this part out yet. So if the LDAP server goes down or is in SIBERIA, your user picker experience will make you want to switch to Cisco ASA. Remember, the UserPick in Dashboard is making queries from YOUR PC!!!!!   So you need to find a LDAP server closer to your PC. The User Picker is pretty darn sensitive to latency so you won’t know if its broke or tired, it just randomly works. It took me forever to figure out how to make the UserPicker wire into an LDAP server that is faster. This is it

1. 1) Note what LDAP server the UserPicker is currently using by expanding the user list. In the example below it is going to the SIBERIA-DC.
2. .Now you have to go through all your AUs and figure out which AU points to the SIBERIA LDAP server. Hopefully you are able to change it to a DC that is more local to your UserPicker. You might have to duplicate this UA and assign the new one to the SIBERIA firewall and keep this one for the UserPicker.
3. If you have multiple DCs in your list, you have to pick the lowest latency one here. This is what decides what DC User Picker will use.
   

Yeah, I know its a hassle but I PROMISE you its fixed in R80. PROMISE!!!.

LDAP OUT!

dreez

routed seems to finally work

We have a fairly huge dynamic routing infrastructure and the new ‘routed’ daemon was pretty flaky for a long time, since its release April 2013.  Last week we finally got a version routed-0.1-cp986005013.i386.rpm  that was finally stable and could handle our environment. So I can finally say its time to move dynamic routing to the firewall and you don’t have to worry about crashing your environment when routed suddenly stops working or fails to come up.

Having said that, the management of routed is still weak.

1) No zero downtime upgrades you have to copy routes between members and then delete them
2)Stop/Start clustering and routing separately
3)Debugging is pretty primitive Linux level skills required

Route ON!

dreez

 

R77.10: Identity Awareness and Groups

I’m on the edge of this so unfortunately don’t know the details, but my buddy will bring me up on the details as it develops. I have a 1/2 blog created on the details and am trying to bring it up to date.

We are a huge AD/Identity Awareness/Captive Portal shop and so we obviously have been breaking it on many fronts. Specifically, it was SmartDashboard picker slow/timeouts, dogging down WAN lines with tons of AD traffic, and most importantly could not work with AD/LDAP groups, not supporting multiple LDAP AU per AD domain (sk92782), having to adjust priorities on hundreds of AU’s over hundreds of firewalls….I’m not sure of all the other issues.

Basically, it didn’t scale.

CP has been working on it for a year and last week huge a breakthrough. Many of the above issues were fixed in the patches they issued to us. Yeah, there are still problems but it is nice to see things finally working after a year of pushing a boulder uphill. These patches were hot from development so not sure they are up for GA yet.

Anyways if your IA is a leaking rowboat note that a fix is on the way and its not your problem. CP knows about it and is working the issue. But you will have to push hard to get to the right people so start pushing.

IA out!

dreez

 

 

 

 

New SmartLog Permissions

UPDATE: 12/1/14: mds_HOTFIX_GYPSY_HF_BASE_748 is the fix and it works in R77.10

UPDATE: 11/13/14: After much hub-bub, this is fixed in R77.30 and they are backporting it to R77.10/20? Will let you know.

With X00 firewalls across X0 domains we live and die by SmartLog. R77.10 SmartLog is awesome, its fast, finally stable. It alone is sufficient reason to chuck any other firewall product.

Except!!!

They changed the way permissions work in R77.10. Now only Domain Super-users and MDS Super users can use  MDS SmartLog. This takes my breath away. Our front-line domain managers (SOC, NOC, Audit, IPS, Security/Risk Management) use SmartLog for debugging not only firewall problems but network problems in general….across all domains. They are not interested what domain the problem is…they just want to know where it is in the enterprise. Domain Super users and MDS Super users only use SmartLog a couple times a week for escalated calls.

So WHY???? restrict permissions to an awesome market changing tool to people that only use it a couple times a week??

Ugh….

dreez

RADIUS Lock Out – Warning

So I was deploying my superuser RADIUS solution to our R75.46 gateways and locked myself out of one box. Could not even log in at the console. Turns out it was a R75.40 unpatched system and RADIUS was broken and ONLY did RADIUS auth and nothing else. Not even local authentication. Something went wrong with the PAM module and bypassed the PAM_UNIX processing.

The secret to get in was to pull the network cable(another guy Dan figured this out). Some sort of race condition between the cable and the console. Geez louise.

Make sure you have these patches.

pam-0.99.6.2-3.26.cp986008001
CPshell-1-986008001

dreez

 

Finally! be able to debug application control!!! New fw ctl debug flag sheet

Long time ago I asked Sergei if he could update this critical document. Sergei spent months updating the fw ctl debug flags. Then he spent more months getting the bureaucracy to release it to us unwashed peasants. Up to today it has been a bitch debugging things like application control because you can’t see inside the monster. Thanks to Sergei we now have a chance of fending for ourselves. If you get a chance please thank Sergei for his work (via linkedin) and provide comments. He will be glad to incorporate them.  I am being dramatic here because the more positive feedback we supply the the more internal information they will release in the future. This is a huge win-win for all of us.

HERE IT IS Sergei’s 2014: FW CTL DEBUG FLAGS.

 

1. 1) Run this command to see all supported/existing flags on your version:
   # fw ctl debug -m
2. 2) After enabling debug flags for a module, run this command to verify:
   # fw ctl debug -m MODULE

 

For those who live in SK land like I do, hopefully you all realize how the quality of the SK’s have improved this past year. Sergei Shir  and his crew:

1) Knowledge Center Manager – Uri Lewitus  (all the major changes in the quality and quantity of solutions are his initiative + he released my debugging document) – https://www.linkedin.com/profile/view?id=103729954
2) Knowledge Center Technology Leader – Ronen Zel (his knowledge of our internal systems is crucial to our team)
3) Knowledge Content Developer – Stella Shteinbuk – https://www.linkedin.com/profile/view?id=972764
4) Knowledge Content Developer – David Kornfield

are knocking them dead. Its a little bit sad that people like Sergei don’t get more credit when people like us are keeping 25% of the world’s economy online (well, CP is in all the top Fortune 100 so made that number up) thanks to people like Sergei having our back. Please make sure and provide comments on the SKs so they can show upper management how important they are to keeping CheckPoint on top of its game. Win-Win-Win.

Check Your gtar versions

Here is an interesting one that doesn’t show up anywhere

Our SOME (not all) R75.47 HAs were failing (and I’m having suspicions about our backups).

• $CPDIR/util/gtar –version    = 1.12
• /bin/tar –version               = 1.15.1

Sure enough some sort of version incompatibility.

• cp /bin/tar $CPDIR/util/gtar

Fixed it.

Taylor found it! Thanks.

dreez

Manually restoring SIC files

Every now and then SIC just tends to disappear so we manually reset. Got this from a colleague (who taught me all I don’t know which is a ton) when they lost SIC on several firewalls  because their restores didn’t work when they tried to upgrade to R77.10 and had massive failures and had to revert. THEY:

– Replaced registry file $CPDIR/registry/HKLM_registry.data
– Also  $CPDIR/conf/sic_cert.p12
– Had to go through several backups to find ‘good’ SIC keys. VERY disconcerting. 

My demo box. $CPDIR/registry/HKLM_registry.data

Be careful out there those of you living on the sharp upgrade end.

dreez

Fun in mds_backup land

mds_backups usually work… but have you tried the restores???? surprise surprise if your ducks aren’t lined up.

1. R75.40-476 (don’t know about the others) they put the customer data  in the wrong directory for open servers (some version of the appliances were hosed too,not sure which)(see my blog on it).
2. If you DO move your customer directory to /var/log/customers (has to be exact name), then make sure you have the patch HOTFIX_FOXX_HF_HA46_184  with the backup/restore magic in it
3. Local GAIA CLI/GUI backups (not mds_backup) will fail if you run out of space because they store the archive in the / partition which has limited space
4. NOTE: That GAIA GUI/CLI backup includes GRUB files in the backup which means you can only restore with a GAIA CLI “set backup restore local XXXXX”
5. NOTE: GAIA will grab its ‘local’ backups from the /var/CPbackup/backups directory… So I hope your partition is big enough if you are planning on copying archives into that directory to restore it.
6. You could store your backups offline…..but dont’ bother reading the  CP instructions for RESTORE. The command line is funky and wrong. And GAIA command completion is screwed up so don’t trust it.set backup restore ftp ip VALUE file VALUE username VALUE password plain ——- OOOOPS check it out there is my password in the file name
7. Oh yeah, just to make it more interesting the backup log is nicely hidden but here it i s:
8. If you want to restore your MDS to a different server for doing upgrades or something like that, then use Unix command line ‘mds_backup -l -d /var/log/CPbackup/backups’ and ‘mds_restore’.
9. SOOOOOO basically if your /var/CPbackup partition is too small you are hosed. Well, there are symbolic links…….but   seems to me  that backups and restores should work out of the box. Try this for symbolic links. I tested this with backups and restores and it seems to work…weirdly. For a locally retained ‘backup’ command, it will actually break the last ‘mv’ command which use to move it into /var/CPbackup/backups and keep it here. Works for ftp backups. I am trying to figure out a better way…please hold. Make sure you test this because may work differently without the magic patch or the version you are on or if its an appliance or open server or the the moon was full and tides where low! mds_backup works regardless, you can specific the directory or us the current working directory.
10. Oh yeah, just noticed that restore did NOT restore my /home/admin directory. Darn, could of really used those scripts I”ve been working on for years. Oh well “se la vie” as the Frenchies say.
11. If for some reason clish cannot see the backups when you do a ‘set backup restore local <backupfile>, try using /bin/restore.
12. Oh you will LOVE this. In /bin/bash mode, make sure you are using the right restore command because there are two of them and your $PATH variable will only pull one of them. There is a snapshot restore and a GAIA backup restore. Look at the full file name paths:
13. Oh yes, did I say that the mds_backup -l switch on MDS will be ignored when it comes to SmartLog index files? See next to exclude.
14. Oh yes, did I say that the $MDSDIR/conf/mds_exclude.dat file has the wrong pathnames in it? They populated it with symbolic link names and you have to use absolute names. Use ‘pwd -P’ to see the real
    
    directory names. NOTE: The exclude names start with the base of the tar command in the script.
    

Grasshopper – What is a Blade you ask?

So I just figured out why I have no clue how licensing works. ‘Blade’ is totally an overloaded term. Seems like CP management, marketing, licensing and GUI people should have lunch together.

I was investigating today’s meaning of Performance Pack PP,  Acceleration and Clustering ACCL, Advance Data Networking ADN or ADNC Advanced Data Networking and Clustering.  These were, are, is, going to be…a ‘blade’ at some point in the marketing branding cycle. CP ADNC blade.

So if today’s definition 12/4/2013 of ACCL is a blade, then what is ClusterXL? Is it a function? Then what is IPS? A blade or a function?

For me, when I hear the term ‘blade’, it should show up in the SmartDashboard GUI with a checkbox next to it.I don’t see PowerPack, ADN, ACCL in the SmartDashboard GUI. ADNC kinda appears in the GUI. Is it a Blade?  A Bundle of Blades in the GUI or a bundle of features? But it is sold as a license ‘blade’.

Seems like CP needs to sync their marketing and licensing with their implementation.

 They overload the term Blade too many times.

At least Blade was a cool movie. They should ask Wesley Snipes for advice.

Dull Blade,

dreez