YAAT- Yet Another Audit Tool – Command Line Auditing

We want to know who is typing which commands into our firewalls. Because we are using RADIUS auth, all users are “non_local” users in the Linus audit facility…and it was a big beast I did not want to dink with. So I wrote my own 1 liner.

AND because we are using GAIA, the syslog file kept getting overwritten on every reboot.

THIS

auditcommands.sh

Solves both problems.

 

Audit ON!

dreez

 

Max Disk Size – The search continues

Update 5/21/15

—————————————

From Check Point……

I would like to clarify:

Gaia:

• Prior to R77.20, GAIA OS supported up to 8TB.
• Since R77.20 (inclusive) Gaia OS support up to 16 TB.

SPLAT:

• OS supports up to 16 TB on all supported versions

—————————————–

This is continuation of SEARCHOFMAXDRIVE

So we have this massive log server. 7-1.8TB RAID-5 drives. (Of course by this time next year my iWatch will have 14TB SSD in it). During an upgrade of our log server, we wanted to resize our tiny root partition.

We tried to use lvm_manager, several hours later……yeaaaaahhhhh. NO. Crash boom bang.

Start from scratch.

Good news: On R77.10, GAIA sees all the drives BUT cannot format more than 8TB instead of 2TB (as it did under pre-R77.10).

WTF? Block size looks like it should support 16TB file systems

Or why can’t GAIA stitch it together with LVM ? Ugh…..

So we still will have to resize by hand using lvm.
HOW???

---

# Format the unused drive
fdisk /dev/cciss/c1d1
# make a EXT3 file system on it
mkfs.ext3 -b 4096 /dev/cciss/c1d1
# label it as a physical volume – making it available to the LVM pool
pvcreate /dev/cciss/c1d1
# extend the volume group vg_splat to include this new drive
vgextend vg_splat /dev/cciss/c1d1
# extend the logical (log) volume to use this new space
lvextend -l +100%FREE /dev/vg_splat/lv_log
# new resize the EXT3 linux file system partition (log partition) to use this space
resize2fs /dev/mapper/vg_splat-lv_log
#
#………. continue for each drive………

---

So the question I ask myself “I just paid $10 Gillion dollars for this state of the art log server (probably enough fuel to fly Gil Schwed from Israel to here in his private jet), do I really want to customize its partitioning? Will I have to do this for every migration? What about DR – ouch!? What about successive admins that may not have my (minimal) Linux talents, they will be totally lost”.

So while geeky and intellectually challenging, I think we will leave well enough alone at this point. I’ll just archive logs every week. Time better spent with my hot German girlfriend Gaby than watching drives format for hours and then crash at 99%.

R80 will fix all.

Partitioning out,
dreez

YADU – Yet Another Debug Utility

I know I know, “What are the chances your SmartDomainManager GUI would crash??” Probably zero. But just in case there happens to be an unheard of bug here is your new debug client. Can debug any open GUI, dynamically identifies it.

R76 and up.

Day 2 – Drinking From the Firehose

Summary: I was asking my friend why CPX was so good this year compared to last years. He said “Less sales RAH RAH, more technical”. He was right. They had about 30 developers in tow that knew the real answers. The “rah rah” team seemed to be in the background and only appeared when you need them which was great.  Oh and the attitude was much more humble this year for whatever reason. Oh, and they even talked about Quality Assurance a bit, wish I could hear more of this! So for the first time I can say I’d spend my own money going to this conference. Next year I think I might head to EuroCPX on my own dime.

41000/61000 – CoreXL/SecureXL

---

The 61000 is basically blades in a cage that cost $1 million. Each blade is 20 processors. In dashboard you only see 1 firewall, not even a cluster. Blades are hot swappable and have a variety of redundancy. The 41000 is a $250K small brother of the 61000. Each blade supports 10-40Gbps?? of throughput. They don’t use any special SecureXL hardware accelerator, they just throw more cores at the problem.

 

R80 SmartEvent/SmartLog Performance

---

SmartLog: They claim they reduced a 5 minute search to 10 seconds in SmartLog. They claim 1:1 index to log size (we were seeing 3:1). SmartEvent is totally rewritten to use its own Smartlog-like index and is suppose to be super fast. You can get R80 now, it is version agnostic works for all versions.

 

SmartDashboard – SubPolicies and Layers

---

I went  back 3 times to the R80 Mgt presentation. I am very excited about the work they are doing and can’t wait to download the EA and try it out.

1. Two types of policies are much more structured in this GUI which is great
   -Access Policies – Rules people write with IP addresses and APPLICATION protocols and User Names. X can get to Y
   -Threat Prevention (dynamic rules responding to threats IPS, AV, Threat emulation,etc).
2. Layers: In the picture you can see the 1/2/3 policies above. Those are called layers. Each can be one of the following: Access Control, Application Control, Compliance, DLP, (and maybe something else). The screen shows we are currently in the data center policy which is an access control policy. This policy is executed first from top to bottom on every packet. Next #2 the  compliance policy is executed and then Next #3 the DLP policy is executed.
3. Cool thing you can install each layer separately from the others so you don’t have to install all of policy and IPS all at the same time. I think I said before that policy and threat measures can be installed separately…finally. You can see the policy installation options in this screen.

1. WITHIN!!! A policy you can create subpolicies. These subpolicies are kinda like the current sections markers we have now except each subpolicy can have its own administrative editors. The policy will be executed from top down including all the subpolicies, but each one has a different editor.
2. Down on the left you can barely see where you can use command line to do everything you want in the GUI. Very cool.
3. They also have a “Web Services” view, where you can build web screens with SOAP/REST scripts to interact with the management station.
4. They also added another column (optional) called data awareness. You can specify what types of files to allow/disallow for upload/downloads. Probably from the DLP blade. In the app control column you can say Frank can access YouTube but only for 60 seconds and 6 meg of data or give a file name they can download.
5. Rules have another action called “Monitor”. They will just log activity but no make enforcement decision so you can play out “What If” scenarios.
6. They do have a view called “Unified” where you can see all policies and threat protection all in 1 pane. Each column per rule is another protection like app control, threat prevention, etc.
7. They finally support multiple concurrent admin audit. You log in and create a session. This session has all your edits. You can save your session, etc. As you work on rules you lock the rule but not the whole DMS. When you are done with your session you publish it. Only after you publish it can it get installed on gateway. When you click on the rule, you can see the history of edits on that rule.
8. You can click on a rule and ask to see all the logs for that rule. Very cool.
9. The gateways can now recognize interfaces as objects like Cisco/Palo/Juniper. You define the interface(s) as a zone and use that as an object.
10. One thing that has me a bit worried. They say they integrated logging, monitoring, smartevent, policy all into one dashboard which would be really cool. But I think in reality you only see a summary in Dashboard and when you click for more detail it kicks off the standalone SmartWhateven client.  Not too impressed, prefer single pane of glass model.
11. You can file Service Requests from SmartDashboard. pretty basic, really hope we don’t have to use it much. PLEASE!!! Not after the last 4 years of pain.
12. Once again, nothing on MDS yet. Still in the thought stage.  But very cool start.
13. I’ll save the best for last. CSV export and import!!!! You can FINALLY import and export objects with CSVs for editing and reimporting. Perfect for enterprises for managing large number of objects. If you are religious, thank you gods.
14. For provisioning, they do have a script manager. Didn’t get to play with it much, seemed pretty basic.
15. Change Control: I guess it will somehow integrate with Change Management systems like HP and Remedy and you can drag and drop from your Change Management on your ticket window into the management station for IP addresses and a column with ticket information like ticket number and comments from the change control ticket.  The links are they, they have to partner up.
16. Web Based Object Management: So if you have an object group that is dynamic and person XXX is responsible for maintaining the group, you can create a web page with WebServices, they log in with AD and manage only that object and nothing else WITHOUT using SmartDashboard.
17. I thought?? I saw CLI access VIA the SIC tunnel port 18191 from SmartDashboard which be very cool. That would supplement WebUI CLI access via 443 and of course CLI via port 22. Helps a ton in case we lock ourselves out of a box somehow, another avenue.

All of the above has the making of THE BEST Enterprise Security Management Environment on the planet. THIS is what makes and differentiates CP from the wannabees. THIS is what makes me so proud to be associated with this product. Two-Thumbs  Plus Up (but please make sure you QA the frigging thing this time. Screw the whiners, take your time and deliver a quality product)

SCADA Demo

---

Kinda anticlimatical. Lots of FUD and when the attack happened I kept asking “what happened” and then it ended. I think I missed the point. I guess they are going after more SCADA traffic signatures for app control and IPS. Not sure how mature it is. If you have SCADA traffic, call them and they are very very ambitious to sniff your traffic and create more signatures.

 

GAIA – Next Steps

---

Nothing really too cool here, incremental which is OK by me as long as they run it through QA. As long as they stabilize the basic firewall features they can go as slow as they like. If the firewall doesn’t work, might as well get a different product.

1. Working on 77.20 for more stability THANK YOU!!!!!
2. They have a routing team in Israel instead of the Nokia crew in California and the Cluster people in Israel. So hopefully routing and clustering will start coming together.
3. More abilities to upgrade from the GUI from the cloud…I missed some of this. I hate upgrading from the GUI because it freezes and dies so I’ll stick with the CLI thanks.
4. You can get detailed reports on HFAs, HFs, versions, etc in the GUI. In future will upload to cloud for more reports. OK start, but need inventory of our whole environment, not just 1 gateway at a time.
5. CPView: Seems to be a really cool CLI tool to view performance issues. Can run on any version NOW. Can see inside the kernel and inside blades to see what they are doing with memory and CPU. Thumbs up.
6. Performance sizer. Runs 24 hours on a system and can tell you if you need a bigger system. We use it and it is so-so. You have to be able to anticipate internal external User base and doesn’t seem to be based on realistic numbers. Neutral.
7. CoreXL and SecureXL can be mostly modified within the GUI instead of the command line. Thank YOU!
8. I saw nothing on fixing licensing hell. Oh well, maybe version R90.
9. LVM Manager. They have a CLI GUI that lets you dynamically change disk partition sizes. Just front end to lvmmanager from Linux but I like it.

Rant and Rave

---

PREQUEL: I had my best discussion with a gentleman from Atlanta who I forgot his name and organization. CP should hire him as their director of marketing because he painted the picture that Gill and marketing have missed for 25 years: “Single Pane of Glass for Policy and Response”.  Right now organizations pick best of breed products. Large ones have 2-3 different firewall products (CP and Cisco), 1 SEIM like Envision (sucks), 1 Threat Emulator like Fireeye (awesome), 1 IPS like SourceFire (awesome). Best of breed. Unfortunately the threats are coming faster than these best of breeds can respond. When SourceFire picks up a DDOS or Fireeye sees a internal compromised system SLOW BUREAUCRATIC UNTRAINED POLITICAL people have to make phone calls and do change control and fight political battles to respond to the threat. Meanwhile the hackers only have one purpose and do not have to fight those political, training, etc battles.

While CP may not have all of the the best of breed point solutions, the do have the best of breed single pane of glass to respond to zero-day threats. It all starts with awesome management and logging which allows organizations to have one political boundary, one trained staff, one  bureaucratic boundaries, one tactical solution solution to react to zero-day threats in one pane of glass.

But does that sell a CSO or CFO? No.

What sells is the ROI. Imagine only having to have X number of security operations personnel instead of 5x, one for firewalls, one for DLP, one for SEIM, one for AV, one for SPAM, one for URL, one for IPS, one for ….. The numbers may be off but you get the idea.

Basically CP marketing is selling technology (performance, appliances, pretty GUIs ). At CPX they pitched their latest theme “Software Defined Protection”. What the heck does that mean? How does that save money? How does that differentiate from competitors? How does that make me want to run to the CP Retail Store and buy 10 610000?? Instead my above description is selling solutions with ROI, and everyone understands ROI. This is the theme The Gill should paint and every talk and demonstration could echo it and every sales and marketing person could lead with. Maybe something like “CP: Your single pane zero-day solution” <<<Rah-Rah do the dance here>>. And then ever year at CPX The Gill should measure and share with us how far they have come to dominating the Security Management market based on their awesome management environment.

(Then again, The Gill has an awesome jet and I drive a 2006 Scion XA. Who Knows Best?)

RADIUS Lock Out – Warning

So I was deploying my superuser RADIUS solution to our R75.46 gateways and locked myself out of one box. Could not even log in at the console. Turns out it was a R75.40 unpatched system and RADIUS was broken and ONLY did RADIUS auth and nothing else. Not even local authentication. Something went wrong with the PAM module and bypassed the PAM_UNIX processing.

The secret to get in was to pull the network cable(another guy Dan figured this out). Some sort of race condition between the cable and the console. Geez louise.

Make sure you have these patches.

pam-0.99.6.2-3.26.cp986008001
CPshell-1-986008001

dreez

 

Defining RADIUS servers in MDS

I know I’m late to the party with this one buthopefully will save others from searching high and low. How do you integrate RADIUS into MDS in R75.40+??? Documentation is sparse

1. Bring up any global policy
2. Click on the Servers and OPSEC tab (below)
3. On Servers create a new RADIUS group
   
4. Add the nodes till you build a group
5. When you assign users, you can specify RADIUS authentication:
   

Done

dreez

Fun in mds_backup land

mds_backups usually work… but have you tried the restores???? surprise surprise if your ducks aren’t lined up.

1. R75.40-476 (don’t know about the others) they put the customer data  in the wrong directory for open servers (some version of the appliances were hosed too,not sure which)(see my blog on it).
2. If you DO move your customer directory to /var/log/customers (has to be exact name), then make sure you have the patch HOTFIX_FOXX_HF_HA46_184  with the backup/restore magic in it
3. Local GAIA CLI/GUI backups (not mds_backup) will fail if you run out of space because they store the archive in the / partition which has limited space
4. NOTE: That GAIA GUI/CLI backup includes GRUB files in the backup which means you can only restore with a GAIA CLI “set backup restore local XXXXX”
5. NOTE: GAIA will grab its ‘local’ backups from the /var/CPbackup/backups directory… So I hope your partition is big enough if you are planning on copying archives into that directory to restore it.
6. You could store your backups offline…..but dont’ bother reading the  CP instructions for RESTORE. The command line is funky and wrong. And GAIA command completion is screwed up so don’t trust it.set backup restore ftp ip VALUE file VALUE username VALUE password plain ——- OOOOPS check it out there is my password in the file name
7. Oh yeah, just to make it more interesting the backup log is nicely hidden but here it i s:
8. If you want to restore your MDS to a different server for doing upgrades or something like that, then use Unix command line ‘mds_backup -l -d /var/log/CPbackup/backups’ and ‘mds_restore’.
9. SOOOOOO basically if your /var/CPbackup partition is too small you are hosed. Well, there are symbolic links…….but   seems to me  that backups and restores should work out of the box. Try this for symbolic links. I tested this with backups and restores and it seems to work…weirdly. For a locally retained ‘backup’ command, it will actually break the last ‘mv’ command which use to move it into /var/CPbackup/backups and keep it here. Works for ftp backups. I am trying to figure out a better way…please hold. Make sure you test this because may work differently without the magic patch or the version you are on or if its an appliance or open server or the the moon was full and tides where low! mds_backup works regardless, you can specific the directory or us the current working directory.
10. Oh yeah, just noticed that restore did NOT restore my /home/admin directory. Darn, could of really used those scripts I”ve been working on for years. Oh well “se la vie” as the Frenchies say.
11. If for some reason clish cannot see the backups when you do a ‘set backup restore local <backupfile>, try using /bin/restore.
12. Oh you will LOVE this. In /bin/bash mode, make sure you are using the right restore command because there are two of them and your $PATH variable will only pull one of them. There is a snapshot restore and a GAIA backup restore. Look at the full file name paths:
13. Oh yes, did I say that the mds_backup -l switch on MDS will be ignored when it comes to SmartLog index files? See next to exclude.
14. Oh yes, did I say that the $MDSDIR/conf/mds_exclude.dat file has the wrong pathnames in it? They populated it with symbolic link names and you have to use absolute names. Use ‘pwd -P’ to see the real
    
    directory names. NOTE: The exclude names start with the base of the tar command in the script.
    

SolarWinds Cattools – Script Manager – The cat’s Meeeeoooow

If you have more than 6 firewalls, I know most of you probably have a script library sitting around. Maybe you’ve been through 10 admins and so the scripts are a hodge-podge of semi supported and hacked up tools that sometimes work…until a new R1000.45 HF 201 comes out and changes formats, so then you hack into your scripts yet one more time.

CP is suppose to come out with some sort of scripting support in R10001.48 HF 132 (R77 has some of this, haven’t seen it). I’ve been begging them to go out and buy SolarWinds Cattools. I’ve started to use it in the past month and I’m not sure how I survived without it all these years.

Cattools is a script manager based on a primitive form of Visual Basic.  You import all your firewalls from an MDS export into an excel spreadsheet and import into the panel on the left. Then on the right panel you have your scripting library. Below you can see I have various scripts for inventorying our firewalls as well as modifying them, etc.

Here is one example of how I inventory our firewalls to make sure they all have snapshots in case we have to rebuild. You highlight the script you want to run and click ‘run’. It will execute this bash script I wrote which downloads a bash script to inventory snapshots and executes the script and dumps the results into standard output on the Cattools management station. From there I use perl, awk, grep to gather the output (I’m working on turning this into a spreadsheet).

Cattools is awesome.  In reality it is fairly good…because its like giving a cup of water to a dying man searing in the Judaean Desert of Scriptland. CheckPoint really needs to regain its lead in supporting and managing large enterprises, and if they bought Cattools and improved it to work best with CP products they could rock the world.  As is…Cattools is designed very specifically for Cisco/Juniper/appliance markets so it had some quirks I had to overcome. It will work out of the package…but I improved it with my own scripts..the results of one you can see above.

Specifically Cattools is designed to work with Cisco like products. So it is hardcoded to expect certain prompts on the CLI. And the GUI is designed around these Cisco  like prompts. As is, it will work with GAIA but you get a lot of errors and timeouts…but in the end it works. So I made it work more generically with Unix and any application like FTP, GAIA clish, Install scripts, SCP with passwords, etc.

When reviewing scripting tools the big hangup is handling prompts. Your script can hang on a “password:” prompt or a prompt from a weird application like installing patches. Cattools with my mods does a great job of handling prompts. Out of the box Unix and GAIA require you install expect scripts to handle the prompts so can be done but a bit of a challenge.

Also look at how wayward processes are killed. Cattools does a great job monitoring and killing off wayward processes. It has several levels of timers where if a response is not forthcoming, it shoots the process.

Also look at how the output is gathered and brought home. The scripting tool should handle this for you. Cattools I feel does a great job of this. It also allows you to post process the output. I am figuring this part out now and will report in the future.

I’ll be talking more about Cattools as I get time. In the mean time you should give it a go…but temper it with knowning about the out-of-box issues. But it is still a cup of water to a dying soul.

Massive script engines are a double edge sword. They can save you incredible amounts of time….or destroy your entire environment. When using tools like these make sure you have control processes around them. For example: On operations that modify the firewalls, those have to undergo review and have 2 people execute them together and on a max of 2,4,,8,16 firewalls as you prove it works.

NOTE: I loved this product so much that my company Midpoint Technologies now sells this product. So I have a strong bias.

Script ON!

dreez

RADIUS Update! Finally have superuser access using RADIUS

You might want to check out the update on my RADIUS post. After 1 year, I finally made some headway to RADIUS superuser access with nonlocal users.

https://dreezman.wordpress.com/2012/11/20/radius-goes-backward-on-gaia-r75-40/

FTP using username password at command prompt

OK this one really pisses me off because I searched high and low for something like this. And then I spent too much time trying to find different ways of doing this without resorting to SCP priv/public key distribution. I’m sure its been around since the dawn of time and I’m the last person on earth to figure it out.

GAIA includes a ‘curl’ application.

UGH!

Download:curl -u ftpuser:ftppass -o localfilename  ftp://ftp_server/public_html/xss.php
Upload:curl -u ftpuser:ftppass -T myfile.txt      ftp://ftp.testserver.com

http://www.thegeekstuff.com/2012/04/curl-examples/

Anyways this will simplify the distribution of patches in a large enterprise environment when using scripts.

NOTE: Only exists on gateways and NOT MDS

FYI curl SCP is NOT supported in GAIA (but it is in Unix)

[Expert@NOCgw1:0]# curl -u user:password scp://1.1.1.1/file.txt
curl: (1) libcurl was built without LIBSSH2, scp: not supported!

Geez I hate being the last one to the party.

dreez