Alternative InfoSec Career Recommendations

Alternative InfoSec Career Recommendations

I have been in InfoSec for a long time, so people seem to think I can dispense sage wisdom on their Information Security careers for themselves or their kids. So instead of answering each one individually I thought I would sum up my 40 years of experience here in order to share and reference it in the future. 

I can summarize my sage wisdom in Two Words “Soft Skills”

In my career some of the best InfoSec colleagues I have known are those that could make me laugh, build trust, serve others, had deep integrity, build consensus, win-win negotiation, improvise and remain calm in tense situations, speak in clear simple short phrases, draw pictures, use analogies, prioritize based on objectives and had my back in good days and bad. 

Why are these skills so important in an InfoSec career? 

If you are about to take on University courses, you can eventually learn to hack, write scripts, click deploy, write a policy document, be the tech hero in an attack scenario, etc. Smart lone wolves are valuable in certain situations but they can never hold back waves of nation state attackers. They are only a tactical defense, a sniper in a WWZ zombie movie.

Whether it’s defending against lone hackers living in their mom’s basement with all the time in the world or a nation state team attacking critical infrastructure, group think is the only enduring defense. For me Information Security is 75% people skills and 25% technical. People that have the ability to encourage cooperative groupthink in both strategic and tense tactical situations are the true InfoSec professionals for whom I uphold the greatest respect.

In the old days, production deployments took 6 months to 1 year. We had time to familiarize ourselves with the product and observe its normal and abnormal behaviors and recommend security modifications. In these days of agile CICD development and 2 week sprints, deployments happen several times a day. Traditional InfoSec defenses can no longer keep up with the velocity of software deployments. The application and the developers are our new firewall. Shifting Left into the development teams is the ONLY enduring strategic defense. As true InfoSec professionals we have to embed ourselves into the development teams or we will be spending our careers believing we are Kim Yo Jong’s issuing decrees that our loyal followers will blindly embrace.

Ask yourself…Can You?

Development	Can you convince a development team that has 2 week sprints to implement their own security reviews on pull requests, include security sensors (not related to their primary objective) in the application, remove secrets from their code, remediate vulnerabilities detected by DAST, SAST, SCA in the CICD pipelines? Remember, developers are financially incentivized to produce functionality. Security tasks just create more bugs, slower code, slower sprints, more testing, more complexity, pissed off marketing people, etc.
Operations	Can you detect abnormal behavior in an app that has a production release every day with new functionality?
Incident Response	Can you guide an incident response team which includes non-security personnel, through an distributed attack scenario where groupthink and info sharing is critical?
Architecture	Can you motivate your architecture team to re-architect a broken authentication infrastructure?
Management	Can you convince management and marketing to incentivize not just functionality but also security efforts? Remember, functionality produces cash flow, security features are a liability.

All the above scenarios require years of experience and patience, so you might as well start developing a collaborative personality now. Python, infrastructure-as-code templates, buffer overflows, CICD pipelines can be mastered in months and as you move through your career you will forget some technical skills as you learn new ones. But soft skills are enduring. You will always build upon and refine soft skills in order to solve the above scenarios.

Of course you will require University InfoSec basics, but as you choose your electives, mentors or outside activities I suggest you think long term with some non-traditional alternatives. These are my recommendations for skills that you will use throughout your career and gain true lasting respect from your peers:

Books
	Negotiation	Look for the long term win-win, not just a path that feeds your ego or people will avoid you in the future
	Building Personal Relationships	Trust is the foundation for any security program. If people do not trust you, they will go around you or ignore you. Look up Dr. Gottman books. Changed my world in life, love and work relationships.
	Motivation and Leadership	Get others to believe they are the masters of their destiny and you will always have their back.
	Business Finance	Money – not logic – rules the world. Get used to it and learn to play within its boundaries.
	Anger Management	Replace your ego with laughter and you will win many battles. Praise in public, admonish in private.
	Priority Management	Learn to prioritize based on cost/benefit/threat/risk/impact
	Sales Techniques	Just because you have a good idea, does not mean people will back it. Learn how to sell your ideas by getting people to buy into it.
Activity
	Improvisation	Get outside your head and learn flexibility, thinking fast, anticipate others thoughts and movements, playing into the big picture
	Comedy classes	Making others laugh is worth 1000’s of pages of security policy manuals. Make people laugh at your faults, believe me you have a lot of material to work with there.
	Rock Climbing	Team building, trust, risk management, focus
	Art classes	Express your ideas in pictures instead of 1000 words
	Listening exercises	You listen but do you hear? Do you really understand what developers and management are struggling with? Until you do, you will not be able to fit your agenda into the business solution.
	Escape rooms	How to work together in teams in tense situations to solve problems. Can you lead the team?
Communication
	Draw cartoons	Express your ideas in pictures to capture attention and tell a story with humor
	Public Speaking	Can you capture a crowd explaining what information security is? Can you make them laugh so hard they cry? Will they go home and tell their friends and family?
	Teach with Pictures	Try teaching your parents, spouse, kids a topic you are interested in only with pictures and no words. Then see if they can repeat the lesson to someone else.
Technical
	Powerpoint	Become a powerpoint expert with pictures. Tell a story. Get rid of the details. Add animation. Capture attention. This is how you sell your ideas.
	Excel	Become an Excel expert. Pivot tables, graphs. You have to put your statistics/probability courses into real life to prove your ideas with numbers. Like Powerpoint, it captures attention but with facts.
	3d animation.	Be a 3d Blender expert. Great skill for both building websites and presentations. Once again, it captures attention.

These soft skills will not only help you in your career but also with personal relationships. And that is what is sorely needed in building the next generation of human firewalls.

Many thanks to Gene Leonard, my mentor for showing me the way.

NSX Training

Greetings Home Boys,

Been digging deep into NSX and SDN and gave several classes. I think I cracked the nut so decided to share with you.

Beware of snake-oil SDN people, I’m not convinced they know what they are talking about. Make sure you don’t replicate your existing environment into The Cloud. Painful.

• 101-Networking for Dummies
• 201-NSX/SDN for Dummies

Microsegmentation OUT!

dreez

 

HTTPS Inspection – The Dark Side

So been playing with HTTPS Inspection on Squid and CP for URL filtering lately and am starting to understand why people avoid it. This is not a CP problem, but an industry problem. Sort of like when IPSEC came out, your IPSEC never talked to my IPSEC. Only HTTPS Inspection has been around since I had hair and I’m not sure we are any closer to nirvana then we were before.

So basically – HTTPS Inspection is where the proxy is a Man-In-The-Middle between a client and a server.  The proxy can then decrypt and inspect all data in the tunnel. Google will give tons of details so I won’t bore you. The key here is the certificates. How does the client get WELLSFARGO.COM server certificate if the proxy is in the middle and acting as the client? Well, the proxy generates a fake WELLSFARGO.COM certificate and sends it to the client. Here is a good diagram and slide show of how the fake cert is created and sent to the client. You might also want to research HTTPS CONNECT where the client connects to the proxy server for the HTTPS tunnel.

All sounds easy-peasy huh??? Not so quick…Before embarking on this journey I tried to google what are all the problems of forcing clients to install a new cert and proxy through a HTTPS Inspecting gateway. Only got bits and pieces. So here is my list of the fun times I am having doing HTTPS Inspection:

 

• How do you work with humans that don’t even know what a cert is? Much less a proxy? How many times do you get the same phone call “What is a cert? Proxy what?”. Yeah deployment was automated via AD and magic just happened….right. Problem is there are too many pieces and unique components..proxies, certs, AD, Linux, OpenSSL, JavaCertStores, pinned certs, that automation will only get 80% and the others are YOUR phone call.
• Client desktops are locked down. Have fun installing those certs or fixing problems for the client in Zambia speaking Bemba on a locked down desktop.
• Not all servers use Windows so installing certs via command line on TinyLinux version .00000003  will be a great time.
• Some applications use Java cert store or OpenSSL cert store or Bob and Jimbo’s cert store version 0.45. So make sure you install the right cert into the right cert stores.
• Some clients apps just will not use fake signed certs signed by the proxy server.  They are pinned  and will only use the original cert and not one generated by the proxy server. 
• BS you say. My ORG has AD or other friggin magic and we already distributed an ORG public cert to all our clients. That’s all you have to do is setup the proxy to sign certs as the ORG CA and wham-bam thank you mam its done. OK, so that means giving the ORG private key to the proxy server. Let me say that again….the ORG PRIVATE key. How do you control use of that private key? How do you know the proxy admin is not deep into hookers and blow and accidentally sends the private key to all his Russian friends? And malware these days are looking for private keys like crazy…is the proxy server secured???
• No problem you say, I’ll just buy some fancy FIPS 140-2 certified  $1billion Hardware Security Module HSM with 126 pretty blinking lights and hook it up and push the GAZUNGA easy button to store all the private keys. HAHAHAHAH. Oh you make me laugh geek boy. Yet more technology on technology on technology. Have fun with that when it breaks. Interoperability, supportability, debugging, etc.
• Speed. Generating fake certs and decrypting SSL kills CPUs. No problem you say, my vendor has XL5Billion hardware accelerator developed by some Nigerian guy for the KGB. HAHAHAHAHAH. You so funny. See above.
• After your vendor wines and dines you with hookers and blow and single malt, in your drunken stupor ask them to allow you to log into their support site and type in HTTPS inspection or SSL decryption.

 

So my cost/benefit analysis is pretty negative right now as I work through all these HTTPS issues. I’ll keep you updated.

 

 

 

R80 – wow

I’m blown away. I’m stunned. I’d sell my kids schoolbooks to use it (I don’t have kids). It is my inner glow.

In the past year I have used:

• Cisco’s new security management GUI
• Palo Alto’s Panorama (sounds like its from the Jetson’s cartoon)
• NSX Distributed Firewall
• R80 Checkpoint

And R80 blows away all the other vendors.

And get this….I think they even tested it before they released it. I know, even I am stunned. OK, there are still some bugs and dealing with CP arrogance is a pain but R80 makes it all OK. They actually thought about the user experience and enhanced its enterprise management capabilities to allow scaling. It is true art.

TESTED: Just basic SmartDashboard on R77.30 gateway. I did not test MDS or R80 gateway which are coming out soon.

[ IN PROGRESS ]

cool things:

• Was in 77.30: Deep inspection of objects. You can search through hierarchies of groups to find a base object like 1.1.1.1. Both in rule base and object finder. The search is like google or you can qualify it. Just beautiful.
• pencils on rules that identify items that were modified
• Local copy of changes that you publish and share with others, finally concurrent access
• SmartLog embedded into Dashboard and interacts with it – very very cool
• 14 second vs 2 minute policy installs – very cool
• From my desktop GUI is API driven. From GUI can console to mgt and issue API statements.
• Add to groups from menu and menu stays up until you are done makes it easy to add to groups. Several ways to group. Grouping is key to scaling a management environment.
• Import/Export domain worked flawlessly
• Can export into spreadsheet rules and objects. Needs a bit of work but step in right direction
• Licensing is actually a bit easier (I thought I’d never say this) to manage

Bad:

• Looks like they will not implement more scoping beyond global/local objects as in the past. I loved PAN’s implementation of global/domain/firewall/zone scoping. When microsegmentation hits, I think we will even need scoping on a per application basis. So application PAYROLL has its own rule/object database and can inherit/export to other databases.
• crashes now and then
• Where is SmartTracker :-(..but you can use R77.30 Tracker!!! Thank YOU!
• For same event – data in Tracker is different that in SmartLog
• vSec integration is pretty basic. You can only see security tags, can’t manipulate them
• Software update notifications are fudged at this time
• Can’t import rules and objects from spreadsheet
• Application-site objects have a flaw that if you use them like groups your rulebase may become corrupted in how it evaluates rules because you might have duplicate application objects and it does not alert you.
• Searching through groups with exceptions doesn’t work right.

I’ll update this as I use it more, but so far kudo’s. For large environments you might want to wait until more bugs are ironed out but for smaller installations you will never look back.

Inner Glow YAAAAH!

dreez

 

How To Change CMA/Domain Name – Preserve SIC

Our local MDS god figured this out. Renaming CMA/Domains

Verified on R77.10.

Note this procedure preserves SIC.

Check Point says this procedure is OK as long as the global policy doesn’t change under you.

 

1. Remove CLM from FW cluster object logging.
2. Old logs:
   1. User Tracker on CLM to issue log switch.
   2. Back up CLM logs (optional).
   3. Make sure to use “p” option to preserve log file timestamps:

“cp –p 2016* /var/temp-logs/”

3. Delete CLM via MDG.
4. Take CMA backup:
   1. mdsenv cma-xyz
   2. cd $FWDIR/bin/upgrade_tools
   3. ./migrate export /var/export_cma-xyz.tgz
5. Delete entire old domain containing CMA.
6. Create new domain & CMA with new names.
   1. Make sure GUI-clients is “any”.
   2. Use same IP address as old CMA so FW still talks to same CMA IP.
   3. Don’t start the CMA till after the import below.
7. Import CMA objects using file /var/export_cma-xyz.tgz.
   1. Click “continue” to the global policy warning.
8. Assign new CMA to appropriate global policy.
9. Create CLM with new names.  Copy logs back in.
10. Tell cluster to send logs to new CLM.
    1. Push policy
    2. Install database.

PA Daily Operations update – From The Trenches

Firewalls have been around for 25+ years and at this point to me is a firewall is a firewall no matter what the label on the box says. I am totally mystified why organizations randomly jump from one vendor to another based on technology alone. I know licensing costs soar, support sucks, platforms are unstable, etc. But in the end a firewall is a firewall and the grass doesn’t get much greener after you make the switch. You probably get 1 year of cheap hardware and kiss ass support before you swirl to the bottom of the toilet as the new vendor pulls in new customers and forgets all the fireworks they promised would fly from your behind.

So yes I am talking the PA vs. CP debacle. PA is a fine firewall (if you don’t turn on all the misc junk). CP is a fine firewall (if you don’t turn on all the misc. junk). PA is a massive marketing machine the earth has never before seen. CP has an incredible enterprise management and logging infrastructure that can’t sell snow to Eskimos.

I just spent 3 hours in a PA hands-on class. Been 5 years since touching one. My reaction : painful.  Why? Because from the marketing rah-rah I was expecting fireworks would fly from my behind. The reality is: Its just another firewall. The web interface reminds me of all the other freeware java/ajax shakey hope-to-god-this-works GUI firewall interfaces. Or maybe they hired a bunch of ex-Cisco CSM programmers and sent them to Web development school for 6 weeks.  I mean its OK, but primitive compared to CP for an enterprise environment. Their logging just sucks compared to SmartLog.

I just don’t get it, but kudos to their marketing machine.

I have friends at another  enterprise corp that spent millions and countless hours to switch. Years ago they had about 300-400 CP firewalls and ~5-10 firewall people. NOW:

• ~50 firewall bodies
• PA management and logging is OK but definitely not as good as CP
• It is stable
• App control is starting to be deployed and mostly works sometimes
• Various 3rd party analysis tools don’t work like Tufin, Firemon, etc. so rule reviews are difficult
• EXPENSIVE
• Support is hot/cold

So millions were spent and countless hours were toiled and they went from Point A to Point A with 5x more bodies. How did that increase security? How did that lower costs?

Summary: Don’t drink the Kool-aid. Understand what your end goal is, don’t just go from Point A to Point A. Spend those funds on more important security projects that have a cost/benefit.

 

 

 

 

Palo Alto Threat Detection review from the trenches

So I have a friend of mine XXX who has been through several iterations/implementations of IPS, DLP, Firewalls, Threat Detection because someone drank Vendor YYYY cool-aid. XXX is much like me — dealing with CheckPoint can sometimes be a pain and its getting real old but CP management and logging (SmartLog) keeps us with the home team.

XXX’s mgt drank the Palo Alto kool-aid so XXX brought me up to date on the good/bad/ugly of PA’s threat detection environment.

So here it is in my words with XXX’s review:

PROS:

• Scoping for objects/rules is great: firewall,zone,global. Wish Checkpoint had this
• Licensing is easier
• Solid as a rock, good quality
• IPS between Palo and CheckPoint is about the same

CONS:

• Logging cannot compare to SmartLog. Some cryptic form of regex
• Trying to correlate logs in centralized logging is very difficult – each log type Firewall, URL, Threat, etc has its own log item and the only way to tie them together is session ID which is reused about every 2 weeks.  Very difficult when there are multiple firewalls that use the same session id.
• They don’t even have a true DLP it is called Data Filtering.  It will not take full regex entries therefore false positive rate is very high for SSN and CC
• Wildfire only scans specific file types and is far less than FireEye.  It also will only scan a file that is 10 mb or lower so some files can get through.  [Getting exact numbers from FireEye.]
• Rules are easy to enter where it becomes difficult is if you want split responsibilities between network and security.  In order to enable URL filtering, IPS, data filtering they need to be added to a rule.
• no “Where Used” function until last release
• Expensive!
• Checkpoint has more of a true DLP, Palo has data filtering
• Support has been poor
• FYI: XXX LOVES!!! FireEye. Every firm I’ve worked with has said the same. What SmartLog is to me, They feel about FireEye

Summary: Detection systems are weak and forensics capabilities (Log searches/correlation) is even weaker.

VMware’s NSX Distributed Firewall Review- Very cool NOT enterprise ready toy

OK folks, the word is out. NSX firewall (Distributed Firewall – DFW) has some seriously cool features

• Security groups
• Security tags
• Dynamic inheritance of policies on VMs, and group membership
• Inheritance hierarchy of firewall rules so you can build subzones
• Micro-segmentation…very cool…per host firewalls
• Ability to monitor traffic on a per nic basis for every host
• REST interface
• Integration into NSX environment for common monitoring (you can see rules and objects in logs)
• Integration into NSX for object lists (object definitions for vSphere items exist for you to consume)
• No separate upgrade for firewalls, part of the whole NSX upgrade package.
• Allow rule inheritance (haven’t tried it but shows potential, lot better then strictly Global/Local like in CP)

…..but will never cut it in an enterprise environment.  It just doesn’t scale, you need an army of analysts to maintain. What I call a ‘write once’ firewall, you write the rules once and never go back to maintain it. It’s times like this that remind me how mature CP management and logging are (and forget about all the bugs).

SUMMARY: Security composer very cool just don’t use it. Easy to write rules, a nightmare to maintain.

I just wish Vmware and CP could merge their firewall efforts. DFW has some really cool ideas, they just need CP’s 25+ year enterprise management mindset.

Here is what I know….

(WORK IN PROGRESS, These are my notes as I am learning. I may update them as I verify the good/bad/ugly. Put grain of salt….)

• vSphere NSX web client is a pretty poor shaky bug ridden java mucus patch, they must of hired some Cisco CSM developers.
  
• Rule administration is done from several different screens so you jump around a lot. Firewall, Security Composer, vCenter VM summary, vCenter VM manage firewall rules, security tags…are all related to rule administration. All in different windows.
• If you are creating a rule in service composer and need to add a unknown service, you have to kill the whole rule creation process and first create the service and then go back and redo your work creating the rule. Not true for the firewall management window, you can create a new service.
• If you want to create a single rule for a single SRC/DST it is easy to use the Firewall manager. But be aware that there is only 1 humongous rule base for all your rules. If you split the one’sies from the more complex rulesets in Security Composer, then you will have 2 rulebases to manage. Write once, no win-win. The micro-segments are broken into SECTION headers but they all still appear as one huge database.
• Basic problem with all new products that don’t scale. Indirection. In order to make a product scale you can’t have tons of base objects. For example in firewalls you can’t have JUST IP addresses in your rules. There would be millions of rules. So you use host objects WebServerA=1.1.1.1 and groups WebServersGroup includes WebServerA/B/C….. and hierarchy of groups EnterpriseServes include WebServersGroup. This is called indirection. You have to go through several references before you can get to 1.1.1.1Well DFW SecurityGroups are very special when it comes to indirection. #1 it can be deeply hierarchical with references to other SecurityGroups and Object groups and #2 it is DYNAMIC, changes at run time. Very powerful and dangerous at same time.With any product, when you see indirection like this you have to look at the back end support. If someone says “1.1.1.1 is slow, fix it” how do you find 1.1.1.1 if its buried in layers of indirection? Lots of new fancy tools have indirection to build the schema, but few good products have tools to unwind and find the base objects. DFW does not have this capability. It cannot search through a hierarchy of SecurityGroups. So debugging is painful. I have spent hours with NSX support looking through backend databases trying to figure out where an object is all defined. Very painful.Write Once Firewall. You write it once, but you can never modify debug it.

  This is where CheckPoint shines. Unwinding indirection is CP’s sweet spot. They have many tools to search for buried objects and manage them. THUS the reason it is my #1 pick for enterprise management.

• Security composer forces you to write rules as:
  # 1 FROM: specificzone TO: whatever
  you always have to specify a zone in the FROM/TO. You can’t specify:
  #2 FROM: any  TO: any
  I think they do this because in the end the compile all rules for NSX into one huge rulebase. So if they allows #2, then it would override all the rules 2+
  NOTE: There is a APPLY TO parameter to make this work. Kind of hidden.
• Because there is 1 humoungous rulebase that applies to all VMs its easy to break micro-segmentation. Imagine you have a zone rule:RULE: 100: FROM ZONEA TO ZONEB DENYYou will have to look through all the rules ABOVE rule 100 to make sure your RULE 100 works as you think it should. Imagine 1 day after you put in RULE 100, someone might create RULE 1 which overrides your microsegmentation.RULE: 1     : ANY ANY ANY ANY DENY
  RULE: 100: FROM ZONEA TO ZONEB DENYImagine trying to do a rule cleanup? After a couple years pulling out RULE 1 would be a nightmare.The problem is rules are in 1 huge database and not broken into individual rulebases for each VM.Write once firewall.
• Some tables you can sort, others not. Security composer you can’t sort.
• Scoping is complex. It is based on the physical world: datacenters and portgroups. So your objects are visible either
  • Globally
  • Per datacenter (group of ESX systems)
  • Per portgroup (group of network ports)
• In reality you want scoping to be based on a logical security constructs
  • Per firewall
  • Per firewall zone (network interface(s))
  • Per management domain
  • Globally
• You can’t comment and log rules by default you have to open a column view window and tell the GUI to insert column and then enter comments and logging option. You can comment when you initially create in the wizard, but after that it is a hassle.
• Security composer: Can’t write comments in individual rules. So making modifications, can’t put in ticket numbers and who/why/owners/ticket numbers/etc.
• Assigning security tags – If you are assigning a security tag to a new VM in a production system, you can’t put a change control comment anywhere. Once you assign the tag, its in production. Hopefully nothing blows up because you won’t be able to track it (although there is an audit function which is nice, just no comments or SR #s etc.) There is a audit function, haven’t tried that yet.
• Security composer: Can’t put section titles on rules.
• HOLD ON THIS NOT SURE: PERFORMANCE: VMware will talk about how the DFW is embedded within the hypervisor so its fast. What they won’t tell you is that the rulebase for ALL the VMs are in 1 rulebase. So if you have 100,000 rules in NSX, then each VM has 100,000 to evaluate PER interface. You can’t assign 6 rules to VM1, 10 rules to VM2, etc.
  NOTE: There is a APPLY TO parameter to make this work. Kind of hidden.
• In FIREWALL, the policy creation works as normal policy editors do. The SRC/DST can be IP, groups, objects, etc. In Security Composer you can only reference Security Groups. So if rule is SRC: vm1 DST: group_of_ips  then you have to embed group_of_ips in a security group which means more clicking and windows. Takes forever to create new rules in security composer.
• No ‘where located’ button to find objects in rules/groups, etc.
• Can’t/limited copy and paste drag/drop policies/objects/services so you can build templates and re-use them. You can copy rules, one at a time.
• Hard to determine relationship between tags and security groups. You can easily build the relationship, but hard to maintain and delete because you can’t understand the relationship. Security groups->Tags is dynamic filter so you are guessing if the tag is included and Tag->Security Group I haven’t figured out how to determine the relationship. Right now have to use naming schema with common names which doesn’t really scale.
• If you are looking at Security Policies, you can’t tell which security policies are tied to which VMs, only security groups. So if you are cleaning up unused rulesets and trying to tell if a Security Policy is actually enforced on any VMs, there is no way to easily tell. You have to visit each VM and look at the ruleset. True, this is all dynamic, but there is no history to show if the policy has been used recently, etc. Write once firewall.
• You can append policies together, but the priority of rule order has to be done on a global basis. You get a list of ALL the policies and you have to UP|DOWN re-arrange them in the window. It should be pick a policy and then chain them together.
•  You never know what all the rules are for a single VM. All the rules for the whole vcenter are in one big long list with ‘assign to VM’ tags on them you so you have to go through them 1 at a time…and there could be thousands. There is one place that lists most of the rules…but not all of them. Only the service composer rules not the firewall rules.
• VM->Monitor->Firewall Rules will show some of a VMs rules, but not all of them. And if the names are long that’s all you get is This_Grou…….. with dots. AND no comments, And you can’t expand the groups, and and and and. Horrible. Write once firewall.
• Security composer – you can’t negate individual objects, you have to negate all the objects in the src/dst column.
• A security policy can be applied to a single VM or a group. When you are editing a policy you can’t tell if you are modifying a policy for a single VM or a group of VMs.
• Sometimes you click on things to open them up, sometimes you have to hunt for the little teeney weenie insey binsey icon…and then try to figure out which icon does what. Other views the object is not clickable. OR you have to highlight something first before you can edit (rule editor to insert a new rule).
• When in service composer and you want to add a service for port 8080, it only searches for names and not port numbers. So you have to guess what the service is if you don’t know it.
• There is minimal policy search. In CP, you can search a policy for port 80, and it will show you all the rules where port 80 appears….even if its embedded in a group. You can in DFW, but you just get the rule headers and not the highlighted items that makes it easy to find like in CP.
• When in Service composer and the SOURCE is P…[finish this]
• When hovering over objects: sometimes you get information about the object in a popup, sometimes not. Sometimes you can expand (firewall), sometimes you can’t (Security Composer, vCenter). Sometimes you click on it (services) sometimes you don’t. Like security groups you really want to know what the description is but you can’t see the info, you can’t click on it so you see a security group name “VERYSECURE_X009383″ and you have to try and figure out what they meant. You HAVE to have a great naming scheme where the name is long and descriptive THISISSECUREGROUPFORSNMPSERVERSINCHINA”.
• Long object names are chopped and even if you expand the column you can’t tell what they are. So if you are looking at rules and you want to find all the CSX_001_AD…………             different AD servers, you aren’t sure if the AD………. is a server or an ADsomething_Network or ADsomethingname. You just get the …… So you have to hover over and it tells you. But you have to look at each one one at a time. You can’t review in bulk. Write once firewall.
• From Service Composer you cannot dynamically create IPs, ranges, services, etc. These all have to be pre-created and THEN used in service composer. In Firewall you can dynamically create these. So to create a service, you first have to use Firewall and edit an old rule and create a new service…but not use it just create it. THEN go into Security composer and use the service. Lots of mouse clicks. In CP it is just 1 or 2 clicks.
• No drag and drop
• Security tags can be created but not edited in vSphere. Have to edit and manage in NSX manager.
• This may seem trivial but huge: Grouping is very basic. CP does a great job of allowing you to group by name, color, drag and drop, etc. DFW you must edit one object at a time. So imagine adding 100 objects to a group…good luck
• SyslogLogging is not stateful…you see every UDP packet. It is overwhelming and hard to search for what happened. Syslog on steroids.
• When review the rules in security composer, you can’t expand the all objects to understand what is inside the groups, services, etc. You have to open the objects separately. Takes at a minimum 5 clicks PER object you want to review.
• Security Composer: If you explicitly add a host into a group, the sum of these does not show up in the summary window. So the group looks empty but it actually has members. Would be a bummer in a rule review to toss it! Write once firewall.
• You can’t stage a policy and then push it during a change window. In firewall rule creation you can insert a new rule and get a “publish now” button…but that is almost like “OK” button. You can SAVE your current config and push later, but that config is for ALL the policies. you can’t stage individual VMs or groups of VMs. So if someone is in the middle of modifying rules in 1 of the 200 sections you will be pushing ALL rules include the 1/2 built ones. In service composer what you change takes effect immediately. if you change security tags rules change immediately (with no possibility to comment) So cross your fingers you changed things correctly, no review process. Write once firewall.
• Some times you create the rule and then have to ‘publish it’..in Security Composer. BUT if you just update a group with new IPs, then it gets automatically pushed out..no ‘publish it’. CHANGE CONTROL!
• If you change policy that has dynamic tags associated with it, you will have a fun time finding what policies are assigned to what VMs you are about to blow up. ‘Write once’. Small scale will work, large scale it is scary.
• Logging Tool – Log Insight : Somewhat cool but SLOOOOOOOOWWW and kludgey to use compared to SmartLog. a 24 hour rules in SmartLog takes seconds. In Log Insight might be 20+ minutes.
• No timeouts on rules. You have to comment the rule and then use rule reviews to delete. Write once firewall
• You can’t expand/collapse rules base, have to expand/collapse each section by hand.
• You can’t freely edit the section title names. They are auto generated for you. So you have to name your security policy with nice names and that is what is used to ID the section titles.  Makes it very hard to figure out where rules are located in the sections, or what the section does. Especially on large rule bases. Remember, the rules are all in 1 LOOOOONG list, you have have 100000 rules, so section titles and searching is critical to finding things. Write once firewall.
• You can’t free text dynamically search the rulebase. Have to fill in filter and then hit apply. Some of the choices are limited, like you can only search for 1 service at a time.
• You can’t search or sort IPsets/services by numbering, only names. You can’t even do a web page search for text, it is disabled. So if you are trying to see if anyone defined port 1234 you have to search through a list of 65535 service items (ok i am exaggerating, maybe 65534) and ranges. Anyways its a long list and will get longer. If a new admin wants to create port 1234 he will just add in Yet Another port 1234 with new name. In 10 years you will have 100 ports @ 1234 all with different names. Write once firewall.
• You can’t create a new host using using the DNS name to resolve to an IP address. You have to enter both separately. So this breaks the tie between DNS and IP address.  If IP addresses change, you can’t go in and click ‘resolvebyIP’. You have to type in new IP address. Write once firewall.
• No NextGen capabilities like allowing User->Server rules.
• You can adjust the placement of the rules in the rulebase,but you have to use this metric number ‘this rule has weight of 5000’ and hope you have all your other weights correct.
• In Security Composer you can have rules log, but in the rule list there is no logging (or comment) column so you can’t tell if you set the flag or not.
•  Because DFW rules are aggregated into one long list that applies to each VM, if you mistakenly insert a ANY ANY ANY ALLOW NOLOG(default) rule (we do this regularly in rule reveiw/cleanup or when building a new site and need to know what traffic is being passed by firewall), you will allow access to ALL your VMs…and good luck finding the problem. Well, it won’t be a problem because everything will work even better than before!
• Rules like:SecurityComposer:
  FROM: SecurityGroupA
  TO: ANYFirewall:
  From: 1.1.1.1
  TO:GeorgeGarageShopGet inserted into EVERY VM. Why? Because the ANY means that all VMs have to evaluate if the packet can enter/leave the VM both on the source VM and dest VM. The 2nd firewall rule does NOT have any objects that tie it to a specific VM so it has to be evaluated by all.SO: Be very careful how you craft these rules because you might think they impact 1 VM, but they may impact ALL VMs.

  ALSO: Bit of performance impact. All VMs will be evaluating this rule even though they may never hit on it.

• If a Security Group statically includes a IPSET, you can’t tell…the security group looks empty unless you open it up and looks at the details. Why do you care? If you are doing cleanup and you see an empty security group, delete it…OOOPS but it is not empty! You need to have a management interface that shows the relationships between objects so you don’t delete things that are in use. WHERE USED in CheckPoint. Another example of “Write Once” firewall.
• Security Composer: In the SRC or DST field one of them has to be a generic POLICYSECURITYGROUP (which you dynamically apply to a security group). You can’t have a rule “FROM: 1.1.1.1 TO: 2.2.2.2 ACTION: Allow” or a specific security group “FROM: HR_security_group TO: SALES_security_group ACTION: allow”. This has to be done with the FIREWALL rulebase. WHY: because negation of objects doesn’t work with the generica POLICYSECURITYGROUP. So you can’t build rules like: FROM: NOT POLICYSECURITYGROUP TO: Sales_security_group ACTION: Allow.
• Security Composer: Basically you can only work with groups and not individual IP addresses (well you can, but you have to bundle into security group). The group is your security zone and how it interacts with other security zones. So if you have mostly zone-to-zone type rules, then SC works. If you have zone to individual-ip OR zone to misc-group-of-random-IPs, then Security Composer is probably not your friend. you might want to create sections in the firewall rulebase. Unfortunately now you have 2 different rulesets for the same zone. So there is no win, hard to maintain. Don’t get me wrong I like the dynamic nature and Securty Composer but it forces you to architect the rules differently.
• When building groups of services, Service Groups, you only get to see the names and not the port numbers. So when you build a group and you want to include port 123, you have to know the name is AREYOUSTUPIDTHISISNTP_POOL_BLAHBLAH_LOSER. You can’t sort or search by port number or see if its UDP/TCP.
• FYI: when you upgrade NSX prepare yourselves for an outage no matter what they say
• Rule numbers that show up in log insight and syslog are GUID and  not the same sequential ones that show up in the GUI. You have to enable GUID in the rule GUI. The GUID seem to be random based on when you entered the rule, so you can’t tell if rule 1812 preceeds/follows 1811. Very difficult to map between the logs and the rule base. Let’s say you are trying to find when a rule is dropped. Rule GUID #1002 drops it. But rule #1893736 allows it. So does 1002 precede/follow 1893736??? Have fun looking through the WHOLE rulebase. Remember, the rule filter will always label rules with #1.So basically you have 5 rule ids. 1) Rule number in Firewall 2) Rule number in security composer 3) Rule number in manage VM 4) Rule GUID 5) Rule number when you filter rules. Why complain…5 for the price of 1!!!They need to have both rule IDs in log insight. Remember that Log Insight searches are painfully slow and its the only source for GUIDs (well, syslog but its ugly). So hunting down rules is just torture.Write Once Firewall.
• Rule GUID does not show up in VM summary of firewall rules. Write once firewall.
• Rule hit counts exist, but ours don’t work. Not sure who’sproblem.
• So when Log Insight decides to return information, you get the GUID and search your rulebase via a filter. Unfortunately you only get the single rule  and its labeled #1 (no matter what its real number is) and you can’t see the whole rule section so you can’t see the context of the rule with the rules above and below.
• The vsphere indexed search is fast, but you cannot search for everything like items in service composer. There are so many menus, you wind up looking through 100 tabs find and object called ‘SOMESTRANGENAMEYOUNEVERSAWBEFORE’  Needs to be extended.
• Log Insight is an IP based SEIM tool. It can’t use security tags or security groups as a filter. When you are doing analysis you want to ask:FROM SECURITYGROUP-A TO SECURITYGROUP-B
  FROM SECURITYTAG-A TO SECURITYTAG-BShow me all the flows.CheckPoint SmartLog is tightly integrated with its management DB. You can query for any objects in the mgt database. AND it is 100x faster.
• Can’t copy multiple rules and use for templates. Write once firewall.
• So you decide to just make changes to the rules and not publish them. Your buddy comes to work and gets a call that says XXX doesn’t work. Buddy looks at the rules and the rule exists and the traffic should flow. How can you tell what has been pushed to the firewalls and what is being staged? Good luck with that. Write once firewall.
• HOLD CHECKING ON THIS: Different NSX controllers have different policies. So if you have similar policies between them, good luck keeping them separated. Make sure you are editing the right one. Same problem with CP when you have multiple policies in a SmartCenter.
• No ‘disable’ rule in Firewall (verify, might only be security composer)
• We are having problems with disappearing vm objects. VMs get modified or removed…but they are still in firewall rules which prevents us from pushing policy. Write once firewall.
• Be aware that NSX and VCENTER are two products jammed into one interface. So the user/admin is still split between two interfaces in the GUI. So if you add a user, you have to add their permissions in 2 spots. Which doesn’t sound bad but because of the poor interface design you wind up clicking a lot.
• DR Danger!: Try taking a snapshot or a Netbackup/snapshot and restore the VM…Is something missing??? Perhaps the security tag? Geez what did label that tag as…hmmmmm.. Write once firewall.
• We added a new subnet. Our security groups trigger off of subnets. So a VM can be included into a security group based on its IP address. We have 100’s of security groups. Which of the 100’s of security groups did this new subnet impact? Have fun going thru them all and looking at each dynamic rule. Write once firewall.
• Security groups can include other security groups. No big deal. Easy to do. X includes Y. X is parent Y is child. What if you wanted to find all the groups that Y is the child? Pretty funny huh? Write once firewall.
• So I get a call “Dreez, what rules are applied to XXX0027 VM. Hey let me look that up…NOT”
  • #1 you can only find the security composer rules
  • #2 If you are searching for RULES that apply to a VM (SRC or DST field), and that VM is  is buried in a group within a group within a group….go fish…search does not work
  • Write once firewall. Putting in the rules is easier then looking them up.
• You can only open 15 ports in a single rule
• Firewall rule Searches are from 1980s. They are not indexed like smartlog or the rest of vcenter. You have to click 5 times to do 1 search. Like if you search for a VM id, you can’t enter it you have to let the GUI fill it in for you. Then it will return ALL rules where that VM lands, like the ANY rules, the encompassing subnet rules, NOT the rules with GROUPS in them….Basically most of the rules. Write once firewall.
• When you add a rule, the default entry is ANY ANY ANY allow. Hope someone does not forget to fill it in
• Let’s say you have a VM with 10 security tags. The tags allow it to belong to 10 security groups. The 10 security groups pull in 10 policies to be applied to that VM. Confused yet?So you get a ticket to add VMXXX to  talk to VMYYYY on port 999. Now this rulebase was created in the civil war and no one remembers what the application does. Which of the 10 rule sets do you want to add the new rule to? If you add this rule, what other set of VMs will you modify because they might also use the same policy?Point is that this approach to micro segmentation works on Day 1, but is difficult to modify on Day 10,000 because the policies are not tightly bounded to the VM. The policies are in a pool that can be used by ANY VM. In a way they are like CheckPoint Global Policies. When you modify it, you impact MANY firewall policies, not just one.So microsegmentation with tags and groups need guardrails to prevent geriatric degradation (I just made that up, sounds kind of official huh?).
• I was asked to find out why LDAP was dropping. Well LDAP rule was named MSAD so couldn’t search for the name or the number 389. Then it was buried in a group. Then it was buried in another group. I couldn’t expand like in CP, so I had to search through several rules expanding several groups one at a time. Then I also had to do this for the SRC and DST groups buried in security composer security groups. Can’t expand. One at a time. It took me about 4 hours, admitedly I had several balls in the air.
• You’ll love this one. This is the only true way of knowing which rules are on a VM.
  1. find host vm is running on
  2. turn on ssh on host
  3. log in
  4. summarize-dvfilter | less
  5. /vm-name
  6. find network interface
  7. copy network interface name
  8. vsipioctl get rules -f <nicname> | less
  9. have funincredidible….
     Write once firewall.

• Beware, I have seen 2 instances where DFW just ‘forgets’ to load policy…and the world is wide open. You won’t find out about it because there is no alert or red sirens. Logging mostly works,  applications are working, no customers are calling so what is the problem? Yeah very scary. Like adding a new logical network, but the firewall wasn’t protecting it. If you do a:

  vsipioctl getrules
  and it comes up empty…you have a problem.

• I just figure out something about many systems as well as DFW. Let’s say you have a rule that reads:From: 1.1.1.1   TO 2.2.2.2  PORT: 80and someone calls you to debug this and says “Do you have a rule that allows 1.1.1.1 to go to 2.2.2.2 on port 80??”. The rule is blatantly obvious. What if the rule is:From: SecurityGroupX To: SecurityGroupY PORT:ManyFunkyPortsThis is obviously more difficult to find. Why? Because:SecurityGroupX->HostGroupZ->ClientHostObject->1.1.1.1There are so many levels of indirection between SecurityGroupX and 1.1.1.1. Almost like a hierarchy of Groups, the final object is hard to find. In addition its dynamic do to the nature of security groups. So one second this relationship exists and the next second its gone because VM 1.1.1.1 got removed from the group for some reason.

  So although security groups and tags are great for building rulesets, debugging them is a bitch because the tools do not exist to allow you drill down into these rule objects. You have to manually expand them through various screens. Write once firewall.

  CheckPoint on the other hand does a great job of allowing you to expand and drill down.

• So you are working with your policy that is 1000000 rules large because for the whole environment there is only 1 rulebase. You try to push policy, but get errors “VMXXX is unknown object”. WTF? Turns out the VMXXX was deleted by someone else. So somewhere buried in your policy are rules with VMXXX. Have fun finding them and deleting them before you are allowed to push policy. The bad thing is that it is not isolated to a single VM, or policy. Deleting a single VM effects the whole platform. Image Amazon AWS. You delete 1 VM and you can’t push policy at all. Write once firewall.
• So a security advisory comes out to not use Shared Global Addrset. No problem, all firewalls get security advisories. Let’s research this a little:
  
  Oh sorry, we didn’t document that. Anyways I’m sure it will eventually show up, but the documentation is pretty poor. If you want to see great documentation check out AWS, wow.

NSX DFW…definitely enterprise OUT,

dreez

 

Redirecting NSX firewall syslogs into SmartLog

 

So we know that NSX DFW is a cool toy, but its logs are invaluable for debugging and forensics. Wouldn’t it be cool if would could see DFW logs along with SmartLogs???So our need is to have single-pane of glass security; Enter vSec and SmartLog.

After 100 VMs we knew that using DFW would just not work. In addition the logging was painful syslog based. So we decided to use the best management and logging tools on the market….and it just so happens this is one of the things CP does extremely good. Management and logging. Today’s talk is on “Having Fun with Logging”.

So we needed to get the DFW syslogs into SmartLog. How to do that. Well, first thing you do is pour salt on the documentation.  Its OK, but only gives you the basics. I have decoded this documentation and will show you how to send DFW logs to SmartLog and make it go a lot faster. By default, you can just turn it on with factory defaults… but will overwhelm your SmartLog server and all the data will be in the ‘info’ field and not parsed out. So it took me a couple months to figure out how to do all the parsing efficiently so as not to bury our log server.

And now for the rest of the story…..

NSX DFW LOGS

NSX is the part of vMWare vSpere that runs its SDN…including the firewalls..Distributed Fire Walls (DFW). These firewalls send their syslog to their eSXI hosts which we forward to a custom syslog-ng server on a single VM.  We customized this syslog-ng server to parse DFW logs into CSV format….no easy task but doable. Why????

Once in CSV, syslog-ng forwards the logs to ‘syslog’ on currently a UTM 4800 with 4 cores. This is a custom syslog built by CP to accept syslog and convert to CP native log format. ‘syslog’ forwards the converted logs to fwd which enables SmartLog and Tracker to read the logs.

Simple enough…….ok, then read no further.

syslog-ng config

Syslog-ng allows you to parse through logs and reformat them. You can do some simple parsing with the basic configuration file, but the fun starts with more complex patterns with the pattern matching databases. Basically you need to convert this:

into this:

Now all you people are tons smarter than this old balding has-been…so I am not going to explain syslog-ng internals to you all. You will have to read the documentation. But I will give you the overview of what I did with it.

syslog-ng has two parts

• General filtering config – Most people use this just for simple filtering of logs
  and re-formatting of logs and redirecting logs to host files, DB’s, other log servers.
• Pattern Database – More complex parsing where you can take pieces of the input line, parse out specific items with regex, and insert those regex patterns into variables/macros to be used later in the General Filtering Config.

General Filtering Config: First thing you need to do only filter on DFW logs and nothing else.  NSX puts labels on these logs and you will have to look in the raw log file for it.

Second thing you have to do is get rid of the TERM log entries (terminate connections for TCP, for UDP you’ll see zillions of these going to 5355 some sort of local subnet DNS resolution. Unfortunately in cloud world we have huge subnets so all the VMs are doing this local link DNS resolution). 75% of the DFW logs are TERM entries so can be ignored.

Pattern Database: Next part is a bit harder. You have to build a pattern database that will parse raw DFW log entries and match each field up to a named macro. For example below is a snippet of a DFW log entry matching pattern. You can see the ‘action’, ‘domain’, ‘protocol’ fields are 3 of many fields that would match a DFW log entry from above.

This can be a delicate task, but there is a tool to help debug … ‘pdbtool match ‘. You can type in a sample line and see if the your pattern database will match it:

General Filtering Config: Next, DFW labels ICMP protocol as ‘PROTO’. So I just translate into normal geek lingo ‘ICMP’.

General Filtering Config:Next is the fun part. We finally get to output CSV to the CP syslog server. Notice how the MACRO names like ‘size’ and ‘protocol’ are used to fill in the CSV?

General Filtering Config: And here is the ‘main’ section that drives all the above phases:

so restart you syslog-ng server:

/etc/init.d/syslog-ng restart

and off you go!

At this point you can ‘tcpdump -X -n s0 port 514’ to verify that in fact the logs are formatted correctly and heading over to CP land.

SYSLOG Testing

Just in the off case my code sucks, here is how you test it. There are syslog generators out there that you can use: I used this one and here you can see how I generated traffic:

One key thing is the “dfwpktlogs–” is what triggers the regex’s to fire in the next section. Do NOT use ‘:’, you have to use ‘–‘ or for some reason the regex won’t parse correctly. More below…

 

CP SYSLOG MDS Config

So now the logs are flowing to port 514 on SmartCenter Log server. So you have to make sure you enable the syslog process (NOT syslogd) to listen.

 

This will start the syslog process on port 514 on SmartCenter.

On a MLS, you have to restart the whole MLS for the config to kick in. On MLS, a syslog daemon will be started PER domain log server IP address for every domain SYSLOG is configured to accept SYSLOGs for that domain.

and syslogng logs will flow magically into Tracker. Actually they will magically flow, but only fill in the INFO field and will not parse …. yet.

Oh yeah, when debugging you can get the syslog process to re-read the syslog config with:

1. mdsenv <DOMAIN>  # MDS only
2. fw kill fwd; fwd -n &

CP SYSLOG Parsing

OK, now it gets fun.

So there is this CP tool: Eventia Log Parser that looks pretty cool. You feed it your SYSLOGS and magic parsing configuration pops out the other end:

 

Yeaaaaaaah…NO. Doesn’t work and the config you output is so complex the SYSLOG engine will run at 100%. Both ELP and SYSLOG were written about the time after the Civil War and are on version 1.0…Typical CP V1 code so don’t get your hopes up. Let me know if you have better luck, I spent days on this.

So I decided to generate the syslog parsing config myself. I saw what ELP attempted and then came up with my own.

Remember that SYSLOG-NG is sending CSV formatted logs:

and CP SYSLOG is taking them in and turning them into SmartLog readable:

So let’s begin.

The debug process is this:

1. mdsenv <DOMAIN> if on MDS
2. Edit the syslog rules with your stuff
3. ‘fw kill fwd; fwd -n &’ restarts the syslog daemon
4. Use the syslog generator
   1. Forget all this parsing junk, just get the SYSLOG-NG to dump into the CP SYSLOG and see the results in TRACKER in the ‘info’ field
   2. Try and get the REGEX rule to fire on ‘dfwpktlogs–‘
   3. Try and get field #1 to parse and fill in some random text field for debugging
   4. Try and get field #1 to output into the official SmartTracker field
   5. Go to #2 for field #2/3/4/5….

So far I have given you enough information to do #4.1. Let’s work on #4.3.

This may look simple…but it took me days and days to fine tune this. The ELP generated config was pages and pages of REGEX expressions. I boiled down to 1 line:

Even if you aren’t a regex geek, you can see that I am parsing the CSV file into 9 fields.

REGEX GEEK OUT (don’t read this if you aren’t a regex geek):

FYI: They don’t implement FULL regex matching! Example: You can only have 9 matching field patterns, I couldn’t get it to recognize ‘:’, you can’t use lazy searches ‘.+?’. It is some limited hack that you will never figure out because there is no documentation other than examples.  AND their regex performance is horrible, so I tried to avoid using ‘.*’ because it is greedy and will scan the whole line and then backtrack. Instead I used the ‘[^,]’ which searches for all characters NOT ‘,’, which is the same as ‘(.*),’…but doesn’t have the backtracking.

 

============= END GEEK OUT===============================

So what is important is the ‘dfwpktlogs–‘.  When the regex sees this pattern, it will fill in the 9 columns of information from the CSV formatted input line. Now there are 3 different types of actions that will be taken depending on the results of a REGEX match:

Rules:

• NO MATCH on regex: Result->log entry with data in INFO field
• MATCH: but no log entry, you matched the  ‘dfwpktlogs–‘ but trying to print out a number into Tracker or Data type error, wrong field name, syntax error on field names, etc.
• MATCH and () field hits: can use index_value(1) to fill out fields (coming next)

OK so next you will try and parse field #1 which is the first ‘([^,]+)’. This is the PASS|DROP field from the CSV. What I did is I first captured the field and then dumped it into a random text field. This told me that 1) I captured it correctly 2) I am able to write to SmartTracker.

Here is the add_field that does this. add_field adds the field to tracker. Field_name is the Tracker field name..in this case it is the ‘rule_name field” field_index is CSV field 1 (from 1-9). Field type is the type of the Tracker field. But beware, I didn’t always get this right and the log entries would just dissappear so I had to experiment. I also looked into:

$FWDIR/conf/syslog/CPdefined/*.C files for examples to see what their field types are.

You can also look into

to see the Tracker field names and types. Its a bit kludgey but between CPDefined examples and this file you’ll get close.

Here you can see ‘rule_name” is the name of the Tracker field defined in svt_fields.C.

 

So now you have 1) You captured field #1 and 2) You wrote it to ‘rule_name’ field. You can now verify that you captured the field you intended to. Once you verify this, you can then write it to the REAL field with:

Here we are writing the PASS|DROP to the ‘action’ field in tracker:

HOLD ON DREEZ: How in the heck did PASS|DROP get converted to Accept|Drop???? you ask.

Grasshopper, I introduce the dictionary file…

This is a second file in the same directory that will do transforms for you. Here you can see “PASS” being converted to ‘accept’.

DREEZ you ask: How do I know what to convert to what?

Oh yes grasshopper. You read the CP documentation…NOT!!! HAHAHAHAHAH. Dreez makes a big funny. HAHAHAHAH. Remember grasshopper, this documentation was written in the Civil War and even now the Lord Developers rarely talk to their lowly documentation peasants slaving in the fields trying to identify nuggets of information to feed their ignorant customers. (If you want real documentation see AWS documentation, you can see where developers talk to the documentation team). That’s what phone support is for —- DUH!!!!

Oh yes, I easily diverge, forgive me.

No grasshopper you do what us old people have been doing for centuries. You look at examples in

$FWDIR/conf/syslog/CPdefined/*.C

OR:

On your log server: fw log

Will print out your logs with the field names in them.

To get a feel of what the valid field types are.

So now that you know how to do field #1, now you go through all the fields one at a time….or……

You can just use my template as a starter!!!

• syslog-ng config to convert DFW to CSV
• CP syslog config that takes CSV and inserts into CP Logging

 

PERFORMANCE

So the first time we did this, we used CP’s syslog ELP generated config on one of our big servers and the server went to 100%++++. Unfortunately the ‘syslog’ process is single threaded so it had no where else to go.

So with 1100 VM’s….each with its own firewall…sending syslog to my config described above to a 4800 (slow) based management station…performance was much better but not ideal. The 4800 has a quad core Q9400 on it. Syslog process is a bit busy but at least not 100%. It will average between 11% and 60%, and burst to 90% now and then.

Now remember that on MLM, each domain/cma/clm has its own ‘syslog’, so you can manually distribute the load amongst multiple logging domains. But I would hate to do domain design based on log loads. For example, at one company 2 of 12 domains had 90% of our logging. So should we split them up further because of logging? Not sure.

Your mileage may very….

 

SUMMARY

Everyone is excited about this because SmartLog is what our org (and all the other orgs I’ve been at) live and die by. Centralized single-pane of glass easy to use and fairly fast security monitoring. It is the gem of all of CP’s products. And it mostly sometimes works!!!

Now we can send logs from other tools like other firewalls, URL filtering, FireEye, etc into this and use SmartLog to get quick answers. I am keeping both flat syslog files as well as sending them to mongodb and SmartLog. YES: SEIM tools exist…but either they don’t work, too expensive or at capacity, etc. ‘grep’ is free. ‘mongodb’ is free, etc.

 

 

LOGOUT!

dreez

 

 

The World is Changing and so am I

The Cloud. Its Coming. Torpedo is in the tube and you are a WWII lumbering oil tanker. You will be automated like a $15 minimum wage McDonalds burger flipper.

I switched gigs and and am now working a cloud security gig. My Check Point role is diminished 50%…but worse I no longer have access to my #1 diamond support guy Taylor who fed me all the good juicy techie fixes that I shared with you all.

SOOOOO, not sure what the future blog will look like at this point, but the CP focused techie posts will be reduced and you’ll see more Cloud security chatter. Worst case you still have the archives.