How to make a Cisco geek cry

 

I work with this awesome Cisco geek who manages ASAs and MARS. Today I walked him through generating a report for his people with SmartLog.

I thought he was going to cry. This normally big tall handsome ex-US-Marine generally stoic person started making squeeky bubbling noises and then just started laughing exclaiming “THAT….IS……AWESOME!!!!!!, It would take me hours or days in MARS and its so beautiful!!!!”

Which brings me back to my rant…..what the hell is Software Defined Protection? CheckPoint will change the world with their awesome new R80 management station and Dudi’s SmartLog that makes Cisco geeks cry. Everyone and their mother has Software Defined Protection…but how many have a “Single Pane Of Glass Security Management”???

Only CheckPoint – god bless their souls.

Dudi…you made a grown man cry. Love you dude,

dreez

PS: PEOPLE: Just make sure ya’ all QA it this time.

How NOT to choose a firewall

{WORK IN PROGRESS}

Over the 20+ years (ugh) I’ve been doing firewalls, I think I’ve seen them all. So I think I’ve answered the question “What firewall should we use??” a hundred times. For the last 5 years I’ve been back supporting Check Point and MDS and being on the front line of deploying and operational support of enterprise environments, I think I now know the answer for how NOT to choose a firewall. It all goes back to my days developing Sidewinder back in the dark ages.

In those days us developers so were so geeked out on developing the most secure firewall that all the other important operational requirements were an afterthought… manageability, monitoring, DR, availability, scalability, etc. Only now do I appreciate how poorly we designed from the bottom up totally blowing off the enterprise.

I can still see this design in a lot of existing firewalls. Cisco ASA in particular has to be the laughing stock of the firewall market. I’m not sure why they just don’t hang it up. The GUI is like Notepad++ for a command line driven router with ACLs. Juniper’s WebGUI must have been designed by the same people that designed CheckPoint’s User Center – high school interns writing their first Java script. I could go on, but all of these have one thing in common. They are all designed from the bottom up. Hardware->Command Line->Crappy GUI-> Even crappier Enterprise Manager for multiple firewalls.  Each layer worse then the previous.

These are the red herrings you should avoid when reading the marketing fluff

1. Security –
   You know what…these days security is the same across all firewalls. Ports, IPs, applications, identities, etc. Firewall X may not have the feature now but it will shortly 
2. Performance/ASIC support –
   In large enterprises there might be 4 or 5 ingress points that really need high performance firewalls.  If one were to spend some time tuning them they probably could support the load…or just use load balancing…or architect the flows a little different.
3. Fancy GUI – GUIs are always pretty but do they support 1000’s of rules. How does one search, delete, add, manage huge rulesets with huge object sets. Day 1 looks like the marketing slide but day 1000 after 10 admins have cycled through what do you have? A ruleset and object base that are too scary to touch so you can’t remove anything and you just keep adding.
4. GUI performance – Day 1 it looks like a marketing slide, but after the 50’th rule it hangs and blows up and corrupts the database
5. Hardware vs software; Who cares? As long as you can setup a test lab in VMware. Biggest problem is for software firewalls is making sure they have a base hardware configuration so you don’t have to play pick-the-network-card-of-the-week game because the device drivers are not in the HCL.

These are the true factors you should consider when evaluating firewalls. Why? Because firewalls are the chockstones of our network. In the time of war the president asks 4 words “Where are the carriers” Same with firewalls. If anything goes wrong in the environment the CEO screams “What are the firewalls doing? (i.e. its a firewall problem). Firewalls are in a unique position in the network. They can not only police the flow of traffic at ALL levels of the network stack, but they can also be a huge network debugging tool. When something blows up, the first thing everyone looks to for information is the firewalls.

1. Centrally manage enterprise environments:
   1. Object sets: Large object sets with scoping rules, coloring, groupings, access controls, search capability, ability to add/modify/delete at all levels. Imagine tracking millions of objects and rules. How do you organize all of them? You have to be able to group, label, sort, search.
   2. Rule sets: large rule sets with scoping rules and generous comment sections, indents, colors,access rules, search capability.
   3. Admins: Large number of admins with various security controls. All from centralized management database like RADIUS. Ability to have multiple admins administrate concurrently read/write.
   4. Provisioning: Ability to set date, passwords, routes, interfaces, debugging commands, scripts across multiple firewalls with 1 command.
   5. Revision controls: ability to store revisions of policy and objects so you can role back/forward in case of problems
2. Monitoring and Logging
   1. Monitoring: Monitor status of whole environment from high level and ability to drill down to detail such as interface performance
   2. Logging: Logging of entire environment and ability to drill down into single log entry
   3. SNMP Integration: Easy integration of product specific parameters into SNMP environment for real-time and historical monitoring of whole environment
   4. Detailed logs at the kernel level, not just network traffic. These are dire in case you have to debug on your own in case support does not answer the phone
3. Staging: Can you stage rules before you deploy them. In the stage environment can you perform some sort of QA workflow before rules are deployed. Ability for separation of duties for admins to create, approve and deploy rulesets.
4. Cluster/Failover: Clustering should be brainless and flawless. Members should never fight for active-active control. You shouldn’t have to understand the minutia or issue 1000 commands to fail the members over. Clustering should work transparently with dynamic routing on failovers so that the firewalls and supporting routes all fail at the same time. Clustering should also work with upgrades so that you can upgrade members and never have downtime as you fail over between the upgraded members.
5. Test Labs: Can you create VM test labs of your environment to test deployment scenarios. Why VM? Because you can quickly snapshot and roleback. In hardware labs you never know what environment you are testing in, you can’t role back.  I can’t tell you how many close calls I have filtered out in test environments that would of been a disaster in a production environment.
6. Command Line Debugging: GUI debugging is so limited because when things go wrong, you never know if the GUI is broken or you are looking at the real problem. I like debugging in Unix where you have a rich set of tools and can debug inside the kernel.
7. Easy to learn GUI: Assume that your company is cheap and will go through admins like toilet paper. When one admin quits, the next newbie will have to pick up the slack (of course we can’t send them to training!) and try and figure out where the last person left off. If the GUI is so complicated you will never be able remove anything, it will only grow more complicated bowl of spaghetti.
8. Network level Packet filtering preferably CLI. Aside from Wireshark, I have not seen a good GUI packet sniffer on a firewall. TCPDUMP is mandatory for any firewall environment and it has to be able to feed into wireshark
9. Virtual Environments: Supports virtual firewalls to for easy deployment
10. Upgrades and cloning: Ability to easily upgrade clusters and/or clone a firewall with minimal downtime
11. Support: This is the tough one.  Monitor blogs or user groups to see what they say about their support environment. May want to stage a test and see if you can call support and see how long it takes to answer a question. How extensive is there knowledge base and forums? Is there a knowledge leader out there you can email and ask their opinion?
12. Licensing: Easy to support licensing model. If complex, this will not scale in an enterprise environment and you will never figure out how many licenses you have and what they do.
13. Backup/Restore: Yes every product can be backed up, but how hard is it to restore?  Cisco probably has the best restore capability here.
14. Provisioning: What are you going to do if you need to set a new password on 1200 firewalls? Can’t do them one at a time. Need some sort of interface to execute local commands once on 1200 firewalls
15. Remote upgrades: So you are upgrading that firewall in Botswana and you have no local console access and the local SalesGuy/IT Guruwho speaks Zamini had a bit too much fire water that morning and it all blows up? Sure hope the firewall can revert back to its previous image automatically or else you are going to be on a plane.
16. Fail open: OK, the security geeks just crapped in their pants. But there should be an option to fail open. How many millions of dollars have been lost because of cluster failure, memory leaks, etc. Security geeks have usually never had to endure the pain of throwing away millions of dollars of product because  a production run was spoiled when an cluster failed. Some firewalls are not the front line of the battle. Many firewalls are nothing more then shopping mall cops that are there to keep general order. But when hell breaks loose they should just alert admins and get the hell out of the way and fail open.
17. Market share: So you have firewall Product X, and your admin quits. Product X has 1% of the market so you can’t find an admin with Product X experience. Of course you have no training budget, so you are forced to find a needle in the haystack and hope it doesn’t cost you $`150K/year and tons of downtime as the new admin tries to figure out Product X on your dime.

Hopefully this will save some of you from the firewall marketing machine.

dreezman