How Check Point and Tufin could win at SDN

Software Defined Networking (SDN) is a game changer and will make most of you reading this blog expensive unemployable boat anchors. Better off learning Cobol. What is it? You can try and read all the blah blah Cisco articles, but they use big words that mean nothing and will leave you with more questions than answers. I took a 1 day Cisco ACI class put on by Dain Deutschman and World Wide Technology, and he did a great job boiling down the good, bad and ugly.

Basically ACI and SDN has this dream that a CIO will have a menu of items:

1. New service offering for health care product
2. Need a forest of web servers
3. Need some backend databases
4. Need some admin stations
5. Need Internet access
6. Need this to have access to SAP
7. Need to comply with HIPAA

push a button and a Python script will spit it all out with no expensive IT whiners telling him/her what a cluster fudge it will be. Cisco ACI is the networking portion of the environment that replace the  expensive IT whiners that now have to touch every switch, router, etc to configure. ACI will put that router geek in the same unemployment line as a PHD Art Majors. It spins up routers, firewalls, switches, rulebases, etc. It can interact with VMWare, by spinning up a special virtual switch in the VM environment that links into the Cisco L2 environment.

So just like VMware, it will be easy to spin up systems and configure them. The cost of deployment will drop like a rock, just as it has in VMware.

And now for the dark side……..

Seeing I am an expensive whiner I’m here to say “Not happening any time soon – but brushing up my resume”. SDN is going to happen, is happening but its going to be a cluster fudge….and CheckPoint has a great shot of being a winner at the game.

Downsides:

1. It will scale exponentially like VMware replacing physical servers
2. Naming schemes will be all over the place because scripts will generate names
3. Lifecycle manageability will be just like a rulebase, rules go in but never come out
4. Debugging will be a nightmare. ACI depends upon tunnels in tunnels. Have you ever tried debugging GRE? Have fun with that.
5. ACI specifically has way too many moving parts, when something goes wrong finding the culprit that was created by a script will be crazy
6. Remember, not only are they integrating networking/servers, you are also spinning up rulebases and firewalls all with one script. Imaging dynamic rulebases. We can’t even debug the ones we have now!
7. Licensing..if you think its bad now…..both technically and asset management will be really bad. You will be buying crap that you already paid for because you’ve been through 10 admins/purchasing people and the new ones have no clue what is going on.
8. Right now ACI only uses IPs and ports. No NexGen. NexGen firewalls will bog the whole thing down and make debugging even worse.
9. Cisco’s ACI management environment is…….a Cisco management environment……a toy. If and when ACI/SDN takes off the scalability will be huge because now scripts will do what it now takes expensive IT whiners. Because the cost of deployment will drop, CIOs will go crazy deploying new environments. So just like a firewall rule base, the environment will explode….and no one will clean up the mess as admins leave. Who would risk taking down an entire network environment that hasn’t been touched for 5 years and has no owner and weird naming schemes and random traffic flows?

So in the end technology on top of technology on top of technology on top of technology with the goal of replacing whining IT boat anchors….will create a new breed of super expensive whining IT boat anchors. These people will be even more critical to the org, because their skillset will control the whole environment, not just a router.

Here are some tips to keeping that paycheck  coming to pay for all your toys:

1. Get VMware or Cisco ACI on your resume or take classes
2. Learn Python, PHP, Pearl like the back of your hand (Software vs. Hardware Config via scripting)
3. Learn SQL like the front of your hand
4. Look for gigs at enterprise shops that are going to virtual data centers
5. If you haven’t boned up on VLANs and routing better start now
6. If you are a router geek, start learning current and Nex Gen firewalls

With that said, there is one more void in the marketplace…the management environment. So here comes the Check Point /Tufin rah-rah. On the off chance that R80 works, Check Point has been the only management environment that I’ve seen that has the potential to manage this crazy new wold. Check Point has always had a great mindset for ‘single pane of glass enterprise management’ that scales. I think Cisco should buy Check Point and Tufin and have them go crazy with R80 to include ACI. Right now R80+++ will be able to integrate with ACI at the fringes with a REST API, but that is child’s play. Go big, all in I say.

Oh yeah, make sure Cisco fires all the people that do licensing. Geez louise I hate licensing.

Aside: My VMware security geek friends say VMware is in similar boat. Their firewall is like IPchains, just sad and won’t scale. Not sure about management environment. CP could play here, but right now just doing REST API which is child’s play. Gotta get inside. I don’t know enough at this point to say more.

Off to CPX.

Peace out,

dreez

PS Special thanks to Cisco/Palo Alto super smart young good looking guy Jacob Durocher who spent hours with me trying to figure out what ACI is and what the future will look like.

Cluster sync cable tricks

Got this from Watcha again.

A sync cable is only used to exchange state information. If you pull it, then the state tables will be out of sync, but the cluster will remain otherwise healthy and be able to failover. So YES – protocols like FTP will hang, but most other resilient protocols like DNS/HTTP will continue to work with no blips.

In a cluster , CCP  udp port 8116 packets are exchanged on all interfaces. CCP are just keep alive statuses and do NOT contain state information. If a member notices that it is not seeing CCP packets from its peer on one interface, the current standby will go to DOWN and the active member will remain active and go to Active Attention. However, failover will still work regardless if a sync cable exists or not. The downside is the state table may be hosed, but the members will failover as always (assuming that both members are otherwise healthy).

Never fully tested the above but makes sense….TBD.

 

 

AV and 100% CPU

We turned on AV and CPU’s went to 100%. Some digging…

1. AV only filters files on HTTP and SMTP. Curious…what happened to FTP, NTFS, SMB, CIFS, etc?
2. AV will look for viruses in HTTP on ALL ports by default.  This search bypasses SecureXL and goes into the medium path where all packets are unwrapped by the worker processes and not just quickly forwarded by the fast path interrupt handlers.
   
3. We then tried to write exceptions and white lists to filter out some of the crap. No luck, they are broken 
4. We were getting random reports of PDF’s getting corrupted. Not possible, we are in detect mode. Unfortunately someone came up with a tcpdump proving the firewalls in AV detect mode are corrupting traffic. Back to the drawing board……

I”m a die hard CP fan because of their management and logging, but when stuff like this hits my desk I just shake my head.

 

Performance Made Easy – Turn off State Sync

Wow, this is an obvious one I got from a CP performance guy Watcha.

DNS/HTTP/icmp…not sure what others have redundancy built into the protocol. If a packet does not go through layer 7 application layer will resend. DNS/HTTP/icmp are probably 75% of most your traffic so they are filling the state tables. WHY???

If a cluster fails over and the state tables are all hosed up…DNS/HTTP/icmp may hesitate for 1 second but will retry at the layer  7 level. So WHY??? keep them in the state table. WHY make the members share HTTP information over the SYNC cable. Get rid of all that crap.

Turn this off for protocols that are redundant at the application 7 layer. SNMP, NTP, etc…but they are probably only a fragment of your traffic. FTP you want to keep enabled and probably database connections (NOT SURE).

Looks like the Chicken is sleeping with the Fox

With Palo Alto eating everyone’s lunch looks like the fox and chicken are checking each other out. Desperate times, desperate measures. No atheists in a foxhole. Keep your friends close, keep your enemies closer…

http://blogs.cisco.com/datacenter/aci-checkpoint

My bets are with CP and R80….assuming they get it right. Future blog.

 

Fun with IA Portal Sharing and Spoofing….But not too much fun

There is this cool feature you can use to:

1. Centralize your portals
2. Limit the number of portals you have to configure and support
3. Users only have to remember 1 portal instead of 100
4. Centralize the portal, but if it breaks you can still authenticate at the local portal

So it looks something like this:

• User authenticates at central Captive Portal
• User can then have access to ANY of the sites whose firewalls share that user database on fw-1:

 

How is this all done?? Well, you first have to allow the fw-1 system to share identities with the remote sites:

And on the remote sites you have to point to the fw-1: corporate identity awareness portal:

Waaaalaaaa done. Works. Ship it. Finished. Chapter Over. Go watch TV and a beer and pizza and chill. OK, seems simple doesn’t it?  Hahahahahaha. Pretty funny.

Let’s look what happened underneath the covers.

PEP and PDP setup connections with each other.

 

 

 

PEP and PDP setup connections with each other. PEP needs to share the user table that PDP is holding on the fw-1: Corporate IA Portal

These are the ports that are being used to do that. You can see there are the main ports and then some random ports it spawns off.

 

 

 

So now comes the fun part. Not sure how your network setup is, but I saw connections come from all interfaces to all interfaces. I didn’t debug but I think it depends on which interface Captive Portal is running. Anyways the result was a ton of spoofing drops. Took me a while to figure out what was going on.

In addition, if you have firewalls between the fw-1 and the remotes, they you have to open these ports so they can talk IA.

So to fix this whole mess, I had to put the firewalls into spoof groups on all interfaces. I also set the spoof to DETECT for a bit until I knew it all worked and then turned it back to PREVENT.

Install policy on both fw1 Corporate IA Portal and fw2 remote.

After I did this CP actually has been working awesome so far. Remember this is with multiple AUs in a single AD domain and a single MDS domain – which is what CP strongly warns you against.