VSX Troubleshooting - CPUG

(c) Valeri Loukine 2010CPUG 2010 Chur Switzerland

VSX Troubleshooting

Q u i c k g u i d e


Agenda

How VSX is built (in brief)

Management scheme

Gateway architecture

Licensing

Issues to fix

Tools and methods

2


Reference Note

Pictures from Check Point publicly available documents are used in this presentation

Information from Check Point troubleshooting documentation used in this presentation

3


Management Side

V S X A r c h i t e c t u r e


MGMT model of VSX

Three tear infrastructure

Two types

SmartCenter

Provider-1

5


SmartCenter Model

Nothing special

6


Provider-1 Model

Virtual Systems are managed by different CMAs

Special so called “Main CMA” to manage VSX cluster objects

“Target CMAs” to manage particular VSs

7


Provider-1 Model

8


MGMT DB objects

Two types instead of one as for regular FW

network_object - security aspects of a Virtual Device

vs_slot_objects - networking aspects of a Virtual Device

network_object - on the Target CMA

vs_slot objects - on Main CMA

9


vs_slot objects

Special DB table called vs_slot_objects

Network interfaces of VS

Routes of VS

Other VS specific attributes, such as reference to hosting VSX object

Vital info for creation and change of VS

10


Network Configuration Scripts

Configuration changes are pushed to VSX from MGMT with NCS

NCS are generated on MGMT

On VSX GW NCS are parsed and executed

11


Types of NCS

local.vs - NCS file - last configuration change

local.vsall - NCS file, full configuration, executed on startup

local.vskeep - contains list of existing VSIDs, used at system startup

12


local.vs

Interfaces lists of each vs_slot: interfaces - interfaces’ configuration to be interfaces_installed - existing interfaces configuration

Each vs_slot object has 2 attributes containing routes lists: routes - routing table to be routes_installed existing routing table

local.vs file is created by comparing and calculating the difference of “interfaces” to “interfaces_installed” and “routes” to “routes_installed”

13


local.vsall and local.vskeep

Each Virtual Device has 2 NCS files kept on the management: VS_name.vsnew - NCS file containing interfaces VS_name.vsrt - NCS file containing routes

These files are updated each time configuration change is done using SmartDashboard.

local.vsall is created by concatenating all the files of all the Virtual Devices.

local.vskeep is created by going over vs_slo_objects table and writing all the Virtual Devices VSIDs to it.

14


Location of NCS files

All .vsnew and .vsrt files - in $FWDIR/conf/vs_repository/VSX_NAME of Main CMA

MGMT: local.vs, local.vsall and local.vskeep files - in $FWDIR/state/VSX_NAME/VSX/

VSX GW: local.vs, local.vsall and local.vskeep files in $FWDIR/state/__tmp/VSX/ If scripts processed successfully, they copied to to $FWDIR/state/local/VSX/ directory

15


Provider-1 forwarding

16


VSX private network

“Funny IPs”

For internal communication

Default cluster private network: 192.168.196.0/22

Cluster Private Network can be changed: - In SmartDashboard, if there is no VDon the VSX - By using “vsx_util change_private_net”

17


gateway Side V S X A r c h i t e c t u r e


Important note

VS is NOT a virtual machine

Common file space

Common kernel

VRFs and kernel contexts are different

19


VRFs

A Multiple routing domain (VRF) has separate:

VRF IDInterfacesUnicast routing tableRouting cacheMulticast forwarding cacheARP tableLoopback interfaceSockets

VRFs enable overlapping IP addresses.

20


VRF: CLI changesMind context when working with ip route, iputils, ping, traceroute, arp, route, ifconfig and netstat:

- traceroute –Z vrfid - ip route vrf vrfid - “-z vrfid” for the rest (arp –z 1, netstat –z 2 -rn)

Use “all” instead of “vrfid” to show information for all VRFs

VRF context can be changed by “vsx set <vsid>” or “vrfctl –s <vrf>” commands.

VRF 0 - physical machine

21


VSX user mode

Processes:

Multi context: fwd, cpd, cplogd and fibmgr

Single context: vpnd and gated

System resources like CPU and HDD space are shared

22


File Structure

VS folders - CTX under $FWDIR / $CPDIR

CTX00xxx - Virtual Device ID xxx.

VSX machine (VSID 0) - $FWDIR / $CPDIR

23


Creating VS object 1

License validation

Update context database with Virtual Device information $CPDIR/conf/ctxdb.C)

Create Virtual Device registry entries (OTP for SIC certificate)

Create Virtual Device directories and soft-links:

$CPDIR/CTX/CTX00xxx/conf

$FWDIR/CTX/CTX00xxx/log, database, …

24


Creating VS object 2

Create initial policy

Create the OS VRF instance

Create the VS instance in the FW kernel and load security policy

Send a message that notifies cpd and fwd that a new context was added. cpd adds the new context to its db.

25


Troubleshooting techniques


Useful knowledge

Management debugging (ref. P-1 lecture)

Gateway architecture and troubleshooting techniques

ClusterXL

SecureXL

27


Things to Check First

Licensing on both MGMT and GW sides

Connectivity between VSX and MGMT

All the jazz:

- local time settings - static routes - IP addressing - mind funny IPs- etc...

28


Management Debugging


Management Issues

Provisioning

Changes

vsx_util operations

policy installation

30


Important

Do not lock Main CMA while working with VSX on Target CMAs

31


Debbuging fwm

TDERROR_ALL_ALL - might be too much

vsx provisioning and vsx_util: TDERROR_ALL_VSXM

Policy installation: TDERROR_ALL_INSTMGR

32


How to set debug flags

Mind context!!!

fw debug fwm on TDERROR_ALL_VSXM=INFO

Or export TDERROR_ALL_VSXM=INFOand restart fwm process

33


Debug output

$FWDIR/log/fwm.elg

34


Turning it off

fw debug fwm TDERROR_ALL_VSXM=0

fw debug fwm off

35


Which CMA?

Most of the cases - Main CMA

Policy installation - Target CMA

36


Gateway Debugging


Common Issues

Connectivity

Policy

Interfaces

Clustering

38


To check first

Connectivity

Topology of VSX cluster and adjacent networks

Local times

Licenses

39


Tools

tcpdump -i <IF name>

fw monitor [–v <vsid>] -e <Your filter> Example: fw monitor –v 2 –e “port(80) and ip_p=17, accept;”

Note: changing context does NOT help “fw monitor” to limit output

41


Acceleration

Most of fw monitor output is accelerated, so you will see just the first packet.

fwaccel [-vs <vsid>] [conns | templates | stat | on | off]

Mind VS number

42


Cluster Issues

Get statuscphaprob [-vs vsid] stat

Interfaces statuscphaprob -a [-vs vsid] if

Problem notification list:cphaprob [-vs vsid] list

Force member UP or DOWN, for failover tests:clusterXL_admin up/down

43


Kernel Debug

One kernel for all!

Massive output, mind performance

Some express debugs:fw ctl zdebug drop | grep <your filter> - to see drop reason on specific traffic

Mind kernel buffer size

44


System Tools

arp, route, netstat, ifconfig - to have “-z X” flag where X is VS number

-z all prints info for all VSs

ping -z...

traceroute -Z... (Capital letter)

45


VS Policy

To fetch the last installed policy:fw [-vs <vsid>] fetch local

Fetching the last policy that failed to be installedfw fetchlocal -d $FWDIR/state/__tmp/FW1/

To unload policy:fw [-vs <vsid>] unloadlocal

Unload policy for all VSs:fw vsx unloadall

46


VS configuration

To fetch configuration: fw vsx fetch

For specific VS: fw vsx fetchvs –vs 2

To see NCS script for a specific VS: fw vsx showncs <vsid>

47


Other tips

Double check topology

If you cannot figure connectivity issues, especially some traffic degradation, suspect ClusterXL before others

48


Questions And

Answers


Thank You For Your Time!

VSX Troubleshooting - CPUG

Documents

Transcript of VSX Troubleshooting - CPUG