VPN troubleshooting hints

VPN Issues

Don’t know where to start!

Note: This FAQ is centered toward site-to-site VPNs. There is some filter hints for client-to-site VPNs that may help, but as C2S is often less business affecting, issues with it are more rare (beyond the initial configuration, IE once working it usually works), and has a plethora of implementations possibilities, a FAQ for troubleshooting such would not be productive.

Solution:

First collect the configuration info as outlined by the VPN Check List FAQ (Linked, can send that one to customer ver batim for the information). The local and remote configurations SHOULD MATCH EXACTLY! Don’t let that Cisco guy on the other side say “that’s not necessary”… Example: Dynamic crypto maps will likely fail unless you are configured for “host to host” in tunnel management (even then its iffy)… If there are any discrepancies in the configuration, it will likely cause instability (most often complete failure per VPN101). It is better to be precise during troubleshooting than to guess whether or not it’s “necessary”.

Personally, I’ve seen that > 70% of all Checkpoint interoperable (IE remote peer is a different vendor) VPNs issues are results of improper VPN domain (aka encryption domain, proxy-id, or cryptomaps) configuration, so if initial analysis finds ANYTHING that hints to a domain issue, then thoroughly check to ensure domain configurations are correct. Easy task, right? No, and due to that I am dedicating another FAQ to an explanation of VPN Domains (linked).

Another semi-domain related issue that occurs often and is a quick check / fix, is to determine whether or not your NAT configuration is correct for the VPN. Is the remote site expecting your local networks as their remote networks, or are they expecting a hide NAT (aka source NAT)? If you expect to not use NATs in this configuration (for the community), then confirm that the VPN’s “Disable NAT inside this VPN Community” is checked. If NAT is required, then ensure NAT rules are configured correctly.

Next: Look at SmartView Tracker logs (this is a great tool in order to get an idea of where to concentrate further troubleshooting):

SmartView Tracker Filter Hints:

For site-to-site VPNs: Set BOTH the “Source” and “Destination” columns’ filters to BOTH the local and remote firewall (both VPN peers) IP.

Are the negotiations completing correctly? Look for records with “Main Mode Completion” (IE phase 1 negotiation successful) and “Quick Mode Completed…. <local and remote networks as negotiated>” (IE phase 2 negotiation successful) in info column. Look for “<Rec’d / Sent> delete <IKE / IPsec> Security Associations…” or other “<IKE / IPsec> messages…” coming from / going to the remote peer. These messages can be searched in order to ascertain an issue with negotiations in the process. IKE debugs may be necessary if either phase fails (see below) to better see why, or ensure you are seeing the correct messages pertaining to the correct negotiations… If no records, then check IPsec traffic (does it try to encrypt outgoing initial connections? explanation below), and tcpdump on external interface to ensure you are receiving IKE traffic from the configured peer (below). Confused due to tunnel management configurations? IKE debugs may help straighten out what conversation / errors belong to which tunnel.

When examining IPsec traffic, it is important to understand how traffic is initiated in the VPN. In the scenario below is for traffic initiated from local networks to remote networks. If traffic initiates remotely, simply swap all “remote” below with “local” and vice versa, and swap all “encrypted” for “decrypted” (lock Icon with a key). If it is a bi-directional initiation, then look for both!

Traffic TO remote networks:
Change the filters in tracker to include traffic TO the remote networks (Src= <local VPN domain>, dst = <remote VPN domain>). If you want to see key exchanges at the same time, you can leave the gateways in the filters, just be aware there may be some local net to gateway traffic that is not related; alternatively, you can “Open in new window…” to have multiple log panels open for review.

1) Is traffic destined for the remote network being encrypted (a lock icon will be in the “type” column)? 2) Are they being permitted? 3) Or are they being blocked?

If yes to 1), then move to next section (traffic FROM remote nets). If NO to 1) but yes to 2) then your firewall does not recognize this as interesting traffic for the VPN — check VPN Domain, firewall policy, and policy install logs (included in SMS cpinfo, search for “adtlog” and ensure customer installed policy since the change, and policy installed correctly). If yes to 3), look in the “info” column to see the reason it is being dropped (if its empty, check “rule” column to see if a rule is dropping it, and correct policy); if it is the “Cleanup rule” then treat as if it is not recognizing the traffic as interesting, so as if 1) No and 2) yes).

If there are any “Invalid SA” messages, then first, keep in mind that this is for outgoing traffic, so if a tunnel is up, and the traffic is interesting, there is only two things that could cause this: a) Tunnel really isn’t up (it just appeared to be) so there is really no valid SA for that traffic (check again for key install issues & IKE or IPsec messages), or b) something is wrong inside the Check Point magic (IE DB corruption, cluster sync issues, etc…) — try a policy install (if there hasn’t been one since the issue started) to clear DB issues, and investigate cluster configuration and status (‘cphaprob ‘ + ‘ -a if’, ‘ stat’, OR ‘ syncstat’ are some good places to start).

Traffic FROM remote networks:
Edit the filters so that the VPN domains are reversed (Src= <remote VPN domain>, dst = <local VPN domain>). This should yield NO results. If you get records for response traffic, carefully inspect these records as the information for their existence may be beneficial in your troubleshooting.

Note: I have rarely seen instances where normal decryption records will exist for response traffic. If the record is a normal decrypt log, then it is ok to ignore (except as proof you are receiving, and decrypting, the peer’s IPsec traffic).

Note for bidirectional VPNs: In order to accurately determine if response traffic is creating logs, limit the filters to only a local client (one that does not service incoming connection) for destination, and a remote server for the source. SmartView Tracker is finicky about displaying response traffic logs when initial connection records exist for a query. This is regardless of transport protocol used.

NO RECORDS for encryption domains, either way? Then we need to ensure IPSec is even getting to the peers from / to each-other. This requires tcpdumps on the external interface (below).

C2S: For Remote user VPN:
Set the “User” column to “Equals” that user’s username. Try source or destination of the remote user’s public IP. Look for dropped packets, and the errors or problems in the info column. If no records exist, best resource will be the client logs (if using a client), and packet captures.

Check Point’s VPN Tunnel Utility:

If you need to quickly print Security Associations (SAs), or need to reset VPN tunnels from a gateway to a remote peer, or need to quickly reset all tunnels, use the command ‘vpn tu’. This tunnel utility is a quick and easy way to do so for troubleshooting. SmartView Monitor has the capability of resetting all VPNs between 2 gateways as well (it calls all VPNs between the two, regardless of IPsec SPIs, a “tunnel”).

IKE debug (IE negotiation issues):

If you see any issues with key installs, and you need better details then what is provided by SV Tracker, then you may want to run the ike debugs for them. In order to do so, issue the command (in expert mode for SecurePlatform / Gaia) ‘vpn debug ikeon’. This will create a file in $FWDIR/log named “ike.elg” (if left running the file will periodically switch over and append a n+1 number to the file name, starting at 0). Best practice follows:

1) Using the ‘vpn tu’ utility, reset all IKE and IPsec SA’s related to the VPN(s) in question. (Optional if constant traffic is present, so you can’t keep tunnel down long enough for step 2)

2) Enable IKE debug: Issue the command (in expert mode for SecurePlatform / Gaia) ‘vpn debug ikeon’.

3) Initiate connections, to initiate VPN negotiation.

4) Reset the SA’s again (vpn tu), and test connections again (you may want to repeat this step a couple times).

5) Issue ‘vpn debug ikeoff’ to disable.

Pull the IKE debug files from the device and load them into IKEView. IKEView is a utility that comes packaged with Check Point’s InfoView. Install InfoView then go to install location to find the IKEView.exe to open (installer does not create a shortcut for it).

This shows a good breakdown of the IKE conversations during key exchanges. This will include any IKE messages, but (unlike SV Tracker) will identify exactly what encryption domains negotiations (IE exact tunnel) such messages are for.

NOTE: This is only IKE debugs and NOT IPsec debugs. I have never found anything useful in IPsec debugs, and thanks to that and its huge verbosity (which includes recording pre-shared keys in plain text) and confusing syntax, this tool should be reserved for Level-3 support only (IE when requested by vendor, through a vendor’s case). See sk33327 for instructions. This is also true for packet captures performed with the ‘vpn debug mon’ command (which also has unencrypted IKE messages).

Packet captures:

The need to verify configuration extends beyond just the configuration of the device itself. Complicated infrastructures may have more components involved with the transport of VPN traffic through a network before it gets to the internet to be routed normally between orginizations. For example, a device may sit between the VPN peer and the internet which performs the orginizations NAT. Futhermore, we can’t trust that a remote peer’s administrator is proficient and their configuration is as they stated. These can create issues where the peer’s IP address is not as expected for the VPN traffic. You can either not be seeing communication from the peer, see “invalid SA” IPsec traffic, or just see successful key exchanges without IPsec traffic. Per VPN101 we know that IKE and IPsec transport via different IP protocols (with the exception of NAT-T: Checkpoint doesn’t support NAT-T as the VPN initiator, but can be configured to support it as a VPN responder). Because of this an orginizations NAT device may NAT the UDP 500 traffic (IKE) one way, but it may NAT the IP protocol 50 (ESP / IPsec) traffic another way… So we need to ensure that the Check Point is getting the traffic as it expects to.

In order to see traffic unaltered by the firewall components of the device, you must do tcpdumps on the recieving interface. Syntax for below examples are as follows: <local / remote-peer> = the IP address of the referenced peer as configured on the opposite peer; <local / remote-host> = a host from the referenced encryption domains; <natof-lhost> the IP address of the NAT for <local-host>; <ext-if> = the external interface or the interface with the <local-peer> IP. Option tags used: -nn this tag disables the tcpdump default of trying to resolve the hostname (first n) and ports to services (second n), which eases resource utilization of the tcpdump (always recommended unless their is an explicit need for name resolution); -i tag specifies a specific interface, which you can use ‘-i any’ to listen to all interfaces, but prefer fw monitor for this use (fw mon identifies the interfaces, tcpdump does not have an option too– closest is -e and matching MACs). For ‘fw monitor’ -e tag simply tells fw monitor to get it’s arguments from following quotation. Finally if you need a wireshark readable form (allows you to use Wireshark to view captures, which gives you timestamps on the captures and the capability to filter further on the captures plus much more), simply append ‘-w <filename>’ (tcpdump) or ‘-o <filename>’ (fw monitor) to the command.

Some useful examples of tcpdumps follows:

tcpdump -nni <ext-if> host <local-peer> and <remote-peer>

Captures all traffic between the two peers, this will show you if both IKE (may display as isakmp) and IPsec (ESP) traffic is making to correct IP. If you don’t see both, then try and capture all IKE / ESP traffic and go through it.

tcpdump -nni <ext-if> ip proto 50 or udp port 500

If Check Point is strictly a VPN responder and supports initiation with NAT-T:

tcpdump -nni <ext-if> udp port 500 or 4500

Alternatively, if you need to limit such traffic captures to just those for your firewall (maybe you have a VPN concentrator behind it for other / unrelated VPNs) you can add the <local-peer> as follows… Note: use of single quotes is required with paranthesis arguments:

tcpdump -nni <ext-if> ‘host <local-peer> and (udp 500 or ip proto 50)’

If you can confirm receipt of both traffics, but suspect something is getting hung up in the fw kernel for the traffic, you can confirm the traffic is passing through the fw chain with the command:

fw monitor -e “accept host(<remote-host>) and host(<local-host>);”

If there are local NATs involved in the VPN you should add that ip as well:

fw monitor -e “accept (host(<remote-host>) and host(<local-host>)) or (host(<remote-host>) and host(<natof-lhost>));”

Advanced: You can move the position of the big-i (I) and little-o (o) capture points to incorpate the actual decryption and encryption points, and this should include the transformation of those packets throughout the chain… This example uses just the local host, as theoritically once a packet has been identified as interesting by the filter it will show all related NATting and encryption (IE should follow that packet throughout the fw chain), but beware that I’ve noticed different versions of CP being finicky about this, and if it doesn’t identify NATs (or encrypted traffic) as interesting to the filter then you will miss that packet. That is why I included above filter for NATs, so you may want to follow that example to ensure traffic filter is complete (IE include any NATting and the following ‘or (host(<remote-peer>) and host(<local-peer>))’ to argument:

fw monitor -pI +vpn -po -vpn -e “accept host(<local-host>);”

 

Comments:

It is not recommended to have Phase 2 rekey timers equal or greater than the Phase 1 rekey timers. This may cause stability issues especially near to such rekey timers.

Anti-spoofing:
Incoming traffic, with a private IP address, regardless of being over a VPN and regardless of VPN domain configurations, is checked by the Check Point’s Anti-spoofing security. You may need to add the remote networks to the anti-spoofing topologies as appropriate (if it is coming through an “External” interface, then you must add the networks or groups of networks to the “Don’t check packets from” of that interface’s Anti-spoofing settings).

Was this article helpful?

Related Articles

Leave A Comment?

You must be logged in to post a comment.