Senior Security Engineer
John Petrucci is a senior security engineer with Optiv’s MSS team. In this role he architects and supports solutions which enable the MSS and SIEM engineering teams to more effectively accomplish their work.
Check Point Kernel Debugging, In-Depth
The following is a look into the features and inner-workings of debugging the Check Point firewall kernel. This information will prepare you to debug Check Point firewalls with more efficiency allowing you to readily identify relevant troubleshooting data. In the first section, I’ll discuss the different tools that allow you to peek under the hood and understand what’s going on inside the firewall. The second section discusses ways to ensure that debugging information doesn’t get lost. The final section walks through reducing the output so you can get to the pertinent information more quickly.
Please familiarize yourself with the following terms, as they will be used frequently throughout this document.
Debugging Tools: zdebug and kdebug
The two tools available to debug a Check Point firewall are zdebug and kdebug. These tools are complimentary to one another. You should look at kdebug as a high precision debugging tool that is useful to diagnose even the most complicated issues. zdebug is best viewed as a tool that can give you a quick look at what’s happening and diagnose simple issues with the firewall such as a rulebase drop.
When you launch zdebug, it automatically sets certain parameters in your debugging session. This behavior is useful because you do not need to manually configure these settings. The following list contains some of these automatic settings:
- Sets the buffer to one MB
- Enables the selected flags
In addition to this default set of options, zdebug also has some automatic behavior including:
- Starts printing the debug buffer contents to the screen
- Resets the buffer and modules to the defaults when you’re finished
The combination of default settings and automatic behavior make zdebug a quick diagnostic tool. However, since the tradeoffs have been made toward ease of use instead of flexibility, its use can be narrow. For issues where zdebug doesn’t allow you to diagnose the issue, Check Point created the kdebug tool.
Compared to kdebug, zdebug requires more interaction, but is much more powerful. Some of the additional features include:
- Setting any combination of module and flags
- Displaying second or microsecond timestamps
- Creating a cyclic set of files to keep disk space from being consumed
- (e.g. debug.0, debug.1, …, debug.n)
- Print additional data fields (columns)
You should think of kdebug as a way to view the kernel message buffer. The actual manipulation of what appears in the buffer: modules, flags, types and frequencies is accomplished with the command `fw ctl debug.` In fact, as long as you set a buffer first with `fw ctl debug` you can start viewing it with kdebug and manipulate the modules and flags while kdebug is running. In comparison, zdebug is quicker to get running but does not allow you to perform the following operations:
- Modify the buffer size
- Specify type/frequency thresholds
- Enable timestamps
Logging Debugging Output with kdebug
kdebug can be used to view the messages on the console in realtime just like zdebug, but it is more useful to redirect the output to a file. You can quickly redirect the output to a file using the following syntax:
fw ctl kdebug –f > debug.out
While the debugging output can be captured within a single file quickly, when you’re debugging over a long duration cyclic files will save you from consuming too much disk space. To set up a cyclic kdebug you can use the following syntax:
fw ctl kdebug –s <Size_of_each> –m <nuMber_rotations> -o <Output_name> -f
When performing longer duration debugging, you should still be wary of the CPU overhead when asking a client to run a resource intensive debugging session during production hours
An interesting and undocumented quirk is seen when the –s and –m options are omitted, but –o is still used to redirect the output. In that case, the output file will contain binary encoded data that can be played back using the following command:
‘fw ctl kdebug –i <file>’
Using this feature, you can switch timestamps on and off, and toggle –p data display during the playback instead of during the collection. In practice there is little reason for using –i in this way, but if you receive a debug output file that appears corrupted in your text editor, then it could be that the file was recorded in a format intended for playback.
fw ctl debug –i filterfile
The ‘fw ctl debug’ command also has an –i argument that is described as defining a filter file.” I have not found a practical way to leverage the –i argument yet. The vendor gives no details regarding this argument.
Buffers and Suppression
When kdebug or zdebug is used to view the messages in the buffer or write them to a file, we introduce a bottleneck. A hard drive can store lots of data, but is not good at storing small pieces of data quickly. The kernel can create debugging messages at an incredible pace and the messages tend to be small pieces of data, resulting in issues for the hard drive to keep up with the pace. In order to allow the messages to be recorded, Check Point firewalls use the concept of buffers. Buffers are areas of RAM allocated to be a temporary storage location that can receive data as quickly as the kernel can create the data, and can keep up with small writes. The idea is to allocate a buffer that’s large enough to store the debugging data, allowing the hard drive to write in bigger chunks. Without a properly sized buffer, the kernel will produce debugging messages faster than can be read eventually resulting in a loss of debugging messages as shown in the following image:
The best advice is to set the buffer size to the maximum size, but be mindful about the CPU utilization. Higher rates of debugging output will cause higher CPU loads, and you’ll have to balance possibly missing messages against overloading the CPU. Be wary whenever you’re expecting a high rate of debugging output such as when you’ve enabled many debugging flags or when a debugging message is displayed at an unusually high rate. In those cases, choose a smaller buffer size such as 500KB to limit CPU utilization.
The buffer size can be set to the maximum of 32,768KB using the following command:
fw ctl debug –buf 32768
Up to this point we’ve been discussing the buffer that the kernel uses to temporarily store debugging messages. There is a second buffer kdebug uses to store debugging messages from the kernel buffer before writing them to your screen or a file. The following image shows the option to set the size of this second buffer.
Changing the size of this secondary buffer has no practical value. However, it’s important to point out since they are easily confused with each other.
Increasing Signal, Decreasing Noise
Check Point firewalls can be configured to print an unintelligible amount of data regarding how the system is functioning. The data is important when trying to diagnose an issue with a firewall, but it’s easy to get lost in a sea of data. One of the most important aspects is filtering the signal from the noise. One method of collecting only the most salient data is filtering based on message type.
For any message type selected in the debugging session, the selection will also include the more critical types. So setting a threshold of Info will include messages grouped into Notice, Wrn, and Err. Likewise, setting a threshold of Wrn will include Err.
Frequency threshold is another discriminator that allows even more fine-grained control over the messages included in the debug output. The two frequency thresholds are Common and Rare. Setting the frequency threshold to common will include the messages in rare. Rare will restrict the output to a smaller set of messages.
Message type and frequency thresholds are a means to control verbosity, but it can be hard to determine the best type and frequency to set for debugging a given problem. Unless we know that a particular debugging message that we’re looking for only appears at a higher “type” it is best to leave these at the lowest, most verbose setting. At the time of this writing the available levels are as follows:
Type and frequency can be changed with the following command:
fw ctl debug -t <type> -f <freq>
Be careful with selecting the NONE message type. Unlike all the other message types, when NONE is set no debugging messages will be printed and there is nothing printed warning you that your debug logs will be empty.
Besides filtering out messages, it can be useful to control what information within a message is printed. kdebug accepts an optional argument, -p, that can be used to select which fields of a debug message will be displayed.
The command line help output shows all possible values for -p:
As an example, take the type and freq values for the –p argument. Selecting these will cause the debug messages to include the type and frequency of the message, allowing us to find out which minimum thresholds to set in order to see those particular messages. The following image shows debugging output with these fields selected.
While we’ve focused on techniques to filter out debug messages, it’s just as important to ensure that debug messages we’re interested in will be printed. Checkpoint uses kernel parameters to control whether certain debugging messages should be printed. These kernel parameters can be adjusted using the “fw ctl set int” command.
We can further enrich these debug message with some flags that print additional data per line. For example, the ‘address’ flag of the module ‘APPI’ will include the source and destination IP address, which would otherwise not be displayed.
To find kernel parameters that can be changed to produce more information, we can list all of the kernel parameters and search for “print,” “dprint,” or similar as we have in the following image. Not all matches will produce debugging information and you should experiment to understand which parameters produce your desired debugging output. Also, you can check out Check Point solution SK33156 for a guide on creating a file that will contain all the kernel parameters and their associated values.
In summary, there is no one size fits all debug that I can recommend, but by learning more about these advanced debug features you will be better equipped to troubleshoot any problems you face.