Setting up the environment

To get started with writing a decoder using the Decoder SDK we need a way of repeatedly executing the decoder against known pcap data. We can then debug and test that it is working correctly before configuring a probe to run the decoder against realtime captured data.

First we need to create an area to write our decoders, so create the directory:

/data/sdk

The "sdk" directory will be the main area where all decoder scripts will be executed from.

Inside the "sdk" directory you create a directory per decoder you wish to write. Lets say we need to create the chapo decoder.

+/data
|
+ /sdk
|
+ /chapo

Once we have this directory setup we add 2 things:

1) PCAP data
2) A shell script to execute the decoder

The shell script executes a retro decode on the pcap data. The important aspect of this is that it is repeatable and the collected output can be checked.

Create a file called "run_decode.sh" in your decoder directory and copy the following script into it:

#!/bin/sh
 
SCRIPTFILE="chapo.js"
 
SCRIPTPATH=$(readlink -f $0)
SCRIPTPATH=${SCRIPTPATH%/*}
 
# use the last two dir names as the prefix for output files
OUTNAME=`echo $SCRIPTPATH | gawk 'BEGIN{FS="/";}{print $(NF-1)"_"$(NF-0)}'`
DECODEDFILE="$OUTNAME.decoded"
OUTPUTFILE="$OUTNAME.output"
 
CONFIG="config:name=$OUTNAME"
CONFIG="$CONFIG filter=\"tcp\""
CONFIG="$CONFIG protocols=ethernet,ip,script_v8:s"
CONFIG="$CONFIG s.script_file=\"$SCRIPTPATH/$SCRIPTFILE\""
CONFIG="$CONFIG msg_collector=capture"
CONFIG="$CONFIG coll_capture.capture_file=$DECODEDFILE"
CONFIG="$CONFIG stat_collector=summary_dumper"
 
# uncomment these 2 lines to specify a UID expression
#CONFIG="$CONFIG coll_capture.uid_gen=variable"
#CONFIG="$CONFIG uid_gen_variable.uid={ftl_nsemd.sequence}"
 
PCAPFILES="--pcap_file $SCRIPTPATH/chapo.pcap"
#PCAPFILES="$PCAPFILES --pcap_file $SCRIPTPATH/another_pcap.pcap"
#PCAPFILES="$PCAPFILES --pcap_file $SCRIPTPATH/and_another_pcap.pcap"
 
# file_packet_decoder appends to this file so make sure its removed each time the shell script is executed
rm $DECODEDFILE
 
echo "pcaps: $PCAPFILES"
echo $CONFIG
/opt/tsa/bin/file_packet_decoder --no_mac_crc ${PCAPFILES} -d /opt/tsa/lib/probe.so "$CONFIG" >& $OUTPUTFILE

You should end up with the following directory and files:

+/data
|
+ /sdk
|
+ /chapo
- chapo.js
- chapo.pcap
- run_decode.sh

Configuring the execution of the decoder

To run your decoder the shell script contains 3 main shell variables which can be modified to suit your requirements.

$SCRIPTFILE

The filename of the javascript decoder script you intend to execute e.g. chapo.js.

$CONFIG

The $CONFIG param passes specific settings to the command line file packet decoder which sequentially processes the pcap data that it is given.

The shell variable is defined over multiple lines in this way so that individual settings can be enabled or disabled by commenting out the start of the line.

The two main settings that are of interest are:

  • protocols
    The pcap data is decoded using the stack of protocols defined with this setting. For TCP and UDP protocols the stack string always starts with "ethernet,ip". The ip layer will pass blocks of data into a buffer to the decoder next to it. So in the script above "protocols=ethernet,ip,script_v8:s", tells the decoder framework to pass the data after the ip layer to a script called "s". "s" is defined in the example above as "s.script_file=\"$SCRIPTPATH/$SCRIPTFILE", which sets the full path and name to the chapo.js script file.
    -
    If a single script decoder is required the configuration in the example above will suffice for most cases. It is possible however to specify more than one script by setting the symbol after the script_v8 into something unique. For example we could use "protocols=ethernet,ip,script_v8:s1,script_v8:s2", and then we would need to add "s1.script_file=..." and "s2.script_file=..." as settings as well. The result of this is that if the "s1" decoder needs to pass a buffer to the next decoder for decoding it will be sent to the "s2" decoder.

$PCAPFILES

You can specify one ore more pcap files to execute the decoder against.

Executing the decoder against pcap data

To execute the decoder against the pcap file and configuration specified in the shell script. First Make sure the shell script has execute file permissions set (chmod 755), then enter:

./run_decode.sh

Once the shell script has finished executing you will see two additional files in the directory:

sdk_chapo.output
sdk_chapo.decoded

The size of these files will indicate how successful the decode was. The ".output" file contains the console output of the execution of the decoder and is never empty. The ".decoded" file contains the datafields collected per message for the decode. If nothing is successfully decoded this file may be empty.

The reason the filenames and configuration are embedded into a script like this is to easily allow the decode to be executed repeatedly. You may modify the shell script to accept parameters to experiment if you wish.


pcaps containing VLAN tags

If the pcap you are using contains vlan tags then the BPF expression has to include "vlan".  So for example a pcap with UDP traffic should be "vlan and udp".  The order of the "vlan" in the expression is significant.

Take a look at: http://man.he.net/man7/pcap-filter

The vlan section explains why "vlan and udp" and "udp and vlan" are different. The expression order alters the offsets for decoding the vlan tags, as long as you specify the vlan tags first and then the protocol it will be correct.


Executing the decoder with realtime data

Create a new passive latency probe and select "SDK" as the Wire Protocol.

Enter the full path and filename of the script file into the "Script File" input box. E.g. /data/sdk/chapo/chapo.js

The new probe can then be started/stopped just like any other probe.