While the Decoder SDK can be used to perform datafield mapping, for simple cases it can often be easier and more performant to use the mapping layer.

This feature allows creation of new datafields and mapping values between them using a JSON configuration file.

Configuration

The configuration is a single JSON file containing an object with 2 required properties plus an optional comment that has no effect on the configuration.

{
"comment": "Example comment",
 
"datafields": {
},
 
"actions": [
]
}

The datafields property is an object that contains a list of new datafields and their types to create. The type names are the same as those used for DDC (dynamic datafield configuration) in some protocol decoders. Datafields may also be created with a generic name by using a JSON object instead of a string.

{
"datafields": {
"string_df": "string",
"integer_df": "int",
"unsigned_integer_df": "uint",
"float_df": "float",
"boolean_df": "bool",
"character_df": "char",
"ip_address_df": "ipaddr",
"ip_port_df": "ipport",
"timestamp_df": "timestamp",
"ethernet_address_df": "ethaddr",
"binary_df": "binary",
"generic_timestamp_df": {
"comment": "This is a timestamp datafield with a generic name",
"type": "timestamp",
"genericName": "TIMESTAMP"
}
},
 
 
"actions": [
]
}

The actions property is an array called an "action list". An action list is a sequential list of objects containing actions to perform. This might be testing the value of a datafield or assigning a value to a datafield, for example. Many actions can include nested action lists, allowing for more complex logic.

Cache Expiration Policies

Some actions include cache expiration policies; these may be one of the following:

none

Never expire values in the cache. This should be used carefully and only when the cache size will remain bounded.

{
"type": "none"
}

maximumSize

Limit the cache to a specific size. The least recently used cache entries will be expired when the size exceeds that specified in the "maximumSize" property.

{
"type": "maximumSize",
"maximumSize": 1024
}

time

Expire items from the cache when they haven't been accessed for a certain length of time. The check for expiration only occurs each time the cache size grows. The “expirationTimeSeconds” property specifies the expiration time in seconds. The optional “timestamp” property specifies a timestamp datafield to use for tracking the time. It defaults to the generic timestamp.

{
"type": "time",
"expirationTimeSeconds": 2.5,
"timestamp": "packet_timestamp"
}

General Actions

comment

The comment action does nothing, but can be used multiple times in an action list to make the configuration more readable.

{
"datafields": {
},
 
"actions": [
{
"comment": "Example comment"
}
]
}

assign

The assign action assigns values to datafields.

{
"datafields": {
"string_df": "string",
"integer_df": "int",
"unsigned_integer_df": "uint",
"float_df": "float",
"boolean_df": "bool",
"character_df": "char",
"ip_address_df": "ipaddr",
"ip_port_df": "ipport",
"timestamp_df": "timestamp",
"ethernet_address_df": "ethaddr"
},
 
"actions": [
{
"assign": {
"string_df": "test string",
"integer_df": "-1234",
"unsigned_integer_df": "1234",
"float_df": "12.5",
"boolean_df": "1",
"character_df": "x",
"ip_address_df": "192.168.1.1",
"ip_port_df": "8080",
"timestamp_df": "20180101T100000.123456789",
"ethernet_address_df": "aa:bb:cc:11:22:33"
}
}
]
}

assignExpr

The assignExpr action assigns an expression to a datafield. The expression format is currently a very limited subset of Spring Expression Language (SpEL), supporting querying the values of datafields (df['datafield name']), unary plus/minus, binary plus/minus/divide/multiply/modulus, numeric and string literals. There is also an extension for specifying integral literals as unsigned with a "u" suffix.

{
"comment": "Examples of assigExpr",
 
"datafields": {
"int1": "int",
"string1": "string",
"string2": "string",
"timestamp1": "timestamp",
"timestamp2": "timestamp",
 
"result_positive_timestamp_calculation_int": "int",
"result_positive_timestamp_calculation_uint": "uint",
"result_positive_timestamp_calculation_timestamp": "timestamp",
"result_negative_timestamp_calculation_int": "int",
"result_negative_timestamp_calculation_uint": "uint",
 
"result_literal_assign_int_from_string": "int",
"result_literal_assign_int_from_int": "int",
 
"result_concatenated_strings_int": "int",
"result_concatenated_strings_string": "string",
"result_concatenated_strings_parentheses_string": "string"
},
 
"actions": [,
{
"comment": "Set up test data"
},
{
"assign": {
"int1": "12",
"string1": "123",
"string2": "456",
"timestamp1": "20210301T100001.000000001",
"timestamp2": "20210301T100000.123456789"
}
},
 
{
"comment": "Evaluate example expressions"
},
{
"assignExpr": {
"result_positive_timestamp_calculation_int": "df['timestamp1'] - df['timestamp2']",
"result_positive_timestamp_calculation_uint": "df['timestamp1'] - df['timestamp2']",
"result_positive_timestamp_calculation_timestamp": "df['timestamp1'] - df['timestamp2']",
"result_negative_timestamp_calculation_int": "df['timestamp2'] - df['timestamp1']",
"result_negative_timestamp_calculation_uint": "df['timestamp2'] - df['timestamp1']",
 
"result_literal_assign_int_from_string": "'1234'",
"result_literal_assign_int_from_int": "-4567",
 
"result_concatenated_strings_int": "df['string1'] + df['string2']",
"result_concatenated_strings_string": "1 + df['int1'] + df['string1'] + df['int1']",
"result_concatenated_strings_parentheses_string": "1 + (df['int1'] + df['string1']) + df['int1']"
}
}
]
}

In the examples, the timestamp subtraction calculations return the delta of the timestamp as nanoseconds. If assigning a negative value to a uint, it will overflow and be a large value. If assigning to a timestamp it will be nanoseconds since the unix epoch:

"result_positive_timestamp_calculation_int": 876543212
"result_positive_timestamp_calculation_uint": 876543212
"result_positive_timestamp_calculation_timestamp": "19700101T000000.876543212"
"result_negative_timestamp_calculation_int": -876543212
"result_negative_timestamp_calculation_uint": 18446744072833008404

If a calculation of 2 uint values (e.g. DatafieldTimestamp) results in a negative number, it will underflow and be a large uint. Assigning this to an int will store the expected negative value. However, if the value is modified to fit as a positive number in an int, it will be stored as a positive value. e.g. “(df['timestamp2'] - df['timestamp1']) / 1000" assigned to an int will be positive. This can be worked around by first calculating the subtraction and storing it in an int (DatafieldInt), then using another expression to do the division on that int.

Assigning literals will behave as if using the "assign" action:

"result_literal_assign_int_from_string":   1234,
"result_literal_assign_int_from_int":      -4567

Binary plus when applied to a string will append strings:

"result_concatenated_strings_int":                123456
"result_concatenated_strings_string":             "1312312"
"result_concatenated_strings_parentheses_string": "11212312"

assignVariableExpr

The assignVariableExpr action assigns an expression to a datafield. The expression is in the “variable UID generator” format.

{
"comment": "Examples of assignVariableExpr",
"datafields": {
"ip.src_host": "ipaddr",
"ip.src_port": "ipport",
"result_int": "int",
"result_string": "string"
},
 
"actions": [
{
"assign": {
"ip.src_host": "10.1.1.1",
"ip.src_port": "123"
}
},
{
"assignVariableExpr": {
"result_string": "{ip.src_host}:{ip.src_port}",
"result_int": "{ip.src_host:?}1|0"
}
}
]
}

In the examples, the results are:

"result_string": "10.1.1.1:123"
"result_int": 1

unset

The unset action will unset datafields specified in the “datafields” array. This removes the value from each datafield. One possible use case is to force a datafield value to be retrieved from a lastKnownValue cache. An optional "comment" property can also be included in the action object.

{
"comment": "Example of unset",
"actions": [
{
"unset": {
"comment": "Just an example, probably no need to ever do this.",
"datafields": [
"ip.dst_host"
]
}
}
]
}

base62Decode

The base62Decode action will decode a base62 encoded string in a string or binary datafield in the "sourceDatafield" property and assign the result as a 64 bit integer datafield in the "destinationDatafield" property (e.g. a numeric datafield type). An optional "comment" property can also be included in the action object.

{
"comment": "Example of base62Decode",
 
"datafields": {
"input": "string",
"output": "uint"
},
 
"actions": [
{
"comment": "We would usually decode a datafield from another protocol, but for a standalone example, we'll assign and decode a string instead"
},
{
"assign": {
"input": "5DIF33YV0"
}
},
{
"base62Decode": {
"sourceDatafield": "input",
"destinationDatafield": "output"
}
}
]
}

In this example, the datafield "output" will have the value of 1138517709214786.

base64Encode

The base64Encode action will base64 encode the binary representation of a datafield in the "sourceDatafield" property and assign the result to a string datafield in the "destinationDatafield" property. It is an error to specify a destination datafield that is not of type string. An optional "comment" property can also be included in the action object.

{
"comment": "Example of base64Encode",
 
"datafields": {
"input": "string",
"output": "string"
},
 
"actions": [
{
"comment": "We would usually encode a binary datafield from another protocol, but for a standalone example, we'll assign and encode a string instead"
},
{
"assign": {
"input": "abcd"
}
},
{
"base64Encode": {
"sourceDatafield": "input",
"destinationDatafield": "output"
}
}
]
}

In this example, the datafield "output" will have the value of "YWJjZA==".

addressToName

The addressToName action maps an IP address datafield, specified in the "addressDatafield" property to a hostname specified in the "nameDatafield" property. The results are cached for performance. The cache size is specified with the "cacheSize" property. The "noFullyQualifiedDomainName" property should be true to omit domain names from hosts on the local network, otherwise false. If the hostname of the IP address datafield can not be resolved, the IP address in dot-notation is stored in the name datafield. If the IP address datafield is not set, the name datafield is not set. An optional "comment" property can also be included in the action object.

{
"comment": "Map source IP address to hostname",
 
"datafields": {
"hostname": "string"
},
 
"actions": [
{
"addressToName": {
"addressDatafield": "ip.src_host",
"nameDatafield": "hostname",
"cacheSize": 100,
"noFullyQualifiedDomainName": true
}
}
]
}

isSet

The isSet action executes action lists depending on whether a datafield, specified in the "datafield" property, is set. If it is set, the action list in the "true" property is executed. If it is not set, the action list in the "false" property is executed. Either "true" or "false" may be omitted if they're not needed. An optional "comment" property can also be included in the action object.

{
"comment": "Set vlan_outer_id_is_set if eth.vlan_outer_id is set",
 
"datafields": {
"vlan_outer_id_is_set": "string"
},
 
"actions": [
{
"isSet": {
"datafield": "eth.vlan_outer_id",
"true": [
{
"assign": {
"vlan_outer_id_is_set": "Yes"
}
}
],
"false": [
{
"assign": {
"vlan_outer_id_is_set": "No"
}
}
]
}
}
]
}


map

The map action executes action lists depending on the value of a key datafield, specified in the "key" property. If no value matches or the datafield is unset, an optional "default" property for the action specifies a default action list to execute. An optional "comment" property can also be included.

{
"datafields": {
"feed_name": "string"
},
 
"actions": [
{
"map": {
"comment": "Map source IP address to feed name",
"key": "ip.src_host",
"mapping": {
"192.168.1.1": [
{
"assign": {
"feed_name": "Feed A"
}
}
],
"192.168.1.2": [
{
"assign": {
"feed_name": "Feed B"
}
}
]
},
"default": [
{
"assign": {
"feed_name": "Unknown"
}
}
]
}
}
]
}

compositeMap

The compositeMap action executes action lists depending on the values of multiple key datafields, specified as a list in the "key" property. The "mapping" property is an array of mappings from a key set to an action list. If a key datafield is not specified in the key set, it must be unset in the incoming message to execute the action list. If no mappings match, an optional "default" property for the action specifies a default action list to execute. Optional "comment" properties can also be included in the action object and mapping objects.

{
"datafields": {
"feed_name": "string"
},
 
"actions": [
{
"compositeMap": {
"comment": "Map IP details to feed name",
"key": [
"eth.vlan_outer_id",
"ip.src_host"
],
"mapping": [
{
"comment": "Mapping with no VLAN (eth.vlan_outer_id is unset)",
"key": {
"ip.src_host": "192.168.1.1"
},
"actions": [
{
"assign": {
"feed_name": "Feed A"
}
}
]
},
{
"comment": "Mapping with VLAN and source IP address",
"key": {
"eth.vlan_outer_id": "21",
"ip.src_host": "192.168.1.1"
},
"actions": [
{
"assign": {
"feed_name": "Feed B"
}
}
]
},
{
"comment": "Mapping with neither a VLAN or a source IP address",
"key": {
},
"actions": [
{
"assign": {
"feed_name": "No source IP address??"
}
}
]
}
]
}
}
]
}

subnet

The subnet action executes action lists depending on the subnet of an IP address datafield, specified in the "datafield" property. The "mapping" property is an array of mapping objects containing netmasks and action lists. Note that these mappings are executed sequentially until one matches, so the order of the objects can be important for both which objects match (e.g. in the case of overlapping mappings), as well as for performance (put more common mappings first, if possible). If no mappings match, an optional "default" property for the action specifies a default action list to execute. Optional "comment" properties can also be included in the action object and mapping objects.

{
"datafields": {
"subnet_name": "string"
},
 
"actions": [
{
"subnet": {
"comment": "Map a subnet on to a name",
"datafield": "ip.src_host",
"mapping": [
{
"comment": "More specific subnet",
"netmask": "192.168.1.0/24",
"actions": [
{
"assign": {
"subnet_name": "Subnet A"
}
}
]
},
{
"comment": "Less specific subnet with some overlap",
"netmask": "192.168.0.0/16",
"actions": [
{
"assign": {
"subnet_name": "Subnet B"
}
}
]
}
],
"default": [
{
"assign": {
"subnet_name": "Unknown"
}
}
]
}
}
]
}

range

The range action executes action lists depending on what range a numeric datafield, specified in the "datafield" property, is in. The "mapping" property is an array of mapping objects containing ranges (a 2 item array of lower bound and upper bounds where lower_bound <= value <= upper_bound to match) and action lists. Note that these mappings are executed sequentially until one matches, so the order of the objects can be important for both which objects match (e.g. in the case of overlapping mappings), as well as for performance (put more common mappings first, if possible). If no mappings match, an optional "default" property for the action specifies a default action list to execute. Optional "comment" properties can also be included in the action object and mapping objects.

{
"datafields": {
"port_range_name": "string"
},
 
"actions": [
{
"range": {
"comment": "Map a source port range to a port range name",
"datafield": "ip.src_port",
"mapping": [
{
"comment": "Mapping for port range A",
"range": [ 2700, 27100 ],
"actions": [
{
"assign": {
"port_range_name": "Port range A"
}
}
]
},
{
"comment": "Mapping for port range B",
"range": [ 27101, 27200 ],
"actions": [
{
"assign": {
"port_range_name": "Port range B"
}
}
]
}
],
"default": [
{
"assign": {
"port_range_name": "Unknown"
}
}
]
}
}
]
}

lastKnownValue

The lastKnownValue action is used for caching datafields that may only be set in some messages and restoring the value of them in messages where they are not set. The "cache" property specifies a list of datafields to cache and restore for each entry in the cache. An optional “comment” property can also be present.

The cache itself can either be defined in the lastKnownValue definition, or in a separate “lastKnownValueCaches” section of the configuration, and then referred to by the “cacheName” property. The "key" property specifies a list of datafields to use as a key for the cache. It may be empty to cache a value for all messages. The "expirationPolicy" property specifies when values should be expired from the cache. See Cache Expiration Policies.

{
"lastKnownValueCaches": {
"exampleSharedCache": {
"comment": "Example shared cache with maximumSize expiration policy",
 
"key": [
"test.msg_type"
],
"expirationPolicy": {
"type": "maximumSize",
"maximumSize": 1024
}
}
},
"actions": [
{
"lastKnownValue": {
"comment": "Example with none expiration policy",
 
"key": [
],
"expirationPolicy": {
"type": "none"
},
"cache": [
"test.possibly_missing_field"
]
}
},
 
{
"lastKnownValue": {
"comment": "Example with a shared cache",
"cacheName": "exampleSharedCache",
"cache": [
"test.possibly_missing_field"
]
}
}
]
}

duplicateIndex

The duplicateIndex action is used for counting duplicate values of a combination of datafield values and assigning the count to a datafield. The first time a unique combination of datafield values is seen, the index will be 0. Unique values are stored in a cache. The "key" property specifies a list of datafields to use as a key for the cache. It may be empty to cache a value for all messages. The "indexDatafield" property specifies a datafield to store the index value in. The "expirationPolicy" property specifies when values should be expired from the cache. See Cache Expiration Policies. An optional “comment” property can also be present.

{
"datafields": {
"index_uint": "uint",
"index_string": "string"
},
"actions": [
{
"duplicateIndex": {
"comment": "Example with none expiration policy",
 
"key": [
],
"indexDatafield": "index_uint",
"expirationPolicy": {
"type": "none"
}
}
},
 
{
"duplicateIndex": {
"comment": "Example with maximumSize expiration policy",
 
"key": [
"test.msg_type"
],
"indexDatafield": "index_string",
"expirationPolicy": {
"type": "maximumSize",
"maximumSize": 1024
}
}
}
]
}

dec_premapper Specific Actions

By default, dec_premapper will forward on the packet payload that it receives to the next decoder layer. If the “nextDecoderDatafield” or “nextDecoderPayload” actions are used in the configuration, this default behaviour will be disabled and data will only be passed to the next decoder layer when those actions are executed.

nextDecoderDatafield

The nextDecoderDatafield action will forward on the binary representation of a datafield to the next decoder layer. This will likely be most useful for binary or string datafields, as other types may have platform dependent binary representations.

{
"comment": "Forward 'hello world' as packet data to the next decoder layer",
 
"datafields": {
"test_payload_data": "string"
},
 
"actions": [
{
"assign": {
"test_payload_data": "hello world"
}
},
 
{
"nextDecoderDatafield": {
"datafield": "test_payload_data"
}
}
]
}

nextDecoderPacketData

The nextDecoderPacketData action forwards on the packet payload that the mapper layer received to the next decoder layer.

{
"comment": "Only forward packet data when eth.vlan_outer_id is set",
 
"actions": [
{
"isSet": {
"datafield": "eth.vlan_outer_id",
"true": [
{
"nextDecoderPacketData": {
}
}
]
}
}
]
}

coll_mapper Specific Actions

messageCollectorBlock

By default, message collectors will be called after transform collectors, if a message reaches the top of the decoder stack. The messageCollectorBlock action will stop this happening. Other transform collectors and statistic collectors will still be called.

{
"comment": "Only call message collectors when eth.vlan_outer_id is set",
 
"actions": [
{
"isSet": {
"datafield": "eth.vlan_outer_id",
"false": [
{
"messageCollectorBlock": {
}
}
]
}
}
]
}