Mappers can be used for mapping values, performing basic arithmetic and conditional logic against fields and their values. You can enrich and transform the data using this layer.

Mappers can be implemented at two different points in the stack probe:

A mapper can be configured as a protocol layer. This is useful because the mapper only needs to process the information passed to it by the lower stack level - it doesn’t need to process message fields which the higher stack levels are working with. It can also ‘intercept’ messages before they reach the higher stack levels. A mapper configured as part of the protocol configuration is known as a premapper. You can see a premapper in action as part of the Market Data worked example for BAM.
A mapper can be configured at the collector stage of the probe by configuring a Transform Collector. A Transform Collector operates on the full data that has been passed up from the protocol decodes.

Here is an example of two mappers configured as part of a transform collector:

  "transform_collector": [
      {
          "type": "module",
          "value": "mapper",
          "id": "mdFeedMapper"
      },
      {
          "type": "module",
          "value": "mapper",
          "id": "protocolNormalizingMapper"
      }
  ],

You can have multiple modules within a transform_collector and they execute sequentially.

Transforming data as it is decoded using high-performance and horizontally scalable probes is a powerful feature that allows data enrichment tasks to be performed as traffic is processed.

Example Mapping Configuration

In this example, the mdFeedMapper might reference a JSON file with the following mapping commands, to map the IP address and port for the feed to something more human-readable:

 "mapping": [
                    {
                        "key": {
                            "ip.dst_host": "233.54.12.111",
                            "ip.dst_port": "26477"
                        },
                        "actions": [
                            {
                                "assign": {
                                    "beeks.line": "nasdaq_totalview_itch_50_New_York_A",
                                    "beeks.channel": "nasdaq_totalview_itch_50_New_York_1_data",
                                    "beeks.side": "A",
                                    "beeks.site": "New_York"
                                }
                            }
                        ]
                    },
                    {
                        "key": {
                            "ip.dst_host": "233.49.196.111",
                            "ip.dst_port": "26477"
                        },
                        "actions": [
                            {
                                "assign": {
                                    "beeks.line": "nasdaq_totalview_itch_50_New_York_B",
                                    "beeks.channel": "nasdaq_totalview_itch_50_New_York_1_data",
                                    "beeks.side": "B",
                                    "beeks.site": "New_York"
                                }
                            }
                        ]
                    }
                ]

We will see more examples in Worked Example: Market Data Gap Detection in Beeks Analytics for Markets (BAM) of how transform collectors are particularly powerful when it comes to analysing information about market data.

Other Data Transformation functions

The term ‘mapping’ suggests that the only transformation function possible is the mapping of values (e.g. via a translation table). This is one function that mappers support, but mappers can also be used for, for example:

data enrichment (adding fields)
data update (editing fields)
simple arithmetic (performing subtraction on timestamps to compute App-to-Wire wiretime latency)

See the protocolNormalizingMapper example in Worked Example: Market Data Gap Detection in Beeks Analytics for Markets (BAM) for examples of data transformation functions, including App-to-Wire wiretime latency calculation.