This configuration example shows how a stack probe is configured to receive market data. Calculations (including when market data gaps occur) are performed in the VMX-Capture stack probes, and are then presented to VMX-Analysis as pre-aggregated statistics.

Mappings are performed so that market data is labelled for the correct channel, feed, and side (A or B).

The example also covers anomalies, which provide extra information about the gaps e.g. the exact sequence numbers that are affected by particular issues. Anomalies are specific to market data gaps.

Stack Probe Configuration for NASDAQ Totalview ITCH

We will take the processing of NASDAQ Totalview ITCH as our example configuration.

The probe setup is simple because traffic is only captured in a single direction. This simplification can be achieved because the Beeks Analytics for Markets configuration only captures market data that is incoming FROM a venue (as opposed to market data that is published and consumed internally, or market data that is published for external consumers).

The extracts presented here would be found in the following file, in a typical BAM deployment:

The stack.json file is laid out using the JSON standard that is common for all stack probes. This is the start of the stack.json file on an example Beeks Analytics for Markets deployment:

In the rest of this worked example, we’ll step through each of these sections in turn.

Stack probes configured per port and Filter Settings

PORT_NAME is used so that the system can be configured to provide different statistics depending on which port is receiving particular market data streams. The stack probe configurations are identical for the different ports, with the following exceptions:

• The probe name has to be unique.

• The BPF traffic filter needs to be able to uniquely identify the traffic for the different ports.
  In our example:

Note that the gap detection exclusion filter and feed mapping parts of the configuration also use IP addresses and ports, so these files also differ between the Port1 and Port2 probe configurations.

Stack layer overview

The following layers are present in the stack:

Premapper layer

A premapper is used because there are only certain channels (defined by the UDP port) that we want to calculate gaps on. It wouldn’t be appropriate to include these in the BPF filter for the stack probe because this would prevent us from calculating ethernet or IP statistics for these other channels. The premapper configuration in the stack probe references a separate filter file. The gap detection exclusion filter file for Port1 is as follows:

Decoder layer (ACD)

The ACD decoder layer is used to decode the actual market data fields. It references the standard ACD decoder for ITCH5, which is provided in a central store as part of the VMX-Capture deployment:

Here is an abbreviated version of the itch5.acd.json file. You can see how it defines the fields that are processed for each message type:

For more details on ACD configuration for different fields, see the Beeks Analytics Decoder Information document.

Transform collector configuration

The transform collector provides the mapper functions. These ensure that the properties that are passed to VMX-Analysis as anomalies (or fed into the pre-aggregator as statistics) are correctly mapped from fields that have been decoded.

Let’s look at the function of each of these mappers in turn.

mdFeedMapper

The mdFeedMapper is used for all market data stack probes in the Beeks Analytics for Markets template. However, there is a different json definition for each stack probe. The definition file (which for BAM can be found in the <confdir>/agent/pmux/<pmux_name>/mdMapper directory) maps IPs and ports to human-readable line, channel, side and site variables. These variables will be used higher up the stack processing. Here is an example of an extract from a Port1_nasdaq_totalview_itch_50.stack.mapper.json configuration file:

The datafields object lists the new fields that are being set.

internalEntitiesMapper

An entities mapper is used for all stack probes in the Beeks Analytics for Markets template. Whereas the mdFeedMapper maps the entities that are public for the external market data feeds that are being monitored, the internalEntities mapper maps the internal IP addresses.

Given that market data is published to a multicast address, and the IP address of the consumer is not seen in the packets, the standard mapping here for all market data probes is a placeholder, which would allow, for example, the static mapping of a particular multicast group to defined consumers. This is what the placeholder mapping looks like:

For a fuller example of the internal entity mapping used for Beeks Analytics for Markets, see the Worked Example: Order Entry Stack Probes in Beeks Analytics for Markets worked example.

protocolNormalizingMapper

The protocolNormalizingMapper is used for both market data and order stack probes in the Beeks Analytics for Markets template. Note that it is only used for stack probes that decode messages - for the TCP decoders (for example), you will only find the entities mapper.

The protocolNormalizingMapper is often shared between different market data protocols. For example, many of the multicast feeds used for BAM reference the <confdir>/agent/global/mapper/acd.stack.mapper.json standard ACD mapper for the protocol normalisation.

This mapper shows the simplicity of the stack configuration when combined with ACD protocol decoding - an 8 line JSON file (excluding parentheses!) calculates App-to-Wire latency for all multicast market data feeds that use the ACD decoder:

For a more complex example of a protocol normalising mapper, see the example of the FIX decoder mapper in Worked Example: Order Entry Stack Probes in Beeks Analytics for Markets .

Stat Collector configuration

The stat collectors specify which statistics to compute for the packets that match the BPF. There are four stat collectors defined for most BAM market data processing:

You will see that 3 of these stat collector modules are of the same type of module - the vmxgenericaggregatorinputeventconnector type (this is sometimes referred as the GAIE connector). One is of a different type - the vmxanomalyconnector.

Generic Aggregator Input Event Connectors (vmxgenericaggregatorinputeventconnector, aka GAIE) for Market Data - Example output to a Preagg

In BAM we have a different VMX GAIE Connector stat_collector module for each aggregator that we want to publish to.

This is an example for the ipStatsExtgrp aggregator definition. The metadata about the aggregator (how often it pushes its statistics etc) is defined in the stack probe configuration:

The agg.json file then defines the structure of the aggregator:

The nodePaths section needs to provide config for breakdown of every level in the aggregator.

The columns section specifies the stats that feed into the Aggregator; these are then used by the Aggregator definitions in VMX-Analysis.

In Beeks Analytics for Markets, multiple probes write to this pre-agg - all of the market data protocol probes, plus the TCP_ingress.stack.json probe (for all of the different Visibility Points). This shows how multiple different probes can consolidate their statistics in a single aggregator structure.

The leaf_nodes object in the preagg configuration file defines the aggregator levels, and the field_sets and field_definitions define the columns for which statistics will be calculated.

Some of the values for the leaf nodes are static values which are defined in the tables section of the stack probe configuration, e.g. in our Nasdaq ITCH example:

The other values (e.g. ip.packets, ip_conversations) are passed up from the lower protocol levels of the stack probe.

Generic Aggregator Input Event Connectors (vmxgenericaggregatorinputeventconnector, aka GAIE) for Market Data - Example calculations

The mdStats module is also a useful example, because it shows how the preagg definition can specify the particular calculations that can be performed on the data. This is a typical MD_stats.stack.agg.json file for a BAM deployment. Note again how this same preagg structure is shared by all of the market data stack probes. This is what the agg.json configuration looks like:

Note how the statistic keyword means that that field is passed directly (without further calculations) as a statistic to the aggregator, whereas you can also use the minimum, mean, maximum and percentile functions to produce be more specific about how the data provided by the stack will be summarised.

Passing market data gap anomalies to VMX-Analysis (vmxanomalyconnector)

An Anomaly is a special type of event that is passed to VMX-Analysis. You can almost think of them as being halfway between the pre-aggregated statistics (which are described above) and the agent events (which we describe in the next worked example). They allow grouped anomalies to be collected by VMX-Analysis into alerts.

The VMX Anomaly Connector has the following parameters set for it in the stack probe config file:

SEQBRK is a specific type of alert rule, configured in VMX-Analysis.

It references a gap writer configuration, which defines the fields which will be passed to the anomalies: