In the previous example, we saw how a stack probe can perform complex functions efficiently, and be configured to send summarised statistics or anomalies to VMX-Analysis and provide an overview of system performance.

This example looks at how a probe for order entry protocols might be created in VMX-Capture. It uses configuration that was generated for the Beeks Analytics for Markets (BAM) templated deployment as an example.

Within BAM order entry traffic is notably different from market data for the following reasons:

• The traffic is bidirectional (BAM makes the assumption that market data is unidirectional, from venue to internal systems).

• Although the traffic is summarised as statistics, individual messages are also sent (as Agent Events) to VMX-Analysis for it to perform more complex correlations (and to persist to the database, for querying in order searches).

  • The fact that individual messages are sent as Agent Events also means that there is more emphasis on normalising key message fields with orders than there is with market data. This allows, for example, FIX orders and OUCH orders to be viewed side-by-side in the Item Trace view.

We will examine how these differences are implemented in the configuration in the following sections.

Stack probe configuration for NYSE FIX

We will take the processing of NYSE FIX data as our example configuration.

As mentioned above, order traffic is captured in both directions. The Beeks Analytics for Markets configuration implements that with two separate stack probes:

• nyse_fix_ingress.stack.json describes the stack probe configuration for processing order messages that are received from NYSE.

• nyse_fix_egress.stack.json describes the stack probe configuration for processing order messages that are received to NYSE.

Both of the above files are found in the standard directory for stack probe configuration for a Beeks Analytics for Markets configuration:

<confdir>agent/pmux/<PMUX_NAME>/

Here’s an example of what the start of the ingress.stack.json configuration file looks like:

Stack probes configured per direction and Filter Settings

Defining the probe by the direction of traffic is achieved easily, by switching around source and destination in the BPF filter for the probe:

Stack layer overview

The following layers are present in the stacks:

With the BAM configuration, the IP and Ethernet modules are only required because some of these fields are important for direction of traffic or for labelling traffic. Actual statistics for IP (or indeed TCP) is relevant to order sessions - but these are captured via separate stack probes.

Decoder layer (FIX)

The dec_payload module has some additional configuration in the stack configuration as follows:

The dynamicDatafield performs the following transformations as part of the protocol decode:

• Converts FIX tag 52 (the SendingTime field) and FIX tag 60 (the Transaction Time field) into a timestamp format, and gives them appropriate names for use elsewhere in the configuration (“beeks.payload_timestamp” and “fix.TransactTime”).

• Converts FIX tag 17 into an integer, with the name ‘fix.ExecID’.

Transform collector configuration

The transform collector provides the mapping functions. These ensure that the properties that are passed to VMX-Analysis as Agent Events or statistics are correctly mapped from fields that have been decoded.

entitiesMapper

Unlike for market data, the entitiesMapper can now map internal IP addresses to specific hostnames, if that will help users to understand the data that they are seeing in the Analytics system. Here is an example of that mapping in the entsMapper/nyse_fix_egress.stack.mapper.json file:

protocolNormalizingMapper

The protocolNormalizingMapper for FIX has two sections:

• A datafields section which defines the datafields and their type for the different fields that will be processed in the FIX messages.

• An actions section which details the transformations that the message will go through. These can be conditional on other fields.

Here is an example of where a mapping is made from a particular order type ID in the message to a more human-readable string which can be passed to VMX-Analysis as part of the Agent Event:

Here is an example of where the mapping deals with a field that isn’t present for every message. In order to display these messages consistently using the same data model, the field is populated with a null value if the field doesn’t exist in the original FIX message:

This is an example from the same file of where FIX messages of Message Type 5 (logout messages) are excluded from the App-to-wire wiretime latency calculation:

Stat Collector configuration (vmxgenericaggregatorinputeventconnector, aka GAIE)

The stat collectors specify which statistics to compute for the packets that match the BPF. There are five stat collectors defined for most BAM order processing:

However, unlike the stat collectors for market data all of these stat collectors are vmxgenericaggregatorinputeventconnector modules. The different collectors are just referenced in order to output the statistics to different aggregators.

For more information on the aggregator structure in Beeks Analytics for Markets templated deployments, please see the Beeks Analytics Data Guide.

Message Collector configuration (vmxconnector)

Market Data stack probes did not have any message collectors defined. For orders, we do need to define this collector, as it will have responsibility for sending the Agent Events to VMX-Analysis:

The coll_vmxconnector configuration is defined later in the configuration file:

The datafield_list is a list of all of the fields which are made available to VMX-Analysis. You can reference these fields in your VEx in order to access these fields in VMX-Analysis.

The uid_gen_agent_routing filter_gen is used to ensure that the message is sent to the correct Agent in the VMX-Analysis configuration.

You also have to ensure that aggregator definitions and agent definitions are all set to match the expected fields.