Correlation sounds easy, and is easy if all Agent Events for the same Item include a pre-defined Correlation ID that is captured by each probe when it sees the message. In practice however, the different parties often use their own custom identifiers for their messages, or each organise the message data in a unique way. In the example below, you can see one item made up of 4 different Agent Events (in this case, one Agent Event per message), each of which has a different identifier.

Diagram showing use of different identifiers by different entities in a Trading system.

identify all Agent Events that belong to an Item, we configure VMX-Analysis to use one of following correlation strategies for each individual Agent. The strategy that’s configured will depend on the attributes available to that Agent.

  • Agent Events have a common Correlation ID
    The easiest way to correlate. Here, a common Correlation ID is provided in every message for the same Item. It’s picked up by the probes and passed to the Agent, which adds it to the Agent Event. VMX-Analysis then uses Correlation ID to group the Agent Events into an Item.

  • Agent Events have a Alternative ID that can be mapped to Correlation ID
    Where some Agents won’t receive a Correlation ID, we can configure chains of Alternative IDs (Alt IDs) to identify the Item, which the Agent then adds to the Agent Event.

    To use this technique, the following must occur:
    A) Every Agent in the Flow maps at least one Capture Attribute (containing an ID) to an Alt ID.
    B) At least one Agent can capture both the Correlation ID and an Alt ID.

How do we create a chain of Alternative IDs, to support more complex correlations?

In the diagram above, the IDs in the messages as follows:

1. Client order ID occurs in both message A and D.
2. Exchange order ID is in both message C and D.
3. Broker order ID is in both message B and C.
3. Correlation ID occurs in message A.

By using the IDs that are in more than one message to link all the IDs, VMX-Analysis creates a chain of Alt IDs, which link to the Correlation ID:

Broker order ID > Exchange order ID > Client order ID > Correlation ID

Alt IDs can be contributed by any Agent and out of temporal sequence

Other Correlation Strategies

Agent Events have common Match Attributes

If no Correlation ID or Alt ID is available in a message, VMX-Analysis can perform fuzzy matching using the Capture Attributes included in the Agent Event.

For example, where an order can be identified by the combination of customer ID, Instrument ID, Quality, and Price, we designate these Capture Attributes as Match Attributes, which instructs VMX-Analysis to use them in correlation.

By default, Beeks Analytics uses a best-match strategy where the Agent Events closest in time is selected where two or more Agent Events share one or more Match Attributes. This strategy can be overridden to meet a specific purpose if needed.

Agent Events are identified by Cross reference beacons

If an Alt ID is provided in a message but there is no direct means of mapping this to the Correlation ID at other points in the Flow, VMX-Analysis can use a cross reference beacon that contains a cross reference from an independent source to map the Alt ID to the Correlation ID and group the Agent Events.

Agent Events are correlated using a Correlation ID assigned by VMX-Analysis

This is called a Tracking ID strategy, and should be used when no Correlation Id is ever provided for Agent Events. Here, VMX-Analysis is responsible for creating a synthetic Tracking ID and assigning it to the Correlation ID.

Ideally, VMX-Analysis should assign the Tracking ID at any of the possible entry points for an item in the Flow. Tracking IDs should not be configured for Agents that will never be entry points in the flow, as although it is possible to make this work, this may have a negative impact on the performance of the correlation engine. In this strategy, VMX-Analysis will assign a given Tracking ID to a message if the message has a specific Capture Attribute and no Correlation ID.

Agent Events are correlated using a custom strategy

Custom Java correlation strategies can be plugged into VMX-Analysis.

Agent Events are correlated using a custom algorithm

VMX-Analysis can be customised to use your own custom algorithms to correlate Agent Events.