We saw in the previous section how stat collectors inform which aggregators probe stats are passed to.

More complex aggregation and calculation can also be performed in VMX-Analysis.

VMX-Analysis also supports persistence of individual messages, as well as business flow correlation between messages.

In order for VMX-Analysis to perform these functions, it must receive the individual messages from VMX-Capture as an input.

Individual messages may also be needed by a system other than VMX-Analysis (for example, a client system that will process individual decoded messages, possibly those that have been enriched by VMX-Capture). VMX-Capture can transmit these messages via the Core Data Feed. See the CDF-M section of the Core Data Feed Guide for more information.

The stack probe uses message collectors to inform where decoded ‘message events’ are sent. The vmxconnector module is used to direct these message events to the VMX-Analysis, where they are received as Agent Events and assigned to a particular Flow. The msg_kafka module is used to direct these messages via the Core Data Feed.

Example Message Collector Configuration

In the example below, the messages will be output to a file via capture collector and to VMX-Analysis via vmxconnector.

"msg_collector": [
    {
        "type": "module",
        "value": "capture",
        "id": "capture_id"
    },
    {
        "type": "module",
        "value": "vmxconnector",
        "id": "coll_vmxconnector"
    }
]

The capture collector logs all messages received. It is useful for debugging, but is not normally used for production deployments.

A separate section of the configuration provides details of the datafields within the events that will be logged (in the case of the capture collector), output via the Core Data Feed (in the case of the Kafka Message collector, used for the CDF-M) or passed to VMX-Analysis (in the case of the vmxconnector).

This example is from a vmxconnector, and it shows not just the fields that will be passed to VMX-Analysis, but also the particular Agents that will be targeted (these are used on the VMX-Analysis side to allocate a message to a particular Flow):

    "coll_vmxconnector": {
        "parameters": {
            "blocking": false,
            "buffer_size": "1224",
            "timestamp": "TIMESTAMP",
            "datafield_list": [
                "ip.src_host",
                "ip.dst_host", 
                "ip.src_port", 
                "ip.dst_port", 
                "beeks.correlationKey", 
                "beeks.request_id", 
                "beeks.request_index", 
                "beeks.previous_request_id", 
                "beeks.session_id", 
                "beeks.request_type", 
                "beeks.response_type", 
                "beeks.exchange_order_id", 
                "beeks.intentity", 
                "beeks.extgroup", 
                "beeks.wiretime", 
                "beeks.gateway", 
                "beeks.vp", 
                "beeks.native_msg_type", 
                "beeks.price", 
                "beeks.side", 
                "beeks.quantity", 
                "beeks.tif", 
                "beeks.symbol", 
                "beeks.unsolicited_id"
            ],
            "filter_gen": {
                "type": "module",
                "value": "variable",
                "id": "uid_gen_agent_routing"
            }
        }
    },
    "uid_gen_agent_routing": {
        "parameters": {
            "uid": "{beeks.target_agent:=Request}Request|{beeks.target_agent:=Response}Response|{beeks.target_agent:=Ack_Response}Ack_Response|{beeks.target_agent:=Reject}Reject|{beeks.target_agent:=Unsolicited}Unsolicited",
            "name": "Request"
        }
    }

For more detailed examples of using the vmxconnector to pass messages to VMX-Analysis, see Worked Example: Order Entry Stack Probes in Beeks Analytics for Markets.