Monitoring Market Data using Beeks Analytics for Markets
In any trading system, it is vital to understand that your incoming market data is of high quality. Beeks Analytics for Markets provides this reassurance.
Market data is particularly prone to burstiness, and can be hard to process given its high volumes. Beeks Analytics allows you to quantify the amount of bandwidth and burstiness of your market data, to assist with future growth planning. It also helps you to understand how current volumes relates to overall historic trends. We have already seen in the section on Monitoring IP and UDP traffic using Beeks Analytics for Markets which metrics (such as microburst) Beeks Analytics provides to assist with market data traffic profiling.
Beeks Analytics also has an optional component called mdPlay which allows current market data traffic captures to be stored longer term, and replayed into a test system. This can be useful for recreating production issues. mdPlay also provides the capability to modify the traffic load so that testing can be performed at multiples of current production traffic levels. See the Analytics Concept Guide or speak to your Beeks representative for more information about mdPlay.
Market data in fast-moving markets often uses multicast protocols, delivered over UDP. It is important to understand when market data messages are not being delivered. Beeks Analytics does this using gap detection.
Market data protocols are typically optimised to ensure that they have the minimum amount of overhead in each message. This reduces bandwidth requirements and helps to ensure lowest latency delivery of the pricing updates. However, it can make it difficult as a result to track which network traffic is associated with which market. Beeks Analytics for Markets supports this by providing up-to-date market mappings alongside its list of decoders. This allows support teams to spend their time focusing on the performance of their market data.
For full details of the types of traffic for which Beeks Analytics has decoders and mappings, see the Beeks Analytics Decoder Information document, and in particular the list of markets for which mappings have been created. Note that even if a particular market is missing from that document, as long as it is a public protocol with an available message specification, the Beeks Analytics team will be able to develop a suitable decoder for it. Beeks Analytics also has unique sequence detection features which allow gap detection to be provided even for markets that do not have an official decoder available for them.
When market data is delivered over TCP, Beeks Analytics will monitor this in the same way that it does for Trading traffic. See Monitoring Trading traffic using Beeks Analytics for Markets and Monitoring TCP traffic using Beeks Analytics for Markets for details.
The table below describes the timeseries metrics for Market Data protocols in Aggregators for Beeks Analytics for Markets. The direction is Rx for all Market Data metrics.
The Beeks Analytics for Markets templated deployment makes the assumption that market data is published by an External Group. If you have a requirement to analyse market data published by an Internal Entity, see the Beeks Analytics Configuration Guide or contact your Beeks account representative for guidance.
Metric name | Description | Displayed in Beeks Portal? |
|---|---|---|
Conversations | The subset of UDP conversations that can be decoded as market data protocols. | No |
Messages per second (pkt/s) | The rate at which TCP packets are being observed at the Visibility Point. The rate is calculated and recorded every 10 seconds. | Yes |
Bandwidth | The rate at which market data is being observed at the Visibility Point. The rate is calculated and recorded every 10 seconds. | No |
Microburst | The rate at which IP data is being sent by this entity, including network processing overhead (network protocol headers, retransmissions, etc). The rate is calculated using a 1 millisecond window, and the maximum rate seen over each 10 second period is measured and recorded. Note that Beeks Analytics is flexible if different microburst calculation measurements are required - 1 millisecond is just the default that is assumed for Beeks Analytics for Markets deployments. | No |
Wiretime Latency (at different percentiles) | App to Wire latency calculation. Market Data latency is the time that the Beeks Analytics calculates between the SendingTime timestamp written within the Market Data message and the time it is observed at the Proximity Cloud Trading Switch (also known as the Wire time). Note that different Market Data protocols may refer to this field using slightly different terminology. The figure can only be as accurate as the time resolution for the SendingTime field. Often this is recorded to the nearest millisecond, whereas the wire time is recorded to the nearest microsecond. As a result, figures of hundreds of microsecond can be considered normal. All latency measurements over a 10 second period are captured, and the (minimum/mean/95th percentile/99th percentile/maximum) value is recorded each 10 second period. | Yes |
Out of sequence messages per second | Unlike TCP, there is no indication in the UDP protocol itself that a message has arrived out of order. Market data feeds therefore usually include a sequence number to allow the receiving application to reorder messages sequentially, if required. The out of order statistic tells you when a message has been delivered which is earlier than the latest sequence number observed for a given conversation. This out of order and gap statistics are related - as an example if a gap is observed, and then filled later, you will always see gap_count >= 1 and out_of_order_msg_count >=1. For any given set of messages, missed_msg_count will always be greater than or equal to the gap_count. This metric does not take into account any ‘deliberate retransmission/gapfill/replay’ flags, which may be implemented by some market data feeds. The statistic is calculated and recorded every 10 seconds. | No |
Gaps in sequence numbers per second | Unlike TCP, there is no indication in the UDP protocol itself that a message has been lost. Market data feeds therefore usually include a sequence number to allow the receiving application to identify missing messages, if required. A gap is recorded where we see data is missing. Note that this statistic does not record how many messages are missed in any given gap. The statistic is calculated and recorded every 10 seconds. | Yes |
These market data metrics are available in the MD_Stats aggregator. The Beeks Portal will display market data metrics per External Group. Breaking down MD_Stats per Internal Entity is not possible in the Beeks Analytics for Markets templated deployment.
Dashboards for viewing Market Data Metrics
Within VMX-Explorer for a Beeks Analytics for Markets deployment, the MD_Stats Aggregator is most commonly viewed within the Market Data Stats by VP dashboard, which shows both gaps and wiretime. This dashboard and associated navigation is described in the User Guide for VMX-Explorer.
Other useful market data dashboards include:
Market Data Gap Stats - focuses on gaps, and their associated alerts and anomaly details (i.e. the specific sequence numbers and feeds for which the gaps occurred).
Side Comparison by Feed - showing A vs B feed comparison.
Feed Comparison and Feed Comparison with Extended Variables - shows side by side comparison of different feeds.