Stack Probe performance for Market Data
The following benchmarks measure VMX-Capture stack probe performance for market data:
Sustained Market Data message rate - IP stats only
Sustained Market data message rate - IP stats + gap detection
Sustained Market Data message rate - full BAM config, IP stats + gap detection + wiretime latency
All of these benchmarks are measured in messages per second.
Market data messages were 75 to 200bytes in size, and the market data protocol was UDP-based.The benchmarks were performed using data from the NYSE ARCA BBO feed, but they have been validated against similar tests which were performed against the OPRA feed.
To simplify the measurements for comparison, there was no order traffic in the traffic profile.
What statistics are available to me with an ‘IP stats only’ configuration?
Metric name | Directions | Description |
---|---|---|
Conversations | n/a | The number of IP pairs that are observed at this Visibility Point. |
Packets per second (pkt/s) | Tx or Rx | The rate at which IP data is being observed at the Visibility Point. The rate is calculated and recorded every 10 seconds. |
Bandwidth | Tx or Rx | The rate at which IP packets are being observed at the Visibility Point. Multicast data will only be included in the bandwidth for a particular Internal Entity if the Internal Entity is explicitly listed as a subscriber to that multicast group. For an External Group, the received rate will also include any multicast traffic sent to a multicast group which this entity is recorded to be publishing to. The rate is calculated and recorded every 10 seconds. |
Microburst | Tx or Rx | The rate at which IP data is being sent by this entity, including network processing overhead (network protocol headers, retransmissions, etc). The rate is calculated using a 1 millisecond window, and the maximum rate seen over each 10 second period is measured and recorded. Note that Beeks Analytics is flexible if different microburst calculation measurements are required - 1 millisecond is just the default that is assumed for Beeks Analytics for Markets deployments. |
In the table above, the ‘Directions’ column tells you whether the statistic is unidirectional or is available on a bidirectional basis. A bidirectional statistic for bandwidth for example would tell you the bandwidth you are transmitting (Tx), as well as the bandwidth which you are receiving (Rx).
The above statistics can be reported for all IP packets, or just for UDP packets.
The concept of Internal and External Groups makes configuring your Beeks Analytics for Markets system really easy, and it provides you with instant metrics which are easy to understand. See the Beeks Analytics Data Guide for more information on Internal and External Groups, and Tx and Rx directions.
What statistics are available to me with an ‘IP stats only + gap detection’ configuration?
As above, but you also get a statistic for ‘gaps in sequence numbers per second’. Market data feeds therefore usually include a sequence number to allow the receiving application to identify missing messages, if required. A gap is recorded where we see data is missing. An anomaly is also produced which will precisely identify the exact time of the gap, and the sequence numbers which were affected.
What statistics are available to me with the ‘full BAM configuration’ for market data?
The following statistics are available with the full Beeks Analytics for Markets (BAM) configuration:
Metric name | Description |
---|---|
Conversations | The subset of UDP conversations that can be decoded as market data protocols. |
Messages per second (pkt/s) | The rate at which TCP packets are being observed at the Visibility Point. The rate is calculated and recorded every 10 seconds. |
Bandwidth | The rate at which market data is being observed at the Visibility Point. The rate is calculated and recorded every 10 seconds. |
Microburst | The rate at which IP data is being sent by this entity, including network processing overhead (network protocol headers, retransmissions, etc). The rate is calculated using a 1 millisecond window, and the maximum rate seen over each 10 second period is measured and recorded. Note that Beeks Analytics is flexible if different microburst calculation measurements are required - 1 millisecond is just the default that is assumed for Beeks Analytics for Markets deployments. |
Wiretime Latency (at different percentiles) | App to Wire latency calculation. Market Data latency is the time that the Beeks Analytics calculates between the SendingTime timestamp written within the Market Data message and the time it is observed at the Proximity Cloud Trading Switch (also known as the Wire time). Note that different Market Data protocols may refer to this field using slightly different terminology. The figure can only be as accurate as the time resolution for the SendingTime field. Often this is recorded to the nearest millisecond, whereas the wire time is recorded to the nearest microsecond. As a result, figures of hundreds of microsecond can be considered normal. All latency measurements over a 10 second period are captured, and the (minimum/mean/95th percentile/99th percentile/maximum) value is recorded each 10 second period. |
Out of sequence messages per second | Unlike TCP, there is no indication in the UDP protocol itself that a message has arrived out of order. Market data feeds therefore usually include a sequence number to allow the receiving application to reorder messages sequentially, if required. The out of order statistic tells you when a message has been delivered which is earlier than the latest sequence number observed for a given conversation. This out of order and gap statistics are related - as an example if a gap is observed, and then filled later, you will always see gap_count >= 1 and out_of_order_msg_count >=1. For any given set of messages, missed_msg_count will always be greater than or equal to the gap_count. This metric does not take into account any ‘deliberate retransmission/gapfill/replay’ flags, which may be implemented by some market data feeds. The statistic is calculated and recorded every 10 seconds. |
Gaps in sequence numbers per second | Unlike TCP, there is no indication in the UDP protocol itself that a message has been lost. Market data feeds therefore usually include a sequence number to allow the receiving application to identify missing messages, if required. A gap is recorded where we see data is missing. Note that this statistic does not record how many messages are missed in any given gap. The statistic is calculated and recorded every 10 seconds. |
See Stack Probe performance for FIX messaging for a fuller description of App-to-wire latency.