The goals behind the Beeks Analytics architecture are as follows:
To operate with an open architecture, which provides the benefit of multiple high volume integration points with your organisation’s own systems.
Allowing your organisation to fully own the data produced by the Analytics system.
Enables the organisation to run the Analytics software on their own hardware.
To allow open scaling, which allows the capacity of the system to handle load to scale up with commercially available server hardware improvements.
To be modular and licenced to support open consumption.
This means that, for example, if you only need the high performing VMX-Capture layer and don’t require the in-depth analytics that the VMX-Analysis layer provides, we’ll ensure the software is licensed and priced accordingly.
It also means that we’re transparent about the drivers of our pricing - core count required for the analysis, which we make clear in our transparent performance metrics.
To support open data by ensuring that you have full access to, and control over, all monitoring data generated by the platform.
Rather than locking data into proprietary interfaces, our architecture emphasises direct and flexible data accessibility.
Our Advanced Configurable Decoder™ (ACD) ensures an agile, lower-cost way to monitor internal messaging data on the wire.
The Kafka-based Core Data Feed provides robust scalability and fault tolerance even under high message rates to provide lossless, low latency data streams. Use of Kafka frees you from vendor lock-in, since Kafka connectors exist for virtually every modern data processing framework, making it straightforward to combine Beeks Analytics output with your broader enterprise data.
For more information about the Beeks Analytics architecture as a whole, see the Analytics Concepts Guide.
VMX-Capture
In Beeks Analytics, VMX-Capture is the layer responsible for data capture and decoding.
VMX-Capture is a set of software components that are specifically designed for high-performance packet capture, decoding, high-volume real-time analysis. It can produce a message output, statistical calculations based on the data, or an output of market data anomalies.
VMX-Capture is most commonly deployed on a Beeks Analytics appliance. However, it can also be installed on the client’s own hardware or on a cloud compute instance.
The simplest task that you can perform with Beeks Analytics is to use it as a packet capture appliance - that is, writing packets to disk and making them available for search and retrieval. This ability to capture high volumes of packets without dropping data acts as the building block on which higher-level services are built.
In Beeks Analytics, VMX-Capture probes capture huge quantities of data at Visibility Points in the network. The data is then mapped to a common format and used in calculations that output easily readable statistics (“stats”) that tell our customers about their network performance.
These statistics are aggregated into hierarchies, so that you can quickly see groupings of latencies whilst also having access to the granular information that you need. For example, aggregations may be provided for:
Overall latency per market data venue, across multiple different feeds
Overall latency for a client or venue, across multiple individual FIX sessions
You might choose to display these higher level aggregations on your dashboard or alerting, but you can drill-down in Beeks Analytics to view which individual feed/FIX session is contributing to the high figure.
In Beeks Analytics, aggregations are performed by both VMX-Capture and VMX-Analysis.
VMX-Analysis
In Beeks Analytics, VMX-Analysis is the layer that analyses the data from VMX-Capture to produce reports and views, and passes aggregated data and individual business items to VMX-Explorer for visualisation.
VMX-Analysis also presents a REST API to users to provide programmatic access to Beeks Analytics data.
Note that:
The REST API is a request/response API. Beeks Analytics also makes available a public/subscribe API called the Core Data Feed which can be more suited to interfacing with VMX-Capture layer directly, or for maintaining a streaming list of updating information from the VMX-Analysis server. The Core Data Feed is outside of the scope of the current document.
Although the REST API is presented by the Analysis Server (part of the VMX-Analysis architectural layer), the REST API is also used to query data from the VMX-Capture architectural layer.
Clients can deploy appliances which are exclusively for packet capture and/or streaming of decoded data to their own systems. These systems do not have the overhead of running a full VMX-Analysis instance running on them. In these cases, a thin deployment of VMX-Analysis which provides a more limited REST API and Grafana data source can be installed on the appliance.
VMX-Analysis can implement a rich item model which can allow multiple different messages to be correlated together. For more information about Items, see the Analytics Concepts Guide.
VMX-Explorer
VMX-Explorer is a web-based user interface that displays analytics views via Grafana. This is a highly configurable tool that enables end users to build views and filter data to provide rich insights into your network traffic and latencies.
VMX-Explorer is fed by the Beeks Analytics REST API.
Beeks Analytics provides pre-configured Grafana dashboards and an easy-to-use Grafana data source to assist with accessing and navigating the analytics data, and to act as guidance for a user who wants to set up their own dashboards.
VMX-Capture and VMX-Analysis - on a single appliance or scaled out for distributed capture
Another key element of the open architecture of Beeks Analytics is the scalability of VMX-Capture and VMX-Analysis. These components can both be deployed on the same hardware, or for higher scale deployments, VMX-Capture can run separately from VMX-Analysis. This allows, for example, VMX-Capture to be scaled locally (close to where the data is captured), with just a subset of the statistics being passed to a central VMX-Analysis server for analysis. VMX-Analysis will then only retrieve full packet data from VMX-Capture on request when a particular timeframe is investigated, reducing the amount of hardware that needs to deployed in order to manage high traffic volumes. This leads to the distributed capture paradigm allowing flexibility in building a capture and aggregation layer. Instead, clients can choose to have multiple capture points – while handling the analysis locally.