The Beeks Core Data Feed (CDF) provides a live streaming feed of data from VMX-Analysis and VMX-Capture. This allows users to further process analysed metrics, events, and alerts for real-time insight or long-term archival/analysis.

Beeks Analytics Architecture showing CDF locations.

As part of its open architecture, Beeks Analytics offers many different points for you to access analytics output. This flexibility provides Beeks Analytics users with the following benefits:

  • Match the interface to your requirements
    Use a RESTful API for portal or reporting style request/response access to analytics, or use a publish/subscribe streaming interface for consumption of individual messages or ongoing big data system integration.

  • Consume high-volume data as close to the source as possible, with minimal processing overhead
    If you want to consume high-volume data cost-effectively (e.g., near real-time message decodes or notifications of gaps), you’ll need to be able to scale your costs and deployments linearly, as opposed to the wasteful step costs involved in appliance-based licencing models.

The Core Data Feed is the Beeks Analytics publish/subscribe interface.

The rest of this section places the Core Data Feed in the context of the other Beeks Analytics interfaces. The following section contains background on Open Telemetry, so that we can show how the Core Data Feed data streams relate Open Telemetry concepts. If you prefer to consider the CDF in isolation from these contexts, you can jump straight to Core Data Feed Overview and the following sections.

Request/response APIs

Beeks Analytics REST API

The REST API provides an interface to programatically access information from within the Beeks Analytics system. The Analysis Server (part of the VMX-Analysis component of the overall Beeks Analytics architecture) presents this REST API to users.

The REST API is a request/response API.

Although the REST API is presented by the Analysis Server, which is part of the VMX-Analysis architectural layer, the REST API is also used to query data from the VMX-Capture architectural layer.

Clients can deploy appliances, which are exclusively for packet capture and/or streaming of decoded data to their own systems. These systems do not have the overhead of running a full VMX-Analysis instance. In these cases, a thin deployment of VMX-Analysis that provides a more limited REST API and Grafana data source can be installed on the appliance.

For more information on the REST API, see the Beeks Analytics REST API Worked Examples Guide, which also contains links to the Swagger documentation.

VMX-Explorer Grafana Datasource

VMX-Explorer presents two Beeks Analytics Grafana data sources to the user. The advantages of a dedicated Beeks Analytics Grafana data source over direct access to the REST API are as follows:

  • Each data source provides its own query editor, which allows you to easily create, edit, and modify custom queries to access the data that you need.

  • Having a data source allows you to use the Grafana Explore interface to understand the underlying data and to help fast-track the development of custom queries and visualisations.

  • The back-end data source allows you to take advantage of Grafana’s rich alerting rules to create custom alerts.

For more information on the VMX-Explorer Grafana datasources, see the Beeks Analytics User Guide for VMX-Explorer.

Using Publish/Subscribe Core Data Feed vs REST API

So when is it better to use the Core Data Feed instead of the REST API? The Core Data Feed is more suited to interfacing with the VMX-Capture layer directly or for maintaining a streaming list of updating information from the VMX-Analysis server.

The Core Data Feed:

  • is a Publish/Subscribe messaging feed.

  • outputs Agent Events, Intervals, Timeseries, Associations, Alerts, and configuration messages.

  • offers pre-filtering of messages to reduce volume.

  • offers reliable rather than guaranteed messaging, to allow for drops if there are problems consuming.

We will go into the Core Data Feed in more detail, but first of all we'll place it in the wider context of how Beeks brings observability to financial markets.