Beeks Analytics has historically had three persistent data repositories for data derived from the incoming wire data that is being captured and analysed:
Capture Repository (always coexists with VMX-Capture). The capture store is the collection of packet capture files written by capture probes.
Item store (always coexists with VMX-Analysis). A database that persists items. The VMX-Analysis process writes events, intervals and associations to this store. The data store uses MariaDB.
Timeseries database (has been co-existing with VMX-Analysis). A timeseries database that stores the aggregated timeseries data produced by VMX-Capture Statistics Collectors or as outputs from VMX-Analysis aggregators).
With version 14 of Beeks Analytics, Beeks has introduced a new QuestDB database that will eventually host all of the timeseries datastores used in the Beeks Analytics product.
For more information about why Beeks chose QuestDB as the database platform for Beeks Analytics, and the benefits that the use of QuestDB brings to our customers, see Appendix C: Why Beeks Analytics chose QuestDB as our database engine of choice to unlock the value of your wire data .
The installation location of QuestDB is usually on VMX-Capture, but it can alternatively be installed on VMX-Analysis for some Beeks Analytics Enterprise deployments:
QuestDB can optionally be installed in QuestDB enterprise mode. This opens up many more deployment possibilities e.g. QuestDB running on both VMX-Capture and VMX-Analysis, with data replication between the two deployments. This mode can also support QuestDB running as a clustered database.
Version 14 starts the migration of the data stores to QuestDB by introducing the following configuration options:
VMX-Capture Statistics Collectors can be revised to output directly to QuestDB in addition to their existing output to VMX-Analysis.
VMX-Capture Message Collectors can be revised to output directly to QuestDB in addition to their existing output to VMX-Analysis.
Until the Item store is fully replaced, the CDF-I feeds item information into QuestDB. This allows these items (and timeseries data derived from them) to be queried using CDF-Q and to be used within Market Edge Intelligence.
Our migration to QuestDB will be phased. Post version 14, VMX-Capture will start to use QuestDB stores for Timeseries data and Messages:
In the final phase, which completes this migration, all packet capture metadata, pre-aggregated statistics, items, intervals and messages will be written directly to QuestDB or derived at query-time from the information stored in QuestDB. Given the now reduced scope of VMX-Analysis by this stage, there will no longer be separate processes for VMX-Analysis and VMX-Prism - these processes will merge, to further simplify the overall architecture.
