Capture Appliance (VMX-Capture)

An appliance running VMX-Capture sits at the heart of Beeks Market Data Packet Services because it provides the foundational capability that everything else depends on: lossless, nanosecond-accurate capture of market data packets.

By combining high-performance FPGA capture cards (e.g. Napatech), precision timestamping (PTP/PPS integration), and robust indexing and storage, the appliance ensures that every packet is captured deterministically and aligned with the exchange’s time source (or with a reliable timesource implemented by the Beeks team and suitable for nanosecond precision). It acts as the authoritative source of truth, giving exchanges confidence that the data they distribute to clients is complete, verifiable, and accurately sequenced.

Moreover, the Capture Appliance is deeply integrated with the broader Beeks Analytics platform — enabling decoded views via mdPublish, structured transformation via mdArchiver, and market data replay services. These other services are explored in later sections of this document.

This central role makes the Beeks Analytics Capture Appliance more than just a recorder — it's the engine of Beeks’ market data transparency and traceability, and the foundation of enabling exchanges to not just manage their service better, but also to monetise the insights that precision-timed packet captures unlock.

mdArchiver

mdArchiver is an optional extra service that can run on the Capture Appliance or on any Linux based compute platform (i.e. it can run on a public cloud instance).

Typically it runs once daily, but it can be scheduled to run more frequently. Where mdArchiver runs daily we assume that PCAPs will be available in the archive from T+1 (i.e. from the start of the the next day after the data is written). However, in rare times of component failure T+2 or T+3 retrieval will be acceptable, until T+1 service is fully restored.

It performs the following post-processing tasks to manipulate PCAP files. Its output is PCAP files that have been processed. The standard file output is PCAP rather than PCAPNG (PCAPs are more efficient at large volumes than the newer PCAPNG format, and the features of the PCAPNG format are generally not required for market data PCAPs).

The output of these post-processing tasks can be local (if the exchange opts for just the mdArchiver solution) or can be…

• synced to external object-based storage

• sent to Beeks scaled storage (if the exchange opts for the mdStore service).

Beeks provides various ways to access the curated PCAP files from the appliance – query and pull via REST API, RSYNC, SFTP, HTTPS.

Alternatively, mdArchiver can be configured to push the PCAP files via RSYNC, NFS or some object storage infrastructure.

Processing the capture files using mdArchiver makes the files easier for clients to manage and recognise on a filesystem.

The following processing can be performed by mdArchiver:

• Reconstructing packet captures to specific time ranges (e.g. 1 hour long, rounded hours, such as 01:00:00 - 02:00:00) and/or regular file sizes.

• Splitting packet capture files so that the files are manageable by clients and can, if necessary, be permissioned separately. For example:

  • 1 file per market

  • 1 file per side

    Split by line/side/channel

  • data channels not retransmission channels.

• De-duplication (by content to provide line arbitration based on first arriving packet, or between two resilient capture points, or between two different capture appliances). Note that in the case where de-duplication is required between captures from two servers, the two jobs can be co-ordinated on the two capture appliances. No third capture appliance is required.

  • With the use of resilient packet capture appliances, a daily post-processing will ensure that the archive contains the superset of all A feed and all B feed packets in the event of any faults at any one capture appliance. The archive will consist of separate A and B archives.

• Anonymisation. Depending on where the capture is taken within the exchange infrastructure, the packet capture may contain information which clients can use to understand system internals that an exchange may not want to make public. For example, the manufacturer of network switches can be understood by understanding the hardware addresses contained in the packet capture. Beeks can replace these sensitive details with alternative details whilst retaining the integrity of the packet checksums.

  • This anonymisation can also include other packet information such as VLAN, IP address or port replacement in the packet capture files.

• Digital Fingerprinting. The act of rewriting the hardware addresses can also act as Copyright Trapping. it will ensure that packet captures that the exchange redistributes can be identified as originating from the exchange (as opposed to being captured by another data vendor on the client side). This can be used as evidence if the exchange suspects that other vendors or clients are redistributing these packet captures without the exchange’s permission.

• mdArchiver can also operate on TCP-based market data.

• All mdArchiver actions ensure accurate segmentation of packet data at defined boundaries. Files will not contain partial packets.

The mdArchiver service is performed after packet captures are initially written to disk, so that the exchange will still have access to the original unmodified captures if needed for troubleshooting or regulatory compliance reasons.

Market Data Replay Services (mdPlay and mdStream)

Market Data Replay is an optional part of Beeks Market Data Packet Services which uses mdPlay technology to provide high fidelity market replay services for exchanges.

This market data replay is valuable for exchanges because it supports:

• Faster participant onboarding & certification. The exchange can schedule a standard T+1 (or fixed-day) replay; members just subscribe to the multicast and run their conformance packs. No per-firm harnesses or bespoke feeds → shorter time-to-go-live.

• Cheaper onboarding with fewer support calls. Centrally scheduled replays (with exchange-set speed and content) mean fewer custom test cycles and less hand-holding by market ops—your team runs one replay that serves many participants at once.

• Smoother post-change assurance. After software/network changes, run a deterministic replay so members can self-verify before market open—reduces back-and-forth and speeds wider readiness sign-off.

• Allows exchanges to increase wallet share and offer a fuller ecosystem for participants. Combined with Exchange Cloud, allows exchanges to offer a value-add service that deepens engagement and can be priced as part of onboarding/certification packages or as a premium test environment.

mdPlay can be offered either as an exchange-hosted offering or as a client-hosted offering. Where the client takes Exchange Cloud, the lightweight mdStream service can also be offered - this is a variant of mdPlay where preselected samples of market data (e.g. the previous trading day’s market data) are replayed regularly over an agreed multicast channel.

Market Data Storage and Distribution Services (mdVault and mdPortal)

mdVault is the Beeks tiered storage solution for market data packet captures.

mdVault provides exchanges and market participants with secure, scalable storage for PCAP files, supporting both short-term and long-term retention needs. It includes options for redundant sites, WORM (Write Once, Read Many) compliance, and integration with direct client access protocols (e.g., sftp, rsync). It forms part of Beeks Market Data Packet Capture Services architecture, often paired with mdPortal for user-facing access.

mdPortal is a web-based client access interface for historical market data packet captures.

mdPortal offers a secure, branded portal for browsing, searching, and downloading PCAP files. It supports role-based access control, granular permissions, and integration with APIs, SFTP, and billing systems. Designed for exchanges offering historical packet data as a service, mdPortal simplifies client access while preserving security and contractual compliance.

mdPublish

mdPublish uses VMX-Capture decoding technology to allow clients to process packet captures into fully decoded, protocol-aware messages or statistics - supporting formats such as ITCH, FIX/FAST, and exchange-specific protocols. 

Exchanges using mdPublish can stream this decoded data directly into high-performance data platforms. In production environments, Beeks has achieved sustained throughput of over 9 million messages per second writing directly into QuestDB, a time-series database optimised for real-time financial data. QuestDB’s high-ingest rate and SQL-style querying make it well-suited for real-time applications and downstream data integration.

For full flexibility, mdPublish supports output via the Core Data Feed (CDF) — Beeks’ high-throughput, open data stream format designed for downstream processing at scale. CDF integrates natively with Kafka, and recent testing has validated compatibility with Redpanda, a high-performance, Kafka-compatible streaming platform optimised for low-latency and high-throughput use cases. This enables exchanges to publish decoded and time-aligned packet data directly into big data platforms or client-facing analytics stacks, with strong support for horizontal scaling and consumption by multiple stakeholders.