PMUX Configuration
A single PMUX can run many different probes. However, it is good practice to separate capture probes and stack probes into separate PMUXs. This ensures that, if something goes wrong with a decode, you have an independent capture from the other PMUX.
These are the PMUX processes that you’d see running in a typical simple system:
[drake]$ vmxadmin vmx-pmux list
Env: drake
----
vmx-pmux VP
vmx-pmux capVP
The naming of VP and capVP is arbitrary - the definition of the probes defines the overall function of the PMUX. VP is short for Visibility Point.
Start by defining the list of PMUXs that will run by editing <confdir>/vmx-servers.cfg
.
For example, to run the above DRAKE_VP PMUX on the valentine01 server, add the following to this configuration file:
valentine01 vmx-pmux DRAKE_VP
Then configure each PMUX in bindir/env-instances.sh
.
For example:
export KALEID_DRAKE_VP_PATH=
"/opt/tsa/bin/nt3_packet_decoder"
export KALEID_DRAKE_VP_XMLRPC_PORT=
8214
export KALEID_DRAKE_VP_ARGS=
"-s 6"
export KALEID_DRAKE_VP_ARGS=
"--max_packet_size_bytes 131072 --max_batch_size 1000 –-pcap_buffer_size 100000"
export KALEID_DRAKE_VP_MEMORY=5g
export KALEID_DRAKE_VP_LOGGING_PROPERTIES_FILE=file:
//${VMX_INSTALL_HOME}/server/config/log4j-kaleid.xml
export KALEID_DRAKE_VP_KALEID_OVERRIDES_FILE=file:
//${VMX_INSTALL_HOME}/server/config/kaleid.properties
export KALEID_DRAKE_VP_PROJECT_ROOT=${VMX_INSTALL_HOME}/server
export KALEID_DRAKE_VP_NIC_ADDRESS=
10.10
.
20.51
export KALEID_DRAKE_VP_JMX_PORT=
7514
export KALEID_DRAKE_VP_SERVER_BROKER_PORT=
3616
export KALEID_DRAKE_VP_LOCATION=NY
export KALEID_DRAKE_VP_CAPTURE_PROBES_EXCLUDE=
"adhoc debug"
export KALEID_DRAKE_VP_STACK_PROBES_EXCLUDE=
"adhoc debug"
export KALEID_DRAKE_VP_INSTANCE_OPTIONS=
"-Dvmx.partitions=${VMX_PARTITIONS:-0}"
In the above example DRAKE_VP
is the name of the PMUX that is being configured.
Kaleid is the name for the communication mechanism that allows statistics and Agent Events to be passed from VMX-Capture to VMX-Analysis. It also acts as the wrapper around the stack probes and capture probes to allow them to be managed by the vmxadmin command. Settings that mention ‘Kaleid’ are often either used for both VMX-Capture and VMX-Analysis configuration, or are used for the configuration of the direct communication between these components.
The -s 6
argument for KALEID_DRAKE_VP_ARGS instructs the PMUX to process the traffic from Napatech stream 6. See VMX-Capture Capture Card Configuration for more information on Napatech stream information.
PMUX Types - File-reading versus Napatech
There are multiple types of PMUX. The type is defined by the value of the KALEID_<INSTANCE ID>_VP_PATH
environment variable.
The two main types that we will consider in this document are the Napatech PMUX and the file-reading PMUX.
The advantage of the Napatech PMUX is that the PMUX reads directly from the card, so it doesn’t need the data to be written to disk in order to process it. The downside of this approach is that the PMUX (and any probes within it) has to be able to keep up with the volume of data sent to it - there is no buffering beyond the deep hardware buffers that the Napatech card provides. This will be fine for a capture PMUX but if your stack probes offer a lot of functionality, they may not keep up with particularly high intensity bursts. For particularly bursty financial markets feeding into a particularly complicated stack probe, we usually recommend a file-reading PMUX.
The Napatech PMUX reads data directly from a Napatech card.
Here is an example env-instances.sh configuration for a Napatech PMUX:
KALEID_<instance_id>_PATH=/opt/tsa/bin/nt3_packet_decoder
KALEID_<instance_id>_XMLRPC_PORT=<xmlrpc port number>
KALEID_<instance_id>_ARGS=”-s
1
”
The ‘-s 1
’ argument tells the PMUX to process packets directly from Napatech stream 1 (see VMX-Capture Capture Card Configuration for details of Napatech stream configuration).
The file-reading PMUX reads data from a PCAP file in a defined read_directory. Here is an example env-instances.sh configuration for a file-reading PMUX:
KALEID_<instance_id>_PATH=/opt/tsa/bin/file_packet_decoder
KALEID_<instance_id>_XMLRPC_PORT=<xmlrpc port number>
KALEID_<instance_id>_ARGS=”--read_directory /data/capture --read_filename_prefix inet --inotify --hardware_decompression”
Normally, the PCAP files would be written there by a Napatech PMUX. However, the file-reading PMUX can also process PCAPs written by other capture processes, or by other capture cards.
Hardware decompression is only possible if a hardware compression card is installed in the appliance. The --hardware_decompression
option for the file-reading PMUX will check which hardware compression cards are available and will choose the appropriate card to use. See VMX-Capture Compression Card Configuration for more details of the compression cards.
It is possible to use the read parameters to set up the file-reading PMUX and instruct it to read the files at regular intervals. However, we recommend the standard configuration, which is to use the ipolling mechanism. The ipolling mechanism prompts the operating system to notify the PMUX as soon as a new file becomes available.
So, if we take our example system above:
[drake]$ vmxadmin vmx-pmux list
Env: drake
----
vmx-pmux VP
vmx-pmux capVP
You would expect to see (as a minimum) the following parameters in the env-instances.sh
file:
KALEID_capVP_PATH=/opt/tsa/bin/nt3_packet_decoder
KALEID_capVP_ARGS=”-s
1
”
KALEID_VP_PATH=/opt/tsa/bin/file_packet_decoder
KALEID_VP_ARGS=”--read_directory /data/capture --read_filename_prefix inet --inotify --hardware_decompression”
This configures the capVP PMUX as a Napatech PMUX (consuming packets from Napatech stream 1) and VP as a file-reading PMUX. More detailed configuration is then performed at the Probe level.
Configuring which probes a PMUX will load
Once a PMUX starts, it loads probes that will perform particular tasks. The PMUX determines which probes will be loaded by checking an exclude list and include list in the env-instances.sh files.
The table below details how to create the exclude and include lists in env-instances.sh.
Parameter Type | Checkpoint for Capture Probes | Checkpoint for Stack Probes |
---|---|---|
Exclude |
Specifies the list of probe names or patterns of probe names to prevent from being automatically loaded. If empty or not set, no probes will be excluded from loading |
Specifies the list of probe names or patterns of probe names to prevent from being automatically loaded. If empty or not set, no probes will be excluded from loading |
Include |
If empty or not set, all *.capture.config will be loaded |
If empty or not set, all *.stack.json will be loaded |
The following formats are supported for the list of probes:
“
probe_name1
” means probe_name1.stack.json will be loaded (or excluded)“
probe_name1 probe_name2
” means probe_name1.stack.json and probe_name2.stack.json will be loaded in that order (or excluded)“
*net*
” means load or exclude all configs that match the glob pattern *net*.stack.json“
*net* *md*
” means load or exclude all configs that match either glob patterns *net*.stack.json or *md*.stack.json
Most typical installations use only the exclude list to explicitly exclude any debug or adhoc probes, which results in the PMUX loading the rest of the probes in the <confdir>/agent/pmux/<PMUX_NAME>
directory without the need for an include list.
For example, if we continue our example where we have a capVP and a VP PMUX, we might leave the include list (KALEID_capVP_CAPTURE_PROBES and KALEID_VP_STACK_PROBES) undefined and have the following filesystem under <confdir>/agent/pmux
:
.
├── capVP
│ └── VP.capture.config
└── VP
├── CBOE_FX_ITCH_egress.stack.json
├── CBOE_FX_ITCH_ingress.stack.json
├── LMAX_FX_FIX_egress.stack.json
├── LMAX_FX_FIX_ingress.stack.json
├── Lucera_UK_FIX_egress.stack.json
├── Lucera_UK_FIX_ingress.stack.json
├── PrimeXM_FX_FIX_egress.stack.json
├── PrimeXM_FX_FIX_ingress.stack.json
├── TCP_egress.stack.json
├── TCP_ingress.stack.json
├── VP_TCP_rtt_Rx_ack.stack.json
├── VP_TCP_rtt_Rx_syn.stack.json
├── VP_TCP_rtt_Tx_ack.stack.json
└── VP_TCP_rtt_Tx_syn.stack.json
This would result in the capVP PMUX loading a single capture probe (using the configuration defined in the ./capVP/VP.capture.config
file). The VP PMUX would have 14 stack probes loaded.