Server design recommendations

This section summarizes the terminology, recommended specifications and deployment guidelines for servers hosting the Pexip Infinity platform. These apply to both on-premises hardware and cloud deployments.

Terminology

The table below provides descriptions for the terms used in this guide, in the context of a Pexip Infinity deployment.

Term Description
Processor The hardware within a computer that carries out the basic computing functions. It can consist of multiple cores.
Core

One single physical processing unit. An Intel Xeon Scalable processor typically has between 8 and 32 cores, although both larger and smaller variants are available.
Socket The socket on the motherboard where one processor is installed.
RAM
 The hardware that stores data which is accessed by the processor cores while executing programs. RAM is supplied in DIMMs (Dual In-line Memory Modules).
Channel A memory channel uses a dedicated interface to the processor, and can typically support up to 2 DIMMs. An Intel Xeon Scalable series processor has 6 channels of memory, older processors have fewer channels.
Virtual CPU (vCPU)

The VM's understanding of how many CPU cores it requires. Each vCPU appears as a single CPU core to the guest operating system.

When configuring a Conferencing Node, you are asked to enter the number of virtual CPUs to assign to it. We recommend no more than one virtual CPU per physical core, unless you are making use of CPUs that support Hyper-Threading — see NUMA affinity and hyperthreading for more details.
NUMA node The combination of a processor (consisting of one or more cores) and its attached memory.

Management Node

Recommended host server specifications

  • 4 cores* (most modern processors will suffice)
  • 4 GB RAM*
  • 100 GB SSD storage
  • The Pexip Infinity VMs are delivered as VM images (.ova etc.) to be run directly on the hypervisor. No OS should be installed.

* Sufficient for typical deployments of up to 30 Conferencing Nodes. For deployments with more than 30 Conferencing Nodes, you will need to increase the number of cores and the amount of RAM on the Management Node. Please contact your Pexip authorized support representative or your Pexip Solution Architect for guidance on Management Node sizing specific to your environment.

Management Node performance considerations

There are a number of factors that can impact the performance of the Management Node, and these should be taken into consideration alongside the recommended specifications described above when determining the size of the Management Node in your deployment.

  • Individual cores can vary significantly in capability.
  • Large numbers of roughly simultaneous join and leave events will increase the load on the Management Node, when compared to the same number of events spread over a broader time range.
  • Different Pexip Infinity features will place different loads on the Management Node, so the overall load will be impacted by the features you have implemented and the frequency with which they are used.
  • Heavy or frequent use of the management API will significantly increase the load on the Management Node.
  • Multiple Live view sessions will increase the load on the Management Node.

Conferencing Node

Below are our general recommendations for Conferencing Node (Proxying Edge Nodes and Transcoding Conferencing Nodes) servers. For some specific examples, see Example Conferencing Node server configurations.

Recommended host server specifications

  • AVX or later instruction set required; we prefer AVX2 or AVX512. We recommend 2nd- or 3rd-generation Intel Xeon Scalable Processors (Cascade Lake / Cooper Lake) Gold 62xx/63xx or 52xx/53xx. We also support Intel Xeon Scalable Processors (Skylake) Gold 61xx generation or E5-2600 v3/v4 Haswell/Broadwell architecture from 2014 or later. Also works with Xeon E5-2600 v1/v2 processors (Sandy Bridge / Ivy Bridge from 2012 or later). AMD processors that support the AVX and AVX2 instruction set are also supported.
  • 2.8 GHz (or faster) clock speed if using Hyper-Threading, 2.3 GHz (or faster) clock speed without
  • Up to 48 vCPU per socket, i.e. 24 cores if using Hyper-Threading, 48 cores without
  • 1 GB RAM for each vCPU that is allocated to the Conferencing Node
  • Populate at least 4 memory channels
  • Storage:
    • 500 GB total per server (to allow for snapshots etc.), including:
    • 50 GB minimum per Conferencing Node
    • SSD recommended, however, this is essential for Management Nodes on large or busy deployments
  • RAID 1 mirrored storage
  • Hypervisors: VMware ESXi 6.5, 6.7 and 7.0; KVM; Xen; Microsoft Hyper-V 2012 or later
  • The Pexip Infinity VMs are delivered as VM images (.ova etc.) to be run directly on the hypervisor. No OS should be installed.

Transcoding Conferencing Nodes versus Proxying Edge Nodes

The specifications and guidelines shown below apply to Transcoding Conferencing Nodes.

The servers hosting Proxying Edge Nodes do not require as high a specification as those servers hosting Transcoding Conferencing Nodes. This is because proxying nodes are not as processor intensive as transcoding nodes. The minimum functional CPU instruction set for a proxying node is AVX, which was first available in the Sandy Bridge generation. You still need multiple proxying nodes for resilience and capacity.

We recommend allocating 4 vCPU and 4 GB RAM (which must both be dedicated resource) to each Proxying Edge Node, with a maximum of 8 vCPU and 8 GB RAM for large or busy deployments.

General deployment recommendations for Transcoding Conferencing Nodes

Cores, CPU and RAM

  • Prefer processors with a high core count.
  • Prefer processors with a high clock speed (2.8 GHz (or faster) clock speed if using Hyper-Threading, 2.3 GHz (or faster) clock speed without).
  • Prefer a smaller number of large Conferencing Nodes over a large number of smaller Conferencing Nodes.
  • When NUMA affinity and Hyper-Threading are enabled, a Conferencing Node should normally be assigned twice the number of vCPUs as there are physical cores on the processor . For more information, see NUMA affinity and hyperthreading. If NUMA affinity and Hyper-Threading are not in use, then the number of assigned vCPU must not exceed the number of physical cores on the processor.

  • For each physical CPU core (or logical thread, if employing NUMA affinity):

    • configure 1 vCPU
    • assign at least 1 GB RAM

    For example, on an Intel Gold 6248 CPU with 20 physical cores (i.e. 40 logical threads) per CPU, either

    • assign 20 vCPU (one per physical core) and 20 GB of RAM, or
    • enable NUMA affinity, and assign 40 vCPU (one per logical thread) and 40 GB of RAM
  • A Conferencing Node must have 4 vCPU and 4 GB RAM as an absolute minimum, and a maximum of 48 vCPU.
  • Do not over-commit (also referred to as over-subscribing or over-allocation) either RAM or CPU resources on hardware hosts. In other words, the Conferencing Node and Management Node each must have dedicated access to their own RAM and CPU cores. Pexip Conferencing Nodes use real-time media, which needs dedicated capacity.
  • Each memory channel should be populated. Intel Xeon Scalable series processors have 6 memory channels per socket, so in a typical dual-socket server, there should be 12 DIMMs installed. Note that each channel usually has two DIMM sockets, only one of which needs to be populated. For older Xeon E5 systems, there are 4 memory channels per socket, so 8 DIMMs should be installed in a dual-socket server. We recommend at least 1GB RAM per vCPU, but small DIMMs are rarely available, so it is usually necessary to install more than 1GB RAM per vCPU in order to populate all the memory channels.

  • When using AMD EPYC processors:

    • We recommend using NPS4 and 32vcpu VMs i.e. four VMs per socket, and that each VM is pinned to one NUMA node.
    • For optimal performance, populate 8 DIMMs for 1 DIMMs per Channel (DPC) configuration, or 16 DIMMs for 2 DPC configuration, per processor.

  • When using Intel-based systems:

    • Disable Sub-NUMA Clustering (SNC) in the BIOS (in this example via chipset configuration > north bridge > uPI configuration > advanced >SNC):

      Disabled Sub-NUMA Clustering (SNC)

Hyper-Threading

  • Hyper-Threading (also referred to as Hyper-Threading Technology), if supported, should always be left enabled by default.
  • When Hyper-Threading is in use, we recommend that Conferencing Nodes are NUMA pinned to their sockets to avoid memory access bottlenecks.

BIOS performance settings

  • Ensure all BIOS settings pertaining to power saving are set to maximize performance rather than preserve energy. (Setting these to an energy-preserving or balanced mode may impact transcoding capacity, thus reducing the total number of HD calls that can be provided.) The actual settings depend on the hardware vendor; some examples are given below:

    Typical HP settings

    • HP Power Profile: Maximum Performance
    • Power Regulator modes: HP Static High Performance mode
    • Energy/Performance Bias: Maximum Performance
    • Memory Power Savings Mode: Maximum Performance

    Typical Dell settings

    • System Profile: Performance Optimized

    Typical Cisco UCS B-Series settings

    • System BIOS Profile (Processor Configuration) - CPU Performance: Enterprise
    • System BIOS Profile (Processor Configuration) - Energy Performance: Performance
    • VMware configuration: Active Policy: Balanced

Network

  • Although the Conferencing Node server will normally not use more than 1-2 Mbps per video call, we recommend 1 Gbps network interface cards or switches to ensure free flow of traffic between Pexip Infinity nodes in the same datacenter. We do not recommend 100 Mbps NIC.
  • Redundancy: for hypervisors that support NIC Teaming (including VMware), you can configure two network interfaces for redundancy, connected to redundant switches (if this is available in your datacenter).

Disk

  • Although Pexip Infinity will work with SAS drives, we strongly recommend SSDs for both the Management Node and Conferencing Nodes. General VM processes (such as snapshots and backups) and platform upgrades will be faster with SSDs.
  • Management Node and Conferencing Node disks should be Thick Provisioned.
  • Pexip Infinity can absorb and recover relatively gracefully from short bursts of I/O latency but sustained latency will create problems.
  • The Management Node requires a minimum of 800 IOPs (but we recommend providing more wherever possible).
  • A Conferencing Node requires a minimum of 250 IOPs (but we recommend providing more wherever possible).
  • Deployment on SAN/NAS storage should in most cases work well. Disk access is only required by the operating system and logs, so a normal fair performance is expected.
  • Redundancy: for RAID 1 mirroring for disk redundancy, remember to use a RAID controller supported by VMware or your preferred hypervisor. The RAID controller must have an enabled cache. Most vendors can advise which of the RAID controllers they provide are appropriate for your hypervisors.

Power

  • Sufficient power to drive the CPUs. The server manufacturer will typically provide guidance on this.
  • Redundancy: Dual PSUs.