Upgrading the Pexip Infinity platform

We have designed the upgrade process to minimize disruption. However, there may be some temporary loss of functionality or unpredictable system behavior due to Conferencing Nodes running conflicting software versions, or some Conferencing Nodes being placed in maintenance mode. For this reason, we recommend upgrading at a time of minimal usage.

The upgrade process

When you initiate an upgrade of the Pexip Infinity software, the following steps occur automatically:

  1. The Management Node software is upgraded, after which the Management Node automatically reboots.
  2. The first 10 Conferencing Nodes are put into maintenance mode. This means that all further incoming calls to those nodes will be rejected.

    (In Pexip Infinity deployments of 10 Conferencing Nodes or fewer, all of the nodes are placed into maintenance mode.)

  3. When all calls have cleared from a Conferencing Node that is in maintenance mode, its software is upgraded and the node is rebooted. This process should take around 2 minutes to complete, but may take longer depending on the connection speed between the Management Node and the Conferencing Node (as the upgrade files have to be transferred between the nodes).

    If all of the calls on a Conferencing Node that is in maintenance mode have not cleared after 1 hour, the node is taken out of maintenance mode and put at the back of the queue of nodes to be upgraded. A further attempt to upgrade that node will be made after all other nodes have been upgraded (or had upgrade attempts made).

  4. After a Conferencing Node has been upgraded successfully (or has been put back in the queue for a later upgrade attempt) and is again available, another Conferencing Node is selected and put into maintenance mode.
  5. The process continues with each subsequent Conferencing Node being put into maintenance mode, and, after all calls have cleared, being upgraded and then rebooted. Up to 10 Conferencing Nodes may simultaneously be in maintenance mode or in the process of being upgraded at any one time.

    Any Conferencing Nodes used for dynamic cloud bursting will be automatically started up and upgraded. Bursting nodes are normally selected for upgrade after the system has upgraded (or attempted to upgrade) the "always-on" nodes.

  6. If the upgrade of the Management Node and all Conferencing Nodes has not completed successfully after 24 hours, the process will stop and all nodes will be left in their existing upgrade state. This is designed to prevent situations where some Conferencing Nodes cannot be upgraded, which would otherwise leave the system in a permanent state of upgrading.

If the upgrade process does not complete successfully and stops after 24 hours, you may have a mix of upgraded and non-upgraded nodes. You will then need to repeat the upgrade process. During a repeat upgrade, only those nodes that have not already been upgraded will be included in the upgrade process.

During the 24-hour period from when an upgrade has been initiated, you cannot re-initiate an upgrade using the Administrator interface. If you must re-initiate an upgrade during this time, you must reboot the Management Node and then start the process again.

When to upgrade

While the upgrade is in progress, some Conferencing Nodes will be running the newer version of the software and some will be running the older version. These Conferencing Nodes will be incompatible until they are all again running the same version. This means that there may be instances where two endpoints dial the same Virtual Meeting Room alias but if they are routed to different Conferencing Nodes that are running different versions of the software, the two endpoints will be in different conferences. It may also mean that, if an endpoint is in an ongoing call on a node that is put into maintenance mode in preparation for an upgrade, some call functionality such as presentation sharing may be limited. For this reason, we recommend upgrading at a time of minimal usage.

Alternatively, to avoid unpredictable system behavior due to Conferencing Nodes running conflicting software versions, you may want to manually put all of your Conferencing Nodes into maintenance mode before initiating the upgrade process. This will allow all existing calls to finish, but will not admit any new calls. You should then actively monitor your Conferencing Nodes' status and manually take each node out of maintenance mode after it has been upgraded to the new software version, so that the system can start taking new calls again on those upgraded nodes.

Upgrading from version 22 or later to version 27

To upgrade Pexip Infinity software from v22 or later to v27:

  1. Before upgrading an on-premises deployment, we recommend that you use your hypervisor's snapshot functionality to take a full VMware/Hyper-V snapshot of the Management Node. You may also want to take a snapshot of each Conferencing Node, although depending on the size and complexity of your deployment it may be easier to simply redeploy these from the Management Node (after it has been rolled back) in the unlikely event that this is required.

    Before upgrading a cloud-based deployment (Azure, AWS, GCP or Oracle), you should backup the Management Node via Pexip Infinity's inbuilt mechanism (Utilities > Backup/Restore).

  2. Download the Pexip Infinity upgrade package for v27 from the Pexip download page.
  3. Before upgrading, ensure that all "always-on" Conferencing Nodes are powered on and are reachable (i.e. no Connectivity Loss errors), and are all running the same version from which you are upgrading. You do not need to power on any cloud bursting nodes.
  4. From the Pexip Infinity Administrator interface, go to Utilities > Upgrade.
  5. Select Choose File and browse to the location of the upgrade package.

  6. Select Continue. There will be a short delay while the upgrade package is uploaded.

    After the upgrade package has been uploaded, you are presented with a confirmation page showing details of the existing software version and the upgrade version.

  7. To proceed, select Start upgrade.

    You are taken to the Upgrade status page, showing the current upgrade status of the Management Node and all Conferencing Nodes (for a definition of each status, see Definition of upgrade statuses). This page automatically refreshes every 5 seconds.

  8. When the upgrade completes, all nodes will show a status of No upgrade in progress and have the new Installed version.

    • If a Conferencing Node fails to upgrade, for example if it remains on a Waiting for calls to clear status, it should be rebooted. The upgrade process will then continue as expected.
    • If the upgrade process completes and there are some nodes that have failed to upgrade, you can restart the upgrade process by uploading the upgrade package to the Management Node again via Utilities > Upgrade. This will skip over any nodes that have already been upgraded.
    • If you are upgrading from v25.0 or v25.1, due to a known issue it is possible that the upgrade will complete on the Management Node but not automatically proceed to the Conferencing Nodes. To resolve this issue, simply upload the upgrade package again via Utilities > Upgrade.
  9. If you have Pexip CVI for Microsoft Teams you must also upgrade your associated Teams Connector deployment in Azure to the same version as your Pexip Infinity deployment (including minor/"dot" releases).

    There are some important steps to be taken when upgrading your Teams Connector to version 27.

    • Version 27 has some specific upgrade procedures that must be completed before you redeploy your Teams Connector:

      1. New Teams Connector API app: you must create a new Azure app (in addition to the existing Pexip CVI app) that is used to secure requests to the Teams Connector APIs. To do this:

        1. Run the following PowerShell command to connect to Azure AD:


          Then follow the prompts to sign in to Azure AD.

        2. Run your variable initialization script to set the required prefix and region name variables.
        3. Run the following commands to create the Teams Connector API app.

          Copy to clipboard
          $teamsConnectorApiApp = New-AzureADMSApplication -DisplayName "${PxBaseConnName}-TeamsConn-${PxVmssRegion} Pexip Teams Connector API" -SignInAudience "AzureADMyOrg"
          Start-Sleep -Seconds 5
          $teamsConnectorApiSp = New-AzureADServicePrincipal -AppId $teamsConnectorApiApp.AppId
          $TeamsConnectorApiApplicationId = $teamsConnectorApiApp.AppId

          Write-Host "`n----------------------------------------`n"
          Write-Host "### Teams Connector API App ID MUST be saved in the variables initialization script ###"
          Write-Host "`$TeamsConnectorApiApplicationId = `"$($TeamsConnectorApiApplicationId)`""
          Write-Host "`n----------------------------------------`n"
        4. When the command runs, it generates some output that lists the Teams Connector API App ID, similar to this:

        5. Copy the output line that defines the API App ID ($TeamsConnectorApiApplicationId = "<your value>") and add it to the bottom of your variable initialization script (the one you ran in step 2). Thus the end of your variables script will then look similar to this:

          Note that your script may have values specified for the $tags variable. If required you may also want to add comment lines (beginning with #) to describe the purpose of the new $TeamsConnectorApiApplicationId variable.

        6. If somebody else will be completing the upgrade (redeployment), ensure that the person who will perform all of the remaining installation steps has Owner permissions for the new API app.

        Note that:

        • This app does not have to be granted any permissions. It does not have access to any resources in the Azure AD tenant. It has no associated credentials.
        • It is different from your existing Pexip CVI App which was created when your originally installed the Teams Connector, and the App ID (with its associated credentials) for that CVI app should already be recorded in your redeploy script.
        • If you have Teams Connectors in multiple Azure regions, you must repeat this process and create an API app in each region, using and then storing the app ID ($TeamsConnectorApiApplicationId) in the relevant variable initialization script for that region.
      2. Variable initialization script: there are two new variables to be added to the variable initialization script:

        1. Add a new $PxBotResourceGroupName variable that defines the name of the resource group for the Azure Bot. We recommend placing it after the existing $PxTeamsConnStaticResourceGroupName variable for consistency. We recommend defining it as follows:

          $PxBotResourceGroupName = "$($PxBaseConnName)-TeamsBot-RG"

        2. You must define a new $TeamsConnectorApiApplicationId variable as described in the previous step for the API app.
      3. Scheduled scaling: this version introduces a new scaling feature to manage the capacity of your Teams Connector instances.

        If you currently use the Azure Event Hub for advanced status reporting, when upgrading your Pexip Infinity platform and Teams Connector to version 27 you should:

        1. Upgrade your Pexip Infinity platform as normal.
        2. Before redeploying your Teams Connector, within the Pexip Infinity Administrator interface, go to Call control > Microsoft Teams Connectors and configure the new Minimum number of instances field.
        3. Redeploy your Teams Connector as normal.
        4. After redeploying the Teams Connector you must update the connection details for the Azure Event Hub in the Pexip Infinity Administrator interface: go to Call control > Microsoft Teams Connectors and change the connection string to the Connection string–primary key associated with the RootManageSharedAccessKey shared access policy.

          The connection string is in the format Endpoint=sb://examplevmss-lltsgzoqun-ehn.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=[string]

          To find this string in the Azure portal:

          1. Go to the static resource group for the Teams Connector (this has a name in the format <prefix>-TeamsConn-<region>-static-RG).
          2. Select the Event Hubs Namespace component (<name>-EHN).
          3. From the left-hand navigation menu, under Settings select Shared access policies.
          4. Select the RootManageSharedAccessKey policy.
          5. Copy the Connection string–primary key.

        Note that:

        • If Minimum number of instances is not configured, the Teams Connector will redeploy with just 1 instance (the default) and you may therefore have less capacity than you had before upgrading, although you can adjust the setting later to reinstate that capacity.
        • If you currently do not use the Azure Event Hub for advanced status reporting then you can ignore the new Minimum number of instances field and upgrade your Teams Connector as normal, and you will retain your existing capacity / number of instances.
        • If you do not update the Azure Event Hub connection string field then advanced status reporting will continue to work but scheduled scaling will not work.
    • The redeployment process has changed in version 27. There are some new steps to perform prior to running the redeploy script:

      • To delete the existing dynamic resource group and then recreate it for the redeployed Teams Connector. Previously these steps were contained within the redeploy script.
      • To ensure that the person performing the redeploy/upgrade has the Azure Owner role for the static and dynamic resource groups, and Contributor role for the Azure Bot resource group.
    • You must use the latest version of the redeploy script as contained within the v27 documentation.

    • You must be using Az module version 4.7.0 or later.
      • To check your installed version you can run:
        Get-InstalledModule -Name Az -AllVersions
      • To install the latest Az version you can run:
        Install-Module -Name Az -MinimumVersion 4.7.0 -AllowClobber -Scope AllUsers
    • If you have deployed multiple Teams Connectors, you must follow the same redeploy process (with the appropriate variable initialization script) for each Teams Connector.
    • As with all upgrades, you can continue to use the Pexip CVI app from your existing deployment.

    Full instructions are available at Upgrading the Teams Connector to the latest software.

No additional actions are required to upgrade the Management Node or Conferencing Nodes individually.

We recommend that you take a fresh backup of your system after upgrading.

Error messages during upgrade

The following Error and Warning messages are expected during an upgrade and do not require any action:

2017-03-03T12:12:02.400+00:00 <manager_hostname> 2017-03-03 12:12:02,456 Level="ERROR" Name="administrator.system.connectivity" Message="Unable to contact node." Src-Node="<node_ip_fqdn>" Node="<node_ip_fqdn>"

2017-03-09T07:04:08.460+00:00 <manager_hostname> 2017-03-09 07:04:08,460 Level="WARNING" Name="administrator.alarm" Message="Alarm raised" Node="<node_ip_fqdn>" Alarm="connectivity_lost" Time="2017-03-09 07:04:08,455" Instance="Source=<node_ip_fqdn>, Destination=<node_ip_fqdn>" Detail=""

Upgrading from versions 16-21 to version 27

If you are running a Pexip Infinity software version between v16 and v21 inclusive, you must first upgrade to version 22 and then upgrade again to version 27. To do this:

  1. Before upgrading, ensure that all "always-on" Conferencing Nodes are powered on and are reachable (i.e. no Connectivity Loss errors), and are all running the same version from which you are upgrading. You do not need to power on any cloud bursting nodes (unless you are upgrading from version 21.0, in which case they must also be powered on at least 15 minutes prior to upgrading from v21.0).
  2. Download the Pexip Infinity v22 upgrade file.
  3. Follow the steps outlined in Upgrading from version 22 or later to version 27, but when asked to Choose File browse to the location of the v22 upgrade file.
  4. Verify that the upgrade has completed successfully.
  5. Download the Pexip Infinity v27 upgrade file.
  6. Follow the steps outlined in Upgrading from version 22 or later to version 27, and when asked to Choose File browse to the location of the v27 upgrade file.

Upgrading configuration-only deployments

The automatic upgrade process described above will update the Management Node and all Conferencing Nodes, including Conferencing Nodes that have been created using the configuration only deployment type.

However, if after the upgrade you wish to deploy new configuration-only Conferencing Nodes, you must download and use a version of the Conferencing Node VM template that matches the version of the Management Node that you have upgraded to. Creating a new Conferencing Node from a VM template containing a different version of the Pexip Infinity software than that which is running on the Management Node is not supported and will not work. For more information, see Deploying a Conferencing Node using a generic VM template and configuration file.

Definition of upgrade statuses

During an upgrade, the Upgrade status page will report the status of each Conferencing Node as follows:

Status Description
No upgrade in progress During a platform upgrade, this is the default state that occurs before a Conferencing Node is upgraded, or after the node has rebooted.
Upgrade pending An upgrade of the platform is in progress and this Conferencing Node is in the queue to be upgraded.
Preparing to upgrade The Conferencing Node is preparing to upgrade. During this time the node is put into maintenance mode, and other services are stopped.
Waiting for calls to clear The Conferencing Node is in maintenance mode and is waiting for existing conferences to complete.
Timeout waiting for calls to clear Not all conferences had cleared after 1 hour. This Conferencing Node has been removed from maintenance mode and the upgrade will be attempted again later.
Upgrade in progress The new software is being unpackaged and installed on the Conferencing Node. During this time the Conferencing Node will not synchronize configuration with the Management Node.
Rebooting The upgrade has completed and the Conferencing Node is rebooting.
Could not communicate with conferencing node This error is reported if the Conferencing Node cannot be contacted.

Downgrading or recovering from a failed upgrade

To downgrade your system to the previous version, or to recover your system after a failed upgrade:

  1. Restore the Management Node from the VM snapshot that you took using your hypervisor at the start of the upgrade process.
  2. Restore the individual Conferencing Nodes by either:
    • redeploying each Conferencing Node from the Management Node - see Deploying new Conferencing Nodes, or
    • restoring the hypervisor snapshot of each Conferencing Node that was taken at the start of the upgrade process.

Checking the installed version

Management Node

To view which software version is running on the Management Node, click on the About link at the top right corner of the Pexip Infinity Administrator interface.

Conferencing Nodes

To view which software version is running on individual Conferencing Nodes, go to Status > Conferencing Nodes. The software version number will be shown in the Installed version column.