You are here: Installation > Amazon Web Services installations > Dynamic bursting to the AWS cloud

Dynamic bursting to the AWS cloud

Pexip Infinity deployments can burst into the Amazon Web Services (AWS) cloud when primary conferencing capabilities are reaching their capacity limits, thus providing additional temporary Conferencing Node resources.

This provides the ability to dynamically expand conferencing capacity whenever scheduled or unplanned usage requires it. The AWS cloud Conferencing Nodes instances are only started up when required and are automatically stopped again when capacity demand normalizes, ensuring that AWS costs are minimized.

You can also use the management API to monitor and manually start up your overflow nodes, if for example, you want to configure a third-party scheduling system to start up all of your additional capacity prior to a large conferencing event starting.

This topic covers:

How it works
Configuring your system for dynamic bursting
Configuring the bursting threshold
Manually starting an overflow node
Viewing cloud bursting status
Deployment guidelines and additional information
Troubleshooting
Detailed example of the overflow process

How it works

Dynamic bursting builds upon Pexip Infinity's standard location capacity overflow logic, which is used to define which overflow location's nodes are used when a particular location reaches its capacity. The dynamic bursting functionality adds to this by automatically starting up those additional AWS cloud nodes when a location is approaching full capacity, so that those overflow nodes will be available if required.

After you have deployed your overflow Conferencing Nodes in AWS, and enabled cloud bursting and configured your bursting threshold in Pexip Infinity, everything is then controlled automatically by the Pexip Infinity Management Node:

Whenever the available capacity in a system location containing your primary (always on) Conferencing Node hits or drops below your configured threshold, a Conferencing Node in a system location containing overflow Conferencing Nodes is automatically started up. That new node can then start handling any additional conferencing requirements if the original location reaches full capacity.
When an overflow Conferencing Node starts to fill up and reaches its own bursting threshold, a further overflow Conferencing Node is started, and so on. The conference graph (right) shows a conference hosted in an on-premises Oslo location that has burst onto two Conferencing Nodes in the "AWS Western Europe" overflow location.
When call levels subside and an overflow AWS Conferencing Node is no longer hosting conferences and is no longer required, the node is automatically shut down again.

This sequence of events is explained in more detail in a worked example.

Configuring your system for dynamic bursting

These instructions assume that you already have a working Pexip Infinity platform, including one or more primary (always on) Conferencing Nodes in one or more system locations. These existing Conferencing Nodes can be deployed using whichever platform or hypervisor you prefer.

Setting up your bursting nodes in AWS and enabling bursting in Pexip Infinity

You must deploy in AWS the Conferencing Nodes that you want to use for dynamic bursting, and then configure those nodes in Pexip Infinity as the overflow destination for your primary (always on) Conferencing Nodes:

In Pexip Infinity, configure a new "overflow" system location e.g. "AWS burst", that will contain your bursting Conferencing Nodes.

(Note that system locations are not explicitly configured as "primary" or "overflow" locations. Pexip Infinity automatically detects the purpose of the location according to whether it contains Conferencing Nodes that may be used for dynamic bursting.)
In AWS, set up a user and associated access policy that the Pexip Infinity Management Node will use to log in to AWS and start and stop the node instances.

See Configuring an AWS user and policy for controlling overflow nodes for more information.
Deploy in AWS the Conferencing Nodes that you want to use for dynamic bursting. Deploy these nodes in the same manner as you would for "always on" usage (see Deploying a Conferencing Node in AWS), except:
1. Apply to each AWS instance to be used for conference bursting a tag with a Key of pexip-cloud and an associated Value set to the AWS tag value that is shown at the bottom of the Platform configuration > Global settings page.
  This tag indicates which VM nodes will be started and shut down dynamically by your Pexip system, and relates to the access policy document configured in the previous step.
2. When adding the Conferencing Node within Pexip Infinity:
  1. Assign the Conferencing Node to the overflow system location (e.g. "AWS burst").
  2. Disable (uncheck) the Enable distributed database setting (this setting should be disabled for any nodes that are not expected to always be available).
3. After the Conferencing Node has successfully deployed, manually stop the node instance on AWS.

In Pexip Infinity, go to Platform configuration > Global settings, and enable cloud bursting and configure your bursting threshold:

Option	Description
Enable bursting to the cloud	Select this option to instruct Pexip Infinity to monitor the system locations and start up / shut down overflow Conferencing Nodes hosted in AWS when in need of extra capacity.
AWS access key ID and AWS secret access key	Set these to the Access Key ID and the Secret Access Key respectively of the User Security Credentials for the user you set up in the AWS dashboard within Identity and Access Management in step 2 above.
Bursting threshold	The bursting threshold controls when your overflow nodes in AWS are automatically started up so that they can provide additional conferencing capacity. When the number of additional HD calls that can still be hosted in a location reaches or drops below the threshold, it triggers Pexip Infinity into starting up an overflow node in the overflow location. See Configuring the bursting threshold for more information.
AWS tag name and AWS tag value	These read-only fields indicate the tag name (always pexip-cloud) and associated tag value (the hostname of your Management Node) that you must assign to each of your AWS instances that are to be used for dynamic bursting.

Go to Platform configuration > Locations and configure the system locations that contain your primary (always on) Conferencing Nodes so that they will overflow to your new "AWS burst" location.

When configuring these locations, you must set the Primary overflow location to the bursting location containing your overflow nodes. (Automatic bursting, and the stopping and starting of overflow nodes only applies to the Primary overflow location; the Secondary overflow location can only be used for standard overflow i.e. to other "always on" nodes.)

Configuring an AWS user and policy for controlling overflow nodes

Within AWS you must set up a user and an access policy to be used by Pexip Infinity to start up and shut down the Conferencing Node overflow instances:

From the AWS dashboard, select Identity and Access Management.
Select Users and create a new user on behalf of the Pexip platform e.g. username "pexip". Ensure that "Generate an access key for each user" is selected.
Either download the user credentials or select Show User Security Credentials and make a note of the Access Key ID and the Secret Access Key — you will enter these values into the Global settings page in the Pexip Infinity Administrator interface.

(You must copy or download these key values when you create the user; you will not be able to access them again later.)
Select Policies and then Create Your Own Policy to set up the access policy for the overflow nodes:
1. Enter a Policy Name and Description.
2. For the Policy Document, copy/paste the following text:
```
{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Action": [
               "ec2:Describe*"
           ],
           "Effect": "Allow",
           "Resource": "*"
       },
       {
           "Action": [
               "ec2:StartInstances",
               "ec2:StopInstances"
           ],
           "Effect": "Allow",
           "Resource": "*",
           "Condition": {
               "StringEquals": {
                   "ec2:ResourceTag/pexip-cloud": "<management-node-hostname>"
               }
           }
       }
   ]
}
```
3. The only element of this policy document that you need to change is to replace <management-node-hostname> with the hostname of your Management Node (as shown via Platform configuration > Management Node). This is the same name as you set as the value of the pexip-cloud tag that you assigned to your AWS overflow Conferencing Nodes.
4. Select Validate Policy and then (if valid) Create Policy.
Attach your "pexip" user to your policy:
1. From the Policies page, select the checkbox next to your policy (you can filter on Customer Managed Policies if necessary).
2. From the Policy Actions dropdown select Attach, select the checkbox next to your pexip user and select Attach Policy.
This policy only allows the pexip user i.e. the Pexip Infinity platform, to retrieve a list of instances and to start and stop existing instances that you have tagged as pexip-cloud. The Pexip Infinity platform cannot (and will not attempt to) create or delete AWS instances.

Configuring the bursting threshold

When enabling your platform for cloud bursting the most important decision you must make is the level at which to set the bursting threshold:

The bursting threshold controls when your overflow nodes in AWS are automatically started up so that they can provide additional conferencing capacity. When the number of additional HD calls that can still be hosted in a location reaches or drops below the threshold, it triggers Pexip Infinity into starting up an overflow node in the overflow location.

For example, setting the threshold to 5 means that when there are 5 or fewer HD connections still available in a location, an overflow node will be started up.
When an overflow location reaches the bursting threshold i.e. the number of additional HD calls that can still be hosted on the Conferencing Nodes in the overflow location reaches the threshold, another overflow node in that location is started up, and so on.

Note that the current number of free HD connections in the original location are not taken into account when seeing if the overflow location needs to overflow further — however, new calls will automatically use any available media resource within those original primary locations that has become available.
The bursting threshold is a global setting — it applies to every system location in your deployment.
Note that it takes approximately 5 minutes for a Conferencing Node instance in AWS to start up and become available for conference hosting. If your primary deployment reaches full capacity, and the overflow nodes have not completed initiating, any incoming calls during this period will be rejected with "capacity exceeded" messages. You have to balance the need for having standby capacity started up in time to meet the expected demand, against starting up nodes too early and incurring extra unnecessary costs.

See Capacity planning for more information on call types and the relationship between Full HD, HD, SD and audio-only calls.

Manually starting an overflow node

If you know that your system will need additional capacity at a specific time due to a predictable or scheduled spike in demand, but do not want to wait for the bursting threshold to be triggered before starting up the overflow nodes, you can manually start up any of your overflow nodes:

via the management API: the cloud_node status resource can be used to list all of the available overflow nodes, the cloud_monitored_location and cloud_overflow_location resources retrieve the current load on the primary locations and any currently active overflow locations respectively, and the start_cloudnode resource can be used to manually start up any overflow node. This means that a third-party scheduling system, for example, could be configured to start up the overflow nodes via the management API approximately 10 minutes before a large conference is due to start.

For example, let's assume that you have:
- a regular spike in conferencing capacity demand at 9:00am every morning
- an even usage of about 20% of that spike level during the rest of the day
- a 30:70 ratio between your "always on" capacity and your overflow cloud capacity
we would recommend:
- configuring a low bursting threshold, such as 10-20% of your "always on" capacity (i.e. if your "always on" capacity is 80 HD calls, then set the bursting threshold to 12)
- getting your scheduling system to call the API to manually start up all of your overflow cloud nodes at 8:50am on weekdays.
via the Pexip Infinity Administrator interface: go to Status > Cloud bursting and select Start for the required nodes (the Start option is in the final column of the Cloud overflow nodes table).

Viewing cloud bursting status

Go to Status > Cloud bursting to see an overview of the media load of your primary locations (that contain your "always-on" Conferencing Nodes), and whether your overflow nodes and locations are in use.

Any issues relating to your cloud bursting deployment will also be shown on this page.
The list of primary locations only includes those system locations that are configured with a Primary overflow location that contains bursting nodes.
An approaching threshold message is displayed in the Available HD connections column for the primary locations when the number of available HD connections is less than or equal to the bursting threshold plus two.

This message changes to bursting threshold reached when the number of available HD connections is less than or equal to the bursting threshold (and therefore overflow nodes are started up).
You can manually start any overflow nodes by selecting Start for the required node (the Start option is in the final column of the Cloud overflow nodes table).
The status page dynamically updates every 15 seconds.

Deployment guidelines and additional information

An overflow Conferencing Node is automatically stopped when it becomes idle (no longer hosting any conferences). However, a node is never stopped until it has been running for at least 50 minutes. This is because AWS charges by the hour, so when a node is started up it is more efficient to leave it running for 50 minutes — even if it is never used — as that capacity can remain on immediate standby for no extra cost.
A location can trigger an overflow node to start up only if there is some existing media load on that location. This is to ensure that incidents such as a temporary network interruption do not inadvertently trigger bursting.
We recommend that you do not mix your primary (always on) Conferencing Nodes and your bursting nodes in the same system location.
As per standard AWS configuration guidelines, all of your AWS Conferencing Nodes must be in the same AWS region.
Typically you should assign all of your overflow Conferencing Nodes to a single overflow system location so that all of your primary locations can make use of the same pool of cloud overflow nodes.

However, if you expect a single conference to require more than three overflow nodes, then you could configure a second cloud overflow location (containing a different set of AWS overflow nodes) and then configure half of your primary locations to overflow to the first cloud overflow location, and the other half of your primary locations to overflow to the second cloud overflow location. In this case, both overflow locations will act (start up and shut down nodes as appropriate) independently of each other.
Do not configure your call control system / DNS to route calls directly to your overflow nodes.
No specific licenses are required for your overflow nodes, but you must ensure that your overall system has sufficient licenses to meet peak conference capacity demand.
If you need to convert an existing "always on" AWS Conferencing Node into an overflow node:
- In AWS:
  1. Apply a tag with a Key of pexip-cloud and an associated Value set to the hostname of your Management Node to the AWS instance.
  2. Manually stop the node instance on AWS.
- In Pexip Infinity:
  1. Change the system location of the Conferencing Node to the overflow system location (e.g. "AWS burst").
  2. Disable the node's Enable distributed database setting. After a node has been deployed, this setting can only be changed via the Management configuration API using the worker_vm resource.
If you need to convert an existing AWS overflow Conferencing Node into an "always on" node:
- In AWS:
  1. Remove the tag with a Key of pexip-cloud from the AWS instance.
  2. Manually start the node instance on AWS.
- In Pexip Infinity:
  1. Change the system location of the Conferencing Node to a location other than the overflow system location.
  2. Enable the node's Enable distributed database setting. After a node has been deployed, this setting can only be changed via the Management configuration API using the worker_vm resource.

Troubleshooting

Checking the administrator and support logs

All log messages that are explicitly related to cloud bursting, such as starting up or shutting down overflow nodes, are tagged with a log module Name of administrator.apps.cloudbursting.

All other log messages related to those overflow nodes or the conferences they are hosting are reported in the same manner as per standard behavior.

No AWS nodes appear in the Cloud overflow nodes area of the Status > Cloud bursting page

Check that the AWS instances have been assigned the pexip-cloud tag and that the tag value is set to the Management Node hostname.

Instance found but no corresponding Conferencing Node

You may see a status issue "Instance <name> (with IP <address>) was found, but no corresponding Conferencing Node has been configured".

This occurs when Pexip Infinity detects an AWS instance with a tag matching your system's hostname but there is no corresponding Conferencing Node configured within Pexip Infinity.

This message can occur temporarily in a normal scenario when deploying a new Conferencing Node and you have set up the VM instance in AWS but you have not yet deployed the Conferencing Node in Pexip Infinity. In this case, the issue will disappear as soon as you have deployed the Conferencing Node.

Connectivity errors in the administrator log

You may see connectivity errors in the administrator log while overflow nodes are being started/stopped. This is normal behavior.

Not authorized to perform an operation

A "Not authorized to perform an operation on <instance ID or region name>. Check the policy created for the AWS user." means that there is a problem with the AWS policy document, or the AWS user is not attached to the policy. See Configuring an AWS user and policy for controlling overflow nodes for more information.

Detailed example of the overflow process

This sequence of actions listed below shows how the process of starting and stopping overflow nodes is managed. It assumes an example scenario where:

the system is configured with a bursting threshold of 5
there is a single primary location (London) containing 2 "always on" Conferencing Nodes with a total location capacity of 40 connections
the primary location is configured to overflow to the cloud overflow location "AWS burst"
the "AWS burst" location contains 2 overflow nodes, each with a capacity of 20 connections.

Primary location: London			Dynamic bursting activity	Cloud overflow location: AWS burst
Action		Call capacity remaining	Dynamic bursting activity	Overflow node A status	Overflow node B status	Call capacity remaining
1.	No conferences in progress	40	none	Not running	Not running
2.	A conference starts and has 33 participants	6*	none
3.	1 more participant joins	5	Primary location threshold reached	Node A starting
4.		5	Overflow capacity becomes available	Running		20
5.	5 more participants join	0	none	↓		20
6.	1 more participant joins	0	Call media handled by overflow node A	↓		18*
7.	12 more participants join	0	All new participants handled by overflow node A	↓		6
8.	1 more participant joins	0	Overflow location threshold reached	↓	Node B starting	5
9.		0	Additional overflow capacity becomes available	↓	Running	25
10.	7 more participants join	0	All new participants handled by overflow nodes A and B	↓	↓	17*
11.	Conference ends	40	Overflow nodes remain available for further bursting if required	↓	↓	40
12.	A new conference starts with 25 participants	14*	none	↓	↓	40
13.			Overflow node A is unused and has been running for 50 minutes	Shutting down	↓	20
14.			Overflow node B is unused and has been running for 50 minutes	Not running	Shutting down	0
15.	Conference ends	40		Not running	Not running
* a Conferencing Node reserves 1 HD connection for the backplane to other nodes in the same conference

Note that if other conferences were running on any of the nodes in these locations, they would consume call capacity and bursting would be triggered in exactly the same way when the remaining capacity within the location reached the threshold.