Enabling live captions

This topic describes how to configure Pexip Infinity and the AI Media Server (AIMS) to enable the live captions feature in Pexip Infinity VMRs, Virtual Auditoriums and gateway calls, including how to enable different languages for different VMRs in your Pexip Infinity deployment.

When live captions are enabled by a participant in a VMR (or other service), the audio from the call is sent from the Conferencing Node that is handling the call, to a Conferencing Node in a dedicated live captions location. That Conferencing Node forwards the audio to AIMS, which transcribes it into text. The text is then returned back to the Conferencing Node in the live captions location, and on to the originating Conferencing Node, which then supplies it as a text overlay to all Connect web app users in the VMR who have enabled live captions.

For AIMS v1 the supported languages are en-US, es-US, de-DE and fr-FR.

Diagram showing connections between Pexip Infinity and AIMS

The sections below describe how to configure Pexip Infinity (including enabling the live captions feature, setting up the necessary system locations, and optionally setting the language for a VMR), and how to configure AIMS with information about the Pexip Infinity deployment connecting to it.

Infinity configuration

In this step you enable live captions on your Pexip Infinity deployment, configure it with the details of the AIMS server, and ensure that your VMRs and system locations are set up appropriately.

Enabling Live Captions on Pexip Infinity

From your Pexip Infinity Management Node:

1. Go to Platform > Global Settings.

2. Scroll down to the Tech preview features section and select the Live Captions checkbox.

   With the release of AIMS v1, live captions is no longer a tech preview feature. The next release of Pexip Infinity will be updated to reflect this.

3. Scroll up to the Live captions section that appears and enter the following information:

   • Enable live captions on VMRs and routing rules by default: enable this option if you want all Virtual Meeting Rooms to be enabled for live captions. You can override this setting on a per-VMR basis.

   • Live captions service API gateway: enter the host address of the AIMS server.

4. Select Save.

   A Live Captions App-Id and Live captions JWT public key are generated.

5. Click on either field to copy both values as a single file to your clipboard. These are the values required when configuring AIMS.

   For Pexip Infinity versions v34.x and earlier, this file will not be in the correct format for uploading to AIMS. See Pexip Infinity v34.x and earlier for information on how to reformat it. Alternatively, contact your Pexip authorized support representative for assistance.

6. Ensure that each VMR that will use the live captions service has the feature enabled:

   1. From Services > Virtual meeting rooms select the VMR.

   2. Scroll down to the Advanced Options section and from the Live Captions drop down, select either Use global live captions setting or Yes.

About system locations and AIMS

Each system location in your Pexip Infinity deployment has one or more nominated Live captions outgoing locations. When live captions are enabled on a VMR, the Conferencing Node that is handling the call will connect to the live caption location configured for the system location that it is in. A Conferencing Node in that live caption location will in turn connect to the AIMS server.

We recommend using a dedicated pool of Conferencing Nodes in their own system location to handle connections to AIMS. All other system locations should then have their Live captions outgoing locations set to these AIMS-specific locations.

Live captions locations should not be used to handle calls, or as overflow locations.

Example

For example, if your deployment has two existing system locations, London and Oslo, you may wish to create two new system locations, Live captions London and Live Captions Oslo. You then add one or more Transcoding Conferencing Nodes to each of these live captions locations.

You then configure the London location to use Live captions London as its first choice and Live Captions Oslo as its second choice for its live captions locations. Likewise, you'd configure the Oslo location to use Live captions Oslo as its first choice and Live Captions London as its second choice.

This would keep live captions traffic geographically local while allowing fallback to a secondary live captions location.

Alternatively, you could create a single Live captions location and configure all other system locations to use it as their Live captions outgoing locations first choice.

Creating and configuring a new live captions location

To create a new system location specifically for use as a live captions outgoing location:

1. From Platform > Locations, select Add system location.
2. Configure the NTP server and DNS server fields as appropriate for your environment. All other fields should be left blank, or with their default settings.
3. In the Live captions outgoing locations section, ensure that all choices are left blank.
4. Select Save.

Configuring existing system locations

To configure each existing system location in your deployment to connect to the live captions outgoing location:

1. From Platform > Locations, select the location and scroll down to Live captions outgoing locations.
2. Select the new live captions-specific location created above as the 1st choice. If you have created other live captions locations, select these as the 2nd choice and 3rd choice.
3. Repeat for all system locations in your deployment.

Setting the language on an individual VMR

AIMS needs to know the language of the incoming audio stream in order to apply the appropriate language model. For AIMS v1, generated live captions text are provided in the same language as the incoming (source) audio stream. Later releases of AIMS will be able to provide live captions text in a different language to the source language.

When configuring AIMS, you can set the default source language for all audio streams from the Pexip Infinity instance. If a default source language is not set, en-US is used.

From Pexip Infinity v36 and later, you can override the source language on a per-VMR / Virtual Auditorium / Call Routing Rule basis using external or local policy in combination with the service's service tag. An example of how to do this using local policy is described below.

Using local policy

The example below uses a combination of a VMR's service tag to specify the source language, and local policy to apply the selected language to the VMR's configuration.

We recommend that you apply the same local policy script to all locations in your Pexip Infinity deployment, so that you can use the service tag to specify the source language for any service.

Before you start, ensure you are familiar with the use of local policy as described in Enabling local policy.

1. Ensure that you have enabled the selected language in AIMS. See Setting the loaded language models.

2. From the Pexip Infinity Administrator interface, go to Call Control > Policy Profiles and select Add Policy profile.

3. Enter a Name and optional Description.

4. In the Service configuration policy section, enable Apply local policy.

5. In the Script section that appears, enter the following (or incorporate this code into an existing local policy script):

   Copy to clipboard

   {
   {% if service_config %}
     "action" : "continue",
     {% if service_config.service_tag.startswith("show_captions:") %}
       "result" : {{service_config|pex_update({
       "source_language":service_config.service_tag.split(":")[1],
       "live_captions_language":service_config.service_tag.split(":")[2]
        })|pex_to_json}}
     {% else %}
       "result" : {{service_config|pex_to_json}}
     {% endif %}
   {% else %}
     "action" : "reject",
     "result" : {}
   {% endif %}
   }

6. Select Save.
7. From Platform > Locations select a Location.
8. From the Policy profile field, select the policy profile created above and select Save.

9. Repeat the above steps to apply the policy profile to all other locations in your deployment.

10. From the Services menu, select the VMR for which you want to set the language.
11. Ensure that live captions are enabled for the VMR. To do this, from the Advanced options section, ensure that the Live Captions field shows either:
    • Yes, or
    • Use global live captions setting, in which case Platform > Global settings > Live captions > Enable live captions on VMRs and routing rules by default must be enabled.

12. From the Advanced options section, in the Service tag field enter the following:

    show_captions:<source-lang>:<captions-lang>

    where <source-lang> is the source language and <captions-lang> is the live captions language. For the current release of AIMS these parameters must be set to the same language.

    This method has been designed to support future releases of AIMS, where the source language and the live captions language may be different.

    For example, if the source language is French:

    show_captions:fr-FR:fr-FR

    For AIMS v1 the supported languages are en-US, es-US, de-DE and fr-FR.

13. Select Save.

To set the source language for any other VMR, Virtual Auditorium or Call Routing Rule in your deployment, ensure that live captions are enabled for the service and repeat the Service tag configuration. You can use different source languages for different services in the same Pexip Infinity deployment.

AIMS configuration

In this step you configure the AIMS server with details of the Pexip Infinity deployment that it will be processing requests from. The process differs depending on the version of Pexip Infinity; please select from the appropriate option below.